Repository: grpc/grpc-go

Issue:
Retry Attempts are not available in `client-stats`
Clients can configure **non** transparent retries either via `XDS` or via `ServiceConfig` however the `stats` handler have no good way to retrieve if a particular `RPC` method is a retry or not.

This type of observability is quite useful to determine if `retry-storms`(due to poor configurations) cause incidents and/or to see if `retries` are actually effective.

I thought of looking at the `retry metadata` header `"grpc-previous-rpc-attempts"` but this one is added by the `http2_client` as a header after the `HandleRPC` method is called so that didnt yield any results.

My first question is: Is this actually true or is there a way to get `o11y` on the `non-transparent` retries from the client?

My second question is: If it is not available, the `numRetries` is available when we construct the `stats.Begin` struct so if you are open to extending the `stats.Begin` struct, we can add it there.

Looking forward to your thoughts and also happy to implement the feature if it is not available

Discussion:
@purnesh42H: > My first question is: Is this actually true or is there a way to get o11y on the non-transparent retries from the client?

We don't have retry metrics yet. There is a proposal in progress https://github.com/grpc/proposal/pull/488/files. A metric label `grpc.client.call.retries` is proposed which has a label `grpc.retry_type` that can take one of three values - "retry", "hedge", or "transparent". Please take a look at the proposal to see if it has everything what you need. Tagging @yashykt in case you have more questions regarding the proposal.

In tracing though we have a way to identify if an rpc attempt was a transparent retry or not. When span is created, an attribute with key `transparent-retry` is set with a boolean value indicating whether the stream is undergoing a transparent retry. Tracing is still in experimental state. You can enable tracing through dial options https://github.com/grpc/grpc-go/blob/master/examples/features/opentelemetry/client/main.go#L65@davinci26: > In tracing though we have a way to identify if an rpc attempt was a transparent retry or not.

Seems reasonable, have to check our implementation of tracing to see if we include this or not but thanks for the pointer.


> We don't have retry metrics yet. There is a proposal in progress https://github.com/grpc/proposal/pull/488/files. A metric label grpc.client.call.retries is proposed which has a label grpc.retry_type that can take one of three values - "retry", "hedge", or "transparent". 

Cool proposal, you guys are one step ahead :D. Everything in the proposal makes sense. 

Havent thought this fully so take this with a bit of a grain of salt but given the proposal above and the fact that the `otel` implementation in `grpc-go` is a wrapper around `StatsHandler` would it make sense to extend `stats.Begin` with:

1. `numRetries`
2. If an `RPC` is a `hedge` or not

That way consumers who have an implementation of the stats handler can implement similar metrics as well. 
@purnesh42H: @davinci26  The OpenTelemetry (OTel) plugin provides trace attributes for RPC attempts that help in tracking retries. As you can see in https://github.com/grpc/grpc-go/blob/master/stats/opentelemetry/trace.go#L50-L51 (in the populateSpan function for *stats.Begin), these include:

- previous-rpc-attempts: This attribute indeed indicates the number of prior attempts for the RPC. The OTel stats.Handler currently calculates this value internally (using ai.previousRPCAttempts).
- transparent-retry: This boolean flag is directly sourced from stats.Begin.IsTransparentRetryAttempt and clearly signals if the attempt is a gRPC-handled transparent retry.

And yes, for retry-specific "metrics", there's a proposal in progress.

`stats.Handler` is experimental and for common observability requirements, it's not recommended as the primary solution for most users. The preferred path is to identify what's missing in the official plugins (like the OTel or OpenCensus ones) and work to add those features if they benefit a broad range of users. For users who need very specific, custom metrics that aren't covered by standard tooling, the stats.Handler interface is precisely the mechanism gRPC-Go provides for such advanced customization. Information like numRetries and the type of retry (e.g., standard vs. hedged) are common requirements. The goal is to make such essential data easily accessible, ideally out-of-the-box via the OTel plugin.@davinci26: > previous-rpc-attempts: This attribute indeed indicates the number of prior attempts for the RPC. The OTel stats.Handler currently calculates this value internally (using ai.previousRPCAttempts).

Does this include `service` level retries? I might be missing something super obvious here but I see that:

```
ai := &attemptInfo{
              ....
}
``` 

is added to the `ctx` when the `RPC` is tagged so retries would have a different `ctx`. Basically I cant understand how https://github.com/grpc/grpc-go/blob/709023de87a25ae63b000139295af10589edffee/stats/opentelemetry/trace.go#L54 can be called twice for the same `RPC` for `non-transparent` retries.

> stats.Handler is experimental and for common observability requirements, it's not recommended as the primary solution for most users.

100% agree, `stats.Handler` is a pretty `low-level` interface that only makes sense for a limited subset of folks and 99% of the users would be better suited with the standard `OTel` plugin. But I am not sure if we are agreeing or not on the next bit. Should we add the `numRetries` in the `stats.Begin` struct? I can see this being helpful both for the `OTel plugin` as it would make the code easier and also easier for us few that work with the `stats.Handler` directly.@purnesh42H: @davinci26 you are right about the problem with attemptInfo. Each attempt of rpc gets a new attempt info so its wrong to maintain the counter of previous rpc attempts within that. This is a bug. I think the right way to do this is to funnel the `clientStream.numRetries` all the way to HandleRPC and record the value in "previous-rpc-attempts".  `clientStream.numRetries` exclusively counts non-transparent retries.

Once we do that, we don't have to make it separately available in stats.Begin. Although i guess its fine even if we add `numRetries` to `stats.Begin` as an exported field? @dfawley do you see any issue with exposing numRetries in stats.Begin?
@dfawley: Retry attempts can be counted by storing data in the context in a per-call interceptor, and incrementing it in `TagRPC`.@purnesh42H: > Retry attempts can be counted by storing data in the context in a per-call interceptor, and incrementing it in `TagRPC`.

What is the issue with funneling cs.numRetries? We need to also make sure to exclude transparent retries https://github.com/grpc/proposal/blob/master/A72-open-telemetry-tracing.md#tracing-information. clientStream.numRetries already have what we want.@dfawley: Sorry, I missed the bit about excluding transparent retries.  You can wait to increment when `stats.Begin` happens, then, and check that struct's `IsTransparentRetryAttempt` field.

> What is the issue with funneling cs.numRetries?

Generally, adding things requires proof that it is either necessary or sufficiently convenient.  So the question should be the other way around: what is the issue with doing it the way that is already possible?

Also...we don't support hedging yet, but when/if we do, those likely wouldn't be included in the `clientStream.numRetries`, and it would be wrong to not count hedging attempts in our tracing instrumentation.@davinci26: > Generally, adding things requires proof that it is either necessary or sufficiently convenient. So the question should be the other way around: what is the issue with doing it the way that is already possible?

Let me code up the suggestion above so take this statement with 75% confidence but where my head is at is that this fairly complicated/non-ergonomic. I appreciate the aversion towards extending the `api` surface but `Retry attempts can be counted by storing data in the context in a per-call interceptor, and incrementing it in TagRPC. and on top  You can wait to increment when stats.Begin happens, then, and check that struct's IsTransparentRetryAttempt field.` seems fairly easy to get it wrong for something that is pretty useful o11y for service owners. `Am I retrying too much`


Edit: I think where I am coming from is that I already got this wrong once and I couldnt think of a way to implement the retries hence my question and then we realized that even the `grpc-go` implementation didn't get this 100% right so maybe we can do better and make it easier for folks to implement this.@dfawley: The `stats` package's API is mainly intended for internal usage or for power users.  Optimal usability is not really a goal (it already has a ton of rough edges).

Our goal is that 99% of users should be able to use the OpenCensus or OpenTelemetry plugins, instead, and that those should be easy to configure and use.@davinci26: Make sense @dfawley, thanks for the feedback and the pointer!

I still have my reservations:
*  Recommending to power users this approach via extending the `context` is going to create an `implicit` api for `grpc-go` that is much harder to test and modify on the `OSS` side. But I think this tradeoff is yours to decide, I am happy and I can support either outcome.
* I think it would simplify the implementation of both `OpenCensus`/`OpenTelemetry` plugins + power users.

I think the main advantage on my side is that this method is available right now with the current release, so I ended up implementing it. It works well so far.

(issue is closed on my side, but I think it is used to track the bug on the `OTEL` plugin so leaving it open)@vinothkumarr227: @purnesh42H I'm looking into this. Please assign the issue to me.@arjan-bal: The PR seems to have introduced a flaky test: https://github.com/grpc/grpc-go/actions/runs/17614152882/job/50042942932?pr=8547

It's being rolled back.@easwars: Part of the discussion here is about exposing retry metrics, which is being tracked as part of gRFC A96 and is a stretch goal for this team for Q4'25.

Part of the discussion is about a bug in the oTel implementation about where the number of retries is being tracked. I've created a [new issue](https://github.com/grpc/grpc-go/issues/8700) for that, to keep the discussion pertinent to the issue on hand.



Solution patch:
diff --git a/stats/opentelemetry/client_tracing.go b/stats/opentelemetry/client_tracing.go
index 6b725fb6..868d6a2f 100644
--- a/stats/opentelemetry/client_tracing.go
+++ b/stats/opentelemetry/client_tracing.go
@@ -21,7 +21,6 @@ import (
 	"log"
 	"strings"
 
-	"go.opentelemetry.io/otel/attribute"
 	otelcodes "go.opentelemetry.io/otel/codes"
 	"go.opentelemetry.io/otel/trace"
 	"google.golang.org/grpc"
@@ -84,10 +83,7 @@ func (h *clientTracingHandler) finishTrace(err error, ts trace.Span) {
 // It creates a new outgoing carrier which serializes information about this
 // span into gRPC Metadata, if TextMapPropagator is provided in the trace
 // options. if TextMapPropagator is not provided, it returns the context as is.
-//
-// Note: The passed attemptInfo pointer (ai) is mutated in-place. Fields such as
-// ai.traceSpan are updated directly. No new attemptInfo is returned.
-func (h *clientTracingHandler) traceTagRPC(ctx context.Context, ai *attemptInfo, nameResolutionDelayed bool) context.Context {
+func (h *clientTracingHandler) traceTagRPC(ctx context.Context, ai *attemptInfo, nameResolutionDelayed bool) (context.Context, *attemptInfo) {
 	// Add a "Delayed name resolution complete" event to the call span
 	// if there was name resolution delay. In case of multiple retry attempts,
 	// ensure that event is added only once.
@@ -102,7 +98,7 @@ func (h *clientTracingHandler) traceTagRPC(ctx context.Context, ai *attemptInfo,
 	carrier := otelinternaltracing.NewOutgoingCarrier(ctx)
 	h.options.TraceOptions.TextMapPropagator.Inject(ctx, carrier)
 	ai.traceSpan = span
-	return carrier.Context()
+	return carrier.Context(), ai
 }
 
 // createCallTraceSpan creates a call span to put in the provided context using
@@ -125,12 +121,7 @@ func (h *clientTracingHandler) HandleConn(context.Context, stats.ConnStats) {}
 // TagRPC implements per RPC attempt context management for traces.
 func (h *clientTracingHandler) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context {
 	ctx, ai := getOrCreateRPCAttemptInfo(ctx)
-	ci := getCallInfo(ctx)
-	if ci == nil {
-		logger.Error("context passed into client side stats handler (TagRPC) has no call info")
-		return ctx
-	}
-	ctx = h.traceTagRPC(ctx, ai, info.NameResolutionDelay)
+	ctx, ai = h.traceTagRPC(ctx, ai, info.NameResolutionDelay)
 	return setRPCInfo(ctx, &rpcInfo{ai: ai})
 }
 
@@ -141,15 +132,5 @@ func (h *clientTracingHandler) HandleRPC(ctx context.Context, rs stats.RPCStats)
 		logger.Error("ctx passed into client side tracing handler trace event handling has no client attempt data present")
 		return
 	}
-
-	// Client-specific Begin attributes.
-	if begin, ok := rs.(*stats.Begin); ok {
-		ci := getCallInfo(ctx)
-		previousRPCAttempts := ci.previousRPCAttempts.Add(1) - 1
-		ri.ai.traceSpan.SetAttributes(
-			attribute.Int64("previous-rpc-attempts", int64(previousRPCAttempts)),
-			attribute.Bool("transparent-retry", begin.IsTransparentRetryAttempt),
-		)
-	}
 	populateSpan(rs, ri.ai)
 }
diff --git a/stats/opentelemetry/opentelemetry.go b/stats/opentelemetry/opentelemetry.go
index 91091c99..cd01f86c 100644
--- a/stats/opentelemetry/opentelemetry.go
+++ b/stats/opentelemetry/opentelemetry.go
@@ -179,9 +179,6 @@ type callInfo struct {
 	// nameResolutionEventAdded is set when the resolver delay trace event
 	// is added. Prevents duplicate events, since it is reported per-attempt.
 	nameResolutionEventAdded atomic.Bool
-	// previousRPCAttempts holds the count of RPC attempts that have happened
-	// before current attempt. Transparent retries are excluded.
-	previousRPCAttempts atomic.Uint32
 }
 
 type callInfoKey struct{}
@@ -242,8 +239,9 @@ type attemptInfo struct {
 	// message counters for sent and received messages (used for
 	// generating message IDs), and the number of previous RPC attempts for the
 	// associated call.
-	countSentMsg uint32
-	countRecvMsg uint32
+	countSentMsg        uint32
+	countRecvMsg        uint32
+	previousRPCAttempts uint32
 }
 
 type clientMetrics struct {
diff --git a/stats/opentelemetry/trace.go b/stats/opentelemetry/trace.go
index 3ee66d1e..40ac7a1b 100644
--- a/stats/opentelemetry/trace.go
+++ b/stats/opentelemetry/trace.go
@@ -17,6 +17,8 @@
 package opentelemetry
 
 import (
+	"sync/atomic"
+
 	"go.opentelemetry.io/otel/attribute"
 	otelcodes "go.opentelemetry.io/otel/codes"
 	"go.opentelemetry.io/otel/trace"
@@ -38,6 +40,18 @@ func populateSpan(rs stats.RPCStats, ai *attemptInfo) {
 	span := ai.traceSpan
 
 	switch rs := rs.(type) {
+	case *stats.Begin:
+		// Note: Go always added Client and FailFast attributes even though they are not
+		// defined by the OpenCensus gRPC spec. Thus, they are unimportant for
+		// correctness.
+		span.SetAttributes(
+			attribute.Bool("Client", rs.Client),
+			attribute.Bool("FailFast", rs.FailFast),
+			attribute.Int64("previous-rpc-attempts", int64(ai.previousRPCAttempts)),
+			attribute.Bool("transparent-retry", rs.IsTransparentRetryAttempt),
+		)
+		// increment previous rpc attempts applicable for next attempt
+		atomic.AddUint32(&ai.previousRPCAttempts, 1)
 	case *stats.DelayedPickComplete:
 		span.AddEvent("Delayed LB pick complete")
 	case *stats.InPayload:



Test patch:
diff --git a/stats/opentelemetry/e2e_test.go b/stats/opentelemetry/e2e_test.go
index 9f345517..2d575bfe 100644
--- a/stats/opentelemetry/e2e_test.go
+++ b/stats/opentelemetry/e2e_test.go
@@ -89,7 +89,13 @@ type traceSpanInfo struct {
 	name       string
 	events     []trace.Event
 	attributes []attribute.KeyValue
-	status     otelcodes.Code
+}
+
+// traceSpanInfoMapKey is the key struct for constructing a map of trace spans
+// retrievable by span name and span kind
+type traceSpanInfoMapKey struct {
+	spanName string
+	spanKind string
 }
 
 // defaultMetricsOptions creates default metrics options
@@ -270,32 +276,45 @@ func validateTraces(t *testing.T, spans tracetest.SpanStubs, wantSpanInfos []tra
 		}
 	}
 
-	// Convert spans to traceSpanInfo for cmp.Diff comparison.
-	actualSpanInfos := make([]traceSpanInfo, len(spans))
-	for i, s := range spans {
-		actualSpanInfos[i] = traceSpanInfo{
-			name:       s.Name,
-			spanKind:   s.SpanKind.String(),
-			attributes: s.Attributes,
-			events:     s.Events,
-			status:     s.Status.Code,
-		}
-	}
-	opts := []cmp.Option{
-		cmpopts.SortSlices(func(a, b traceSpanInfo) bool {
-			if a.name == b.name {
-				return a.spanKind < b.spanKind
-			}
-			return a.name < b.name
-		}),
-		cmpopts.SortSlices(func(a, b trace.Event) bool { return a.Name < b.Name }),
-		cmpopts.IgnoreFields(trace.Event{}, "Time"),
-		cmpopts.EquateComparable(attribute.KeyValue{}, attribute.Value{}, attribute.Set{}),
-		cmpopts.IgnoreFields(tracetest.SpanStub{}, "InstrumentationScope"),
-		cmp.AllowUnexported(traceSpanInfo{}),
+	// Constructs a map from a slice of traceSpanInfo to retrieve the
+	// corresponding expected span info based on span name and span kind
+	// for comparison.
+	wantSpanInfosMap := make(map[traceSpanInfoMapKey]traceSpanInfo)
+	for _, info := range wantSpanInfos {
+		key := traceSpanInfoMapKey{spanName: info.name, spanKind: info.spanKind}
+		wantSpanInfosMap[key] = info
 	}
-	if diff := cmp.Diff(wantSpanInfos, actualSpanInfos, opts...); diff != "" {
-		t.Errorf("Spans mismatch (-want +got):\n%s", diff)
+
+	// Compare retrieved spans with expected spans.
+	for _, span := range spans {
+		// Check that the attempt span has the correct status.
+		if got, want := span.Status.Code, otelcodes.Ok; got != want {
+			t.Errorf("Got status code %v, want %v", got, want)
+		}
+
+		// Retrieve the corresponding expected span info based on span name and
+		// span kind to compare.
+		want, ok := wantSpanInfosMap[traceSpanInfoMapKey{spanName: span.Name, spanKind: span.SpanKind.String()}]
+		if !ok {
+			t.Errorf("Unexpected span: %v", span)
+			continue
+		}
+
+		// comparers
+		attributesSort := cmpopts.SortSlices(func(a, b attribute.KeyValue) bool {
+			return a.Key < b.Key
+		})
+		attributesValueComparable := cmpopts.EquateComparable(attribute.KeyValue{}.Value)
+		eventsTimeIgnore := cmpopts.IgnoreFields(trace.Event{}, "Time")
+
+		// attributes
+		if diff := cmp.Diff(want.attributes, span.Attributes, attributesSort, attributesValueComparable); diff != "" {
+			t.Errorf("Attributes mismatch for span %s (-want +got):\n%s", span.Name, diff)
+		}
+		// events
+		if diff := cmp.Diff(want.events, span.Events, attributesSort, attributesValueComparable, eventsTimeIgnore); diff != "" {
+			t.Errorf("Events mismatch for span %s (-want +got):\n%s", span.Name, diff)
+		}
 	}
 }
 
@@ -838,10 +857,26 @@ func (s) TestMetricsAndTracesOptionEnabled(t *testing.T) {
 
 	wantSpanInfos := []traceSpanInfo{
 		{
-			name:       "Recv.grpc.testing.TestService.UnaryCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
+			name:     "Recv.grpc.testing.TestService.UnaryCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
 			events: []trace.Event{
 				{
 					Name: "Inbound message",
@@ -882,8 +917,15 @@ func (s) TestMetricsAndTracesOptionEnabled(t *testing.T) {
 		{
 			name:     "Attempt.grpc.testing.TestService.UnaryCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -933,29 +975,50 @@ func (s) TestMetricsAndTracesOptionEnabled(t *testing.T) {
 		{
 			name:       "Sent.grpc.testing.TestService.UnaryCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
-			name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
-			events:     nil,
+			name:     "Recv.grpc.testing.TestService.FullDuplexCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
+			events: nil,
 		},
 		{
 			name:       "Sent.grpc.testing.TestService.FullDuplexCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
 			name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -1018,10 +1081,26 @@ func (s) TestSpan(t *testing.T) {
 
 	wantSpanInfos := []traceSpanInfo{
 		{
-			name:       "Recv.grpc.testing.TestService.UnaryCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
+			name:     "Recv.grpc.testing.TestService.UnaryCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
 			events: []trace.Event{
 				{
 					Name: "Inbound message",
@@ -1054,8 +1133,15 @@ func (s) TestSpan(t *testing.T) {
 		{
 			name:     "Attempt.grpc.testing.TestService.UnaryCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -1097,29 +1183,50 @@ func (s) TestSpan(t *testing.T) {
 		{
 			name:       "Sent.grpc.testing.TestService.UnaryCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
-			name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
-			events:     nil,
+			name:     "Recv.grpc.testing.TestService.FullDuplexCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
+			events: nil,
 		},
 		{
 			name:       "Sent.grpc.testing.TestService.FullDuplexCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
 			name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -1184,10 +1291,26 @@ func (s) TestSpan_WithW3CContextPropagator(t *testing.T) {
 
 	wantSpanInfos := []traceSpanInfo{
 		{
-			name:       "Recv.grpc.testing.TestService.UnaryCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
+			name:     "Recv.grpc.testing.TestService.UnaryCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
 			events: []trace.Event{
 				{
 					Name: "Inbound message",
@@ -1220,8 +1343,15 @@ func (s) TestSpan_WithW3CContextPropagator(t *testing.T) {
 		{
 			name:     "Attempt.grpc.testing.TestService.UnaryCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -1263,29 +1393,50 @@ func (s) TestSpan_WithW3CContextPropagator(t *testing.T) {
 		{
 			name:       "Sent.grpc.testing.TestService.UnaryCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
-			name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			attributes: nil,
-			events:     nil,
+			name:     "Recv.grpc.testing.TestService.FullDuplexCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(false),
+				},
+				{
+					Key:   "previous-rpc-attempts",
+					Value: attribute.IntValue(0),
+				},
+				{
+					Key:   "transparent-retry",
+					Value: attribute.BoolValue(false),
+				},
+			},
+			events: nil,
 		},
 		{
 			name:       "Sent.grpc.testing.TestService.FullDuplexCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			attributes: nil,
 			events:     nil,
 		},
 		{
 			name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			attributes: []attribute.KeyValue{
+				{
+					Key:   "Client",
+					Value: attribute.BoolValue(true),
+				},
+				{
+					Key:   "FailFast",
+					Value: attribute.BoolValue(true),
+				},
 				{
 					Key:   "previous-rpc-attempts",
 					Value: attribute.IntValue(0),
@@ -1399,10 +1550,10 @@ const delayedResolutionEventName = "Delayed name resolution complete"
 // only once if any of the retry attempt encountered a delay in name resolution
 func (s) TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
 	tests := []struct {
-		name          string
-		setupStub     func() *stubserver.StubServer
-		doCall        func(context.Context, testgrpc.TestServiceClient) error
-		wantSpanInfos []traceSpanInfo
+		name      string
+		setupStub func() *stubserver.StubServer
+		doCall    func(context.Context, testgrpc.TestServiceClient) error
+		spanName  string
 	}{
 		{
 			name: "unary",
@@ -1425,136 +1576,7 @@ func (s) TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
 				_, err := client.UnaryCall(ctx, &testpb.SimpleRequest{})
 				return err
 			},
-			wantSpanInfos: []traceSpanInfo{
-				{
-					name:       "Recv.grpc.testing.TestService.UnaryCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Error,
-					attributes: nil,
-					events: []trace.Event{
-						{
-							Name: "Inbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				// RPC attempt #1
-				{
-					name:     "Attempt.grpc.testing.TestService.UnaryCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Error,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 0),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Delayed LB pick complete",
-						},
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Recv.grpc.testing.TestService.UnaryCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Error,
-					attributes: nil,
-					events: []trace.Event{
-						{
-							Name: "Inbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				// RPC attempt #2
-				{
-					name:     "Attempt.grpc.testing.TestService.UnaryCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Error,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 1),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Recv.grpc.testing.TestService.UnaryCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Ok,
-					attributes: nil,
-					events: []trace.Event{
-						{
-							Name: "Inbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				// RPC attempt #3
-				{
-					name:     "Attempt.grpc.testing.TestService.UnaryCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Ok,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 2),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-						{
-							Name: "Inbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Sent.grpc.testing.TestService.UnaryCall",
-					spanKind:   oteltrace.SpanKindClient.String(),
-					status:     otelcodes.Ok,
-					attributes: nil,
-					events: []trace.Event{
-						{Name: delayedResolutionEventName},
-					},
-				},
-			},
+			spanName: "Sent.grpc.testing.TestService.UnaryCall",
 		},
 		{
 			name: "streaming",
@@ -1598,106 +1620,7 @@ func (s) TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
 				}
 				return nil
 			},
-			wantSpanInfos: []traceSpanInfo{
-				{
-					name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Error,
-					attributes: nil,
-					events:     nil,
-				},
-				// RPC attempt #1
-				{
-					name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Error,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 0),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Delayed LB pick complete",
-						},
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Error,
-					attributes: nil,
-					events:     nil,
-				},
-				// RPC attempt #2
-				{
-					name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Error,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 1),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-					spanKind:   oteltrace.SpanKindServer.String(),
-					status:     otelcodes.Ok,
-					attributes: nil,
-					events: []trace.Event{
-						{
-							Name: "Inbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				// RPC attempt #3
-				{
-					name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
-					spanKind: oteltrace.SpanKindInternal.String(),
-					status:   otelcodes.Ok,
-					attributes: []attribute.KeyValue{
-						attribute.Int("previous-rpc-attempts", 2),
-						attribute.Bool("transparent-retry", false),
-					},
-					events: []trace.Event{
-						{
-							Name: "Outbound message",
-							Attributes: []attribute.KeyValue{
-								attribute.Int("sequence-number", 0),
-								attribute.Int("message-size", 0),
-							},
-						},
-					},
-				},
-				{
-					name:       "Sent.grpc.testing.TestService.FullDuplexCall",
-					spanKind:   oteltrace.SpanKindClient.String(),
-					status:     otelcodes.Ok,
-					attributes: nil,
-					events: []trace.Event{
-						{Name: delayedResolutionEventName},
-					},
-				},
-			},
+			spanName: "Sent.grpc.testing.TestService.FullDuplexCall",
 		},
 	}
 
@@ -1753,15 +1676,38 @@ func (s) TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
 			if err := tt.doCall(ctx, client); err != nil {
 				t.Fatalf("%s call failed: %v", tt.name, err)
 			}
-			spans, err := waitForTraceSpans(ctx, exporter, tt.wantSpanInfos)
+
+			wantSpanInfo := traceSpanInfo{
+				name:     tt.spanName,
+				spanKind: oteltrace.SpanKindClient.String(),
+				events:   []trace.Event{{Name: delayedResolutionEventName}},
+			}
+			spans, err := waitForTraceSpans(ctx, exporter, []traceSpanInfo{wantSpanInfo})
 			if err != nil {
 				t.Fatal(err)
 			}
-			validateTraces(t, spans, tt.wantSpanInfos)
+			verifyTrace(t, spans, wantSpanInfo)
 		})
 	}
 }
 
+func verifyTrace(t *testing.T, spans tracetest.SpanStubs, want traceSpanInfo) {
+	match := false
+	for _, span := range spans {
+		if span.Name == want.name && span.SpanKind.String() == want.spanKind {
+			match = true
+			if diff := cmp.Diff(want.events, span.Events, cmpopts.IgnoreFields(trace.Event{}, "Time")); diff != "" {
+				t.Errorf("Span event mismatch for %q (kind: %s) (-want +got):\n%s",
+					want.name, want.spanKind, diff)
+			}
+			break
+		}
+	}
+	if !match {
+		t.Errorf("Expected span not found: %q (kind: %s)", want.name, want.spanKind)
+	}
+}
+
 // TestStreamingRPC_TraceSequenceNumbers verifies that sequence numbers
 // are incremented correctly for multiple messages sent and received
 // during a streaming RPC.
@@ -1810,23 +1756,27 @@ func (s) TestStreamingRPC_TraceSequenceNumbers(t *testing.T) {
 		{
 			name:       "Sent.grpc.testing.TestService.FullDuplexCall",
 			spanKind:   oteltrace.SpanKindClient.String(),
-			status:     otelcodes.Ok,
 			events:     nil,
 			attributes: nil,
 		},
 		{
-			name:       "Recv.grpc.testing.TestService.FullDuplexCall",
-			spanKind:   oteltrace.SpanKindServer.String(),
-			status:     otelcodes.Ok,
-			events:     wantInboundEvents,
-			attributes: nil,
+			name:     "Recv.grpc.testing.TestService.FullDuplexCall",
+			spanKind: oteltrace.SpanKindServer.String(),
+			events:   wantInboundEvents,
+			attributes: []attribute.KeyValue{
+				attribute.Bool("Client", false),
+				attribute.Bool("FailFast", false),
+				attribute.Int("previous-rpc-attempts", 0),
+				attribute.Bool("transparent-retry", false),
+			},
 		},
 		{
 			name:     "Attempt.grpc.testing.TestService.FullDuplexCall",
 			spanKind: oteltrace.SpanKindInternal.String(),
-			status:   otelcodes.Ok,
 			events:   wantOutboundEvents,
 			attributes: []attribute.KeyValue{
+				attribute.Bool("Client", true),
+				attribute.Bool("FailFast", true),
 				attribute.Int("previous-rpc-attempts", 0),
 				attribute.Bool("transparent-retry", false),
 			},