Repository: getsentry/sentry-kafka-schemas
Pull request #389: fix(spans): Align the spans topics with reality
Author: login: jan-auer
Created: 2025-03-05T16:00:11Z | Merged: 2025-03-06T10:28:57Z

Various fixes to the `ingest-spans`, `buffered-segments`, and `snuba-spans`
schema definitions:

- Remove `_metrics_summary`, as this is no longer produced and supported
- Make `parent_span_id` and `segment_id` optional everywhere
- Remove `exclusive_time_ms` from `ingest-spans` and `buffered-segments`
- Remove `is_segment` and `segment_id` from `ingest-spans`

Also exposes `SegmentSpan` as named type from `buffered-segments` for the consumer




Diff:
diff --git a/examples/snuba-spans/1/basic_span.json b/examples/snuba-spans/1/basic_span.json
index 96317bd8..350a44f2 100644
--- a/examples/snuba-spans/1/basic_span.json
+++ b/examples/snuba-spans/1/basic_span.json
@@ -39,39 +39,5 @@
       "value": 100.0,
       "unit": "byte"
     }
-  },
-  "_metrics_summary": {
-    "c:sentry.events.outcomes@none": [
-      {
-        "count": 1,
-        "max": 1.0,
-        "min": 1.0,
-        "sum": 1.0,
-        "tags": {
-          "category": "error",
-          "environment": "unknown",
-          "event_type": "error",
-          "outcome": "accepted",
-          "release": "backend@2af74c237fbd61489a1ccc46650f4f85befaf8b8",
-          "topic": "outcomes-billing",
-          "transaction": "sentry.tasks.store.save_event"
-        }
-      }
-    ],
-    "c:sentry.events.post_save.normalize.errors@none": [
-      {
-        "count": 1,
-        "max": 0.0,
-        "min": 0.0,
-        "sum": 0.0,
-        "tags": {
-          "environment": "unknown",
-          "event_type": "error",
-          "from_relay": "False",
-          "release": "backend@2af74c237fbd61489a1ccc46650f4f85befaf8b8",
-          "transaction": "sentry.tasks.store.save_event"
-        }
-      }
-    ]
   }
 }
diff --git a/schemas/buffered-segments.v1.schema.json b/schemas/buffered-segments.v1.schema.json
index 135a4ef6..7018a299 100644
--- a/schemas/buffered-segments.v1.schema.json
+++ b/schemas/buffered-segments.v1.schema.json
@@ -5,21 +5,22 @@
   "additionalProperties": true,
   "properties": {
     "spans": {
-      "$ref": "#/definitions/Spans"
+      "$ref": "#/definitions/SegmentSpans"
     }
   },
   "required": ["spans"],
   "definitions": {
-    "Spans": {
+    "SegmentSpans": {
       "type": "array",
       "minItems": 1,
       "items": {
-        "$ref": "#/definitions/Span"
+        "$ref": "#/definitions/SegmentSpan"
       }
     },
-    "Span": {
+    "SegmentSpan": {
       "type": "object",
       "additionalProperties": true,
+      "title": "segment_span",
       "properties": {
         "event_id": {
           "$ref": "#/definitions/UUID"
@@ -39,11 +40,11 @@
           "description": "The span ID is a unique identifier for a span within a trace. It is an 8 byte hexadecimal string."
         },
         "parent_span_id": {
-          "type": "string",
+          "type": ["string", "null"],
           "description": "The parent span ID is the ID of the span that caused this span. It is an 8 byte hexadecimal string."
         },
         "segment_id": {
-          "type": "string",
+          "type": ["string", "null"],
           "description": "The segment ID is a unique identifier for a segment within a trace. It is an 8 byte hexadecimal string."
         },
         "profile_id": {
@@ -62,10 +63,6 @@
           "$ref": "#/definitions/UInt32",
           "description": "The duration of the span in milliseconds."
         },
-        "exclusive_time_ms": {
-          "$ref": "#/definitions/PositiveFloat",
-          "description": "The exclusive time of the span in milliseconds."
-        },
         "retention_days": {
           "$ref": "#/definitions/UInt16"
         },
@@ -95,17 +92,10 @@
         },
         "measurements": {
           "$ref": "#/definitions/Measurements"
-        },
-        "_metrics_summary": {
-          "type": "object",
-          "additionalProperties": {
-            "$ref": "#/definitions/MetricsSummary"
-          }
         }
       },
       "required": [
         "duration_ms",
-        "exclusive_time_ms",
         "is_segment",
         "project_id",
         "received",
@@ -205,35 +195,6 @@
         }
       },
       "required": ["value"]
-    },
-    "MetricsSummary": {
-      "type": "array",
-      "items": {
-        "$ref": "#/definitions/MetricSummaryValue"
-      }
-    },
-    "MetricSummaryValue": {
-      "type": "object",
-      "properties": {
-        "min": {
-          "type": "number"
-        },
-        "max": {
-          "type": "number"
-        },
-        "sum": {
-          "type": "number"
-        },
-        "count": {
-          "type": "number"
-        },
-        "tags": {
-          "type": "object",
-          "additionalProperties": {
-            "type": "string"
-          }
-        }
-      }
     }
   }
 }
diff --git a/schemas/ingest-spans.v1.schema.json b/schemas/ingest-spans.v1.schema.json
index 08dbe377..74f92314 100644
--- a/schemas/ingest-spans.v1.schema.json
+++ b/schemas/ingest-spans.v1.schema.json
@@ -26,21 +26,13 @@
           "description": "The span ID is a unique identifier for a span within a trace. It is an 8 byte hexadecimal string."
         },
         "parent_span_id": {
-          "type": "string",
+          "type": ["string", "null"],
           "description": "The parent span ID is the ID of the span that caused this span. It is an 8 byte hexadecimal string."
         },
-        "segment_id": {
-          "type": "string",
-          "description": "The segment ID is a unique identifier for a segment within a trace. It is an 8 byte hexadecimal string."
-        },
         "profile_id": {
           "$ref": "#/definitions/UUID",
           "description": "The profile ID. It is an 16 byte hexadecimal string."
         },
-        "is_segment": {
-          "type": "boolean",
-          "description": "Whether this span is a segment or not."
-        },
         "start_timestamp_ms": {
           "$ref": "#/definitions/UInt",
           "description": "The start timestamp of the span in milliseconds since epoch."
@@ -57,10 +49,6 @@
           "$ref": "#/definitions/UInt32",
           "description": "The duration of the span in milliseconds."
         },
-        "exclusive_time_ms": {
-          "$ref": "#/definitions/PositiveFloat",
-          "description": "The exclusive time of the span in milliseconds."
-        },
         "retention_days": {
           "$ref": "#/definitions/UInt16"
         },
@@ -93,18 +81,10 @@
         },
         "data": {
           "$ref": "#/definitions/Data"
-        },
-        "_metrics_summary": {
-          "type": "object",
-          "additionalProperties": {
-            "$ref": "#/definitions/MetricsSummary"
-          }
         }
       },
       "required": [
         "duration_ms",
-        "exclusive_time_ms",
-        "is_segment",
         "project_id",
         "organization_id",
         "received",
@@ -209,35 +189,6 @@
         }
       },
       "required": ["value"]
-    },
-    "MetricsSummary": {
-      "type": "array",
-      "items": {
-        "$ref": "#/definitions/MetricSummaryValue"
-      }
-    },
-    "MetricSummaryValue": {
-      "type": "object",
-      "properties": {
-        "min": {
-          "type": "number"
-        },
-        "max": {
-          "type": "number"
-        },
-        "sum": {
-          "type": "number"
-        },
-        "count": {
-          "type": "number"
-        },
-        "tags": {
-          "type": "object",
-          "additionalProperties": {
-            "type": "string"
-          }
-        }
-      }
     }
   }
 }
diff --git a/schemas/snuba-spans.v1.schema.json b/schemas/snuba-spans.v1.schema.json
index 08dbe377..6e51ead4 100644
--- a/schemas/snuba-spans.v1.schema.json
+++ b/schemas/snuba-spans.v1.schema.json
@@ -26,11 +26,11 @@
           "description": "The span ID is a unique identifier for a span within a trace. It is an 8 byte hexadecimal string."
         },
         "parent_span_id": {
-          "type": "string",
+          "type": ["string", "null"],
           "description": "The parent span ID is the ID of the span that caused this span. It is an 8 byte hexadecimal string."
         },
         "segment_id": {
-          "type": "string",
+          "type": ["string", "null"],
           "description": "The segment ID is a unique identifier for a segment within a trace. It is an 8 byte hexadecimal string."
         },
         "profile_id": {
@@ -93,12 +93,6 @@
         },
         "data": {
           "$ref": "#/definitions/Data"
-        },
-        "_metrics_summary": {
-          "type": "object",
-          "additionalProperties": {
-            "$ref": "#/definitions/MetricsSummary"
-          }
         }
       },
       "required": [
@@ -209,35 +203,6 @@
         }
       },
       "required": ["value"]
-    },
-    "MetricsSummary": {
-      "type": "array",
-      "items": {
-        "$ref": "#/definitions/MetricSummaryValue"
-      }
-    },
-    "MetricSummaryValue": {
-      "type": "object",
-      "properties": {
-        "min": {
-          "type": "number"
-        },
-        "max": {
-          "type": "number"
-        },
-        "sum": {
-          "type": "number"
-        },
-        "count": {
-          "type": "number"
-        },
-        "tags": {
-          "type": "object",
-          "additionalProperties": {
-            "type": "string"
-          }
-        }
-      }
     }
   }
 }


Reviews:
user: login: kylemumma
body: * What are these files? Are these schema definitions for what data looks like on a kafka topic?
* What is the difference between the 3: buffered_segment, ingest-spans, and snuba-spans? The only topic I am familiar with is the snuba-spans topic that snuba consumes spans off of.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T18:12:15Z
user: login: kylemumma
body: The change to snuba-spans to allow parent_span_id and segment_id to be null should be safe for snuba consumers because they already handle the null case for these fields. https://github.com/getsentry/snuba/blob/b6f874b16bc533ae233ebfac39110c3d473bc907/rust_snuba/src/processors/spans.rs#L39-L62
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T19:28:59Z
user: login: jan-auer
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T10:07:55Z
user: login: Dav1dde
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T10:28:51Z

Review comments:
diff_hunk: @@ -93,12 +93,6 @@
         },
         "data": {
           "$ref": "#/definitions/Data"
-        },
-        "_metrics_summary": {
-          "type": "object",
-          "additionalProperties": {
-            "$ref": "#/definitions/MetricsSummary"
-          }
path: schemas/snuba-spans.v1.schema.json
user: login: kylemumma
body: why is this removed?
created_at: 2025-03-05T19:24:52Z
author_association: MEMBER
diff_hunk: @@ -209,35 +203,6 @@
         }
       },
       "required": ["value"]
-    },
-    "MetricsSummary": {
-      "type": "array",
-      "items": {
-        "$ref": "#/definitions/MetricSummaryValue"
-      }
-    },
-    "MetricSummaryValue": {
-      "type": "object",
-      "properties": {
-        "min": {
-          "type": "number"
-        },
-        "max": {
-          "type": "number"
-        },
-        "sum": {
-          "type": "number"
-        },
-        "count": {
-          "type": "number"
-        },
-        "tags": {
-          "type": "object",
-          "additionalProperties": {
-            "type": "string"
-          }
-        }
-      }
path: schemas/snuba-spans.v1.schema.json
user: login: kylemumma
body: why are these removed?
created_at: 2025-03-05T19:26:04Z
author_association: MEMBER
diff_hunk: @@ -93,12 +93,6 @@
         },
         "data": {
           "$ref": "#/definitions/Data"
-        },
-        "_metrics_summary": {
-          "type": "object",
-          "additionalProperties": {
-            "$ref": "#/definitions/MetricsSummary"
-          }
path: schemas/snuba-spans.v1.schema.json
user: login: jan-auer
body: See the PR description - metrics summaries do not exist anymore and will not come back.
created_at: 2025-03-06T10:07:55Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T16:01:03Z
body: <!-- This PR has a migration; here is the generated SQL for  `json-schema-changes` () -->
**versions in use:**

*The following repositories use one of the schemas you are editing. It is recommended to roll out schema changes in small PRs, meaning that if those used versions lag behind the latest, it is probably best to update those services before rolling out your change.*

**latest version:** 1.1.1

<details><summary><strong>benign changes</strong></summary>

**schemas/buffered-segments.v1.schema.json**
- Removed a property `_metrics_summary` from `.spans.?`, but it is still accepted via `additionalProperties=true`

  ```
  {"path": ".spans.?", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  ```

- Removed a property `exclusive_time_ms` from `.spans.?`, but it is still accepted via `additionalProperties=true`

  ```
  {"path": ".spans.?", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "exclusive_time_ms"}}}
  ```

- ```
  {"path": ".spans.?.parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  ```

- ```
  {"path": ".spans.?.segment_id", "change": {"TypeAdd": {"added": "null"}}}
  ```

- ```
  {"path": ".spans.?", "change": {"RequiredRemove": {"property": "exclusive_time_ms"}}}
  ```

**schemas/ingest-spans.v1.schema.json**
- Removed a property `_metrics_summary` from ``, but it is still accepted via `additionalProperties=true`

  ```
  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  ```

- Removed a property `exclusive_time_ms` from ``, but it is still accepted via `additionalProperties=true`

  ```
  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "exclusive_time_ms"}}}
  ```

- Removed a property `is_segment` from ``, but it is still accepted via `additionalProperties=true`

  ```
  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "is_segment"}}}
  ```

- Removed a property `segment_id` from ``, but it is still accepted via `additionalProperties=true`

  ```
  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "segment_id"}}}
  ```

- ```
  {"path": ".parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  ```

- ```
  {"path": "", "change": {"RequiredRemove": {"property": "exclusive_time_ms"}}}
  ```

- ```
  {"path": "", "change": {"RequiredRemove": {"property": "is_segment"}}}
  ```

**schemas/snuba-spans.v1.schema.json**
- Removed a property `_metrics_summary` from ``, but it is still accepted via `additionalProperties=true`

  ```
  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  ```

- ```
  {"path": ".parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  ```

- ```
  {"path": ".segment_id", "change": {"TypeAdd": {"added": "null"}}}
  ```


</details>

✅ This PR should be safe to roll out to **consumers** first. Make sure to bump
the library in the following repos first:

```
getsentry/sentry
getsentry/snuba
getsentry/super-big-consumers
```

...then in the other repos:

```
getsentry/sentry
getsentry/relay
```

Take a look at the README for how to release a new version of sentry-kafka-schemas.
        

author_association: NONE
user: login: jan-auer
created_at: 2025-03-05T19:08:01Z
body: > * What are these files? Are these schema definitions for what data looks like on a kafka topic?

These files define minimum requirements for valid data on the topics required by the consuming side. The schema can be enforced in consumers. I'm not sure if Snuba uses them, at least Sentry does. @untitaker may have more information on that.

> * What is the difference between the 3: buffered_segment, ingest-spans, and snuba-spans? The only topic I am familiar with is the snuba-spans topic that snuba consumes spans off of.

`snuba-spans` is the only topic we use in production today. Relay produces and Snuba consumes. We're preparing a change in the pipeline that introduces two more intermediate steps, hence the new topics:
- Relay produces to `ingest-spans` and the trace buffer in Sentry consumes.
- The trace buffer produces to `buffered-segments` and a segment processor consumes.
- The segment consumer produces to `snuba-spans`, which Snuba/EAP consume from. 

There are differences between these topics, as each of the pipeline steps adds more attributes to the spans. From Snuba's perspective nothing changes, as Snuba will still consume the same payloads from `snuba-spans`. 

The reason for the change in this schema is that it was incorrect. Specifically, `parent_span_id` and `segment_id` can be `null`. My understanding is that the Snuba consumer already handles this case, as Relay can produce such values, but I'd ask for confirmation on that.
author_association: MEMBER