Event Platform/Schemas/Guidelines

This page intends to document conventions, guidelines and rules for designing and evolving Event Schemas.
= WMF Schema Repositories =
WMF maintains 2 separate git schema repositories:
* schemas/event/primary
* schemas/event/secondary
You can think of these as one single schema repository (and namespace), as softwares that use schemas at WMF generally use both. The only reason we have more than one schema repository is to allow for different git merge permissions. schemas/event/secondary is more permissive and should be used for schemas that are not used by critical production systems.
= Rules =
=== Required fields ===
All event schemas must have the following fields.
; $schema : A URI identifying the JSONSchema for this event. Its value should match an event schema $id in a schema repository. E.g. /schema_name/1.0.0
; meta : Event metadata sub-object. This field contains common metadata fields for an event. See also Event Platform/Flaws#meta field
; meta.stream : Name of the stream/queue that this event belongs in. This is used for routing incoming events to specific streams and downstream 'datasets'. This will likely correspond with a Kafka topic and a Hive table.
; meta.dt : This should be the event ingestion|processing|system time in ISO-8601 format. This will be filled in by EventGate or a producer library if a client does not set it. If you are using EventGate, you should leave this field blank in your event data and allow EventGate to set it to server side receive time.
; dt : This should be set by your client and represents the 'event time' of the event in ISO-8601 format. This is the actual time the event happened.
NOTE: As of 2020-12, these meta.dt and dt conventions are not fully adopted in existent schemas, but all new schemas should use these conventions. See T240460 and T267648.
See also: Event Platform/Schemas#Timestamps: meta.dt and dt, Event Platform/Schemas#meta.stream and Event Platform/Schemas#$schema.
=== No union types / No null values ===
Union types are not allowed. E.g. type: [string, integer] and type: [string, null] are not allowed. This means that null JSON values in data are not useful, as to set a field to null, its type would have to be exactly type: "null", which means that all values are null. See the Optional Fields Guidelines documentation below. See also https://json-schema.org/understanding-json-schema/reference/null.html.
Using oneOf is allowed, but only if you are using it for value validations, not for variable types. For example, the following is not allowed: 
oneOf:
 - type: object
 properties:
 f1:
 type string
 - type: object
 properties:
 f1:
 type: integer
This will fail, as the type of the f1 field is not deterministic.
=== No object additionalProperties ===
The full structure of your data must be known ahead of time. The exception to this is map type fields (see below), since maps fully specify the data types of keys and values.
=== arrays ===
Arrays must specify their items type, and all items must have exactly the same type.
array_field:
 type: array
 items:
 type: string
The array items type may be a complex object, as long as that object also explicitly declares all properties and types.
 
links_hovered:
 type: array
 items:
 type: object
 properties:
 link_url:
 type: string
 hover_time_ms:
 type: integer
Before using a complex type in an array, please read Event_Platform/Schemas/Guidelines#Complex_array_element_and_map_value_type_evolution_is_not_well_supported
=== Identifier Naming Rules ===
==== No Capital letters - Use snake_case ====
All identifiers (schema names, field names, etc.) should be in snake_case and should be all lower case. Event fields are often imported into case-insensitive RDBMS SQL systems. Mixing captial and lower case letters in e.g. Hive or MySQL table and field names can be confusing and cause issues in systems and code that access those SQL systems. 
Keys in map type objects may use mixed case. However, unless totally necessary to use capital letters it is recommended that your producer convert map keys to lowercase. Doing so will avoid annoying case insensitive multi-case SQL queries later.
See also: camelCase is bad
==== No special characters ====
Avoid using characters that would require special quoting in SQL systems. E.g. do not use hyphens '-', spaces ' ', '@', etc. in your schema or field names.
==== Acceptable identifier regex ====
jsonschema-tools uses this regex to ensure identifiers are properly named.
/^[$a-z]+[a-z0-9_]*$/
Avoid using the allowed dollar sign '$'. This is a special case used only for JSONSchema's $schema URI field.
== Backwards compatible modifications only ==
This means that the only allowed schema modification is to add new optional fields. 
==== No type changes ====
Type changes are the worst kind of backwards incompatible change. They can severely break downstream data consumers.
==== Do not remove fields ====
Field removals may cause downstream components to break.
While field removals may be possible if they are done carefully, the best practice is just not to do them.
==== Do not rename fields ====
Renaming a field is the same as removing a field and adding a new one, and removing fields is not allowed.
==== Do not delete schemas ====
Even if your producer code is no longer producing events, your event data might be out there somewhere, used by a consumer. We want to be able to always map from an event to its schema, and this is not possible if the schema has been deleted.
It may be possible to delete schemas with a lot of collaboration with ALL consumers of data, but the best rule is just not to do it.
=== Exceptions ===
==== Major schema version upgrades ====
It is technically allowed to make backwards incompatible schema changes between major version changes. However, the tooling does not support it. Doing so will require manual migrations for any producers and consumers (including downstream Hive tables) of streams that use the schema. 
See Event_Platform/Stream_Configuration#Stream_versioning for more details.
To make a change like this, please coordinate with all the users of your data. An incompatible schema change is like an incompatible API change, and should be handled accordingly.
==== Incompatible changes to schemas not yet in use ====
If your schema does not yet have a corresponding production declared stream, then making an incompatible change will be fine. There are no real usages of the schema anywhere, so there are no producers or consumers to migrate. In this case, you should probably edit the latest version of the schema directly. That is, edit the current.yaml file without changing the version in the schema's $id field, and rematerialize the versioned schema file with npm run build-new ./jsonschema/path/to/your_schema/current.yaml.
==== Incompatible changes to WMF analytics schemas ====
If you are sure that the only consumer usage of your schema is by a stream only used for analytics (i.e. in analytics-hadoop Hive), then the Data Engineering team can help you make your change. If there is data in the relevant Hive table(s), the tables may need to be dropped first. They can review your schema change, and then manually alter or drop the relevant Hive table(s).
= Conventions =
=== Optional / Missing fields ===
As long as a field is not in the list of required fields, it is 'optional'. There may be times when you don't want to set a field's value. To do so, just leave the field out of the event data when you send it. In downstream SQL systems (i.e. Hive), missing fields will be set to NULL during ingestion.
Note that null is not a valid value for any field, unless it has type: null, which is useless. See also: #No_union_types
=== examples ===
JSONSchema has an examples annotation. Your schema should include at least one example of an event that validates with the schema. Data Engineering uses these examples to produce canary (AKA heartbeat) events into event streams for monitoring of ingestion. jsonschema-tools tests will ensure that your examples validate with your schema.
You should always add at least one event example. If you don't, jsonschema-tools will generate one with random values for you. It is preferred that you add your own example.
=== datetimes / timestamps ===
All date time / timestamps should be serialized in JSON data as ISO-8601 datetime strings, ideally in UTC time, suffixed with the 'Z' (Zero) timezone qualifier. E.g. 2015-12-20T09:10:56Z.
Datetime fields should be named suffixed with '_dt'.
session_start_dt:
 type: string
 format: date-time
 maxLength: 128
Note the maxLength: 128. This is a security constraint that limits the amount of work a JSONSchema validator has to do in order to validate a format (all fields that use format or pattern will need to specify maxLength.
There may be cases where sending an integer unix epoch timestamp instead of an ISO-8601 string is required. In this case the field type should be integer, the timestamp should be sent as integer milliseconds, and the field name should be suffixed with '_ts_ms'. If you must send seconds instead of milliseconds, then you should suffix your field name with '_ts_s'.
session_start_ts_ms:
 type: integer
 description: Unix epoch in milliseconds
=== Elapsed time fields ===
If possible, time duration fields should be sent as integer milliseconds. Please suffix these fields with the time unit, e.g. '_ms', '_ns', '_s', etc.
page_preview_visible_time_ms:
 type: integer
 description: Time the page preview popup was shown to the user
=== map types ===
There may be times when you don't know all field names ahead of time. Since neither union types or schemaless object (AKA structs) are supported, you'll want to use a 'map type'. JSON serialization does not differentiate between a 'map' and an 'object', but downstream systems need to know the difference. A map type in JSONSchema can be represented by an object with an additionalProperties schema specified for which all properties have the same type.
map_field:
 type: object
 additionalProperties:
 type: string
This would allow you to emit event data like:
{
 "map_field": {
 "key1": "value1",
 "key2": "value2"
 "}
}
Complex value types are supported, e.g.
map_field:
 type: object
 additionalProperties:
 type: object
 properties:
 p1:
 type: string
 p2:
 type: integer
Which would allow you to emit event data like:
{
 "map_field": {
 "key1": {"p1": "foo", "p2": 123}, 
 "key2": {"p1": "bar", "p2": 456}
 }
}
In this example, the map_field has string keys, but complex struct (object) types.
==== Declaring specific fields in a map ====
Since JSON does not differentiate between and object and a map, and our JSONSchema convention is treat any object type with additionalProperties defined as a schema as a map, there is nothing preventing the declaration and specific keys in the map field. 
You might want to do this if you want to add value validation for specific map fields, or perhaps to improve documentation about commonly used keys in the map. To do this, declare the map as describe above, and also declare specific properties. The property types MUST match the additionalProperties type.
map_field:
 type: object
 additionalProperties:
 type: string
 properties:
 start_dt:
 type: string
 format: date-time
In this case, map_field can have arbitrary keys with string values, but if start_dt is one of those keys, it must be formatted as a date-time. The event will fail validation if map_field.start_dt is set but not in date-time format.
=== Complex array element and map value type evolution is not well supported ===
Because event data is often used in SQL systems, we need to make sure our changes to event schemas are supported by SQL systems. Unfortunately, modern SQL DDL (at least in Spark and Hive) does not support altering the 'type' of a nested 'struct' inside of an array or a map field.
If you choose to use a complex type in an array or map, you will not be able to change it later. If you put a nested struct field inside of a map value, you will not be able to add any fields to the inner struct in a backwards compatible way.
See T366487 Event Platform schemas should not support type changes to structs as array element or map value types for more details.
=== Modeling state changes ===
We model state changes by providing the current (new) state as well as the previous state in the same event, rather than attempting to provide a diff of the change. We do so by providing the current state as data fields in the main event body, and providing the previous state as a subobject called 'prior_state' with the same state fields from the main event body.
Example (from mediawiki/user/blocks-change):
 ### user/blocks_change specific fields
 blocks:
 description: >
 The current state of blocks for the target user of this user change event.
 type: object
 properties:
 name:
 description: Whether the name or IP should be suppressed (hidden).
 type: boolean
 email:
 description: Whether sending email is blocked.
 type: boolean
 user_talk:
 description: Whether the user is blocked from editing their own talk page.
 type: boolean
 account_create:
 description: Whether the user/IP is blocked from creating accounts.
 type: boolean
 expiry_dt:
 description: >
 The timestamp the block expires in ISO8601 format.
 If missing, the blocks do not expire.
 type: string
 format: date-time
 maxLength: 128
 prior_state:
 description: >
 The prior state of the entity before this event. If a top level entity
 field is not present in this object, then its value has not changed
 since the prior event. For user blocks changes, if prior_state is not
 present, then the User or IP did not have any existing blocks in place
 at the time this event was emitted. This does not mean this User
 or IP never had any blocks. It is possible that the User's block
 had automatically expired and were no longer in place when this event
 was emitted.
 type: object
 properties:
 blocks:
 description: >
 The prior state of blocks for the target user of this user change event.
 type: object
 properties:
 name:
 type: boolean
 email:
 type: boolean
 user_talk:
 type: boolean
 account_create:
 type: boolean
 expiry_dt:
 type: string
 format: date-time
 maxLength: 128
== Schema fragments ==
jsonschema-tools allows us to reference and include fields from other schemas by their schema URI. To keep dependencies easy to reason about, we only reference 'fragment' schemas stored in the fragment namespace/directory in our schema repositories.
* Events should never use fragment schema as their $schema URI
* Concrete (non fragment) schemas should only ever $ref fragment/ schemas
* Avoid making fields required in fragment schemas. Concrete schemas should be able to decide what fields are required
* Concrete (non fragment) schemas should define all fields at the root level, and only use allOf for $refs. E.g.
This is the preferred way of $refing and merging fragment schemas and defining concrete properties:
allOf:
 - $ref: /fragment/common/2.0.0#
properties:
 test:
 type: string
 default: default value
You may see some schemas that do this in a less than ideal way. Please avoid this:
allOf:
 - $ref: /fragment/common/2.0.0#
 - properties:
 test:
 type: string
 default: default value
== Frequently used fields ==
This section documents some fields that may appear in many different schemas. These fields may DRYed up into a $ref-able subschemas, and/or they may be moved into a 'Data Dictionary'.
=== http information ===
Events are often associated with a HTTP request/response. It can be useful for the event to record some information about the HTTP request and response. If you need to do this, for consistency we suggest $ref-ing the fragment http schema:
allOf:
 - $ref: /fragment/common/2.0.0#
 - $ref: /fragment/http/1.2.0#
Note that in that schema, client_ip is not captured. If you want to collect the client IP address of the producer of the event, the http.client_ip property is the appropriate place to do it. There is a separate fragment http/client_ip schema for this purpose. $ref it along with the fragment/http schema:
allOf:
 - $ref: /fragment/common/2.0.0#
 - $ref: /fragment/http/1.2.0#
 - $ref: /fragment/http/client_ip/1.0.0#
(See this discussion for more context.)
=== MediaWiki state fragments ===
==== Overview ====
In 2022-2023, we revisited some MediaWiki event data model conventions. Previously, reusable MediaWiki schema fragments were stored in fragment/mediawiki/**/common directories. While these fragments are still in use, they should not be used in new schemas. Instead, please use the fragment/mediawiki/state/entity and fragment/mediawiki/state/change fragments instead. See this README for more info.
The fragment/mediawiki/state/entity directory contains reusable schema fragments that provide basic data models for MediaWiki entities. As of 2023-06, these include:
* page
* user
* revision
* revision_slots
* content
* page_links
The fragment/mediawiki/state/change change directory has fragments that represent a state change of an entity (or entities). These exist so that they can also be used as reusable fragments, e.g. mediawiki/page/change and mediawiki/page/prediction_classification_change.
==== JSON Integer limitations and MediaWiki IDs ====
As part of T367856, MediaWiki IDs for revision, actor, and comment changed from UNSIGNED INT to UNSIGNED BIGINT. This change presents challenges, as our JSON serialization of events depends on JavaScript's support for integers.
As an example, the revision fragment currently defines the rev_id field as:
 rev_id:
 description: The (database) revision ID.
 type: integer
 maximum: 9007199254740991
 minimum: 1
The 9007199254740991 value comes from Number.MAX_SAFE_INTEGER and is added to the jsonschema maximum via the jsonschema-tools enforcedNumericBounds config. MAX_SAFE_INTEGER is way larger than max UNSIGNED INT, which is the prior max for MediaWiki, but way smaller than the new UNSIGNED BIGINT. We do have a long runway before this is a real problem, as the highest recorded revision id for one of our biggest wikis, wikidatawiki, is currently halfway to UNSIGNED INT:
 Highest rev for wikidatawiki: 2274147444
 UNSIGNED INT max: 4294967295
 MAX_SAFE_INTEGER: 9007199254740991
 UNSIGNED BIGINT max: 18446744073709551615
In the future, we may want to consider using a JavaScript BigInt instead, which has support on recent version of this language, and allows JSON serialization via STRINGs.
See more details of this at T379703.
=== Common analytics fields ===
In the schemas/event/secondary repository, there are several fragment subschemas for standardized fields in analytics events (e.g. identifiers such as device_id, session_id, pageview_id and many others).
See Event Platform/Analytics/Fragments for documentation on these.
== Automatically populated fields ==
If your events are produced through a Wikimedia EventGate service (as most events are), some fields can be auto-populated if they are present in the schema, but not present in the event. This is done through custom code in the eventgate-wikimedia repository. For up to date info on what fields are populated, read the code documentation for the makeSetWikimediaDefaults function there. As of 2020-10, if your schema has the following properties defined, but the fields are not present in the event, they will be populated by EventGate as follows:
; $schema : To value of schema_uri query param if provided in HTTP request.
; meta.stream : To value of stream query param if provided in HTTP request.
; meta.dt : To current ISO-8601 UTC timestamp
; meta.id : To new uuid
; http.client_ip : To value of X-Client-IP request header if set.
; HTTP request headers specified by producers.eventgate.enrich_fields_with_http_headers stream config setting: See Event Platform/Stream Configuration#enrich fields with http headers for details.
== Event Data Modeling and Schema Naming ==
Data modeling is hard, and these guidelines are intentionally vague about proscribing any data modeling rules. It'd be difficult to cover all the use cases out there. However, there is one guideline we'd like to note. We are modeling events here, so we should try to design these schemas as events!
An event should represent something happening at a specific time, so it is good practice to name your event schema as an action happening to something.
We've got many examples where we've named schemas after the entity (noun) and verb the event represents. E.g. mediawiki/api/request: a MediaWiki API endpoint was requested. Or, mediawiki/user/blocks-change: a MediaWiki User's blocks have changed. 
When attempting to model a user's interaction with a website, perhaps you'd choose to model an entity as a specific interface (a button, and text-input-element, a sidebar, a popup) and then describe the action with a verb (button/clicked, text-input-element/entered, popup/displayed or popup/hidden). Or you might choose to use the same schema to model several types of state changes to the same entity, e.g. popup/visibility-change with a field action: displayed or action: hidden. Either of these ways is fine, as are many variations on them. What is important is that an event data model and schema name represent a real event happening.
When you name a schema, the question 'What is a <schema_name> event?' should make sense. E.g. What is a 'user/create' event makes sense. What is a 'user' event does not. (What would it mean for a user to be an event?)
Examples of good event model / schema names
* entity create, entity delete, entity state change
* button click, button hover, UI element display
* interface feature user interaction
* funnel user state change
* search request
Examples of bad event model / schema names
* mobile app
* user
* page
* recommendation
Category:Event Platform