SCHEMA-DRIVEN DISTRIBUTED DATA PROCESSING

One embodiment of the present invention sets forth a technique for performing schema-driven data processing. The technique includes detecting a first change to a first producer schema for a first dataset produced by a first data processor. The technique also includes performing a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema includes a set of fields required by the second data processor. The technique further includes modifying an operation of the second data processor based on a result of the compatibility check.

BACKGROUND

Field of the Various Embodiments

Embodiments of the present disclosure relate generally to data processing platforms and, more specifically, to techniques for performing schema-driven distributed data processing.

DESCRIPTION OF THE RELATED ART

Increasing amounts of data are collected, stored, transported, and processed by organizations and other entities. For example, an organization could collect petabytes of data related to millions or billions of users or devices and store the data in tens or hundreds of thousands of datasets across multiple types of data stores. The organization could also query the data from the data stores and/or process the data within multi-stage data pipelines to generate recommendations, alerts, or other types of output or actions based on the data.

However, this increased collection, storage, transport, and processing of data has led to a corresponding increase in resource overhead and complexity in using, tracking, and auditing the data. Continuing with the above example, the organization may fail to track or otherwise manage the datasets and/or data pipelines. As a result, teams within the organization could generate the datasets and develop the data pipelines in isolation. Further, because a given team is unable to easily discover data that is generated or processed by other teams, the same types of data or data pipelines could be duplicated by multiple teams within the organization. This duplication of datasets or data pipelines increases the consumption of computational, storage, and network resources.

As the foregoing illustrates, what is needed in the art are more effective techniques for managing and tracking the generation and processing of data.

SUMMARY

One embodiment of the present invention sets forth a technique for performing schema-driven data processing. The technique includes detecting a first change to a first producer schema for a first dataset produced by a first data processor. The technique also includes performing a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema includes a set of fields required by the second data processor. The technique further includes modifying an operation of the second data processor based on a result of the compatibility check.

One technical advantage of the disclosed techniques relative to the prior art is that data processors and data schemas are created, tracked, and managed in a centralized manner. Accordingly, the disclosed techniques improve the discoverability and reusability of the data and/or data processors, compared with conventional techniques that lack a centralized mechanism for tracking data sources, data schemas, and/or data pipelines. The improved discovery and reuse of the data and/or data processors additionally reduces resource overhead associated with duplication of data sources and/or data pipelines, in contrast to conventional approaches that involve multiple teams or entities generating data or data processors in isolation. Another technical advantage of the disclosed techniques is that changes to the schema of a first data processor are automatically checked for compatibility with other data processors that depend on the first data processor. Schema changes that are compatible with another data processors can automatically be propagated to the other data processor, while schema changes that are incompatible with another data processor can be used to prevent the other data processor from incorrectly processing data produced by the first data processor. Consequently, the disclosed techniques can be used to efficiently operate and update data pipelines composed of multiple data processors. These technical advantages provide one or more technological improvements over prior art approaches.

DETAILED DESCRIPTION

Data stores such as databases and data warehouses are used to store increasing quantities of data across increasing numbers of datasets. Complex multi-stage data pipelines are also used to transport the data between data stores, convert the data to different formats, generate predictions or recommendations related to the data, and/or perform other types of processing related to the data. For example, an organization could store petabytes of data related to users, devices, events, sensor readings, and/or other entities across tens or hundreds of thousands of datasets. As new data sources (e.g., applications, devices, sensors, repositories, etc.) are added, the organization could create new datasets to store the data. The organization could additionally develop data pipelines to transport, aggregate, analyze, and/or otherwise process the data.

However, this increased collection, storage, transport, and processing of data has lead to a corresponding increase in resource overhead and complexity in using, tracking, and auditing the data. Continuing with the above example, the organization could lack a centralized mechanism for tracking the datasets and/or data pipelines. As a result, teams within the organization could generate the datasets and develop the data pipelines in isolation. Further, because a given team is unable to easily discover data that is generated or processed by other teams, the same types of data or data pipelines could be duplicated by multiple teams within the organization. This duplication of datasets or data pipelines increases the consumption of computational, storage, and network resources and diverts time and attention away from other tasks to be performed by the teams.

To address at least these issues, a distributed data-processing system includes multiple reusable and configurable data processors. Each data processor performs data-processing operations with respect to one or more input datasets to produce one or more output datasets. A series of data processors can also be linked within a data pipeline, so that the output of a given data processor is used as the input into the next data processor.

Each data processor produces or consumes data in accordance with one or more schemas. More specifically, each data processor includes a producer schema that represents the data generated by the data processor and/or a consumer schema that represents the data that is required for consumption by the data processor. When a producer schema for a first data processor changes, a controller in the distributed data-processing system performs compatibility checks between the producer schema and the consumer schemas of any other data processors that consume data generated by the first data processor. During the compatibility checks, the controller determines that the change to the producer schema is incompatible with a consumer schema when the change includes a field that has been removed from the producer schema and the same field is included in the consumer schema. After the controller identifies the change to the producer schema as incompatible with the consumer schema, the controller discontinues execution of the data processor associated with the consumer schema.

On the other hand, the controller determines that the change to the producer schema for the first data processor is compatible with a consumer schema for another data processor if the change to the producer schema does not interfere with consumption of the data generated by the first data processor by the other data processor. For example, the controller could determine that the change to the producer schema is compatible with the consumer schema for the other data processor if the change includes adding a field, renaming a field, and/or removing a field that is not included in the consumer schema. If the change to the producer schema is compatible with the consumer schema, the controller allows the other data processor to continue executing.

When the producer schema for the first data processor is compatible with the consumer schema for another data processor that consumes data generated by the first data processor, the controller selectively propagates some or all changes to the producer schema to the other data processor. If the other data processor is configured to “opt in” to schema propagations from the first data processor, the controller propagates all fields from the producer schema for the first data processor to another producer schema for the other data process. If the other data processor is configured to “opt out” of schema propagations from the first data processor, the controller propagates fields that are found in the consumer schema for the other data processor from the producer schema for the first data processor to the other producer schema for the other data processor.

One technical advantage of the disclosed techniques relative to the prior art is that data processors and data schemas are created, tracked, and managed in a centralized manner. Accordingly, the disclosed techniques improve the discoverability and reusability of the data and/or data processors, compared with conventional techniques that lack a centralized mechanism for tracking data sources, data schemas, and/or data pipelines. The improved discovery and reuse of the data and/or data processors additionally reduces resource overhead associated with duplication of data sources and/or data pipelines, in contrast to conventional approaches that involve multiple teams or entities generating data or data processors in isolation. Another technical advantage of the disclosed techniques is that changes to the schema of a first data processor are automatically checked for compatibility with other data processors that depend on the first data processor. Schema changes that are compatible with another data processors can automatically be propagated to the other data processor, while schema changes that are incompatible with another data processor can be used to prevent the other data processor from incorrectly processing data produced by the first data processor. Consequently, the disclosed techniques can be used to efficiently operate and update data pipelines composed of multiple data processors. These technical advantages provide one or more technological improvements over prior art approaches.

System Overview

FIG.1illustrates a system100configured to implement one or more aspects of the present disclosure. In some embodiments, system100is configured to perform distributed processing of data associated with a number of sources102(1)-102(X) and a number of sinks112(1)-112(Z). Each of sources102(1)-(X) is referred to individually as source102, and each of sinks112(1)-112(Z) is referred to individually as sink112.

In one or more embodiments, each source102and each sink112corresponds to a different data store. A given source102acts as an external source of data that is imported into and processed within system100. For example, sources102could include (but are not limited to) one or more “ground truth” data stores, such as relational databases, non-tabular databases, column stores, key-value stores, and/or other types of data stores that act as primary sources of data. A given sink112acts as an external recipient of data that has been processed within system100. For example, sinks112could include one or more of the same data stores as sources102. Sinks112could also, or instead, include other types of data stores, such as (but not limited to) data warehouses, analytics data stores, search engines, and/or other components for storing, retrieving, and/or managing datasets produced by system100.

In some embodiments, system100includes different types of data processors that implement various stages of data processing between sources102and sinks112. Each data processor receives one or more sets of input data and generates one or more sets of output data. As shown inFIG.1, these data processors include, without limitation, a number of source connectors104(1)-104(X), a number of sink connectors110(1)-110(Z), and a number of intermediate processors108(1)-108(N). Each of source connectors104(1)-104(X) is referred to individually as source connector104, each of sink connectors110(1)-110(Z) is referred to individually as sink connector110, and each of intermediate processors108(1)-108(N) is referred to individually as intermediate processor108.

Each source connector104retrieves data from a corresponding source102for subsequent processing within system100. For example, each source connector104could detect changes to a corresponding data store by reading from a transaction log for the data store on a continuous or periodic basis. Each source connector104could then write the changes as change data capture (CDC) events within system100.

Intermediate processors108perform processing of data from source connectors104and/or other intermediate processors108. For example, each intermediate processor108could retrieve input data generated by one or more source connectors104and/or one or more other intermediate processors108within system100. Each intermediate processor108could then perform aggregation, transformation, filtering, joining, windowing, partitioning, and/or other types of operations on the input data to generate one or more types of output data.

Each sink connector110performs writes related to data from system100to an external sink110. For example, each sink connector110could receive, as input, data generated by one or more source connectors104and/or one or more intermediate processors108. Each sink connector110could then write the data to a corresponding sink112, thereby replicating data from system100to the corresponding sink112.

Source connectors104, intermediate processors108, and sink connectors110are used to form a number of data pipelines for processing data within system100. Each data pipeline includes a series of data processing and data transport operations performed by one or more source connectors104, one or more optional intermediate processors108, and one or more sink connectors110. The source connector(s) import data from one or more sources102into the data pipeline, the intermediate processor(s) perform a series of data-processing operations on the data, and the sink connector(s) export the data-processing results to one or more sinks112.

In addition, source connectors104, intermediate processors108, and sink connectors110transport and process data within system100via a number of data streams106(1)-106(Y). Each of data streams106(1)-106(Y) is referred to individually as data stream106.

In one or more embodiments, data streams106are created and managed via a distributed streaming-processing platform. Within the distributed stream-processing platform, each stream106includes one or more sequences of messages that are identified by the same topic. A data processor that produces data within system100(e.g., source connector104or intermediate processor108) publishes the data as one or more streams106of messages to one or more topics. A data processor that consumes data within system100receives the data by subscribing to one or more topics and reading the messages published to the topic(s) by one or more other data processors. By decoupling generation of the messages by producers of data from receipt of the messages by consumers of the data, the distributed stream-processing platform allows topics, streams, and data processors to be dynamically added, modified, replicated, and removed without interfering with the transmission and receipt of messages using other topics, streams, and data processors.

Within the distributed stream-processing platform, each source connector104exports changes out of a corresponding source102by writing events (e.g., CDC events) that capture the changes to one or more topics within the distributed streaming platform. Each intermediate processor108subscribes to a given topic within the distributed streaming platform to receive data that is written to the topic by a source connector and/or a different intermediate processor. Each intermediate processor108also writes output data that is generated after processing the input data to one or more other topics within the distributed streaming platform. Each sink connector110receives data from one or more source connectors104and/or one or more intermediate processors108by subscribing to the corresponding topics. Each sink connector110then replicates the data on a corresponding sink112by performing writes of the data to the corresponding sink112.

While the operation of system100has been described with respect to streams106in a distributed stream-processing framework, those skilled in the art will appreciate that system100can use other types of frameworks or platforms to import, process, and/or export data. For example, system100could use a distributed messaging system, event-based monitoring system, CDC pipeline, batch-processing system, and/or another type of data transportation system to transmit data across source connectors104, intermediate processors108, and sink connectors110.

In one or more embodiments, source connectors104, intermediate processors108, and sink connectors110are configured for reuse by multiple entities. For example, source connectors104, intermediate processors108, and sink connectors110could be implemented by developers and include templates for configuration or customization by other users. System100could also provide a user interface for creating, updating, and/or managing source connectors104, intermediate processors108, sink connectors110, and/or data pipelines via the corresponding templates. Within the user interface, a user could search for existing source connectors104, intermediate processors108, sink connectors110, and/or datasets produced or consumed by the existing source connectors104, intermediate processors108, and/or sink connectors110. The user could also interact with the user interface to specify fields in a template that is used to configure a new source connector104, intermediate processor108, and/or sink connector110. The user could further interact with the user interface to create and/or modify a data pipeline by connecting graphical representations of one or more source connectors104, intermediate processors108, and/or sink connectors110with directed edges that denote the flow of data between the corresponding components. Consequently, each source connector104, intermediate processor108, and/or sink connector110can be created once and adapted for different uses by other users. Further, the user interface and templates for source connectors104, intermediate processors108, and/or sink connectors110allow the other users to configure source connectors104, intermediate processors108, sink connectors110, and/or data pipelines without requiring the other users to possess deep knowledge of the underlying data transport and/or data-processing frameworks.

As shown inFIG.1, system100additionally includes a controller114that is coupled to source connectors104, intermediate processors108, sink connectors110, and/or other components of system100. As described in further detail below, controller114performs centralized tracking and management of data processors in system100and schemas for data produced or consumed by the data processors. Consequently, controller114improves the reusability and discoverability of data and data pipelines within system100and minimizes overhead and disruptions caused by changes to the operation of individual components within system100.

FIG.2is a more detailed illustration of controller114ofFIG.1, according to various embodiments. It is noted that controller114described herein is illustrative and that any other technically feasible configurations fall within the scope of the present invention. For example, the hardware and/or software components of controller could be implemented on source connectors104, intermediate processors108, sink connectors110, and/or other components of system100. In another example, multiple instances of controller114may execute on a set of nodes in a data center, cluster, or cloud computing environment to implement the functionality of controller114.

As shown, controller114includes, without limitation, a central processing unit (CPU)202and a system memory204coupled to a parallel processing subsystem212via a memory bridge205and a communication path213. Memory bridge205is further coupled to an I/O (input/output) bridge207via a communication path206, and I/O bridge207is, in turn, coupled to a switch216.

In operation, I/O bridge207is configured to receive user input information from input devices208, such as a keyboard or a mouse, and forward the input information to CPU202for processing via communication path206and memory bridge205. Switch216is configured to provide connections between I/O bridge207and other components of controller114, such as a network adapter218and various add-in cards220and221.

I/O bridge207is coupled to a system disk214that may be configured to store content, applications, and data for use by CPU202and parallel processing subsystem212. As a general matter, system disk214provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROM (compact disc read-only-memory), DVD-ROM (digital versatile disc-ROM), Blu-ray, HD-DVD (high definition DVD), or other magnetic, optical, or solid state storage devices. Finally, although not explicitly shown, other components, such as universal serial bus or other port connections, compact disc drives, digital versatile disc drives, film recording devices, and the like, may be connected to the I/O bridge207as well.

In various embodiments, memory bridge205may be a Northbridge chip, and I/O bridge207may be a Southbridge chip. In addition, communication paths206and213, as well as other communication paths within controller114, may be implemented using any technically suitable protocols, including, without limitation, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

In some embodiments, parallel processing subsystem212includes a graphics subsystem that delivers pixels to a display device210, which may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. In such embodiments, parallel processing subsystem212incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. Such circuitry may be incorporated across one or more parallel processing units (PPUs) included within parallel processing subsystem212. In other embodiments, parallel processing subsystem212incorporates circuitry optimized for general purpose and/or compute processing. Again, such circuitry may be incorporated across one or more PPUs included within parallel processing subsystem212that are configured to perform such general purpose and/or compute operations. In yet other embodiments, the one or more PPUs included within parallel processing subsystem212may be configured to perform graphics processing, general purpose processing, and compute processing operations. System memory204includes at least one device driver configured to manage the processing operations of the one or more PPUs within parallel processing subsystem212.

In various embodiments, parallel processing subsystem212may be integrated with one or more of the other elements ofFIG.2to form a single system. For example, parallel processing subsystem212may be integrated with CPU202and other connection circuitry on a single chip to form a system on chip (SoC).

It will be appreciated that the system shown herein is illustrative and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs, and the number of parallel processing subsystems, may be modified as desired. For example, in some embodiments, system memory204could be connected to CPU202directly rather than through memory bridge205, and other devices would communicate with system memory204via memory bridge205and CPU202. In other alternative topologies, parallel processing subsystem212may be connected to I/O bridge207or directly to CPU202, rather than to memory bridge205. In still other embodiments, I/O bridge207and memory bridge205may be integrated into a single chip instead of existing as one or more discrete devices. Lastly, in certain embodiments, one or more components shown inFIG.2may not be present. For example, switch216could be eliminated, and network adapter218and add-in cards220,221would connect directly to I/O bridge207. In another example, display device210and/or input devices208may be omitted for some or all computers in a cluster.

In some embodiments, controller114is configured to run an analysis engine222and a management engine224that reside in system memory204. Analysis engine222and management engine224may be stored in system disk214and/or other storage and loaded into system memory204when executed.

Analysis engine222maintains a logical representation of dependencies across source connectors104, intermediate processors108, and sink connectors110in system100. Analysis engine222also stores and tracks schemas for data that is produced or consumed by each source connector104, intermediate processor108, and sink connector110. When a change is made to a given schema, analysis engine222uses the logical representation and schemas to perform compatibility checks between the schema and downstream components that are affected by the change.

Management engine224manages the deployment and execution of source connectors104, intermediate processors108, and sink connectors110within system100. More specifically, management engine224deploys, terminates, or configures the operation of source connectors104, intermediate processors108, and sink connectors110based on the corresponding configurations, schemas, and compatibility checks performed by analysis engine222. Consequently, analysis engine222and management engine224perform centralized tracking and management of source connectors104, intermediate processors108, and sink connectors110that supports the discovery, reuse, integrity, and update of data processors and data pipelines within system100. The operation of analysis engine222and management engine224is described in further detail below.

Schema-Driven Distributed Data Processing

FIG.3is a more detailed illustration of analysis engine222and management engine224ofFIG.2, according to various embodiments. As mentioned above, analysis engine222and management engine224track and manage the operation of a number of data processors302(1)-302(M) in system100. Each of data processors302(1)-302(M) is referred to individually as data processor302and can include a source connector104, intermediate processor108, and/or sink connector110.

Analysis engine222maintains metadata320related to data processors302. For example, analysis engine222could receive metadata320and/or updates to metadata320in real-time or near-real-time from data processors302, user interfaces for configuring data processors302, developers or administrators associated with data processors302, application-programming interfaces (APIs) associated with data processors302, and/or other sources. Analysis engine222could store metadata320in an in-memory cache, one or more files, a database, and/or another type of data store. Analysis engine222could also, or instead, process queries of metadata320, generate user interface output that includes metadata320, write metadata320to one or more files, and/or otherwise provide metadata320to users or other entities for the purposes of discovering, using, or auditing data processors302and/or data generated or consumed by data processors302.

As shown inFIG.2, metadata320includes a pipeline directed acyclic graph (DAG)322, a set of producer schemas324, a set of topic schemas326, and a set of consumer schemas328. Pipeline DAG322includes a logical representation of data pipelines within system100. Nodes in pipeline DAG322represent data processors302, and directed edges between pairs of nodes in pipeline DAG322denote the flow of data between the corresponding data processors302. Thus, a given pipeline can be identified within pipeline DAG322as a sub-graph that includes a sequence of directed edges from one or more source connectors104to one or more sink connectors110. The sequence of directed edges optionally connects the source connector(s) to one or more intermediate processors108and each intermediate processor108to one or more other intermediate processors108and/or one or more sink connectors110.

For example, a data pipeline could include one source connector104and one sink connector110. The source connector could export data out of a relational database into system100, and the sink connector could write the data to an analytics data store. The source connector and sink connector would be represented by a first node and a second node, respectively, in pipeline DAG322. The flow of data from the source connector to the sink connector would be represented by a directed edge from the first node to the second node within pipeline DAG322.

In another example, a data pipeline could include three source connectors104that export data out of three relational database tables into system100. The data pipeline could also include a first intermediate processor108that consumes the exported data and produces a union of the data and a second intermediate processor108that consumes the unioned data outputted by the first intermediate processor108and produces enriched data based on the unioned data and an external data source. The data pipeline could additionally include one sink connector110that writes the enriched data outputted by the second intermediate processor108to an analytics data store. Within pipeline DAG322, the three source connectors104could be represented by a set of three nodes, the first and second intermediate processors108could be represented by two nodes, and the single sink connector110could be represented by one node. The flow of data across the data pipeline could be represented by three directed edges from the three nodes representing the three source connectors104to the node representing the first intermediate processor108, one directed edge from the node representing the first intermediate processor108to the node representing the second intermediate processor108, and one directed edge from the node representing the second intermediate processor108to the node representing the sink connector110.

Producer schemas324include logical representations of data that is generated or outputted by data processors302, and consumer schemas328include logical representations of data that is consumed by data processors302. As a result, producer schemas324are defined for data processors302that output data within system100(i.e., source connectors104and intermediate processors108), and consumer schemas328are defined for data processors302that consume data within system100(i.e., intermediate processors108and sink connectors110). Each producer schema and each consumer schema includes (but is not limited to) a schema name, a schema namespace, one or more field names, one or more field types, and/or one or more primary keys. Each producer schema or consumer schema optionally includes a “nullability” attribute that identifies a field as capable or not capable of having null values.

Each of producer schemas324represents a dataset that is produced by a corresponding data processor302. For example, a producer schema for a given data processor302could specify field names and field types for fields included in messages written by the data processor to a corresponding stream106. The producer schema could also identify each field as required or optional and specify one of the fields as a primary key.

Each of consumer schemas328specifies a set of fields that is required for consumption by the corresponding data processor302. For example, a consumer schema for a given data processor302could include field names and field types for one or more fields that must be consumed by the data processor to generate output data and/or perform writes to an external data store (e.g., one or more sinks112).

In one or more embodiments, a given data processor302includes a separate consumer schema for each set of data that is consumed by the data processor (and therefore produced by an upstream data processor). For example, a data processor that consumes data produced by three upstream data processors302could include three consumer schemas328, one for each of the upstream data processors. Each consumer schema could identify one or more fields from the producer schema for the corresponding upstream data processor as required by the data processor.

As the operation of a given data processor302changes over time, the producer and/or consumer schemas for the data processor are updated to reflect these changes. For example, a data processor could initially be configured to output fields A and B at a first time t1. At a time t2>t1, the data processor could be updated to output fields A, B, and C. At a time t3>t2, the data processor could be updated to output fields A, B, and D. As a result, three different versions of a producer schema for the data processor could be included in metadata320. Each version of the producer schema is associated with a unique identifier, version number, or another attribute that differentiates the version from other versions of the producer schema for the same data processor. A first version of the producer schema could be associated with a time range of t1to t2and include fields A and B. A second version of the producer schema could be associated with a time range of t2to t3and include fields A, B, and C. A third version of the producer schema could be associated with a start time of t3and include fields A, B, and D.

In another example, a data processor could initially be configured at time t4to consume field E from another data processor. At a time t5>t4, the data processor could be updated to consume fields E and F from the other data processor. As a result, two different versions of a consumer schema for the data processor could be included in metadata320. Each version of the consumer schema is associated with a unique identifier, version number, or another attribute that differentiates the version from other versions of the consumer schema for the same data processor. A first version of the consumer schema could be associated with a time range of t4to t5and include field E. A second version of the producer schema could be associated with a start time of t5and include fields E and F.

As mentioned above, each data processor302that produces data within system100writes the data to a corresponding stream106that is identified by a topic. To track data written to individual topics in system100, analysis engine222aggregates one or more versions of each producer schema into a single topic schema (e.g., topic schemas326) for the corresponding topic. A given version of the topic schema is backward compatible with all versions of the producer schema up to the point at which the version of the topic schema was created. The version of the topic schema can also be used to read all messages in the topic that conform to the versions of the producer schema from which the version of the topic schema was created. As a result, the topic schema can be used to configure data processors302that consume from the topic and/or create consumer schemas for those data processors302.

For example, analysis engine222could receive three versions of a producer schema for a data processor over time. The first version includes a required field A and an optional field B; the second version includes the required field A, the optional field B, and a required field C; and the third version includes the required field A, the optional field B, and a required field D, respectively. Analysis engine222could also generate three corresponding versions of a topic schema for the topic that stores data associated with the producer schema. A field is listed as required in the topic schema if the field is present and required in all versions of the producer schema that are aggregated into the topic schema. A field is listed as optional in the topic schema if the field is optional in any version of the producer schema or is not present in all versions of the producer schema. Thus, the first version of the topic schema could be created based on the first version of the producer schema and include the same required field A and optional field B. The second version of the topic schema could be created based on the first and second versions of the producer schema and include required field A, optional field B, and optional field C. The third version of the topic schema could be created based on all three versions of the producer schema and include required field A, optional field B, optional field C, and optional field D.

In one or more embodiments, producer schemas324, consumer schemas328, and/or other metadata320associated with data processors302is referenced from or stored in the corresponding nodes in pipeline DAG322. For example, each node in pipeline DAG322could include one or more producer schemas324for data generated by the corresponding data processor302and/or one or more consumer schemas328for data consumed by the corresponding data processor302. Each node in pipeline DAG322could also, or instead, include an identifier, path, link, or another data element that can be used to identify or retrieve the producer and/or consumer schemas. In another example, each node in pipeline DAG322could store, link to, or otherwise identify a configuration for the corresponding data processor302.

As shown inFIG.3, analysis engine222uses metadata320to analyze the effects of schema changes318made to producer schemas324and/or consumer schemas328for individual data processors302on other data processors302in the same data pipelines. For example, schema changes318could be made to producer schemas324and/or consumer schemas328by users associated with the corresponding data processors302. Schema changes318could include (but are not limited to) adding or removing an optional field, adding or removing a required field, renaming a field, changing the field type (e.g., data type) of a field, and/or changing the primary key in a given producer or consumer schema.

As with other metadata320, schema changes318can be received in real-time or near-real-time from data processors302, user interfaces for configuring data processors302, developers or administrators associated with data processors302, APIs associated with data processors302, and/or other sources. After a schema change is received, analysis engine222validates the schema change, transmits an acknowledgment of the schema change, and updates the corresponding producer schema or consumer schema in metadata320.

Next, analysis engine222uses pipeline DAG322to determine any schema dependencies330associated with the schema change. In one or more embodiments, schema dependencies330include other data processors that are affected by the schema change. When a schema change is made to a producer schema for a given data processor302, analysis engine222traverses pipeline DAG322to identify schema dependencies330as any other data processors that consume data represented by the producer schema.

Analysis engine222also performs compatibility checks332that determine whether the schema change to a producer schema interferes with the consumption of data represented by the producer schema by downstream data processors302included in schema dependencies330. As shown inFIG.3, compatibility checks332are used to classify schema changes318as destructive changes340, incompatible changes342, and/or compatible changes344.

In one or more embodiments, destructive changes340include schema changes318to producer schemas324that disrupt the consumption of the corresponding data by all downstream data processors302. For example, analysis engine222could determine that a schema change to a producer schema is disruptive when the schema change includes changing a field type, changing a primary key, and/or making another type of change that interferes with the integrity or ordering of the resulting data.

While destructive changes340affect all downstream data processors302associated with a given producer schema, incompatible changes342and compatible changes344are determined with respect to individual data processors302that consume data represented by a change to a producer schema. A schema change to a producer schema is an incompatible change with respect to a downstream data processor302when the schema change interferes with the downstream data processor's consumption of the corresponding data. Conversely, a schema change to producer schema is a compatible change with respect to a downstream data processor302when the schema change does not interfere with the downstream data processor's consumption of the corresponding data.

In one or more embodiments, analysis engine222determines that a schema change that adds or removes an optional field, adds a required field, or renames a field in a producer schema is compatible with all consumer schemas for downstream data processors302that consume data represented by the producer schema. When a schema change removes a required field in a producer schema, analysis engine222determines that the schema change is compatible with a consumer schema if the field is not included in the consumer schema and incompatible with a consumer schema if the field is included in the consumer schema.

Analysis engine222also determines schema propagations334based on the results of compatibility checks332. In one or more embodiments, schema propagations334include propagation of some or all fields from a first producer schema for a first data processor to a second producer schema for a second data processor that consumes data generated by the first data processor. As a result, schema propagations334can be performed to automatically synchronize schema changes318made to producer schemas324with downstream data processors302, when these schema changes318are compatible with consumer schemas328for the downstream data processors302.

In some embodiments, analysis engine222determines schema propagations334based on configurations for data processors302that are included in schema dependencies330for a given schema change. Each configuration includes a flag or another type of attribute that specifies whether the corresponding data processor has “opted in” to schema propagations334from one or more upstream data processors. For example, a data processor could be set to “opt in” to schema propagations334from the upstream data processor(s) if the data processor performs a “pass through” of all fields from the upstream data processor(s) (e.g., if the data processor writes the fields to a sink or performs filtering of values in the fields). When a given data processor opts in to schema propagations334from the upstream data processor(s), a consumer schema that specifies the data processor's requirements for consuming data from the upstream data processor is not required.

On the other hand, a data processor could be set to “opt out” of schema propagations334from one or more upstream data processors if the data processor consumes a specific subset of fields from a topic to which the upstream data processor(s) write data (e.g., if the data processor performs projection, enrichment, and/or another operation based on the subset of fields from the upstream data processor(s)). When a given data processor opts out of schema propagations334, the configuration for the data processor includes a consumer schema that identifies the subset of fields from the topic that are consumed by the data processor.

If the data processor is set to “opt in” to schema propagations334and the consumer schema for the data processor is compatible with a given set of schema changes318to a topic schema for a topic consumed by the data processor, analysis engine222determines that all fields in the topic schema to which the set of schema changes318are made are to be propagated to the data processor. If the data processor is set to “opt out” of schema propagations334and the consumer schema for the data processor is compatible with a given set of schema changes318to the topic schema, analysis engine222determines that any schema changes318that apply to fields included in the consumer schema for the data processor are to be propagated from the topic schema to the data processor.

While the operation of analysis engine222has been described above with respect to schema changes318to producer schemas324or topic schemas326, analysis apparatus222can also determine schema dependencies330and perform compatibility checks332for schema changes318to consumer schemas328. For example, when a schema change is made to a consumer schema for a given data processor302, analysis engine222could traverse pipeline DAG322to identify schema dependencies330as a topic schema for a topic consumed by the given data processor302. Analysis engine222could also perform compatibility checks332to determine if the schema change made to the consumer schema renders the consumer schema incompatible with the topic schema. If the schema change includes adding a field to the consumer schema that is not found in the topic schema, analysis engine222could determine that the schema change is incompatible with the topic schema. If the schema change does not include adding a field to the consumer schema that is not found in the topic schema, analysis engine222could determine that the schema change is compatible with the topic schema.

In some embodiments, management engine224manages the deployment, execution, and termination of data processors302. First, management engine224deploys each data processor302with a fixed configuration that includes one or more producer schemas324for data produced by the data processor, one or more consumer schemas328for data consumed by the data processor, and a set of configuration fields that define the operation of the data processor and/or the data processor opting in or out of schema propagations334from upstream data processors. The fixed configuration ensures that the data processor processes only compatible data and produces data with a fixed producer schema. Management engine224can subsequently redeploy the data processor to change the configuration and operation of the data processor.

Management engine224also performs actions that address destructive changes340, incompatible changes342, and compatible changes344identified by analysis engine222. First, management engine224generates new topics310and corresponding topic schemas in response to schema changes318that are identified as destructive changes340. After management engine224creates a new topic and a corresponding topic schema in response to a destructive schema change to a producer schema, a data processor that produces data according to the producer schema can write data that reflects the destructive schema change to the new topic. At the same time, downstream data processors302that consume data produced by the data processor can use an older version of the topic schema to read and process messages304from an existing topic. Consequently, new topics310preserve the integrity and ordering of data produced by data processors302, even after destructive changes340are made to producer schemas324for these data processors302.

Second, management engine224carries out processor discontinuations312for data processors302that are associated with incompatible changes342. More specifically, management engine224discontinues the execution of any data processor with a consumer schema that is determined by analysis engine222to be incompatible with a corresponding upstream topic schema. Management engine224also generates an alert, notification, and/or other output that communicates the incompatibility to a developer, administrator, and/or another user associated with the discontinued data processor. These processor discontinuations312prevent the corresponding data processors from attempting to read or process data that is incompatible with the operation of the data processors.

Third, management engine224performs schema updates314that carry out schema propagations334associated with compatible changes344. As described above, analysis engine222determines that all fields from a producer schema for a data processor are to be propagated to a producer schema for a downstream data processor when the downstream data processor “opts in” to schema propagations334from the data processor and the topic schema for the topic to which the data processor writes data is compatible with a corresponding consumer schema for the downstream data processor. When both conditions are met, management engine224performs schema updates314that copy all fields in the producer schema for the upstream data processor to the producer schema for the downstream data processor.

Alternatively, analysis engine222determines that a subset of fields from a topic schema for the topic to which a data processor writes data are to be propagated to a producer schema for a downstream data processor when the downstream data processor “opts out” of schema propagations334from the data processor and the producer schema for the data processor is compatible with a corresponding consumer schema for the downstream data processor. In this instance, management engine224copies any schema changes318that apply to fields in the consumer schema for the downstream data processor from the producer schema for the data processor to the producer schema for the downstream data processor.

After schema updates314are made to the producer schema for a given data processor302, management engine224redeploys the data processor with schema updates314to allow the data processor to operate based on schema updates314. The redeployment of the data processor for an updated producer schema is detected by analysis engine222as another set of schema changes318to the producer schema for the data processor. Analysis engine222then repeats the process of determining schema dependencies330, performing compatibility checks332, and determining schema propagations334associated with the data processor and any downstream data processors, and management engine224performs actions that affect the downstream data processors based on the results of compatibility checks332and schema propagations334. Consequently, schema propagations334can be applied recursively by analysis engine222and management engine224across stages of a data pipeline until the end of the data pipeline and/or a data processor that has opted out of schema propagations334is reached.

Those skilled in the art will appreciate that a schema change can be made to a producer schema for a data processor before a downstream data processor that consumes data produced by the data processor is made aware of the schema change. More specifically, the data processor can begin writing messages that reflect the schema change before the schema change is propagated to the downstream data processor. As a result, the downstream data processor can encounter a message that conforms to the new producer schema before the downstream data processor is redeployed with a configuration that includes the new producer schema.

In one or more embodiments, each data processor302includes functionality to perform “schema-aware” message processing308that accounts for changes in upstream producer schemas306for messages304consumed by the data processor. This “schema-aware” message processing308is performed differently by data processors302that “opt in” to schema propagations334from the upstream producer schemas306and data processors302that “opt out” of schema propagations334from the upstream producer schemas306.

First, a data processor that opts out of schema propagations334performs message processing308of messages304associated with a new producer schema by attempting to deserialize messages304using a consumer schema in the configuration for the data processor. If the data processor is able to deserialize a message using the consumer schema (e.g., if the message includes all fields required in the consumer schema), the data processor continues processing the message using the configuration with which the data processor was deployed. If the data processor is unable to deserialize a message using the consumer schema (e.g., if the message does not include all fields required in the consumer schema), the data processor does not process the message. As mentioned above, if the new producer schema is incompatible with the consumer schema, management engine224discontinues execution of the data processor once the incompatibility is detected to prevent the data processor from incorrectly consuming messages304associated with the new producer schema. Consequently, both the data processor and management engine224include safeguards for preventing the data processor from consuming data that is incompatible with the consumer schema for the data processor.

Second, a data processor302that “opts in” to schema propagations334is deployed with a configuration that includes a whitelist of upstream producer schemas306for each upstream data processor302. If a message is associated with a new producer schema that is not in the whitelist, the data processor stops processing the message. This whitelisting of producer schemas on individual data processors302allows management engine224to “pause” processing of data that adheres to the new producer schema on a given data processor. During this pause, management engine224applies schema propagations334to downstream data processors302and redeploys the downstream data processors302with the corresponding schema updates314. Once the downstream data processors302have been redeployed with the new producer schema and are “ready” to accept messages that adhere to the new producer schema, management engine224redeploys the data processor with the new producer schema in the whitelist to enable processing of data that adheres to the new producer schema by the data processor. At the same time, the reconfiguration and redeployment of the data processor, downstream data processors302, and/or sinks112does not block the generation of messages that conform to the new producer schema by the upstream data processor.

FIG.4Aillustrates an exemplar set of schemas402-404and422-424associated with a first data processor, according to various embodiments. Schema402is a first version of a producer schema for the first data processor, and schema404is a second version of the producer schema for the first data processor. Schema422is a first version of a topic schema for a topic to which the first data processor writes data, and schema424is a second version of the topic schema. Schema402indicates that data produced by the first data processor up to a certain point includes a required field A, a required field B, and an optional field C. Schema404indicates that data produced by the first data processor after that point includes the required field A and optional field C, lacks field B, and has a new optional field named D. Schema422is identical to schema402, and schema424includes fields that are in both versions of the producer schema. Schema424indicates that field A is required to reflect the requirement of field A is required in both versions of the producer schema. Schema424also indicates that fields B, C, and D are optional because these fields are optional in one or both versions of the producer schema or are not present in all versions of the producer schema.

FIG.4Billustrates an exemplar set of schemas406-408associated with a second data processor that consumes data produced by the first data processor ofFIG.4A, according to various embodiments. Schema406is a consumer schema for the second data processor, and schema408is a first version of a producer schema for the second data processor.

As shown inFIG.4B, schema406indicates that field B is required for consumption by the second data processor. Schema408indicates that data produced by the second processor includes required field B and optional field C. Thus, while the first data processor is configured to produce data that conforms to schema402, the second data processor consumes the data outputted by the first data processor and produces additional data that conforms to schema408.

Because field B is required in schema402and missing from schema404, schema406is compatible with schema402and incompatible with schema404. This incompatibility can be determined by examining the topic schema424corresponding to schema404and determining that field B is not required in schema424. This incompatibility between schema406and schema404additionally causes the second data processor to discontinue execution after the first data processor produces data that conforms to schema404. Consequently, the second data processor lacks a second version of a producer schema that corresponds to schema404. The operation of the second processor can then be resumed after the incompatibility is resolved (e.g., by removing field B from schema406).

FIG.4Cillustrates an exemplar set of schemas410-412associated with a third data processor that consumes data produced by the first data processor ofFIG.4A, according to various embodiments. More specifically,FIG.4Cillustrates a set of schemas410-412for a third data processor that has “opted out” of schema propagation from the first data processor. As described above, this “opt out” can be specified in a configuration for the third data processor.

As shown inFIG.4C, schema410is a consumer schema for the third data processor, and schema412is a first version of a producer schema for the third data processor. Schema410indicates that field A is required for consumption by the third data processor. As a result, schema410is compatible with both producer schemas402and404for the first data processor.

Schema412includes a required field A and an optional field E. Because schema propagation from the first data processor to the third data processor is not performed, schema412differs from either of the producer schemas402or404for the first data processor. Further, the creation of a second producer schema404for the first data processor does not result in the creation of a corresponding producer schema for the third data processor. In other words, schema410-412indicate that the data produced by the third data processor is independent of the data produced by the first data processor, as long as the data produced by the first data processor is compatible with the required fields specified in the consumer schema410for the third data processor.

FIG.4Dillustrates an exemplar set of schemas416-420associated with a fourth data processor that consumes data produced by the first data processor ofFIG.4A, according to various embodiments. More specifically,FIG.4Dillustrates a set of schemas416-420for a fourth data processor that has “opted in” to schema propagation from the first data processor. As discussed above, this “opt in” can be specified in a configuration for the fourth data processor.

As shown inFIG.4D, schema416is a consumer schema for the fourth data processor, schema418is a first version of a producer schema for the fourth data processor, and schema420is a second version of the producer schema for the fourth data processor. Schema416indicates that field A is required for consumption by the third data processor. As a result, schema416is compatible with both producer schemas402and404for the first data processor. Further, schema416can be omitted while the fourth data processor opts in to schema propagation from the first data processor.

Schemas418and420include the same fields as the corresponding topic schemas422and424for the topic to which the first data processor writes data. As a result, schemas418and420reflect the automatic propagation of topic schemas422and424to the fourth data processor. Subsequent changes to the producer schema for the first data processor are also propagated to corresponding versions of the producer schema for the fourth processor via corresponding topic schemas for the topic, as long as the fourth data processor is configured to “opt in” to schema propagation from the first data processor and the topic schema for the topic to which the first data processor writes data is compatible with the consumer schema416for the fourth data processor.

FIG.5is a flow diagram of method steps for performing schema-driven data processing, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-3, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, analysis engine222detects502a change to a producer schema for a dataset produced by a first data processor. For example, analysis engine222could receive the change via a user interface, API, user, data store, and/or another mechanism associated with the first data processor. Analysis engine222could also, or instead, detect the change within a topic schema for a topic to which the first data processor writes data. The change could include (but is not limited to) adding or removing a required field, adding or removing an optional field, renaming a field, changing a field type of a field, and/or changing the primary key of the dataset.

Next, analysis engine222determines504if the change is destructive. For example, analysis engine222could determine that the change to the producer schema is destructive if the change involves changing a field type and/or primary key in the dataset. If the change to the producer schema does not involve changing a field type or changing a primary key in the dataset, analysis engine222could determine that the change is not destructive.

If the change is determined to be destructive, management engine224creates506a new topic for data associated with the schema change. Management engine224also creates a new topic schema for the new topic, where the new topic schema includes the change to the producer schema. After the new topic is created, management engine224, a user associated with the first data processor, and/or another entity can configure the first data processor to write the data to the new topic to prevent the data from interfering with the ordering or integrity of older data in the dataset. At the same time, other data processors that consume the dataset outputted by the first data processor are able to continue reading messages from an existing topic associated with the dataset using older versions of the producer schema.

If the change is determined to not be destructive, analysis engine222identifies508a set of additional data processors that consume the dataset. For example, analysis engine222could use a DAG and/or another logical representation of a data pipeline that includes the first data processor to identify the additional processors that consume the dataset produced by the first data processor.

Analysis engine222also performs510compatibility checks between the producer schema and a consumer schema for another data processor in the set to determine512if the change is compatible with the consumer schema. For example, analysis engine222could determine that the change is compatible with the consumer schema if the change involves adding or removing an optional field, adding a required field, renaming a field, and/or removing a required field that is not included in the consumer schema. Conversely, analysis engine222could determine that the change is incompatible with the consumer schema if the change involves removing a required field from the producer schema when the field is also included in the consumer schema.

If the change is incompatible with the consumer schema, management engine224discontinues514execution of the other data processor. A user associated with the first data processor and/or the other data processor can then resolve the incompatibility by changing the producer schema for the first data processor and/or changing the consumer schema for the other data processor.

If the change is compatible with the consumer schema, analysis engine222determines516if the other data processor has opted in to schema propagation from the first data processor. For example, analysis engine222could determine whether the other data processor has opted in to schema propagation from the first data processor by examining a flag, field, and/or another attribute in a configuration for the other data processor. If the other data processor has opted in to schema propagation from the first data processor, management engine224propagates518the change to the producer schema for the other data processor. For example, management engine224could apply the change to the producer schema for the other data processor, so that the producer schema for the other data processor matches the producer schema for the first data processor. Management engine224could also redeploy the other data producer with the updated producer schema to allow the other data producer to process the dataset according to the updated producer schema.

If the other data processor has opted out of schema propagation from the first data processor, management engine224does not automatically propagate the change to the producer schema for the other data processor. Instead, management engine224propagates the change to the producer for the other data processor if the change is made to a field that is listed in the consumer schema for the other data processor. For example, management engine224could propagate a change to a field name from the producer schema for the first data processor to the producer and/or consumer schemas for the other data processor when the field associated with the field name is included in the consumer schema for the other data processor. Management engine224could also redeploy the other data processor with the updated producer and/or consumer schemas to allow the other data processor to perform processing based on the updated field name.

After processing related to the first data processor and a given other data processor in the set is complete, analysis engine222determines520whether any processors remain in the set. If no other processors remain in the set, no additional processing related to the change is performed. If other processors remain in the set, analysis engine222and management engine224repeat operations510-518for each remaining processor to adjust the operation of the other data processor based on the change to the producer schema for the first processor.

In sum, the disclosed techniques perform schema-driven data processing via reusable and configurable data processors. Each data processor performs data-processing operations with respect to one or more input datasets to produce one or more output datasets. A series of data processors can also be linked within a data pipeline, so that the output of a given data processor is used as the input into the next data processor.

Each data processor includes a producer schema that represents data generated by the data processor and/or a consumer schema that represents data that is required for consumption by the data processor. When a change is made to a producer schema for a first data processor, a controller performs compatibility checks involving the change and the consumer schemas of any other data processors that consume data generated by the first data processor. During the compatibility checks, the controller determines that the change is incompatible with a consumer schema when the change includes a field that has been removed from the producer schema and the same field is included in the consumer schema. After the controller identifies a change to the producer schema as incompatible with the consumer schema, the controller discontinues execution of the data processor associated with the consumer schema.

During the compatibility checks, the controller can also determine that the change to the producer schema for the first data processor is compatible with a consumer schema for another data processor if the change to the producer schema does not interfere with consumption of the data generated by the first data processor by the other data processor. For example, the controller could determine that the change to the producer schema is compatible with the consumer schema for the other data processor if the change includes adding a field, renaming a field, and/or removing a field that is not included in the consumer schema. If the change to the producer schema is compatible with the consumer schema, the controller allows execution of the other data processor to continue.

When the producer schema for the first data processor is compatible with the consumer schema for another downstream data processor that consumes data generated by the first data processor, the controller selectively propagates some or all changes to the producer schema to the downstream data processor. If the downstream data processor is configured to “opt in” to schema propagations from the first data processor, the controller propagates all fields from the producer schema for the first data processor to another producer schema for the downstream data processor. If the downstream data processor is configured to “opt out” of schema propagations from the first data processor, the controller propagates fields that are found in the consumer schema for the downstream data processor from the producer schema for the first data processor to the other producer schema for the downstream data processor.

One technical advantage of the disclosed techniques relative to the prior art is that data processors and data schemas are created, tracked, and managed in a centralized manner. Accordingly, the disclosed techniques improve the discoverability and reusability of the data and/or data processors, compared with conventional techniques that lack a centralized mechanism for tracking data sources, data schemas, and/or data pipelines. The improved discovery and reuse of the data and/or data processors additionally reduces resource overhead associated with duplication of data sources and/or data pipelines, in contrast to conventional approaches that involve multiple teams or entities generating data or data processors in isolation. Another technical advantage of the disclosed techniques is that changes to the schema of a first data processor are automatically checked for compatibility with other data processors that depend on the first data processor. Schema changes that are compatible with another data processors can automatically be propagated to the other data processor, while schema changes that are incompatible with another data processor can be used to prevent the other data processor from incorrectly processing data produced by the first data processor. Consequently, the disclosed techniques can be used to efficiently operate and update data pipelines composed of multiple data processors. These technical advantages provide one or more technological improvements over prior art approaches.

1. In some embodiments, a computer-implemented method comprises detecting a first change to a first producer schema for a first dataset produced by a first data processor; performing a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema comprises a set of fields required by the second data processor; and modifying an operation of the second data processor based on a result of the compatibility check.

2. The computer-implemented method of clause 1, further comprising generating a topic schema for a topic based on one or more versions of the first producer schema for the first dataset; and transmitting the topic schema to the second data processor, wherein the topic schema is used by the second data processor to read one or more messages written to the topic by the first data processor.

3. The computer-implemented method of any of clauses 1-2, wherein generating the topic schema comprises specifying, within the topic schema, that a first field is required when the first field is required in each of the one or more versions of the first producer schema; and specifying, within the topic schema, that a second field is optional when the second field is not required in at least one version of the first producer schema.

4. The computer-implemented method of any of clauses 1-3, further comprising determining that a second change to a second producer schema for a second dataset is to be propagated to a third data processor that consumes the second dataset; and propagating the second change to a third producer schema for a third dataset produced by the third data processor.

5. The computer-implemented method of any of clauses 1-4, further comprising: deploying the third data processor with the second change propagated to the third producer schema for the third dataset; and based on the deployed third data processor, propagating the second change to a fourth producer schema for a fourth dataset produced by a fourth processor that consumes the third dataset.

6. The computer-implemented method of any of clauses 1-5, wherein determining that the second change is to be propagated to the third data processor comprises determining that the third data processor consumes the second dataset based on a logical representation of a data pipeline; and determining that the third data processor has opted into schema propagation from the second dataset based on metadata associated with the third data processor.

7. The computer-implemented method of any of clauses 1-6, wherein modifying the operation of the second data processor comprises determining an incompatibility between the first change and the first consumer schema; and in response to the determined incompatibility, causing execution of the second data processor to discontinue.

8. The computer-implemented method of any of clauses 1-7, further comprising outputting a notification of the incompatibility to an entity associated with at least one of the first data processor or the second data processor.

9. The computer-implemented method of any of clauses 1-8, wherein the first change comprises a removal of a field from the first producer schema.

10. The computer-implemented method of any of clauses 1-9, wherein the first producer schema and the first consumer schema comprise at least one of a schema name, a schema namespace, a field name, a field type, a field nullability, or a primary key.

11. In some embodiments, a non-transitory computer readable medium stores instructions that, when executed by a processor, cause the processor to perform the steps of detecting a first change to a first producer schema for a first dataset produced by a first data processor, wherein the first change comprises a removal of a field from the first producer schema; performing a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema comprises a set of fields required by the second data processor; and modifying an operation of the second data processor based on a result of the compatibility check.

12. The non-transitory computer readable medium of clause 11, wherein the instructions further cause the processor to perform the steps of determining that a second change to a second producer schema for a second dataset cannot be used with a first topic to which the second dataset is written; and creating a second topic associated with the second dataset, wherein a second processor that produces the second dataset writes one or more messages that reflect the second change to the second topic.

13. The non-transitory computer readable medium of any of clauses 11-12, wherein the second change comprises at least one of a change to a field type included in the second producer schema or a change to a primary key in the second producer schema.

14. The non-transitory computer readable medium of any of clauses 11-13, wherein the instructions further cause the processor to perform the steps of determining that a second change to a second producer schema for a second dataset is to be propagated to a third data processor that consumes the second dataset; propagating the second change to a third producer schema for a third dataset produced by the third data processor; and deploying the third data processor with the second change propagated to the third producer schema for the third dataset.

15. The non-transitory computer readable medium of any of clauses 11-14, wherein the third data processor stops processing of the second dataset after the second change is made to the second producer schema and before the third data processor is deployed with the second change propagated to the third producer schema.

16. The non-transitory computer readable medium of any of clauses 11-15, wherein modifying the operation of the second data processor comprises determining a compatibility between the first change to the first producer schema and a second consumer schema associated with processing of the first dataset by a third data processor; and in response to the determined compatibility, causing execution of the third data processor to continue.

17. The non-transitory computer readable medium of any of clauses 11-16, wherein the compatibility is determined based on an omission of a field associated with the first change from the second consumer schema.

18. The non-transitory computer readable medium of any of clauses 11-17, wherein modifying the operation of the second data processor comprises determining an incompatibility between the first change and the first consumer schema; and in response to the determined incompatibility, causing execution of the second data processor to discontinue.

19. The non-transitory computer readable medium of any of clauses 11-18, wherein the first data processor comprises at least one of a data source or a first intermediate data processor, and wherein the second data processor comprises at least one of a second intermediate data processor or a data sink.

20. In some embodiments, a system comprises a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to detect a first change to a first producer schema for a first dataset produced by a first data processor; perform a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema comprises a set of fields required by the second data processor; modify an operation of the second data processor based on a result of the compatibility check; determine that the first change is to be propagated to a third data processor that consumes the first dataset; propagate the first change to a second producer schema for a second dataset produced by the third data processor; and deploy the third data processor with the first change propagated to the second producer schema for the second dataset.