Contributions and parts data pipeline framework

Techniques for managing a parts catalog of part objects corresponding to media content items are provided. Upon receiving a request to build a new part object for a part provider management system for a new content item, one embodiment collects a plurality of contribution references corresponding to a plurality of contributions relating to the new content item. A new part object is built based on the plurality of contributions and using a part creation graph corresponding to a type of the new content item. The new part object is published to a parts catalog for the part provider management system, for consumption by one or more downstream systems.

BACKGROUND

The present invention relates to a data ingestion framework, and more specifically, to a data ingestion framework for ingesting data for a digital video streaming service.

Modern digital video streaming service offer digital video streaming for a substantial number of different content items. Additionally, such a digital video streaming service may maintain a variety of different files for each of these different content items, resulting in a substantial number of files that need to be maintained and organized. For example, for a given digital content item, multiple different video encodings may be maintained in various encoding rates and formats. Moreover, different audio tracks in different languages and even in different encoding rates and formats may also be maintained, as well as corresponding subtitle files and various metadata files. Complicating the problem even further, different geographic regions may have their own video files and/or audio files, due to national or regional preferences and cultures. As a result, digital video streaming services may contain significantly more data than can be manually verified. However, as an erroneous encoding or a mislabeled or misclassified file (e.g., an audio track corresponding to a particular episode of a show being mislabeled as the audio track for a different episode of the same show) can severely impact the user experience, as these issues can render the streaming digital video content unwatchable by users.

DETAILED DESCRIPTION

Digital video streaming services can provide digital video streaming services for a vast catalog of different content items. Digital video streaming services may also maintain a variety of different files for each of these different content items, resulting in a substantial number of files that need to be maintained and organized. As a given digital content item may have multiple different video encodings in various encoding rates and formats, different audio tracks in different languages and even in different encoding rates and formats, as well as corresponding subtitle files and various metadata files, even a single content item can be associated with a significant number of different files. Complicating the problem even further, different geographic regions may have their own video files and/or audio files, due to national or regional preferences and cultures, adding to the number of files that need to be maintained. As an erroneous encoding or a mislabeled or misclassified file (e.g., an audio track corresponding to a particular episode of a show being mislabeled as the audio track for a different episode of the same show) can severely impact the user experience, since these issues can negatively affect the streaming digital video experience or even render the digital video content unwatchable by users, a technical problem clearly exists in storing and organizing these various files associated with various content items. Such a technical problem is amplified, as the content library for a digital video streaming service may change over time, as content items are added to or removed from the content library.

As such, embodiments described herein provide a part provider management system that can provide an Application Programming Interface (API) configured to receive input documents in any schema and persist them to configurable storage locations. In one embodiment, the part provider management system can validate the input documents with syntax and light semantic checks. Additionally, the part provider management system can send notifications to one or more subscribers, informing the subscribers when the data has arrived (e.g., when an input document has been successfully ingested into the system). Embodiments described herein can allow upstream contributors to provide interesting video-related data asynchronously from the development work needed to process that data and present it to the streaming customer.

In one embodiment, the part provider management system can receive a new schema or schema version and can validate the new schema or schema version. Upon successfully validating the new schema or schema version, the part provider management system can store the new schema or schema version at a schema management system. The part provider management system can subsequently use the new schema or schema version to validate input documents (or, more generally, new content) using the new schema or schema version stored at the schema management system. By providing a schema management system and requiring content providers and other contributors to register schemas for their input documents, embodiments can enable content providers and other contributors to add new schemas and modify existing schemas over time, and can use the defined schemas to validate submitted content of the corresponding type.

Upon successfully validating the new content, the part provider management system can store the new content (or input document) at a content storage system. The part provider management system can then initiate a workflow to create a new part object representing the new content item within a parts catalog of the part provider management system. In doing so, the part provider management system can collect a number of contributions relating to the new content and the part provider management system can filter these contributions down to a subset of contributions. Additionally, the part provider management system can provide one or more validation operations (or a sequence of validation operations, such as a high-level validation operation followed by a deep validation operation) on the contributions in the subset of contributions to further filter out any invalid contributions. The part provider management system could then build a part object from the remaining contributions and can publish the new part object to a parts catalog, for consumption by one or more downstream systems. Doing so provides a framework through which content producers and other contributors can submit new content for ingestion and this content (and any related contributions) can be represented within the parts catalog using a part object(s) for consumption by downstream consumers (e.g., video streaming service applications, advertising systems, digital assistant services, etc.), in a manner that provides for the automatic validation of the content (e.g., using the pre-registered schema(s)) and that standardizes the intake process for content for all downstream consumers of the parts catalog.

FIG. 1Ais a block diagram of a system that includes a part provider management system, according to one embodiment described herein. As shown, the system100illustrates a plurality of content provider systems110, a content provider API115, a schema management system120, a content storage system125, a part provider management system130and a parts catalog. The illustrated workflow begins at operation112, where one of the content provider systems110submits to the content provider API115new content for ingestion by the part provider management system130. In one embodiment, the content provider systems110can submit a link, reference or other identifier through which the new content can be downloaded (directly or indirectly) to the content storage system125. Additionally, the content provider system110can register a new schema or schema version with the schema management system120(operation114). For example, such a schema could be expressed using an extensible markup language (XML) format or, more generally, any suitable format for representing a schema, consistent with the functionality described herein. Upon receiving the new schema or schema version, the schema management system120can perform a validation operation to confirm that the registration request and the new schema or schema version are syntactically correct, and can then store the new schema or schema version for subsequent use.

Upon receiving the submission of new content for ingestion, the content provider API115can perform a validation operation together with the schema management system120to validate the new content's schema (operation116). For example, the content provider API115could retrieve the corresponding, pre-registered schema from the schema management system120and could confirm that the new content conforms to the syntax specified in the corresponding schema. Generally, each content provider can define their own schema(s) for use with their content, and can further define different schemas for use with different types of content. For example, a particular content producer (e.g., a movie studio) could define a first schema for use with source video content from the particular content producer, and could define a second schema for use with subtitle content from the particular content producer. Continuing the example, a different content producer could define a different schema for use with their source video content and a different schema for use with their subtitle content, and so on. Once the content provider API115confirms that the new content conforms with the corresponding schema, the content provider API115stores the validated new content in the content storage system125(block118).

In one embodiment, the content provider API115can initiate an operation to store the validated new content in the content storage system125, but in some instances such an operation (e.g., a download of the new content across a secure Internet channel) can take a significant amount of time to complete. As such, the content provider API115can allow one or more other components to subscribe to notifications relating to the new content, and the content provider API115can publish a notification to all such subscribers when the copy operation is completed.

The content provider API115can initiate an operation with the part provider management system130to generate a new part object within the parts catalog145for the ingested content (operation122). The part provider management system130contains a processor(s)132and a memory134, which in turn contains a part provider management application135and an operating system142. The part provider management application135contains a part creation API137and a part provider visibility API140. In one embodiment, the content provider API115initiates the operation122by making one or more calls to the part creation API137of the part provider management application135. In a particular embodiment, the part creation API137can determine that the storage operation118of the new content has not yet completed, and the part creation API137can subscribe to notifications relating to the storage of the new content from the content provider API115.

Upon determining that the new content has been successfully stored within the content storage system125(e.g., upon receiving a notification from the content provider API115), the part creation API137can initiate an operation to create a new part object for the parts catalog145that represents the new content. In one embodiment, the operation to create the new part object specifies as an input a reference to the storage location of the new content within the content storage system125, rather than using the raw data of the new content. For example, the part creation API137could initiate the new part object creation process using a path to the new content stored in the content storage system125, rather than the raw data of the new content itself (which could be very large in size).

Once the new part object is built, the part creation API137publishes the new part for the new content to the parts catalog145(operation124). Throughout the process of building the new part object, the part provider visibility API140can provide one or more exposed functions that can be called (e.g., by the content provider API115) to determine a status of the part object creation workflow. For example, one of the content provider systems110could transmit a status request to the content provider API115, which in turn could submit a status request to the part provider visibility API140of the part provider management application135. The part provider visibility API140could then return a current status of the creation of the new part object, which could in turn be returned by the content provider API115to the content provider system110.

In one embodiment, once the new part has been created and published to the parts catalog145, the part is made available for consumption by one or more downstream systems or consumers. As used herein, downstream systems or consumers refer to software applications and/or computer systems that are allowed to access the parts catalog145to retrieve part objects and their corresponding data. For example, in the event the new part corresponds to new video content (e.g., encoded video data for an episode of a particular show), a downstream consumer such as a digital video streaming service could consume the published new part and its associated information to populate a graphical user interface for the digital streaming service site and even to retrieve video data for transmission to streaming video clients. As another example, a downstream web application server for a web site that features descriptions and other attributes of movies and other video content could consume the part object to extract relevant information for use in populating web pages for the site. More generally, however, any number of different types of downstream consumers can make use of the new part object, consistent with the functionality described herein.

FIG. 1Bis a block diagram of a system that includes a part provider management system, according to one embodiment described herein. As shown, the system150depicts one more content providers110submitting files and associated metadata to a video central/reception service152. The video central/reception service152provides partner title (PT) information and contribution information to the content ingestion services152. Contribution information can include, without limitation, any information that can influence the distributed catalog, such as rights information, metadata contributions, asset contributions and so on. In one embodiment, the PT represents information that uniquely identifies a given content entity (e.g., an episode of a show, a movie, etc.) within a content provider namespace.

The content ingestion services154transmit the PT information to an entity manager component156. The entity manager component156is configured to generate a reconciled title (RT) based on the received PT, and transmits the RT to the entity mappings data store172. In one embodiment, the RT represents information that uniquely identifies the given content entity corresponding to the PT, within the entity manager namespace. The content ingestion services154further transmit the contribution information to the contributions data store170. The entity manager156can then retrieve this contribution information, as well as any related contribution information, from the contributions data store170.

In the depicted embodiment, an entity matcher component168is configured to retrieve the contributions information from the contributions data store170and transmits the contributions information to a title matching service166. Generally, the title matching service166is configured to identify entities that correspond to the same title. For example, video data for a particular movie could be submitted multiple times (e.g., by different providers110), and the entity manager156could represent each of these separate submissions with a separate entity. The title matching service166could determine that these entities are related with one another based on the contributions information for the entities; for example, the title matching service166could determine that two entities are related based on comparing the contributions information for the two entities and identifying similarities in the contributions information. The title matching service166can return match results specifying any existing entities that are determined to match the new entity to the entity matcher component168, and the entity matcher component168can store the matching entities and contributions information within the contributions data store170.

In one embodiment, the title matching service166is configured to receive normalized title records belonging to a catalog as inputs. The title matching service166can match related title records using machine learning techniques asynchronously, and the title matching service166can publish the results as a cluster of related records. In one embodiment, the title matching service166can use machine learning techniques to identify records that match together in work and expression levels.

Additionally, a part provider component174can retrieve the contributions information from the contributions data store170and can retrieve the RT information from the entity mappings data store172. The part provider component174can use the contributions information and RT information to generate a new part object for the content, and the part provider component174can store the new part object in the part store145. The new part object and RT information can be published by the publishing component158to a title release component160. The title release component160can further publish the new part object to a vending component162, and ultimately the new part object can be consumed by the downstream consumers164.

FIG. 2is a block diagram illustrating components of a part provider management application, according to one embodiment described herein. As shown, a plurality of contributing systems210can interface with an input API220(e.g., the part creation API137) provided by the part provider management application135. The Input API220in turn interfaces with a Dev Tools Website215as well as a Document Store Service225. The Dev Tools Website215can provide operational tooling for users to view and manage contributions objects and related workflows. The Document Store Service225, in turn, interacts with an Entity Manager230, a Listener component250of the part provider management application135and a Common Activities Workflows245. When interfacing with the Listener component250, the Document Store Service225may publish Contribution and Reconciled Entity notifications235to one or more subscribers. The Common Activities Workflows245co-manages a plurality of Part Creation Graphs275, and interfaces with a Feedback Service255. The Feedback Service255interfaces with Domain Activities Workflows260, which also co-manage the Part Creation Graphs275together with the Common Activities Workflows245and the Listener250. The Domain Activities Workflows260interface with Domain Services265and engage in selection and part building operations270.

In the depicted embodiment, the Part Creation Graphs275can correspond to a number of different types of content and can define a variety of types of information. For example, as such, the Part Creation Graphs275can be used for part generation for Video on Demand (VOD) video, live video, and images, and can define information such as base metadata, genre, offers and so on.

In one embodiment, the Content Provider API115implements the Ingest stage310of the workflow shown inFIG. 3.FIG. 3Ais a workflow for generating a new part object within a parts catalog for a new content entity, according to one embodiment described herein. Generally, an entity refers to an abstract concept that represents an entry in the catalog of the part provider management application135. For example, an entity could be a traditional instance of digital video content (e.g., an episode of a show, a movie, etc.), a live event, a real-world object (e.g., a sports team, a player, etc.) or an aggregation of the above or other entities. One embodiment provides three types of entities: a Partner Entity (PE), a Reconciled Partner Entity (RPE) and a Reconciled Entity (RE).

The Input API220can allow upstream contributing systems210to create and inspect contributions. In one embodiment, a contribution refers to a set of tightly-coupled, highly-cohesive attributes that are delivered as a unit by a contributor via a contributing system210. In a particular embodiment, an individual contribution is treated as immutable and is associated with exactly one PE. A PE represents a single Partner's view of an entity, and a PE can encapsulate one external identifier (e.g., a particular alias plus a stock keeping unit (SKU) value, paired with a content identifier). In one embodiment, all contributions recognized by the part provider management application135are associated with a PE. In a particular embodiment, two or more different contributing systems210may submit to the same PE. Generally, each PE will have one parent RPE value, and can be referenced by a globally-unique PE identifier (PEID) value.

In one embodiment, a RE refers to a node in a customer-facing browse graph (e.g., a television (TV) series, a TV season, a TV episode, a movie, a trailer video, video of a sports tournament, video of a sporting event, video of an event highlight, an actor, an athlete, etc.). In one embodiment, meaningfully-different versions of an entity (e.g., a Director's Cut and a Theatrical Version of the same movie) can be modeled using separate REs. Generally, an RE will have one or more child RPEs and can be referenced using a globally-unique RE identifier (REID) value.

In one embodiment, a RPE represents a rights-centric version of a RE with fundamentally the same video content. In such an embodiment, multiple unique video files can be considered fundamentally the same when the only variations between the video files would not typically be noticed by viewers. In one embodiment, a single RPE may have multiple videos with variations such as different legal bumpers (e.g., a legal warning about the unauthorized copying of video content), content overlays (e.g., no smoking symbols in certain geographic locations), censored scenes (e.g., which can vary, depending on geographic region, nationality and so on) or device-targeting variations (e.g., different aspect ratios). Generally, a RPE will have a single parent RE and one or more child PEs, and an RPE can be referenced by a globally-unique RPE identifier (RPEID) value.

Returning toFIG. 3, the ingestion stage310generally encapsulates the operations involved in or closely related to receiving inputs into the pipeline300. In one embodiment, contributing systems310, acting on behalf of contributors, can create new PEs and contribution documents by interfacing, directly or indirectly, with one or more APIs (e.g., content provider API115, part creation API137, etc.). As discussed above, content provider systems110can upload asset files, metadata, and other information, and the content provider API115can validate such files and facilitate their storage in the content storage system125. The content provider API115can also provide functions through which the content provider systems110can submit corrections for mistakes or provide missing attributes for high priority content.

As discussed above, the content provider systems110can interface with the schema management system120to define schemas. Additionally, these content provider systems110may be empowered to iterate on those schemas to provide new attributes. As discussed above, the content provider API115can use such schemas to perform validation for contribution payload syntax during ingestion. Semantic validation that is limited to the contribution document itself (e.g., offer end date is after the start date) can also be defined for specific schemas.

In one embodiment, asset, metadata, and rights contributions use a PEID as the partition key. In such an embodiment, entity relationship contributions can use a concatenated pair of PEIDs. In various other embodiments, contributions may be keyed by REID, RPEID, file URI, contract ID, or other non-PEID values.

Contribution unique IDs may vary across content provider systems110. In one embodiment, all contributions will include the contributor identity as the prefix, to avoid multiple contributors overwriting each other's contributions for the same PE. In a particular embodiment, contributions can also include the territory string (i.e., a unique value identifying a geographic territory) to, for example, facilitate scalable partitioning in the Collect stage. The ID may also include domain-specific discriminators, e.g., metadata contributions may include locale, assets will include a file Uniform Resource Identifier (URI), and rights contributions can include a type of the offer in question. These domain discriminators may be used when a contributor provides multiple contributions with the same schema and partition key, but different data targeting different sub-elements of the target entity.

In one embodiment, the Ingest stage310uses the schema configuration store of the schema management system120to decouple content provider systems110from internal details of contribution persistence. Each schema can be configured with a Lambda function to transform the contribution payload into the partition key and unique ID. The destination domain and library can also be retrieved from the schema configuration. Such encapsulation allows the contributing system to only provide the payload and contributor identity.

The part provider management application135can begin the Listen stage320when an input action is taken by a contributing system210, e.g., a contribution is created, updated, or deleted, or the relationship between a Partner Entity, Reconciled Partner Entity, or Reconciled Entity changes. To determine when such input actions are taken, the part provider management application135can subscribe and process many notification queues across multiple domains and libraries. When a notification is received, the part provider management application135can resolve the target entity from the triggering notification, query for previously-published parts, and choose the correct domain-specific code path, before proceeding to the Collect stage330.

Generally, the target entity refers to the RE or RPE for the entity closure that is relevant to this workflow instance. For example, if the target entity is an alias (i.e., not the primary RE or RPE), the part provider management application135can determine that there is only a single next step before the workflow is terminated, e.g., a workflow where any existing parts published to the merged entity are deleted. Doing so ensures that downstream systems do not continue to use these parts as they become increasingly out-of-date, versus the parts published to the primary entity.

For primary entities, the part provider management application135can avoid race conditions through the use of an optimistic lock. In doing so, the part provider management application135can query for all published parts under the target entity and store the version strings for later use in the workflow. The part provider management application135can also use the set of territory strings from the parts to ensure they are unpublished, even if all their contributions have been deleted.

Generally, different workflows require different code paths with different owners. For example, an “image language” contribution could be routed to an image part creation code path, while a “genre” contribution could go to a metadata part creation code path. The Listen stage320generally encapsulates the configuration involved in such routing. In one embodiment, the Listen stage ends by initiating further processing in the Collect stage330. Doing so guarantees the execution of at least one workflow that incorporates all contributions for a target entity, regardless of the order of ingestion.

In one embodiment, the Collect stage330begins with the part provider management application135processing the target entity identifier (RPEID or REID) and the territories of existing parts. Generally, the Collect stage330involves collecting all contributions and subdividing them by territory. As part of the Collect stage330, the part provider management application135can next perform a strongly-consistent query against a document store (e.g., content storage system125). Such a two-stage query can include all contributions in all relevant contribution domains for all PEs under the target entity. In one embodiment, such a query ignores deleted contributions.

One challenge of the Collection stage330is scaling to the number of contributions. A single RPE may have only a handful of PEs, but each PE could have thousands of contributions. For example, for a single image, 200 locales, with 10 variations per locale, and 5 attributes per image and 1 contribution per attribute can result in 10,000 different contributions for the single image. The part provider management application135can simplify the Collection stage330by only collecting contribution references, without collecting the entire payload. In one embodiment, the part provider management application135can achieve this while still allowing territory partitioning by requiring the territory be in the contribution ID. After collecting the contributions, the part provider management application135in the Collections stage330can create a child workflow for each territory with a contribution or a previously-published part. Each child workflow can execute separately during the Select stage340, Build stage350and Publish stage360.

At the Select stage340, the part provider management application135can determine the subset of contributions for a particular territory that are both consistent and valid. As used herein, a set of contributions is said to be consistent when none of the contributions within the set contain conflicting information, e.g., two contributions specifying two different language attributes for the same image file. As used herein, valid contributions are contributions that pass semantic validation.

Generally, a contribution may contain top-level attributes for the entity, or the contribution could have information about sub-elements that will lead to a more complex part structure. Examples of complex part structures include the list of image assets in an Image Part, or a complex nested playlist in a VOD Video Part. The part provider management application135can key the contributions themselves to avoid collisions, e.g., as described in the Ingest stage310section. In the Select stage340, the part provider management application135can determine the full set of contributions using the schema-specific unique IDs, and at the Build stage350the part provider management application135can assemble the full set of contributions into the complex part.

In one embodiment, the part provider management application135during the Select stage340can simply choose the “best” contribution of each type for the relevant partition. For instance, the part provider management application135could initially choose the “best” contribution based on contribution timestamp and contributor trust (e.g., the contribution with the higher trust and latest timestamp wins). As an example, if the part provider management application135determines there are two “image language” contributions for the same image file, the part provider management application135could choose the contribution with the latest timestamp and the earlier contribution would not be included in the selected set. The part provider management application135can also include some level of semantic validation, to ensure the part provider management application135ignores invalid contributions even if they are otherwise the “best”.

At the Build stage350, after the part provider management application135has selected the ideal subset of contributions for the target entity, the part provider management application135can aggregate the data into a Part object. As such, in one embodiment, the Build stage350is considered the most domain-specific of any stage in the workflow300. During the build stage, the part provider management application135can perform deep validation (e.g., subtitle drift checks) and call lateral dependencies (e.g., to encode video files, upload images, transform metadata, merge rights and contracts).

Once the Build stage350is complete, the part provider management application135can write the output of the Build stage350to the Document Store, which is represented in the workflow300by the Publish stage360. Once written to the document store, the new part can be discovered and published to customer-facing data stores by, for example, a Title Assembly Service (TAS) or a Catalog Production system. In one embodiment, if the part provider management application135determines that a part cannot be built, the part provider management application135refrains from publishing a partial or incomplete part to the same library as successful parts. Moreover, in such an embodiment, the part provider management application135may not unpublish previously published parts, as such parts represent a successful earlier state of the target entity's contributions. In one embodiment, the part provider management application135only unpublishes a part upon determining that the part was generated in error, either from a bug or from all source contributions being deleted.

In one embodiment, when publishing a new part, the part provider management application135is configured to account for race conditions. That is, contributions and entities may have changed in the time elapsed since the Listen stage320, and the part provider management application135can protect against this by performing an optimistic write to the Document Store. If the part provider management application135determines that the existing part does not have a version that matches the version acquired during the Listen stage320, the write fails and the workflow300restarts. If the part provider management application135determines no part existed for a territory when the workflow300was initiated, the part provider management application135can publish the new part with the CREATE_ONLY flag.

In one embodiment, the part provider management application135publishes each part with the REID or REID+RPEID as the partition key and territory as the unique ID. Doing so allows easy lookup by the Listen stage320(e.g., for unpublishing) and by downstream systems like TAS and Catalog Production.

Once the workflow300is complete and a new part has been published, downstream consumers such as a Title Assembly Service and/or the Catalog Production system can aggregate the part into the target entity's manifest. For example, the Catalog Production can incorporate additional data (that may not be deterministic), such as pricing data or current encoding status. The part provider management application135can pass the entity manifest and part payloads to a Catalog Release service for cross-part validation. In one embodiment, the cross-part validation process is where cross-domain checks occur during the workflow300. Finally, the part provider management application135can make the parts available to Catalog Distribution for vending to downstream clients.

FIG. 3Bis a workflow for generating a new part object within a parts catalog for a new content entity, according to one embodiment described herein. As shown, the workflow370begins, where the part provider management system130performs an operation to collect data according to a collection graph (operation381). In the depicted embodiment, the part provider management system130collects a piece of new content310, together with content schemas378corresponding to the new content and territory information379corresponding to the new content. As shown, the new content371can include asset file(s)372, which include video file(s)373, audio file(s)374and subtitle file(s)375, as well as associated metadata376and content provider identifier(s)377. Of note, the depicted data types are shown for illustrative purposes only and without limitation, and more generally any suitable types of data can be ingested by the part provider management system130, consistent with the functionality described herein.

Upon collecting the data, the part provider management system130retrieves a part creation graph from the part creation graph repository385(operation380). The part provider management system130then uses the retrieved part creation graph to generate a new part from the collected data (operation383). The part provider management system130publishes the new part to the parts catalog (operation382). As discussed above, prior to publishing the new part and making it available for the downstream consumers164, the part provider management system130can perform one or more validation operations on the new part to ensure the part's validity. Once the part in the parts catalog145is made available to the downstream consumers164, the downstream consumers164can consume the new part according to their own application-specific workflow (operation384). For example, a video streaming service downstream consumer164could consume the new part to facilitate the streaming of video data to client devices. As another example, a web application server could consume the new part and could use the information within the part to populate one or more web pages to be served to client devices.

FIG. 4is a block diagram illustrating an entity management system, according to one embodiment described herein. As shown, the system400includes the content ingestion services154, which are configured to provide an API through which content providers can submit new content for ingestion. The content ingestion services154can forward a request to ingest new content to an entity manager service405. Generally, the entity manager service405is configured to generate a new entity corresponding to the new content and to store information describing the new entity within the entity manager database415.

As discussed above, the entity matcher component168can transmit entity data and metadata to the title matching service166, and the title matching service166can return information identifying relationships between entities. For example, the title matching service166could determine that two content items are related, when the content items both represent the same movie but submitted from different content providers110. The entity matcher component168can receive the information identifying the relationships between entities from the title matching service166, and could store this relationship information within the document store425.

Additionally, the entity manager service405can transmit the new content and related metadata to a relation ingestion component410. The relation ingestion component410can determine when the related metadata for the new content identifies a relationship with one or more existing entities, and the relation ingestion component410can store data representing such a relationship within the document store425.

In the depicted embodiment, the entity manager service405for the entity manager system156can transmit new content and related metadata to a reconciliation component420for processing. In one embodiment, the reconciliation component420is responsible for merging or splitting a title within the entity management namespace. For example, the reconciliation component420can perform a merging operation by creating “merged to” links between REs/RPEs and linking REs and PEs to create merged REs or RPEs. Likewise, the reconciliation component420could perform a split operation by deleting “merged to” links between REs/RPEs and unlinking the REs and PEs within a merged RE or RPE.

As discussed above, the part provider management system130can use a part creation graph to generate a new part. Generally, part creation graphs can be used to coordinate the Collect stage330, Select stage340, Build stage350and Publish stage360of the workflow300. In one embodiment, at least some part creation graphs are configured to publish territory-specific parts using child graphs and work items.FIG. 5illustrates an exemplary part creation graph template, according to one embodiment described herein. As shown, the part creation graph template500begins with a collection stage510, where the part provider management system130reads territories and versions of currently-published parts under the target entity. In one embodiment, the part provider management system130acquires an optimistic write lock on the target entity. The part provider management system130can query an Entity Manager component for all PEIDs under the target entity. The part provider management system130can perform a strongly-consistent read of all active contributions for the determined PEIDs, to read all libraries passed in as an input to the work item. The output of the Collect stage330can be a list of all determined contributions, partitioned by territory.

At the Select stage515, the part provider management system130can select activities that will reduce the set of collected contributions and resolve any outstanding conflicts. In doing so, the part provider management system130can determine that only the latest version of contributions should be used. After the Select stage515, the part provider management system130at the Build stage520can create a Part object from the final set of selected contributions. If the part provider management system130determines there are any ambiguities in the contributions, the part provider management system130can fail the Build stage520and transmit one or more determined issues to a Feedback Service. At the Publish stage525, the part provider management system130updates the part store with the newly generated Part object. In one embodiment, the part provider management system130is configured to use a version of a child workflow acquired during the Collect stage510, together with the new Part object from the Build stage520, for each territory child workflow. In select instances, e.g., where a territory has no contributions or a target entity is aliased or deprecated, the part provider management system130can perform an unpublish operation530to unpublish one or more parts from the part store.

In a particular embodiment, at a high level, the part provider management system130is configured to use two conceptual polling components and five types of graphs. An example of this is shown inFIG. 6, which is a flow diagram illustrating a work flow for generating a new part for a parts catalog, according to one embodiment described herein. As shown, the part provider management application135at the Listen stage320can subscribe to Document Store EventBus Queues610in order to receive contribution change notifications and RE change notifications. Specifically, in the illustrated embodiment, a set of poller components615of the part provider management application135subscribe to these notifications, with a contribution poller component subscribing to contribution change notifications and a RE poller component subscribing to RE change notifications.

The part provider management application135in the depicted embodiment is configured with a number of domain-agnostic collection graphs620for processing the notifications received by the set of poller component615. These domain-agnostic collection graphs include, without limitation, a RE+Territory Reaction Graph for RE-level parts, a RE+Territory Reaction Graph for RPE-level parts, a RE Reaction Graph for RE-level parts and a RE Reaction Graph for RPE-level parts. More generally, however, these examples are provided for illustrative purposes only and without limitation. At operation625, the part provider management application135then uses a part creation graph for RE- or RPE-level parts to generate parts objects for the relevant territories, with the parts objects having Object IDs that are determined depending on their corresponding domain-agnostic collection graph620. For example, in the depicted example, parts objects generated based on the RE+Territory Reaction Graph for RE-level parts would have an object ID calculated by combining the associated REID and the territory identifier.

FIG. 7illustrates a reaction graph for processing contribution change notifications, according to one embodiment described herein. As shown, the illustrated workflow700begins, where the part provider management application135receives a contribution change notification specifying a Domain, a Library, a PEID, a Document ID and a Document Version. The part provider management application135retrieves graph configuration information for the library (block715) and retrieves the REID corresponding to the PEID from an Entity Manager component (block720). The part provider management application135reads territory information from payload data of the updated contribution (block725) and initiates a RE+Territory workflow (block730). In initiating the RE+Territory workflow, the part provider management application135can place a mutually exclusive lock on the RE+Territory, to ensure that only one workflow is running at a time for the RE+Territory. If the part provider management application135determines that the RE+Territory is already locked when the part provider management application135attempts to initiate the RE+Territory workflow at block730, the part provider management application135can refrain from initiating the workflow at that point in time and can schedule to initiate the RE+Territory workflow at a subsequent point in time (e.g., at a time a predefined period of time after a current time). At the subsequent point in time, the part provider management application135could again attempt to initiate the RE+Territory workflow (i.e., if the RE+Territory is unlocked, the part provider management application135will place a mutually exclusive lock on the RE+Territory and proceed with executing the workflow).

FIG. 8illustrates a reaction graph for processing reconciled entity change notifications, according to one embodiment described herein. As shown, the workflow800begins, where the part provider management application135receives a RE change notification specifying a REID, a RE status (e.g., primary, alias, deleted, etc.), one or more REPIDs and one or more PEIDs. As discussed above, the entity manager component156can manage entity objects and can perform merge and split operations on such entity objects. For example, where the entity manager component156determines that two separate entity objects refer to the same title (e.g., a particular movie), the entity manager component156could perform a merge operation to combine the two entity objects. In doing so, the entity manager component156can define a relationship between the entity objects and can update the RE status of the entity objects, based on the relationship. For example, in doing so, the entity manager component156could designate one of the entity objects as a “primary” status, and the other could be defined as an “alias” status with a pointer to the “primary” object. In one embodiment, the part provider management application135is configured to publish a part object only for the “primary” entity object, as opposed to, e.g., publishing separate part object for each of the two entity objects.

The workflow800then enters a loop at block815, where the part provider management application135, for each active graph configuration, determines a scope of the RE change notification (block820). In the depicted embodiment, if the part provider management application135determines that the scope of the change is at the RE level, the part provider management application135starts a RE Reaction Workflow for an RE Part (block825). On the other hand, if the part provider management application135determines that the scope of the change is at the RPE level, the part provider management application135starts a RE Reaction Workflow for an RPE Part (block830).

FIG. 9illustrates a domain-agnostic reaction graph for collecting contributions and starting a single work item for a contribution's reconciled entity and territory, according to one embodiment described herein. As shown, the workflow900begins at operation910, where the part provider management application135receives workflow orchestration engine execution data. Such workflow orchestration engine execution data includes a REID, a territory identifier, a destination part store designator, one or more source contribution libraries, a part creation graph client identifier and a part creation graph name.

The part provider management application135retrieves a RE document (block915) and determines a status of the RE document (block920). If the part provider management application135determines the RE document has been merged or deleted, the part provider management application135takes no further action (block925). On the other hand, if the part provider management application135determines the status is primary, the part provider management application135gets an existing part for the RE and territory from the parts catalog. The part provider management application135then performs an operation to collect active contributions keyed by territory, where the part provider management application135first retrieves PEs from the RE document (block935). The uses a strongly-consistent read operation and retrieves active contributions for PEs from all source contribution libraries (block940). The part provider management application135filters out any contributions for territories that do not match the input territory received at block910(block945).

Once the active contributions are collected, the part provider management application135calculates a Part Document ID using the REID and territory identifier (block950). The part provider management application135then initiates a part creation workflow (block955) and the workflow900ends. As discussed above, the part provider management application135can execute the part creation workflow to generate a new part object for the part catalog. For example, if no part object corresponding to the updated or new content currently exists within the part store, the part provider management application135could create a new part object for the new content. As another example, if a part object corresponding to the content currently exists within the part store, the part provider management application135could create a new version for the existing part object.

FIG. 10illustrates a domain-agnostic reaction graph for collecting contributions and starting a single work item for a contribution's reconciled partner entity and territory, according to one embodiment described herein. As shown, the workflow1000begins at block910, where the part provider management application135receives workflow orchestration engine execution data. Such workflow orchestration engine execution data includes a REID, a RPEID, a territory identifier, a destination part store, a source contribution library, a part creation graph client ID and a part creation graph name. The part provider management application135retrieves a RE document (block1015) and determines a status of the RE document (block1020). If the part provider management application135determines the RE document has been merged or deleted, the part provider management application135takes no further action (block1030). On the other hand, if the part provider management application135determines the status is primary, the part provider management application135gets an existing part for the RE, RPE and territory from the parts catalog (block1025).

The part provider management application135then initiates a workflow to collect active contributions keyed by territory, where the part provider management application135retrieves PEs from the RE document for the input RPEID (block1045). The part provider management application135then uses a strongly-consistent read operation to get active contributions for PEs from all source contribution libraries (block1050). The part provider management application135filters out any contributions whose territory identifiers do not match the input territory identifier (block1055).

The part provider management application135calculates the part object identifier using the REID, RPEID and territory identifier (block1060). The part provider management application135then initiates a part creation workflow (block1065), and the workflow1000ends. For instance, in the depicted embodiment, the part creation workflow takes as input parameters the REID, RPEID, and territory of the new content, as well as one or more active contribution references for the territory, destination part store information, destination part document identification information and a designation of an existing part version (if any) of a corresponding part object already within the parts catalog.

FIG. 11illustrates a domain-agnostic graph for collecting contributions and starting work items in appropriate territories, according to one embodiment described herein. The operation1100begins at block1110, where the part provider management application135receives a workflow execution engine work item specifying a REID, a source contribution library, a destination part store and a part creation graph configuration. The part provider management application135retrieves a RE document (block1115) and gets any existing part objects from the part store (or part catalog) for the RE (block1120).

The part provider management application135determines the RE status at block1125, and if the part provider management application135determines the status is merged or deleted, the part provider management application135deletes all parts under the RE in the part store (block1130). On the other hand, if the part provider management application135determines the RE status is primary, the part provider management application135initiates a workflow to collect active contributions keyed by territory. In doing so, the part provider management application135gets PEs from the RE doc (block1135) and gets active contributions for PEs from all source contribution libraries using a strongly-consistent read operation (block1140). The part provider management application135groups contributions by territory (block1145).

Once the active contributions are collected, the part provider management application135, for each entry in the collected contributions map (block1150), calculates a part document identifier using the REID and territory (block1155). The part provider management application135then initiates a part creation workflow (block1160), and the workflow1100ends. In the depicted embodiment, the part provider management application135initiates the part creation workflow using the REID, territory information, active source contribution reference(s), part document ID and existing part version (if any) as input parameters to the part creation workflow.

FIG. 12illustrates a domain-agnostic graph for collecting contributions and starting work items for appropriate territories and deleting existing parts for reconciled partner entities and territories that are no longer associated with the reconciled entity, according to one embodiment described herein. As shown, the workflow1200begins at block1210, where the part provider management application135receives a workflow orchestration engine work item specifying a REID, a source contribution library, a destination part store and a part creation graph configuration. The part provider management application135retrieves a RE document (block1215) and retrieves any existing part objects from the part store (or part catalog) for the RE (block1220).

The part provider management application135determines the RE status at block1225, and if the part provider management application135determines the status is merged or deleted, the part provider management application135deletes all parts under the RE in the part store (block1230). On the other hand, if the part provider management application135determines the RE status is primary, the part provider management application135initiates a workflow to collect active contributions keyed by territory. In doing so, the part provider management application135gets PEs from the RE doc (block1235) and gets active contributions for PEs from all source contribution libraries using a strongly-consistent read operation (block1240). The part provider management application135groups contributions by territory (block1245).

The part provider management application135then enters a loop at block1250, where for each entry in the collected contributions map, the part provider management application135calculated a part document identifier using the REID, RPEID and territory identifier (block1255). The part provider management application135initiates a part creation workflow (block1260). The part provider management application135unpublishes parts for the RPE and territory pairs with existing pairs that (i) are not in the RE document, (ii) have a merged status, or (iii) have zero contributions (block1265), and the workflow1200ends.

FIG. 13illustrates a part creation graph, according to one embodiment described herein. The workflow1300begins at block1310, where the part provider management application135receives workflow orchestration engine execution data including a REID, a RPEID, a territory identifier, active source contribution references, a destination part store, a part document ID, and an existing part version (if any). The part provider management application135validates each contribution corresponding to the active source contribution references (block1315) and selects a subset of contributions for use in building a part object (block1320). The part provider management application135then builds the part object using the selected contributions (block1325). At the decision block1330, the part provider management application135determines whether the part was created successfully, and if so, the part provider management application135publishes the part with a version write constraint to the parts catalog (block1335). If the part was not created successfully (e.g., insufficient valid contributions or an error occurred), the part provider management application135unpublishes the part from the parts catalog with a version write constraint (block1340). At block1345, the part provider management application135publishes the “failed part” document for visibility purposes only (e.g., to effectively log the unsuccessful part creation), and the workflow1300ends.

FIG. 14is a flow diagram, illustrating a method for publishing a part to a parts catalog, according to one embodiment described herein. As shown, the method1400begins at block1410, where a schema management system120receives, from a contributing system, a request to register a new schema or schema version with a schema management system. Upon successfully performing a validation operation for the new schema or schema version, the schema management system120stores the new schema or schema version at the schema management system (block1420). The content provider API115receives a submission of new content for ingestion (block1430). Upon successfully validating the new content using the new schema or schema version stored at the schema management system, the content provider API115stores the new content at a content storage system (block1440).

The content provider API115initiates a workflow with the part provider management application135to ingest the new content item into a part provider management system as a new part object (block1450). The part provider management application135collects a plurality of contribution references corresponding to a plurality of contributions, wherein a first contribution in the plurality of contributions comprises a set of attributes corresponding to the new content item (block1460). The part provider management application135builds the new part object based on a subset of contributions from the plurality of contributions (block1470). The part provider management application135then publishes the new part object to a parts catalog for the part provider management system, for consumption by one or more downstream systems, and the method1400ends.

FIG. 15is a flow diagram, illustrating a method for publishing a part to a parts catalog, according to one embodiment described herein. As shown, the method1500begins at block1510, where upon validating a new schema or schema version, the schema management system120stores the new schema or schema version at a schema management system. Upon successfully validating the new content using the new schema or schema version stored at the schema management system, the content provider API115stores the new content at a content storage system (block1520). The part provider management application135then ingests the new content item into a part provider management system as a new part object (block1530), and the method1500ends.

FIG. 16is a flow diagram, illustrating a method for publishing a part to a parts catalog, according to one embodiment described herein. As shown, the method1600begins at block1610, where the schema management system120receives, from a contributing system, a request to register a new schema or schema version with a schema management system. The schema management system120performs a validation operation to validate the new schema or schema version (block1620), and the schema management system120stories the new schema or schema version at the schema management system (block1630).

Upon receiving a submission of new content for ingestion, the content provider API115validates the new content using the new schema or schema version stored at the schema management system (block1640) and stores the new content at a content storage system125(block1650). The part provider management application135builds a new part object based on a plurality of contributions (block1660) and publishes the new part object to a parts catalog for the part provider management system, for consumption by one or more downstream systems (block1670), and the method1600ends.

Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g. an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present invention, a user may access applications (e.g., part provider management application135) or related data available in the cloud. For example, the part provider management application135could execute on a computing system in the cloud and could provide the part creation API137through which users can request for new content to be ingested as a parts object. In such a case, the part provider management application135could execute a parts creation graph workflow to generate a new part object and could insert the new part object into a parts catalog145, for consumption by downstream computing devices and services. Doing so allows a user to access the part provider management application135(e.g., via the part provider API137) from any computing system attached to a network connected to the cloud (e.g., the Internet).