Patent Description:
Data security is vital for computing systems connected to public networks, such as the Internet. Computer systems are often protected from unauthorized access and data breaches using network security technologies, such as firewalls.

A virtual machine provides an emulated version of a computer system. A virtual machine can include emulated processing units (e.g., a central processing unit (CPU)), memory, network interfaces, and/or other computing components.

<CIT> describes a virtual sandbox environment that enables a publisher to publish rules for supplemental content, such as third party advertising, displayed on a page or other grouping of content from the publisher. One or more rule sets can be applied or enforced in a number of ways, such as through active script on a page, as part of a browser or other such application or interface, or via a third party such as an anti-virus application or service. A rule set can specify behavior that is allowed or prohibited, in various circumstances, as well as actions to be taken in response to any unapproved or prohibited behavior. Rule sets may be maintained on, and enforced by, or cached on a client device such that any number of pages or other groupings of content can utilize the same rule set without reloading the set.

<CIT> describes methods and systems for providing a framework to securely integrate third-party logic into electronic transaction processing workflow. Third-party programming code that implements different third-party logic may be obtained and stored in a repository. A transaction processing request is received from a third-party server, and an instance of a transaction processing module is instantiated within an operating runtime environment to process a transaction according to a workflow. When the instance of the transaction processing module has reached an interruption point, the instance of the transaction processing module is suspended, and a third-party programming code is executed within an isolated runtime environment. The third-party programming code is configured to provide an output value based on attributes of the transaction. The instance of the transaction processing module then determines whether to authorize or deny the transaction based in part on the output value.

<CIT> describes methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting a template for a content item. In one aspect, a method includes receiving a content item request that includes context data. A content item that is eligible to be presented in response to the content item request is identified. A determination is made that the content item includes a template variable that is resolved based on the context data. A template feed including a set of templates is identified based on the template variable. A template is selected from the template feed for the content item. The template can be selected based on the context data. The selected template is populated with content for the content item to create a formatted content item. The formatted content item is provided in response to the content item request.

<CIT> describes systems and methodologies for accessing resources associated with a Web-based application, including a browser that obtains at least first resources from a first domain and second resources from a second domain and a resource management component that facilitates controlled communication between the first resources and the second resources and prevents the first resources and the second resources from accessing other resources that the first resources and the second resources are not permitted to access. The resource management component may be further operable to contain restricted services in a sandbox containment structure and/or to isolate access-controlled resources in a service instance. In addition, the resource management component may be operable to facilitate the flexible display of resources from disparate domains and/or controlled communication therebetween.

This specification describes technologies related to securely performing workflows that enable non-disclosed and otherwise proprietary customization of the stages of the workflow in ways that prevent other parties from accessing the customization. A workflow is a set of executable stages through which a unit of work passes from initiation to completion. The technologies include performing workflows in isolated environments, e.g., in virtual machines, that provide secure sandboxes while still supporting full-function workflows. The techniques can further include constraints, including conditional constraints, on inputs to and/or outputs from workflows or portions thereof to maintain user privacy, prevent access to confidential customizations, and enhance system integrity.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods including the operations of receiving, from a client device, a digital component request that can include a set of data. In response to receiving the digital component request, multiple operations can be performed. A multi-stage workflow for selecting a digital component from candidate digital components of multiple content platforms can be identified based on the set of data, and the multi-stage workflow can include one or more customizable stages. Each stage of the multi-stage workflow can be executed in a sequence defined by the multi-stage workflow, and each stage can include executable instructions. Execution can include, for each customizable stage: (i) initiating, for each content platform, an isolated environment in which executable instructions provided by the content platform is executed; (ii) receiving, from at least one isolated environment, a data access request requesting data stored outside of the isolated environment; (iii) for each data access request: (a) obtaining at least one access rule that is associated with the data access request; (b) processing each access rule that is associated with the data access request to determine whether to retum, to the isolated environment, the data requested by the data access request; and (c) whenever it is determined to return the data requested by the data access request, providing the data to the isolated environment. Executing can further include, receiving output data from one or more of the customizable stages; and selecting, using a selection stage of the multi-stage workflow and based on the output data received from each of the one or more customizable stages, the digital component from the candidate digital components. The digital component can be sent to the client device. Other implementations of this aspect include corresponding apparatus, systems, and computer programs, configured to perform the aspects of the methods, encoded on computer storage devices.

These and other implementations can each optionally include one or more of the following features. Whenever it is determined to not return the data requested by the data access request, access to the data can be denied. In response to determining not to return the data requested by the data access request, a value can be provided to the executing stage that indicates that data access request is not authorized. Each access rule that is associated with the data access request can include a condition. Determining whether to return the data requested by the data access request can include evaluating the condition of each access rule, and determining to return the data requested only when each condition of each access rule evaluates to TRUE. A condition can specify, for a given user, at least one content platform that can receive data of the user. The condition can further specify the type of data of the user that can be received by the content platform. The condition can specify one or more customizations that can access the data. At least one access rule that is associated with the data access request can be precompiled. Precompiling the at least one access rule can include precompiling the at least one access rule to programming instructions. Precompiling the at least one access rule can include factoring common expressions of the at least one access rule.

The techniques described in this document can be used to select digital content, e.g., digital components, from a variety of content providers (e.g., content platforms) while preserving the privacy of the requestor. In addition, the techniques enable such digital components to be provided by content platforms while also preserving the confidentiality and integrity of techniques and proprietary logic used by the content platforms to select and/or customize content for requesters. As described further below, the system can execute stages of a workflow that is used to determine digital components, with stages that involve sensitive user data and/or confidential techniques and/or logic being executed in a virtual machine or in other appropriate isolated and/or sandbox environments. Executing code in a virtual machine protects the privacy of the content requestor since the virtual machine can constrain access to information about the requestor, e.g., by preventing the logic of a content platform from sending sensitive data from the system that hosts the virtual machine. Executing code in a virtual machine protects the content platform that supplied the code since the virtual machine can ensure that the content platform's proprietary customizations remain isolated such that other content platforms never have access to the content platform's customizations. The techniques can include encrypting code for the customizations, which ensures the security, confidentiality, and integrity of the code. The techniques can also be used to ensure that data produced by stages of a workflow meet validity criteria. Such criteria can further protect requestor privacy by ensuring that stages only provide data to other stages and to content platforms when privacy constraints are satisfied. The techniques can ensure that each instance of a stage is provided access only to data authorized for that instance, thereby enhancing user privacy. In addition, the access can be controlled at a granular level, and can depend on the party attempting to access the data, the purpose of the data access as defined by the customization logic requesting the data, the type of data requested, the data provider, and so on. Further, data access authorization can depend on explicit permissions provided by users and the aforementioned granular controls provide the users the same granular control over when, how, and by which content platforms their data is used. Also, since enforcement of the data authorization rules is performed before data are transmitted over the network, computing resources are used efficiently, e.g., by preserving network bandwidth and reducing the processor cycles and memory required to download and store the data.

In general, this document describes systems and techniques for selecting and distributing digital components to client devices in ways that protect user privacy and confidential data of content platforms and/or digital component providers. A secure distribution system can be configured to perform digital component selection processes that use sensitive user data so that the user data is not provided to any other entity. The secure distribution system can host and execute selection logic of various content platforms (e.g., supply side platforms and demand side platforms) when selecting digital components based on user data in manners that ensure that no other entity can access the selection logic of the content platform. In this way, both the data of the users and the content platforms is kept secure.

Ensuring the privacy of personal data is a requirement of many computing systems, especially those connected to public networks such as the Internet. Some consumers who do not trust that strong privacy protection will be enforced by a system will simply choose not to use that system. In addition, some jurisdictions have regulations that protect privacy. Such privacy guarantees can include not only how data is stored, but also processes that control data sharing with third parties.

However, some data sharing can provide utility to data consumers, especially when a consumer authorizes specific sharing. For example, private data, including aggregate private data, can be used to locate content that is both relevant and interesting to the data consumer, if the user authorizes such uses of the user's data. Absent information about the data consumer, it can be challenging for a system to provide relevant information.

One solution to balancing privacy and utility is to enable a "data custodian" to retain private data with a guarantee that the data are only used for authorized purposes. Such authorization can include which content platforms can use the data, how often, for what purpose, and so on. When a data consumer requests content, a data custodian can execute code provided by content platforms, and the result can be the selection and presentation of content that is of interest or otherwise relevant to the data consumer. In such cases, the data custodian can execute code on behalf of content platforms, and that code can use authorized subsets of the private data, but the content platforms never actually receive the private data.

However, code or other types of logic (e.g., rules) of content platforms can be a valuable business asset of the content platform. Such code can contain differentiating innovations that result in the selection of more relevant content for data consumers. For that reason, content platforms can be reluctant to provide their code to a data custodian.

In addition, executing code from multiple content platforms can create integrity issues. For example, one content platform might attempt to share data with another content platform that is not authorized to access the data. In another example, a content platform might attempt to use their code to determine how the code of another content platform operates. Thus, a need exists to ensure overall system integrity while still allowing non-disclosed, proprietary code to operate on private data.

Further, data privacy of users and content platforms must be preserved, and each content platform can have differing restrictions on content access. For example, a content platform will typically have access to data that it itself provides, but to protect the privacy of its users and of its own data, can restrict access to the data by other content platforms. In another example, one content platform can have a relationship that enables it to access summarized, anonymized data of another content platform, but a second content platform might lack that relationship, and therefore not have similar access.

This specification describes a workflow system that enables content platforms to execute code for each stage of the workflow, or for a subset of stages, in a virtual machine, and the result of the workflow can be recommended content. Such code supplied by a content platform to implement a stage can be called a "customization. " To preserve privacy, the system can ensure that each customization is permitted to access only data that does not violate privacy constraints, and that the results generated by each customization satisfies validity constraints. In addition, the system can execute each customization in a separate virtual machine, ensuring that each customization is executed in isolation. Further, content platforms can encrypt their customizations before providing them to the system. Encrypting customizations, then executing them in isolation provides protection to these valuable data assets. In addition, encryption ensures the integrity of the customization by ensuring that customizations cannot undergo tampering before arriving at the workflow system.

<FIG> is a block diagram of an example environment <NUM> in which a secure distribution system <NUM> distributes digital components to client devices <NUM> in a privacy preserving manner. The environment <NUM> includes a data communication network <NUM>, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. The data communication network <NUM> connects client devices <NUM> to the secure distribution system <NUM> and connects the secure distribution system <NUM> to content platforms, such as supply side platforms (SSPs) <NUM> and/or demand side platforms (DSPs). The network <NUM> can also connect the various content platforms to one another and/or to digital component providers <NUM>, e.g., to servers of the digital component providers <NUM>.

A client device <NUM> is an electronic device that is capable of communicating over the network <NUM>. Example client devices <NUM> include personal computers, server computers, mobile communication devices, e.g., smart phones and/or tablet computers, and other devices that can send and receive data over the network <NUM>. A client device can also include a digital assistant device that accepts audio input through a microphone and outputs audio output through speakers. The digital assistant can be placed into listen mode (e.g., ready to accept audio input) when the digital assistant detects a "hotword" or "hotphrase" that activates the microphone to accept audio input. The digital assistant device can also include a camera and/or display to capture images and visually present information. The digital assistant can be implemented in different forms of hardware devices including, a wearable device (e.g., watch or glasses), a smart phone, a speaker device, a tablet device, or another hardware device. A client device can also include a digital media device, e.g., a streaming device that plugs into a television or other display to stream videos to the television, a gaming system, or a virtual reality system.

A client device <NUM> can include applications <NUM>, such as web browsers and/or native applications, to facilitate the sending and receiving of data over the network <NUM>. A native application is an application developed for a particular platform or a particular device (e.g., mobile devices having a particular operating system). Although operations may be described as being performed by the client device <NUM>, such operations may be performed by an application <NUM> running on the client device <NUM>.

The applications <NUM> can present electronic resources, e.g., web pages, application pages, or other application content, to a user of the client device <NUM>. The electronic resources can include digital component slots for presenting digital components with the content of the electronic resources. A digital component slot is an area of an electronic resource (e.g., web page or application page) for displaying a digital component. A digital component slot can also refer to a portion of an audio and/or video stream (which is another example of an electronic resource) for playing a digital component.

An electronic resource is also referred to herein as a resource for brevity. For purposes of this document, a resource can refer to a web page, application page, application content presented by a native application, electronic document, audio stream, video stream, or other appropriate type of electronic resource with which a digital component can be presented.

As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, image, text, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component. For example, the digital component may be content that is intended to supplement content of a web page or other resource presented by the application <NUM>. More specifically, the digital component may include digital content that is relevant to the resource content (e.g., the digital component may relate to the same topic as the web page content, or to a related topic). The provision of digital components can thus supplement, and generally enhance, the web page or application content.

When the application <NUM> loads a resource that includes a digital component slot, the application <NUM> can generate a digital component request <NUM> that requests a digital component for presentation in the digital component slot. In some implementations, the digital component slot and/or the resource can include code (e.g., scripts) that cause the application <NUM> to request a digital component from the secure distribution system <NUM>.

A digital component request <NUM> sent by a client device <NUM> can include sensitive user data related to a user of the client device <NUM> and/or non-sensitive data, such as generic keywords and/or a query string. The sensitive user data can include, for example, data identifying user groups that include the user as a member. The user groups can include interest-based groups. Each interest-based group can include a topic of interest and a set of members identified (e.g., determined or predicted) to be interested in the topic. The user groups can also include, for example, groups of users that performed particular actions at electronic resources (e.g., websites or native applications) of publishers. For example, a user group can include users that visited a website, users that requested more information about an item, interacted with (e.g., selected) a particular digital component and/or added an item to a virtual cart to potentially acquire the item. The user data for a user can also include user profile data and/or attributes of the user.

Further to the descriptions throughout this document, a user may be provided with controls (e.g., user interface elements with which a user can interact) allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user's social network, social actions, or activities, profession, a user's preferences, or a user's current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user" geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

A digital component request <NUM> can also include contextual data, which is generally considered non-sensitive. The contextual data can describe the environment in which a selected digital component will be presented. The contextual data can include, for example, coarse location information indicating a general location of the client device <NUM> that sent the digital component request <NUM>, a resource (e.g., website or native application) with which the selected digital component will be presented, a spoken language setting of the application <NUM> or client device <NUM>, the number of digital component slots in which digital components will be presented with the resource, the types of digital component slots, and other appropriate contextual information.

The secure distribution system <NUM> can be implemented using one or more server computers (or other appropriate computing devices), that may be distributed across multiple locations. In general, the secure distribution system <NUM> receives requests for digital components from client devices <NUM>, selects digital components based on data included in the requests, and sends the selected digital components to the client devices <NUM>.

As the secure distribution system <NUM> receives sensitive user data, the secure distribution system <NUM> can be operated and maintained by an independent trusted party, e.g., a party that is different from the users of the client devices, the parties that operate the SSPs <NUM> and DSPs <NUM>, and the digital component providers <NUM>. For example, the secure distribution system <NUM> can be operated by an industry group or a governmental group.

As described in more detail below, the secure distribution system <NUM> can select a digital component from a set of digital components stored in a digital component repository <NUM> and/or a set of digital components received from an SSP <NUM>. The digital component repository <NUM> stores digital components received from content platforms (e.g., from SSPs <NUM> and/or DSPs <NUM>) and additional data (e.g., metadata) for each digital component. The metadata for a digital component can include, for example, distribution criteria that defines the situations in which the digital component is eligible to be provided to a client device <NUM> in response to a digital component request received from the client device <NUM> and/or a selection parameter that indicates an amount that will be provided to the publisher if the digital component is displayed with a resource of the publisher and/or interacted with by a user when presented. For example, the distribution criteria for a digital component can include location information indicating which geographic locations that digital component is eligible to be presented, user group membership data identifying user groups to which the digital component is eligible to be presented, resource data identifying resources with which the electronic resource is eligible to be presented, and/or other appropriate distribution criteria. The distribution criteria can also include negative criteria, e.g., criteria indicating situations in which the digital component is not eligible (e.g., with particular resources or in particular locations). Other data that can be used to select a digital component can also be stored in the digital component repository with a reference (e.g., a link or as metadata) to its digital component.

An SSP <NUM> is a technology platform implemented in hardware and/or software that automates the process of obtaining digital components for the resources. Publishers of resources can use an SSP <NUM> to manage the process of obtaining digital components for digital component slots of its resources. Each publisher can have a corresponding SSP <NUM> or multiple SSPs <NUM>. Some publishers may use the same SSP <NUM>.

A DSP <NUM> is a technology platform implemented in hardware and/or software that automates the process of distributing digital components for presentation with the resources and/or applications. A DSP <NUM> can interact with multiple supply-side platforms SSPs on behalf of digital component providers <NUM> to provide digital components for presentation with the resources of multiple different publishers. Digital component providers <NUM> can create (or otherwise publish) digital components that are presented in digital component slots of publisher's resources.

In this example, user data does not cross a trust boundary that separates the client device <NUM>, the secure distribution system <NUM>, and the digital component repository from the SSP <NUM>, DSP <NUM>, and digital component providers <NUM>. In this way, no entity other than the client device <NUM> and the secure distribution system <NUM> receives the user data that is included in a digital component request <NUM>. This preserves user privacy and data security, especially when compared to techniques that employ third party cookies to send user data across the Internet.

An example process for selecting and providing a digital component for presentation at a client device <NUM> is illustrated in stages A - I, which illustrate a flow of data between the components of the environment <NUM>.

In stage A, the application <NUM> sends a digital component request <NUM> to the secure distribution system <NUM>. As described above, the application <NUM> can send a digital component request to request a digital component for presentation in a digital component slot of a resource being presented by the application <NUM>. The digital component request <NUM> can include user data and contextual data.

In stage B, the secure distribution system <NUM> sends a context-based digital component request to an SSP <NUM>. The context-based digital component request <NUM> can include the contextual data of the digital component request <NUM> received from the application <NUM>. However, the context-based digital component request <NUM> does not include any of the user data. The secure distribution system <NUM> can temporarily store the user data while waiting for a response from the SSP <NUM>. The secure distribution system <NUM> can send the context-based digital component request <NUM> to an SSP <NUM> for the publisher of the resource being presented by the application <NUM>.

In stage C, the SSP <NUM> forwards the context-based digital component request <NUM> to one or more DSPs <NUM>. In stage D, each DSP <NUM> sends, to the SSP <NUM>, one or more selection parameters for one or more digital components, e.g., digital components stored in the digital component repository <NUM>. For example, the DSP <NUM> can select a digital component based on the contextual data of the context-based request and determine a selection parameter for the digital component based on the contextual data. The DSP <NUM> can also provide a digital component and selection parameter, e.g., a digital component that is not stored in the digital component repository <NUM>. Each DSP <NUM> can send a selection parameter with data indicating the digital component to which the selection parameter applies. The digital components for which selection parameters are provided by the DSPs <NUM> can be referred to as context-based digital components.

In stage E, the SSP <NUM> sends the digital components and/or selection values to the secure distribution system <NUM>. In some implementations, the SSP <NUM> can filter digital components and/or selection parameters prior to sending the digital components and/or selection values to the secure distribution system <NUM>. For example, the SSP <NUM> can filter digital components and/or selection parameters based on publisher controls specified by the publisher of the resource being presented by the application <NUM>. In a particular example, a publisher of a web page about a particular event may define, as a publisher control, that digital components related to another event may not be presented with this web page. The SSP <NUM> can filter based on rules or other data provided by the publisher.

In stage F, the secure distribution system <NUM> queries the digital component repository <NUM> for a set of user-based digital components that are selected based on the user data of the digital component request <NUM>. For example, the secure distribution system <NUM> can submit a query that defines, as conditions of the query, the user data of the digital component request <NUM>. In some implementations, the query can also include context-based conditions. For example, a query can request retrieval of digital components that include, as distribution criteria, a particular user group and/or a particular geographic location. Although shown after stages B - E, the secure distribution system <NUM> can query the digital component repository in parallel with these stages to reduce the latency in selecting and providing a digital component to the application <NUM>.

In stage G, the secure distribution system <NUM> receives a set of user-based digital components from the digital component repository <NUM> and a selection parameter for each user-based digital component. The set of user-based digital components can include those having distribution criteria that matches the conditions of the query.

In stage H, a selection engine <NUM> of the secure distribution system <NUM> selects a digital component to provide to the application <NUM> for presentation in the digital component slot. The selection engine <NUM> can select a digital component from the set of context-based digital components and the user-based digital components. The selection engine <NUM> can select the digital components from the two sets based on the selection parameter for each digital component in the two sets. For example, the selection engine <NUM> can select the digital component having the highest selection parameter.

In stage I, the secure distribution system <NUM> provides the selected digital component to the application <NUM>. The application <NUM> can then present the digital component with the resource being presented by the application <NUM>.

<FIG> shows components of the secure distribution system <NUM> of <FIG> in more detail. In general, the secure distribution system <NUM> can receive digital component requests <NUM> from client devices <NUM>, securely execute workflows <NUM>, and provide digital components <NUM> to client devices <NUM>. The secure distribution system <NUM> can include an interface engine <NUM>, a workflow management engine <NUM>, a virtual machine (VM) management engine <NUM>, an output validation engine <NUM>, a request management engine <NUM> and an access rule repository <NUM>.

The interface engine <NUM> is configured to accept workflow specifications 240a, customizations <NUM> and digital component requests <NUM>, and can provide digital components <NUM> and/or references to digital components <NUM>, e.g., Uniform Resource Locators (URLs), which enable client devices <NUM> to download the referenced digital components <NUM> from servers. The interface engine <NUM> can include an application programming interface (API) that is configured to accept data (e.g., workflow specification 240a, customizations <NUM> and digital component requests <NUM>) provided to the secure workflow system <NUM> and to provide data (e.g., digital components <NUM>) to other components in the environment <NUM> of <FIG>.

A workflow specification 240a, 240b (collectively workflow specification <NUM>) can define the structure of a workflow, and can include a description of stages 252a, 252b (collectively stages <NUM>) of a workflow, including interconnections among the stages <NUM> and constraints on the inputs and outputs to a stage <NUM>.

A workflow specification <NUM> can be instantiated into a particular workflow <NUM> by the workflow management engine <NUM>, and the same workflow specification <NUM> can be instantiated multiple times into multiple workflows <NUM>, e.g., for multiple content platforms - i.e., workflow management engine <NUM> can execute workflows <NUM> instantiated from a single workflow specification <NUM> for each of multiple content platforms. Each content platform can provide customizations <NUM> that provide implementations for stages <NUM> of the workflow <NUM> executed by the secure distribution system <NUM> on their behalf. As described further below, to provide privacy of user data, in some implementations, each stage <NUM> of a workflow <NUM> can be executed in a separate virtual machine instance <NUM>, and in some implementations, all stages of a workflow <NUM> executed on behalf of a content platform are executed in a single virtual machine instance <NUM>. The workflow specification <NUM> can use any appropriate technique for defining a workflow structure. For example, a workflow can be described as a graph where nodes of the graph are stages <NUM> of the workflow, and edges in the graph show inputs and outputs to stages <NUM>. In some workflows, outputs from one stage become inputs to another stage <NUM>, or to multiple other stages <NUM>, and stages <NUM> can also obtain inputs for sources other than another stage <NUM>. In addition, output from a stage <NUM> can be processed before it becomes input to another stage <NUM>. For example, output can be validated before it is passed to another stage <NUM>, as described further below.

A workflow specification <NUM> can include default code for any subset of the workflow stages <NUM>, for none of the stages <NUM>, or for all of the stages <NUM>. The default code provides a base implementation of the stage, reducing the coding burden for parties providing customizations <NUM> that do not wish to provide customizations <NUM> for all stages <NUM>. A workflow specification <NUM> can also include indications that particular stages cannot be overridden or otherwise customized, and the default code must always be used for the stage. Stages in a workflow specification <NUM> that can be overridden can be called "customization points.

For example, the secure distribution system <NUM> can perform a digital component selection process to select digital components to provide to client devices <NUM> based on user data and/or other data, such as distribution criteria for each digital component, contextual data, and/or other appropriate data. The digital component selection process can have multiple stages <NUM> defined by a workflow specification <NUM>. The overall sequence of stages can be rigid such that there are no customizations by content platforms (e.g., SSPs <NUM> and/or DSPs <NUM>). However, the processes performed in some stages may be customized by the content platforms. For example, the digital component selection process can have a stage in which a digital component request <NUM> is processed to extract data from the request <NUM>. This stage may be a default stage in which default code that cannot be customized by content platforms is used by the workflow management engine <NUM>. A later stage can include obtaining candidate digital components and corresponding selection parameters. At this stage, the workflow management engine <NUM> can execute customizations provided by the content platforms to select the candidate digital components and generate the corresponding selection parameters. As the logic provided by the content platforms is typically considered confidential, this logic can be securely stored by the secure distribution system <NUM> and can execute in isolated environments, e.g., within content platform specific virtual machines, as described in more detail below. After the candidate digital components are obtained, the workflow management system <NUM> can perform another default stage to select a digital component <NUM> from the candidate digital components to provide to the client device <NUM> from which the digital component request <NUM> was received.

In some implementations, metadata can be included in, or associated with, the workflow specification <NUM>. Such metadata can include, but is not limited to, the author of the workflow specification <NUM>, its version, time of creation, owner, etc. The metadata can also include criteria that indicate whether the workflow defined by the workflow specification <NUM> is appropriate for a digital component request <NUM>. For example, one workflow specification <NUM> might be appropriate for requests for textual digital components <NUM> while a different workflow specification <NUM> is appropriate for requests for multimedia digital components <NUM>. The secure distribution system <NUM> can use such metadata to select a workflow specification <NUM> appropriate for a particular digital component request <NUM>. The metadata can also include an identifier for the workflow specification <NUM>.

A customization <NUM> can be computer-executable instructions that define the operation of a stage <NUM> of a workflow <NUM>. The instructions for a customization <NUM> can be expressed as executable instructions (e.g., bytecodes) and/or in any appropriate programming language, including scripting languages, and different customizations <NUM> can be expressed in any combination of executable instructions and programming languages, which can be different programming languages. For example, customizations <NUM> can be specified as C, C++, Python, and/or other example types of code.

Customizations can be grouped into workflow descriptions, where a workflow description is an implementation of a workflow specification <NUM>, or a portion of a workflow specification <NUM>, and more specifically, workflow descriptions can include one or more customizations <NUM>, each implementing a customization point in a workflow specification <NUM>. A customization <NUM> can also include one or more identifiers that specify the stage in a workflow specification that the customization implements. For example, if a workflow specification <NUM> includes stages A, B, C and D, a content platform can provide workflow descriptions that contain customizations <NUM> for stages A, B, C and D or some portion thereof. When the secure distribution system <NUM> instantiates, on behalf of a content platform, a workflow specification <NUM> into a workflow <NUM>, the secure distribution can use the customizations <NUM> for each of stages A, B, C and D provided by the content platform. To simplify the management of the customizations, a content provider can group the customizations <NUM> into a workflow description or the content provider can provide the customizations <NUM> individually.

Customizations <NUM> can include a wide range of metadata. For example, metadata can include the author of the customization <NUM>; its version; time of creation; owner; relationships between the author and other entities providing data, customizations, workflow specifications, etc.; descriptive text; and so on. Metadata related to customizations can be used by access rules <NUM> to enforce limits on data access, as described below.

Workflow descriptions (i.e., groups of customizations) can also have metadata that can be explicitly associated with the workflow description, or associated with the workflow description indirectly. , a workflow description can inherit the metadata data of the customizations <NUM> included in the workflow description. As with metadata related to customizations, metadata related to workflow descriptions can be used by access rules <NUM> to enforce limits on data access,.

The workflow management engine <NUM> can execute workflows as specified by workflow specifications <NUM> and customizations <NUM>. The workflow management engine <NUM> can obtain a virtual machine (VM) instance <NUM> from the VM management engine <NUM>, and instruct the virtual machine instance <NUM> to execute the instructions as specified by the customization <NUM>. A VM instance <NUM> can emulate the operation of a physical computing device, while isolating the instructions performed by the VM from the computing device and from other VM instances. In cases where no customization <NUM> exists for a stage <NUM>, the VM instance <NUM> can execute default code for the stage <NUM> as specified by the workflow specification <NUM>. Executing customizations <NUM> in VM instances <NUM> provides isolation of customization <NUM> both from customizations <NUM> provided by other content providers, and from resources in the secure distribution system <NUM> to which a customization <NUM> is forbidden to access.

The workflow management engine <NUM> can include a data Access gating module <NUM>. When a stage <NUM> requests access to external data, the data access gating module <NUM> can determine whether the stage <NUM> will be permitted access to that data. The data access gating module <NUM> can provide a data access API that evaluates specific data access requests <NUM> that are made by customization <NUM> using the access rules <NUM>, and can determine whether the data access requests <NUM> are authorized. The data access gating module <NUM> can use the information in the data access request <NUM>, metadata associated with the customization, metadata associated with workflow descriptions, and/or other data available to the secure distribution system <NUM> to determine whether the data access request <NUM> is authorized. If authorized, the data access gating module can obtain the requested data and provide the requested data to the customization that requested the data, e.g., by passing the data to the virtual machine running the customization using a virtual network that connects the data access gating module <NUM> to each virtual machine. The API provided by the Data Gating Module <NUM> can be of any appropriate form, such as a WebService API, a Remote Procedure Call (RPC), etc., and the API can support multiple formats (e.g., WebServices and RPC).

The data access gating module <NUM> can include one or more rule evaluation components that are configured to evaluate the access rules <NUM>. The data access gating module <NUM> can use the information in a data access request <NUM> and other available data (as described above) to evaluate the access rules <NUM> to determine whether the request is authorized. Authorized requests can pass through the data access gating module <NUM> and reach the provider of the requested data.

In some implementations, the data access gating module <NUM> can retain information from prior data access requests <NUM>, and use the information from prior requests in the evaluation of the access rules <NUM>. For example, an access rule <NUM> might specify that any party can access a first piece of data, a second piece of data, but not both. Information from prior data access requests <NUM> can be used to enforce such rules. In some implementations, the API of the data access gating module <NUM> can permit requests for multiple pieces of data by accepting multiple data access requests <NUM> in one API call. In such cases, the system will evaluate the data access rules <NUM> individually for each data access request <NUM> and permit access only for the data access requests <NUM> that are authorized. Enabling customizations <NUM> to request multiple pieces of data in a single API call can reduce the number of networking operations required to download the data that is authorized.

A data access request <NUM> can include broad range of data describing the data access request, such as the specific data requested, the party attempting to access the data (e.g., the content platform corresponding to the customization requesting the data), the purpose of the data access (e.g., the customization <NUM> requesting the data), the type of data requested, the data provider that provides the requested data, a description of data accessed previously by the party attempting to access the data, and/or other appropriate data. The specific data requested can be indicated by a Uniform Resource Location (URL) or Uniform Resource Identifier (URI), among other techniques. A data access request <NUM> can specify the information requested in any appropriate form. For example, a data access request <NUM> can be expressed as Extensive Markup Language (XML) or JavaScript Object Notation (JSON).

In some implementations, fields with a data access request <NUM> can be provided by and/or verified by the data access gating module <NUM>. For example, to prevent a customization <NUM> from falsely claiming that it is requesting access to data on behalf of a party that did not provide the customization <NUM>, the data access gating module <NUM> can determine the party that provided the customization, and use that party as the requestor.

The VM management engine <NUM> can create VM instances <NUM>, destroy VM instances <NUM>, and assign VM instances <NUM> to workflow stages <NUM>, among other VM management operations. The VM management engine <NUM> can include an API, which, when called by the workflow management engine <NUM>, causes the VM management engine <NUM> to create a VM instance <NUM> and provide a reference to the VM instance <NUM> to the workflow management engine <NUM>. The workflow management engine <NUM> can assign the VM instance <NUM> to a stage <NUM> of a workflow <NUM>, and execute the instructions for the stage <NUM> (which can be a customization <NUM>) in the VM instance <NUM>.

In some implementations, VM management engine <NUM> can deploy each virtual machine or other isolated environment in a sandbox environment. In this way, the VM management engine <NUM> can control the data that is provided to each virtual machine or sent from each virtual machine, which can prevent logic of a content platform from sending sensitive data, e.g., user data, outside of the secure distribution system <NUM>. This further enhances user privacy in the generation of dynamic digital content by preventing leakage of sensitive user data from the secure distribution system <NUM>.

The output validation engine <NUM> can accept output from the stage <NUM> of a workflow, which can be implemented by a customization <NUM>, and determine whether the output satisfies criteria imposed on the output. In various implementations, the criteria are included in the workflow specification <NUM>; the criteria are configured into the workflow management engine <NUM>, e.g., by an authorized administrator of the secure distribution system <NUM>; and/or the workflow management engine <NUM> obtains the criteria using other techniques, such as reading them from a storage device. If the output validation engine <NUM> determines that an output does not satisfy the criteria, the output validation engine can indicate to the workflow management engine <NUM> that the workflow <NUM> should be halted.

The criteria can be expressed in any suitable format and can be expressed using any property of the output and/or any property of the system. For example, the criteria can be expressed as Boolean expressions. Criteria can include types of values produced (e.g., the output must be an integer, floating point number, string, array, Boolean, etc.), ranges of values produced (e.g., a value must be between a minimum and maximum value), limits of values produced (e.g., a value cannot exceed a threshold value), and so on. Criteria can be expressed for specific content platforms, for classes of content platforms (e.g., providers of images or providers of multimedia), for all content platforms, etc..

The request management engine <NUM> can accept digital component requests <NUM> from the interface engine <NUM>, and provide the request <NUM> (or data from the request <NUM>) to the workflow management engine <NUM>. A digital component request <NUM> can be any appropriate specification of content. For example, a digital component request <NUM> can be a query string, and can include user data and/or context data provided by a client device <NUM>, as described above. The workflow management engine <NUM> can use the digital component request <NUM> and metadata associated with workflow specifications <NUM> to determine an appropriate workflow specification <NUM> for the digital component request <NUM>. For example, if the request is for an image, the workflow management engine <NUM> can select a workflow configured to select digital components <NUM> that are in the form of, or that include, images.

The access rule repository <NUM> can store access rules <NUM> that enable the secure distribution system <NUM> to enforce limits on data access by stages <NUM>, including customizations <NUM>. The access rule repository <NUM> can include any appropriate storage system, or combinations of storage systems, such as a relational database, a file system, a block storage system, and so on. The access rule repository <NUM> can obtain rules from authorized parties, such as systems administrators. In some implementations, the access rule repository <NUM> can receive access rules <NUM> that have been encrypted by the party providing the access rules <NUM>, e.g., using a public key provided by the access rule repository <NUM>, and the access rule repository <NUM> can decrypt the access rules <NUM> using its private key.

An access rule <NUM> can include a condition and an action. A condition can include a Boolean expression that can include fields in the data access request <NUM> and other properties available to the secure distribution system <NUM>. For example, a condition can state that if a particular property (e.g., requestor of the data) has a particular value (e.g., an authorized users), then the Boolean expression (or a subcomponent of the Boolean expression) will evaluate to TRUE. Information in the data access request <NUM> and other properties can relate to various entities, such as the user providing the digital component request <NUM> (whose identity can be anonymized), the digital component requested, user consent provided, user data relevant to the digital component request <NUM>, the steward of the user data, authorized relationships between the data steward and other parties. When the condition evaluates to TRUE, the action can indicate that data access is permitted or denied.

An access rule <NUM> can be assigned to a class of data items or multiple classes of data items, and the access rule <NUM> can apply to all data items in the class. The association can be based on any metadata associated with the data items in the class. For example, a class of data items can relate to a set of broad categories such as users in a location or age range, all data items belonging to a steward, data items belonging to a corporation or other organization, and so on. The class of data items can be represented using a string of characters. For example, if the data steward is "ExampleOrg", the class can be represented as "DataSteward: ExampleOrg". There can also be multiple categories associated with a rule, e.g., "DataSteward: ExampleOrg; AgeRange: <NUM>-<NUM>", indicating that the access rule <NUM> applies when the data steward in ExampleOrgor when the user age range is <NUM> to <NUM>. In some implementations, all access rules <NUM> associated with a data item must evaluate to 'permit' before access to the data item is allowed.

A data access rule <NUM> can also include information relating to user consent that the data can be released, and under what circumstances. For example, a user might authorize release only of anonymized data, only to a particular party (e.g., a particular content platform) or set of parties (e.g., a particular organization such as a content platform and the organization's affiliates), and only for a purpose explicitly authorized by the user as defined by the customization <NUM> making the request. The data access rule <NUM> can encode these criteria as part of the condition, and can further include, or include a link to, proof that the authorization was granted.

In some implementations, the secure distribution system <NUM> can provide a user interface that enables users to define which parties can access their data, what type of the users' data each party can access and, for each type of data, and for what purpose(s) each party can access that type of data. This provides granular control for users to define how their data is used and not used. The secure distribution system <NUM> can receive user consent data from the user interface and generate data access rules for each user based on the user consent data. For example, a user can indicate that any content platform can use their user group membership data to generate custom digital components, but cannot use location data for any purposes. If a customization requests user group membership data from a customization of a content platform for the purpose of generating a custom digital component, the data access gating module <NUM> can evaluate the data access rule(s) for the user for which the digital component is being customized. In this example, the data access gating module <NUM> would provide the user group membership data for the user to the customization based on the rule. However, the location data for the user would not be provided to the customization.

In some implementations, the data access gating module <NUM> can precompile all data access rules <NUM>, or a subset of the data access rules <NUM>, to improve the evaluation efficiency. For example, the data access gating module <NUM> can translate the rules into an efficient encoding such as programming instruction (e.g., in C++, which can then be compiled to assembly language). In another example, the data access gating module <NUM> can precompile data access rules <NUM> by factoring common expressions. , if a particular condition (e.g., User is Jane Doe) appears in multiple conditions, the data access gating module <NUM> can factor that expression such that it is evaluated once, and the results are provided to multiple rules.

One data access request <NUM> can require multiple levels of authorization, all of which will be included in conditions. For example, authorizations can include a user consent to receive personalized digital components <NUM>, a user's consent that specifies which, if any, user information can be shared (e.g., a user's identity, coarse or fine location, device type, etc.), a user's consent that one organization share information with an authorized affiliate, among many other examples. Only if all authorizations are satisfied will the data access requests <NUM> be permitted by the data access rules <NUM>.

<FIG> is a flow diagram of an example process <NUM> for executing secure workflows for content selection. For convenience, the process <NUM> will be described as being performed by a system for executing secure workflows for content selection, e.g., the secure distribution system <NUM> of <FIG> and <FIG>, appropriately programmed to perform the process. Operations of the process <NUM> can also be implemented as instructions stored on one or more computer readable media, which may be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of the process <NUM>. One or more other components described herein can perform operations of the process <NUM>.

The system obtains (<NUM>) a workflow specification. For example, the system can include an API that is configured to accept workflow specifications. In another example, the system can accept data containing the workflow specification over a network, e.g., data passed using Transmission Control Protocol / Internet Protocol (TCP/IP), Hypertext Transfer Protocol (HTTP) or HTTP-Secure (HTTP-S). In still another example, the system can obtain a workflow specification from a storage device using an API appropriate for the storage device or from a database using SQL. As noted above, the workflow specification can define the structure of a workflow and can include, in some examples, default code for stages of the workflow.

The system obtains (<NUM>) customizations, which can be included in one or more workflow descriptions. The system can obtain customizations using any appropriate technique. For example, the system can accept customizations included in data received over a networking protocol such as TCP/IP, HTTP or HTTP-S. In another example, the system can include an API, which, when called by parties such as content providers, enables the parties to provide customizations. As described above, each customization can be configured to implement a stage of a workflow specification. Multiple parties, which can be content platforms, can provide customizations, and parties can provide multiple workflow descriptions, each configured to implement the stages of a different workflow specification.

In some implementations, to secure the workflow description, a customization can be encrypted by the submitting party and decrypted by the system using any appropriate encryption technique. For example, the system can publish a public key (e.g., by storing the public key at a known, accessible location). Submitting parties can obtain the public key, encrypt the customization (or multiple customizations in a workflow description) using the public key, and provide the encrypted customization and/or encrypted workflow description to the system. The system can decrypt the encrypted customization and/or workflow description using its private key that corresponds to the public key used by the submitting party.

The system can create a workflow from the workflow specification and the customizations. As described above, customizations can include an identifier of the stage of a workflow specification that the customization implements. For each stage included in the workflow specification, the system can include the default code for the stage from the workflow specification, and for customization points (i.e., stages of the workflow specification that can be overridden), replace the code for the stage with code included in the corresponding customization. If the system obtains a customization for a stage that is not a customization point, the system can ignore the customization and/or provide an error message.

The system receives (<NUM>) a request for a digital component using any appropriate technique. For example, the system can receive the digital component request <NUM> over HTTP or SQL. As described above, a digital component request <NUM> can contain a description of the digital component requested (e.g., a search string), user data, and/or contextual data, as described above.

In response to receiving the request, the system determines (<NUM>) a multi-stage workflow for selecting personalized content from candidate content of multiple content platforms. As described above, the metadata associated with a workflow specification can include criteria that indicate whether the workflow is appropriate for a particular digital component request <NUM>. The system can determine whether the criteria are satisfied to determine a workflow for execution. For example, criteria might specify that a workflow specification is appropriate for requests that are received from particular geographical regions where location-specific regulations apply. In such cases, the system can evaluate the criteria associated with the workflows to select a workflow appropriate for requests from that region.

The system executes (<NUM>) each stage of the multi-stage workflow to determine at least one output value. Executing the workflow can include various operations, as described below.

For at least a subset of the content platforms that provided stages, which can be customizations, relevant to a digital component request <NUM>, the system can execute the stages of the workflow determined in operation <NUM>. Executing a stage can include instantiating (<NUM>) a VM instance configured to isolate a set of executable instructions provided by the content platform (e.g., the customization for the stage). The system can instantiate a VM instance using a hypervisor that is configured to instantiate VM instances or using any other suitable techniques for instantiating VM instances. The system can then execute the code for the stage in a VM instance. Thus, in some implementations, the system executes each stage in a separate, isolated VM instance, and the isolation ensures that the VM instance can access only the system resources for which the VM instance has access permission. For example, the system can execute, for each customizable stage, the customization of each content platform for that stage in its own separate and isolated VM. Such a system provides isolation among all stages, regardless of whether they were provided by a single content platform or multiple content platforms. In addition, in some implementations, the system can execute multiple stages provided by a given content platform in a single VM instance. Using a single VM instance to execute multiple stages can reduce the computational requirements of the computing system by reducing the number of required VM instances.

The system performs (<NUM>) the executable instructions of the stage within the VM to determine the at least one output value. The system can include one or more execution environments (e.g., Java Virtual Machines, JavaScript engines, and Python engines, among other alternatives) that are configured to perform executable instructions, which can determine an output value used to select the digital component. In some implementations, the workflow can include multiple stages, where the end stage selects a digital component directly from a set of candidate digital components, and the output of intermediate stages can influence the operation of the end stage (e.g., by selecting the candidates and/or generating selection parameters for the candidates), thereby influencing the selection of the digital component. As described above, the workflow specification and customizations can include code specified using any appropriate programming language or as executable instructions such as bytecodes. The results of each stage can be returned to the system for delivery to the next stage and/or for use selecting a digital component. The results of a stage can also be evaluated using validity criteria, as described below.

When executing within a VM, implementations of stages can access input data necessary for performing the stage. The input data can be stored in any appropriate format and can be stored on any appropriate storage device. For example, the input data can be stored as (key, value) pairs, where the stage accesses data by providing a key, and the system can provide the value associated with the key. In various implementations, the input data can be stored in a database and obtained using SQL queries and/or stored in a file system and accessed using a file system API.

The system can provide access control for the (key, value) pairs. For example, each (key, value) pair can be associated with an access control list, and the system can provide the (key, value) pair only if the content platform that provided a customization is included in the access control list. The system can obtain the access control list from an authorized administrator or using other suitable techniques for obtaining access control lists. In some implementations, a content platform can provide a collection of (key value) pairs, and the system permits access to such collections only for the content platform that provided the collection, or to other content platforms explicitly authorized by the content platform that provided the collection.

In some implementations, access control for data requested by customization can be controlled by configurable access criteria. <FIG> is a flow diagram of an example process <NUM> for secure data access while executing secure workflows for content selection. This process <NUM> can be performed for one or more stages of the process <NUM> for which data is requested by a customization. For convenience, the process <NUM> will be described as being performed by a system for executing secure workflows for content selection, e.g., the secure distribution system <NUM> of <FIG> and <FIG>, appropriately programmed to perform the process. Operations of the process <NUM> can also be implemented as instructions stored on one or more computer readable media, which may be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of the process <NUM>. One or more other components described herein can perform operations of the process <NUM>.

The system can inspect (<NUM>) an instruction to be performed by a stage of a workflow to determine data that will be used to perform the instruction. For example, the instruction can include a data access request (e.g., the data access request <NUM> of <FIG>), and the data access request can specify the data used to perform the instruction. Since stages are executed in isolation within a virtual machine that prohibits general data access, the system (e.g., a data access gating module component of the system) can provide a data access API that enables specific data requests, and only provides the requested data when such data requests are authorized for the stage. For example, the system can provide an API that allows stages to request data (e.g., as expressed as a data access request) from a particular data provider, and that requires the stage to provide the metadata relating to the data requested. Continuing the example above, a stage might request data from "Steward: ExampleOrg," with an indication of the specific data requested. When a stage calls a data request API, the system can inspect the data access request associated with the request to determine the data that will be used to perform the instruction. In some implementations, the data access API is the sole mechanism for stages to obtain data from data providers.

The data access API can be any appropriate API type. For example, a data access API can accept as parameters, such as a data access request, that indicate the data requested, e.g., as specified by a URL or. Data may only be retrieved when the API call is approved, as described below. In some implementations, the API can accept search string parameters that indicate the data to be retrieved instead of, or in addition to, URLs and/or URIs. The system can provide multiple APIs to support different programming languages. For example, the system can include an API for Python and an API for C++. The APIs can differ syntactically, but provide equivalent semantics. This enables content platforms to configure their customizations in their preferred code, which can then generate the API calls in that same language without requiring additional computations to translate the request to a common language.

If the system determines (<NUM>) that no data is requested (e.g., the data request API is not invoked), the system can proceed to operation <NUM>; if the system determines that data is requested, the system can proceed to operation <NUM>.

The system can obtain (<NUM>) access rules appropriate to the data access request determined from the inspection of operation <NUM>. In some implementations, the system can obtain all access rules. In some implementations, the system can segment access rules, e.g., according to the data requestor, and only obtain access rules appropriate to the particular segment. Generally, the system can obtain the access rules using any appropriate technique, e.g., the system can obtain the access rules from an access rules repository using a technique appropriate for the repository. For example, if the repository includes a relational database, the system can use SQL operations to obtain the access rules. In another example, if the repository includes a file system, the system can use file system operations to obtain the rules.

The system can evaluate (<NUM>) the access rules obtained in operation <NUM>. The system can evaluate a first access rule of the access rules by evaluating the condition portion of the rule in light of the data requested (e.g., a data access request). If the first access rule evaluates to TRUE, the system can evaluate subsequent access rules until a rule evaluates to FALSE, or no further rules exist.

The system can determine (<NUM>) whether access to the data item is permitted. If all rules have evaluated to TRUE in operation <NUM>, access is permitted, and the system can proceed to operation <NUM>. If any rule has evaluated to FALSE in operation <NUM>, access is denied, and the system can proceed to operation <NUM>.

In some implementations, the system can accept requests for multiple pieces of data, e.g., as expressed by multiple data access requests. In such cases, each data access request can be evaluated separately. For data access requests in which all rules evaluate to TRUE, the system can proceed to operation <NUM>; for data access requests for which as least one rule evaluates to FALSE, the system can proceed to operation <NUM>. Note that, in some cases, the system will proceed to operation <NUM> for a subset of the data access requests and to operation <NUM> for a different subset of the data access requests. In some implementations, the system can defer proceeding to operations <NUM> and/or <NUM> until all data access requests have been evaluated.

The system can provide (<NUM>) the data. The system can execute the data access API invoked by the stage. As described above, the API can include as a parameter a specification of the data requested. In some implementations, the API can be executed in the virtual machine to ensure that data remains isolated. In some implementations, the API can be provided by a component of the system that is outside a virtual machine. Once the data has been provided (e.g., by executing the data access API), the system can continue at operation <NUM>.

The system can deny (<NUM>) access to the data. In some examples, the system can provide a notification to the party that provided the customization indicating that data access was denied. In some implementations, the system can provide a NULL value to the customization, and the customization can determine how to proceed without the requested data. For example, a customization can be configured to generate output data using a default process in the absence of requested data. In some implementations, the customization can be terminated. In some implementations, the customization can specify how to proceed if data access is denied. For example, the customization can specify that it should attempt to proceed with a NULL data value or terminate.

The system can perform (<NUM>) the instructions as described above in reference to operation <NUM>. In some cases, the customization will include instructions that process the requested data, and the system can perform those instructions. As noted above, the instructions can be specified in any appropriate programming language such as C or Python.

Returning to <FIG>, to limit system resources consumed on behalf of a content platform, resources consumed on behalf of any subset of content platforms, or on behalf of all content platforms, the system can enforce limits on resources consumed by the entire workflow, by a subset of stages in a workflow, or by any workflow stage. Such limits can be valuable, for example, when the system is executed on a server or group of servers that provide a multi-tenant environment, and the server or servers ensure that computing resources consumed by any given tenant does not exceed a configured threshold, leaving sufficient computing resources available for other tenants. This can also reduce the overall workload on the system, which can extend the life of processing components of the system. Resources can include any operational metric such as processor time (e.g., by a Central Processing Unit(s) (CPUs), Graphics Processing Units (GPUs), Digital Signal Processor(s) (DSP), Application Specific Processor(s) (ASPs), etc.), memory use, storage use, and network use, among other operational metrics. The system can enforce limits, which can be expressed as constraints, on any aspect of resource use such as aggregate use, peak use, rate of use over some time period, and so on. Limits can also be combined in any logical combination. For example, a limit can specify a maximum peak use and a limit on aggregate use. The limits can be enforced on a particular stage, on a group of stages, on an entire workflow, on all workflows provided by a content provider, by collections of content providers, and/or on other singular or aggregate entities. If a limit is met, the system can perform a remediating action. Such actions can include, but are not limited to, terminating or delaying operation of a stage or workflow.

When a stage completes execution, the system determines (<NUM>) the validity of one or more of the output values produced by the stage. In situations in which multiple VMs are used in a stage, e.g., when there is a separate VM for each content platform for that stage, the system can determine the validity of the output(s) of each VM. As described above, criteria used to determine the validity of output can be included in the workflow specification and/or configured into the system. Such criteria can protect requestor privacy by ensuring that stages only provide data to other stages and to content providers that satisfies constraints, which can include privacy constraints. The system can evaluate the criteria using the output value to determine whether the criteria are satisfied. As described above, criteria can be expressed as function of the type output, value or values of the output, and so on.

The VM management engine can deactivate (<NUM>) the VM using any appropriate technique for deactivating a VM. For example, the system can instruct a hypervisor to deactivate the VM. Deactivating the virtual machine removes all state of the computation done by the stage, thereby ensuring the stage does not provide unauthorized content to another stage.

In response to determining that the output value is valid (e.g., as determined in operation <NUM>), the virtualized workflow engine determines (<NUM>) a result at least in part based on the first output value from each VM. The result can be a digital component or a descriptor for a digital component. The descriptor can include a reference to one or more digital components (e.g., as indicated by a URL or URI) that are present at, or provided by, a content platform, and/or a descriptor that can be used by a content platform to determine a digital component. For example, the descriptor can include any metadata describing the digital component, such as a category of information requested (e.g., sports, fashion, news, science, etc.), depersonalized information about the requestor (e.g., general categories of demographic data such as age range, location, etc.), characteristics of the device (e.g., phone or laptop), and so on.

The system can provide (<NUM>) the result to the client device of the requestor, to the content provider or to both. When the result is a digital component provided to the client device of the requestor, the client device can display the digital component on the user interface of the client device. When the result is a descriptor for a digital component, the client device can retrieve the digital component (e.g., using HTTP to retrieve a digital component specified by a URL) and display the digital component on the user interface of the client device.

In addition, in cases where the descriptor is provided to the requestor and not to the content provider, the system can provide to the content platform an indication that the content from the provider was selected. Such information can allow a content provider to determine the popularity of digital components provided by the provider. The content provider can also use such information to refine the operation of its customizations.

<FIG> is a block diagram of an example computer system <NUM> that can be used to perform operations described above. The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> can be interconnected, for example, using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In one implementation, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM>.

In one implementation, the input/output device <NUM> can include one or more of a network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-<NUM> port, and/or a wireless interface device, e.g., and <NUM> card. In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other input/output devices, e.g., keyboard, printer and display devices <NUM>.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented using one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer-readable medium can be a manufactured product, such as a hard drive in a computer system or an optical disc sold through retail channels, or an embedded system. The computer-readable medium can be acquired separately and later encoded with the one or more modules of computer program instructions, such as by delivery of the one or more modules of computer program instructions over a wired or wireless network. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them.

The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a runtime environment, or a combination of one or more of them. In addition, the apparatus can employ various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, software application, script, or code) can be written in any suitable form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any suitable form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Processors suitable for the execution of a computer program include, by way of example, special purpose microprocessors. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computing device capable of providing information to a user. The information can be provided to a user in any form of sensory format, including visual, auditory, tactile or a combination thereof. The computing device can be coupled to a display device, e.g., an LCD (liquid crystal display) display device, an OLED (organic light emitting diode) display device, another monitor, a head mounted display device, and the like, for displaying information to the user. The computing device can be coupled to an input device. The input device can include a touch screen, keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computing device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any suitable form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any suitable form, including acoustic, speech, or tactile input.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any suitable form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Intemet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

While this specification contains many implementation details, these should not be construed as limitations on the scope of what is being or may be claimed, but rather as descriptions of features specific to particular embodiments of the disclosed subject matter. Thus, unless explicitly stated otherwise, or unless the knowledge of one of ordinary skill in the art clearly indicates otherwise, any of the features of the embodiments described above can be combined with any of the other features of the embodiments described above.

In certain circumstances, multitasking and/or parallel processing may be advantageous.

Claim 1:
A computer-implemented method comprising:
receiving (<NUM>), from a client device (<NUM>), a digital component request (<NUM>) comprising a set of data; and
in response to receiving the digital component request (<NUM>):
identifying (<NUM>) a multi-stage workflow for selecting a digital component from candidate digital components of multiple content platforms based on the set of data, wherein the multi-stage workflow comprises one or more customizable stages;
executing (<NUM>) each stage of the multi-stage workflow in a sequence defined by the multi-stage workflow, each stage comprising executable instructions, the executing comprising:
for each customizable stage:
initiating (<NUM>), for each content platform, an isolated environment in which executable instructions provided by the content platform is executed (<NUM>);
receiving, from at least one isolated environment, a data access request (<NUM>) requesting data stored outside of the isolated environment;
for each data access request:
obtaining (<NUM>) at least one access rule (<NUM>) that is associated with the data access request;
processing (<NUM>) each access rule that is associated with the data access request to determine whether to return, to the isolated environment, the data requested by the data access request; and
whenever it is determined to return the data requested by the data access request, providing (<NUM>) the data to the isolated environment;
receiving output data from one or more of the customizable stages; and
selecting (<NUM>), using a selection stage of the multi-stage workflow and based on the output data received from each of the one or more customizable stages, the digital component from the candidate digital components; and
sending (<NUM>) the digital component to the client device (<NUM>).