Patent Description:
The Internet facilitates the exchange of information between users across the globe. This exchange of information enables distribution of content to a variety of users. In some situations, content from multiple different providers can be integrated into a single electronic document to create a composite document. For example, a portion of the content included in the electronic document may be selected (or specified) by a publisher of the electronic document. A different portion of content (e.g., digital component) can be provided by a third-party (e.g., an entity that is not a publisher of the electronic document and/or does not have access to modify code defining the electronic document).

In some situations, the digital component is selected for integration with the electronic document after presentation of the electronic document has already been requested and/or while the electronic document is being rendered. For example, machine executable instructions included in the electronic document can be executed by a client device when the electronic document is rendered at the client device. The executable instructions can enable the client device to contact one or more remote servers to obtain a digital component that will be integrated into the electronic document while presented at the client device.

<CIT> describes systems and methods for associating a plurality of Internet-enabled devices with a common user profile for targeting Internet content or advertising. One method includes: receiving, from a plurality of Internet-enabled devices, a plurality of requests for electronic content or advertising; extracting, from each of the plurality of requests, a source IP address and a unique identifier associated with the respective Internet-enabled device; identifying each possible pair of devices from which requests were received; calculating for each possible pair of devices a probability that the pair of devices are owned or operated by a common user; and prompting a user to either confirm a characteristic of a prior browsing session or to log-in to an account associated with the common user based on a comparison of the calculated probability to one or more thresholds.

The scope of the invention is defined by a method according to independent claim <NUM>, a computing system according to independent claim <NUM>, and one or more non-transitory computer-readable mediums according to independent claim <NUM>. Further advantageous aspects of the invention are set out in the dependent claims.

The method according to the disclosure includes identifying, by a back-end computing server, an opportunity to transmit a digital component to a client device; determining, by the back-end computing server, a first probability of a given outcome occurring following user interaction with the digital component assuming that the digital component is transmitted to the client device in response to the identified opportunity; and determining, by the back-end computing server, a second probability of the given outcome occurring assuming that the digital component is not transmitted to the client device in response to the identified opportunity.

The method also includes, generating, by the back-end computing server, an outcome incrementality factor for the digital component, including determining a ratio of the first probability relative to the second probability; triggering, by the back-end computing server, adjustment of an eligibility value that controls transmission of the digital component based on the outcome incrementality factor for the digital component; and controlling, by the back-end computing server, transmission of the digital component to the client device using the adjusted eligibility value.

These and other implementations can each optionally include one or more of the following features. For example, in some implementations, generating the outcome incrementality factor includes determining an extent to which transmission of the digital component changes a probability that the user will perform an action that results in the given outcome.

In some implementations, the back-end computing server includes multiple different predictive modeling systems, and determining the first probability and the second probability includes: generating, by a first predictive modeling system, a first parameter that indicates the parameter value for the first probability; and generating, by a second predictive modeling system, a second parameter that indicates the parameter value for the second probability.

In some implementations, determining the first probability of the given outcome occurring includes: determining, by a first model, an interaction probability that quantifies a probability that user interaction with the digital component will occur if the digital component is transmitted to the client device in response to the identified opportunity; determining, by a second model, a post-interaction outcome probability that quantifies a probability that the given outcome occurs following the user interaction with the digital component if the digital component is transmitted to the client device in response to the identified opportunity; and determining the first probability as a product of the interaction probability and the post-interaction outcome probability.

In some implementations, determining the second probability of the given outcome includes determining, by a third model, a baseline probability that quantifies a probability that the given outcome occurs absent transmission of the digital component in response to the identified opportunity; and generating the outcome incrementality factor includes dividing the first probability by the baseline probability.

In some implementations, the method further includes: determining, by the back-end computing server and based on the first parameter, a first score for the given outcome in response to transmitting the digital component to the client device; determining, by the back-end computing server and based on the second parameter, a second score for the given outcome absent transmission of the digital component to the client device; and generating, by the back-end computing server, an incrementality score based on a difference between the first score and the second score.

In some implementations, the eligibility value corresponds to a particular target score, and triggering adjustment of the eligibility value includes: determining a difference between the incrementality score and the particular target score; and adjusting the eligibility value to achieve the particular target score based on the difference between the incrementality score and the particular target score.

Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A computing system of one or more computers or hardware circuits can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation cause the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

According to one aspect of the present disclosure there is a computing system, comprising: a digital component server that stores items of media content and that transmits, to client devices, particular media content included in a digital component; a back-end computing system, including at least one server, that is in communication with the digital component server, where the at least one server includes one or more non-transitory computer-readable mediums storing instructions executable by one or more processors to cause performance of operations comprising: identifying, by the back-end computing system, an opportunity to transmit the digital component to a client device; determining, by the back-end computing server, a first probability of a given outcome occurring following user interaction with the digital component assuming that the digital component is transmitted to the client device in response to the identified opportunity; determining, by the back-end computing server, a second probability of the given outcome occurring assuming that the digital component is not transmitted to the client device in response to the identified opportunity; generating, by the back-end computing system, an outcome incrementality factor for the digital component, including determining a ratio of the first probability relative to the second probability; triggering, by the back-end computing system, adjustment of an eligibility value that controls transmission of the digital component based on the outcome incrementality factor for the digital component; and controlling, by the back-end computing system, transmission of the digital component to the client device using the adjusted eligibility value.

In some implementations, generating the outcome incrementality factor comprises determining an extent to which transmission of the digital component changes a probability that the user will perform an action that results in the given outcome.

In some implementations, the back-end computing system includes multiple different predictive modeling systems, and determining the first probability and the second probability comprises: generating, by a first predictive modeling system, a first parameter that indicates the parameter value for the first probability; and generating, by a second predictive modeling system, a second parameter that indicates the parameter value for the second probability.

In some implementations, determining the first probability of the given outcome occurring comprises: determining, by a first model, an interaction probability that quantifies a probability that user interaction with the digital component will occur if the digital component is transmitted to the client device in response to the identified opportunity;
determining, by a second model, a post-interaction outcome probability that quantifies a probability that the given outcome occurs following the user interaction with the digital component if the digital component is transmitted to the client device in response to the identified opportunity; and determining the first probability as a product of the interaction probability and the post-interaction outcome probability.

In some implementations, determining the second probability of the given outcome comprises determining, by a third model, a baseline probability that quantifies a probability that the given outcome occurs absent transmission of the digital component in response to the identified opportunity; and generating the outcome incrementality factor comprises dividing the first probability by the baseline probability.

In some implementations, the operations further comprise: determining, by the back-end computing system and based on the first parameter, a first score for the given outcome in response to transmitting the digital component to the client device; determining, by the back-end computing system and based on the second parameter, a second score for the given outcome absent transmission of the digital component to the client device; and generating, by the back-end computing system, an incrementality score based on a difference between the first score and the second score.

In some implementations, the eligibility value corresponds to a particular target score, and triggering adjustment of the eligibility value comprises: determining a difference between the incrementality score and the particular target score; and adjusting the eligibility value to achieve the particular target score based on the difference between the incrementality score and the particular target score.

According to another aspect of the present disclosure there is provided one or more non-transitory computer-readable mediums storing instructions that, when executed by one or more processors, cause performance of operations comprising: identifying, by a back-end computing server, an opportunity to transmit a digital component to a client device; determining, by the back-end computing server, a first probability of a given outcome occurring following user interaction with the digital component assuming that the digital component is transmitted to the client device in response to the identified opportunity; determining, by the back-end computing server, a second probability of the given outcome occurring assuming that the digital component is not transmitted to the client device in response to the identified opportunity; generating, by the back-end computing server, an outcome incrementality factor for the digital component, including determining a ratio of the first probability relative to the second probability; triggering, by the back-end computing server, adjustment of an eligibility value that controls transmission of the digital component based on the outcome incrementality factor for the digital component; and controlling, by the back-end computing server, transmission of the digital component to the client device using the adjusted eligibility value.

In some implementations, the back-end computing server includes multiple different predictive modeling systems, and determining the first probability and the second probability comprises: generating, by a first predictive modeling system, a first parameter that indicates the parameter value for the first probability; and generating, by a second predictive modeling system, a second parameter that indicates the parameter value for the second probability.

In some implementations, the operations further comprise: determining, by the back-end computing server and based on the first parameter, a first score for the given outcome in response to transmitting the digital component to the client device; determining, by the back-end computing server and based on the second parameter, a second score for the given outcome absent transmission of the digital component to the client device; and generating, by the back-end computing server, an incrementality score based on a difference between the first score and the second score.

In some implementations, the particular target score is received at a front-end computing server and specifies a desired performance for the given outcome in response to user interaction with the digital component when the digital component is transmitted to the client device.

According to the described subject matter, distribution of a digital component can be triggered using probabilities determined using machine learning systems. In particular, the probabilities of a given outcome can be used to determine whether to trigger distribution of a digital component that directs a user to a publisher's website in response to user interaction with the digital component.

For example, a first probability can indicate a likelihood that a given outcome will occur after the user is directed to the publisher's website through interaction with the digital component (assuming that distribution of the digital component is triggered), and the second probability can indicate a likelihood that the given outcome will occur even if the digital component is not distributed to the user (i.e., the distribution is not triggered). Triggers for distribution of the digital component can be set based on a computed relationship involving the first and second probabilities (e.g., a difference between the probabilities, or a ratio of one probability relative to another probability).

Using the computed relationship to trigger distribution of the digital component helps to reduce (or eliminate) transmission of the digital component in situations that the given outcome is going to occur irrespective of whether the digital component is transmitted. This can prevent unnecessary distribution of the digital component, which reduces the computational resources (e.g. processor and/or memory resources) required to obtain the given outcome. As such, the described technology improves the efficiency of the computer system operation, which is an improvement to the computer system itself. Hence, operation of the system itself is improved by providing more efficient processes for controlling transmission of digital components. Furthermore, preventing unnecessary distribution of the digital component may enable using less bandwidth of a network. Embodiments may therefore address problems associated with transmission of data to a client device in a networked environment.

This document discloses methods, systems, apparatus, and computer readable media that facilitate data processing and system modeling of techniques used to transmit digital components over a communications network. As discussed in more detail below, a machine learning system executes predictive models for determining probabilities that a given outcome will occur if a given digital component is transmitted in response to a particular request for content that is received from a client device. The given outcome can be specified (or selected) as appropriate for the environment in which the technology discussed throughout this document is implemented.

For example, the given outcome in the context of the internet can be a publisher specified user action (e.g., downloading specified content, completing a transaction, interacting with a specific icon, widget, or application, launching a specific script, or some other specified action) being performed at the publisher's website. The probabilities of the given outcome can be used to determine whether to trigger distribution of a digital component that directs the user to the publisher's website in response to user interaction with the digital component.

In some implementations, the determination as to whether to trigger distribution of the digital component in response to a particular request (and/or the outcome value/score that should be obtained for the distribution) can be based on a difference between the probability that the given outcome will occur after the user is directed to the publisher's website through interaction with the digital component (assuming that distribution of the digital component is triggered), and the probability that the given outcome will occur even if the digital component is not distributed to the user (i.e., the distribution is not triggered).

For example, the machine learning system can determine a first probability that the given outcome will occur following user interaction with the digital component assuming that the digital component is transmitted for presentation in a webpage responsive to the request. The machine learning system can also determine a second probability that the given outcome will occur assuming that the digital component is not transmitted for presentation in the webpage.

In some implementations, distribution of the digital component may not be triggered (e.g., distribution is prevented) when the first probability does not exceed the second probability by at least a specified amount. In some implementations, the outcome value that is required for distribution of the digital component can be set based on the amount by which the first probability exceeds the second probability. For example, the outcome value required to be submitted by the publisher can increase as the first probability increases relative to the second probability.

Using the difference between the first probability and the second probability as a basis or condition for triggering distribution of a digital component helps to reduce (or eliminate) transmission of the digital component in situations that the given outcome is going to occur irrespective of whether the digital component is transmitted, which can prevent unnecessary distribution of the digital component, thereby reducing the computational resources (e.g. processor and/or memory resources) required to obtain the given outcome. As such, the technology described in this document improves the efficiency of the computer system operation, which is an improvement to the computer system itself. Furthermore, preventing unnecessary distribution of the digital component may enable using less bandwidth of a network. Embodiments may therefore address problems associated with transmission of data to a client device in a networked environment.

As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, image, text, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component. The phrase "digital component" can refer to a portion of digital content that is embedded into an electronic document that is provided by an entity different from the provider of the digital component.

<FIG> is a block diagram of an example environment <NUM> in which digital components are distributed for presentation with electronic documents. The example environment <NUM> includes a network <NUM>, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network <NUM> connects client devices <NUM>, modeling system <NUM>, third party interface <NUM>, and digital component servers <NUM>. The example environment <NUM> may include many different user/client devices <NUM> that can receive a variety of electronic documents that include multiple types of digital components transmitted by digital component servers <NUM>.

A client device <NUM> is an electronic device that is capable of requesting and receiving resources over the network <NUM>. Example client devices <NUM> include personal computers, mobile communication devices, and other devices that can send and receive data over the network <NUM>. A client device <NUM> typically includes a user application, such as a web browser <NUM>, to facilitate the sending and receiving of data over the network <NUM>. Native applications executed by the client device <NUM> can also facilitate the sending and receiving of data over the network <NUM>.

An electronic document is data that presents a set of content at a client device <NUM>. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices <NUM> by an example electronic document servers or from an application server that provides content to a native application. In some implementations, a conversational response from a voice-powered assistant device, or from a virtual assistant application, is an electronic document for presenting content at client device <NUM>.

For example, the electronic document servers can include servers that host publisher websites. In this example, the client device <NUM> can initiate a request for a given publisher webpage, and the electronic server that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device <NUM>. In another example, the electronic document servers can include app servers from which client devices <NUM> can download apps. In this example, the client device <NUM> can download files required to install an app at the client device <NUM>, and then execute the downloaded app locally. Note that while the description refers to "requests" for purposes of example, the techniques described in this document are deeply applicable to other opportunities to provide content that do not require a request to be submitted by the client device <NUM>.

Electronic documents can include a variety of content. For example, an electronic document can include static content (e.g., text or other specified content) that is within the electronic document itself and/or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document.

In this example, the given electronic document can include a tag or script that causes the client device <NUM> to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device <NUM>. The client device <NUM> integrates the content obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

In some situations, a given electronic document can include a digital component tag or digital component script. In these situations, the digital component tag or digital component script is executed by the client device <NUM> when the given electronic document is processed by the client device <NUM>. Execution of the digital component tag or digital component script causes the client device <NUM> to generate a request for digital components (referred to as a "component request <NUM>"), which is transmitted over the network <NUM> to the digital component server <NUM>. In some implementations, rather than executing a digital tag or script to request digital components from a third-party system, digital components and other third-party content can be requested and incorporated at a first party content server (e.g., server-side or publisher server).

For example, the digital component tag or digital component script can enable the client device <NUM> to generate a packetized data request including a header and payload data. The component request can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device <NUM>), and/or information that the digital component server <NUM> can use to select one or more digital components provided in response (e.g., as a "reply <NUM>") to the component request <NUM>.

Digital component server <NUM> selects digital components that may include items of media content (e.g., video files, audio files, images, or text, and combinations thereof, which can all take the form of advertising content, e.g., <NUM>-A, or non-advertising content, e.g., <NUM>-B) that will be presented with a given electronic document in response to receiving the component request and/or using information included in the component request.

Modeling system <NUM> includes a front-end server <NUM> and a back-end server <NUM>. In some implementations, modeling system <NUM> may be a data modeling and eligibility value adjustment system. For example, system <NUM> can use data models to determine probability values for predicting the occurrence of a given outcome. The system <NUM> can then use these probability values to trigger adjustment of an eligibility value that controls transmission of digital components from digital component server <NUM> to client device <NUM>, as discussed in more detail below.

Generally, digital components are transmitted based on some eligibility value. For example, the eligibility value can be based on how relevant distribution criteria for particular digital component is to a search query submitted by a client device <NUM>, or content of an electronic document being provided to the client device <NUM>. In some situations, the eligibility value can also be based on how likely it is that a given outcome (e.g., a specified user interaction occurring at a landing page that is linked to by the digital component) will occur when the digital component is distributed in response to a request.

However, the likelihood of the given outcome occurring when the digital component is distributed does not take into account or differentiate between the likelihood that the given outcome would have occurred in the absence of the digital component being distributed. In other words, basing the eligibility value on the likelihood that the given outcome will occur if the digital component is distributed does not provide information regarding the incremental increase in the likelihood of occurrence of the given outcome that is caused by the distribution of the digital component.

As such, systems that base the eligibility value on the likelihood of the given outcome occurring when the digital component is distributed do not consider the incremental increase in the likelihood that is caused by the distribution of the digital component, which can lead to distribution of the digital component even in situations where the likelihood of the given outcome occurring does not significantly change irrespective of whether the digital component is distributed or not. For brevity, the likelihood of the given outcome occurring will be referred to as the "outcome likelihood.

The techniques described in this document take into account the incremental outcome likelihood that is provided by distribution of the digital component for purposes of determining whether to distribute the digital component and/or the outcome value that should be submitted for distribution of the digital component. The phrase "incremental outcome likelihood" refers to an amount by which the outcome likelihood changes because of a specified action. For example, the incremental outcome likelihood provided by distribution of the digital component refers to an amount by which the outcome likelihood differs when the digital component is distributed relative to the outcome likelihood when the digital component is not distributed.

As described in more detail below, outcome likelihood predictions provided by machine learning models can be used to determine the incremental outcome likelihood before the digital component is distributed in response to a particular request. This determination can then be used to determine whether to distribute the digital component in response to the particular request and/or what eligibility value will be used for purposes of determining whether to distribute the digital component in response to the particular request.

Referring again to modeling system <NUM>, in some implementations, system <NUM> includes predictive modeling systems that analyze a variety of data for purposes of determining the incremental outcome likelihood that will be provided by distribution of a digital component (e.g., the likelihood of the given outcome occurring when the digital component is distributed relative to the likelihood of the given outcome occurring absent distribution of the digital component). A back-end server <NUM> of the system <NUM> can include multiple different predictive modeling systems that receive and analyze various types of data (described below) in order to generate predictions as to the outcome likelihood under certain conditions. The data can be received via network <NUM> and may relate to user interaction with digital components and/or electronic documents generated for display as a webpage viewable using web browser <NUM>.

Regarding data analysis, each modeling system of back-end server <NUM> can execute computing logic for training an artificial neural network that performs deep-learning analytics based on analysis of the data. Trained neural networks can be used as data models for generating predictions and inferences about user tendencies. As described in more detail below, these predictions and inferences are used by modeling system <NUM> to determine probabilities that quantify a likelihood that a given outcome will occur given a particular set of conditions. While artificial neural networks are described herein as an example machine learning technique, multiple other techniques, such as heuristic, statistical, or machine learning models, can be used to perform the data predictions and data modeling functions described in this document.

Front-end server <NUM> can be a component such as a client computer having a graphical user interface or other application program(s) (e.g., web browser) through which a user/third-party can interact with system <NUM>. For example, front-end server <NUM> is configured to exchange data communications with third-party interface <NUM> to receive data inputs such as target data <NUM> that are received from digital component providers. Third-party interface <NUM> can be an example user/mobile computing device, client computer, or any combination of software, middleware, or related front-end components.

Target data <NUM> can be specified in the form of a target score or other performance metric that is used to adjust the eligibility value based on the incremental outcome likelihood determined by machine learning models of system <NUM>. In some instances, the target score can be a percentage value of the eligibility value. For example, the eligibility value can be <NUM> and the target score can be <NUM>, where the target score is <NUM>% of the eligibility value (e.g., <NUM>). In some implementations, the target score indicates an expected gain (e.g., commercial gain such as revenue return) from a given outcome and the eligibility value defines a threshold value for controlling transmission of a digital component to the client device <NUM>.

The relationship between the target score and the eligibility value can be structured such that a specified target score (e.g., a revenue return from occurrence of a given outcome) is obtainable through more efficient resource spend. The improved efficiencies are realized when an eligibility value is adjusted, as described herein, to better account for incremental outcome likelihoods that are provided by distribution of a digital component. For example, a content provider can expend commercial resources to cause transmission of a digital component to client device <NUM> to increase the likelihood that a given outcome will occur.

The eligibility value can represent the amount of commercial resources being expended (e.g., <NUM>). The content provider can obtain or realize a desired revenue return (e.g., target score of <NUM>%) in response to the outcome occurring. However, in some situations, transmission of the digital component may have a negligible impact on causing occurrence of the given outcome. Because eligibility values can represent a resource spend to cause, or control, transmission of digital components, the described teachings provide options for adjusting eligibility values to obtain desired target scores (or revenue returns) with improved resource spend efficiency.

<FIG> is a block diagram of an example computing system <NUM> of the environment <NUM>. System <NUM> can correspond to a sub-system of system <NUM> and includes at least one back-end server <NUM>. System <NUM> can be a digital content provisioning network (e.g., an advertiser network) that analyzes user data and, based on this analysis, controls provisioning or transmission of digital components to client device <NUM>. As described in more detail below, system <NUM> is configured to use data models to predict the occurrence of certain item or conversion related outcomes based on analysis of the user data.

System <NUM> includes a first predictive model <NUM>, a second predictive model <NUM>, and computing logic <NUM>. First predictive model <NUM> and second predictive model <NUM> correspond to model outputs produced when predictive modeling systems of back-end server <NUM> train artificial neural networks of system <NUM>. Hence, first predictive model <NUM> and second predictive model <NUM> may be trained neural networks that are used as data models to generate predictions and inferences for determining the incremental outcome likelihood provided by distribution of a digital component <NUM>. As noted above, in addition to artificial neural networks, multiple other techniques, such as heuristic, statistical, or machine learning models, can be used to perform the data predictions and data modeling functions described herein.

The neural networks (or other learning systems) can be trained based on multiple datasets that include data about user interaction with electronic documents and digital components <NUM> displayed using web browser <NUM>. For example, the data can include specific keywords associated with the electronic document or entities (e.g., people, places, or things) that are referenced by the electronic document. In some implementations, system <NUM> transmits a request <NUM> to multiple different client devices <NUM> to obtain data about how different users interact with the electronic documents and digital components <NUM>.

The client devices <NUM> can respond by providing or transmitting user data <NUM> to system <NUM> in response to receiving the request <NUM>. In some implementations, conversion data is provided to modeling system <NUM> as user data <NUM> to facilitate training a machine learning system to predict whether a user is likely to convert, or initiate conversion of, certain items displayed using web browser <NUM>. In some instances, the conversion data can also include a search query that was submitted from the client device <NUM> to obtain a search results page. The data can also include conversion data related to other information, such as information that a user of the client device <NUM> has provided, or geographic information indicating a state or region in which the user resides.

In some implementations, the user data <NUM> includes other information that provides context for the environment in which digital component <NUM> will be displayed. For example, this context data can include a time of day of a component request <NUM>, a day of the week of the component request <NUM>, a type of client device <NUM> at which the digital component <NUM> will be displayed, such as a mobile device or tablet device. Upon completion of the neural network (or other learning system) training, models <NUM> and <NUM> can be used to accurately predict outcome likelihoods as it relates to a digital component <NUM> being shown, or not shown, to a user.

System <NUM> uses models <NUM> and <NUM> to generate probability values that quantify a likelihood that a given outcome will occur. For example, system <NUM> can use models <NUM> and <NUM> to predict the outcome likelihood of a given outcome, e.g., a user interaction with a particular portion of a web page or completion of a specified transaction at the web page. The predictions generated by the system <NUM> can include the likelihood of occurrence of an outcome with, and without, user interaction with a particular digital component <NUM>.

In some implementations, the digital component <NUM> may include graphical representations of data such as items of media content <NUM>-A and <NUM>-B. Upon interaction with the digital component <NUM>, graphical data included in media content <NUM>-A, <NUM>-B may influence or trigger the user to perform an action (e.g., user interaction with the media content <NUM>-A, <NUM>-B) which directs the user to a landing page linked to by the media content <NUM>-A, <NUM>-B and leads to occurrence of the outcome (e.g., completing a purchase at the landing page or another specified outcome such as downloading an application).

Outcomes that are modeled can include item conversions or procurements that would occur at a webpage following user interaction with a digital component (e.g., a digital advertisement) when the component is transmitted to client device <NUM> in response to an identified opportunity. Likewise, outcomes that are modeled can also include item procurements that occur at a webpage without user interaction with a digital component being transmitted to client device <NUM> in response to an identified opportunity.

For example, modeling system <NUM> can run predictive models <NUM> that include a predicted click-through rate model (pCTR) and a predicted Conversion Rate model (pCvR). Both models (pCTR and pCvR) are used to generate a probability value that quantifies a predicted likelihood of an outcome (e.g., item conversion) given distribution of a digital component <NUM> followed by user interaction with the digital component <NUM>. For example, the outcome can be user purchases of an item within a particular time window, e.g., <NUM> days, following the user interaction with the digital component <NUM> that was displayed at the client device <NUM>.

Modeling system <NUM> can also run a predictive model <NUM> that includes a pCvR model that generates a probability value quantifying a predicted likelihood of an outcome (e.g., item conversion) given no digital component <NUM> being displayed at the client device <NUM>. For example, the pCvR model can predict a likelihood of a conversion from any given user within the same time window (e.g., <NUM> days), given no presentation of the digital component <NUM>.

An example use case can include an online advertisement auction where, for each time a user is eligible to be presented a digital component <NUM> at the client device <NUM>, modeling system <NUM> generates two prediction parameters. A first parameter, produced by model <NUM>, indicates a first probability of user conversion given the digital component <NUM> being presented to the user and the user clicking or otherwise interacting with the digital component <NUM>. A second parameter, produced by model <NUM>, indicates a second probability of user conversion given no presentation of the digital component <NUM> at the client device <NUM>.

The first probability and the second probability can be used by system <NUM> to account for, or differentiate between, the likelihood of a given outcome occurring when the digital component <NUM> is distributed and the likelihood that the given outcome would have occurred in the absence of the digital component <NUM> being distributed. For example, modeling system <NUM> can use back-end server <NUM> to determine a ratio of the first probability relative to the second probability. In particular, the first probability of a conversion given by model <NUM> (e.g., the first parameter value) is divided by the second probability of a conversion given by model <NUM> (e.g., the second parameter). This ratio represents the "incremental outcome likelihood" discussed above and can correspond to an outcome incrementality factor (or parameter) that quantifies an incrementality of a conversion.

Incrementality of a conversion corresponds to how much more likely a user is to convert or procure an item if digital component <NUM> is presented to the user (and interacted with by the user) at the client device <NUM> versus if digital component <NUM> is not presented to the user. As used herein, user conversion or procurement can be incremental if system <NUM> determines that a user is less likely to convert without interaction with digital component <NUM> being presented to the user at the client device <NUM>. In some implementations, truly "incremental" conversions correspond to outcomes that have a significantly higher probability of occurring given that a user was presented and/or interacted with the digital component <NUM> than a scenario in which digital component <NUM> is not presented to the user.

As discussed above, digital components <NUM> can be transmitted based on some eligibility value received at system <NUM>, e.g., value <NUM> received from interface <NUM>, as shown in <FIG>. System <NUM> can adjust the eligibility value <NUM> to account for incremental increases in the likelihood of an event occurring that is caused by the distribution of the digital component <NUM>. In some implementations, system <NUM> provides an adjusted eligibility value <NUM> (<FIG>) to interface <NUM> for presentation to digital component providers and for use in controlling transmission of digital components <NUM> for presentation at the client device <NUM>.

System <NUM> can use the outcome incrementality factor (e.g., the computed ratio of the first parameter and the second parameter) to trigger adjustment of an eligibility value that controls transmission of digital components <NUM> to client device <NUM>. In some situations, an entity (e.g., an advertiser) can submit eligibility values, e.g., value <NUM> of <FIG>, to cause digital components <NUM> to be transmitted to the client device <NUM>.

The eligibility value be can a pecuniary amount (e.g., a bid amount) an entity will expend to transmit the digital component <NUM> to achieve a desired conversion outcome (e.g., earn a target score or realize some commercial gain). The conversion outcome is achieved when a user converts an item following interaction with the digital component <NUM> presented at client device <NUM>. System <NUM> executes computing processes to adjust eligibility values to better account for conversions that are more "incremental. " Such adjustments enable entities to efficiently expend pecuniary resources to cause transmission of digital components <NUM> to achieve a given outcome.

Providers of digital content can enter, and use, eligibility values for incremental conversions in multiple ways. First, eligibility values for incremental conversions can be received by interface <NUM> and used as a bid to indicate a pecuniary amount the digital content provider is willing to expend for a conversion that is deemed to be truly "incremental. " As indicated above, truly "incremental" conversions can be outcomes that have a significantly higher probability of occurring given that a user was presented and/or interacted with the digital component <NUM> than a scenario in which digital component <NUM> is not presented to the user.

Second, eligibility values for incremental conversions can be received by interface <NUM> and a computed incrementality factor can be used as a bid multiplier that is adjustable based on a predicted incrementality of a conversion. An incrementality factor that can be used as a bid multiplier offers the advantage of allowing digital content provider to gradually tune a multiplier to adjust resource expenditures for more incremental conversions. For example, use of an adjustable bid multiplier allows content providers to specify percentage increases to a submitted eligibility value for more incremental conversions (e.g., bidding up to <NUM>%, <NUM>%, or <NUM>% more for conversions that are believed to be truly incremental).

Third, eligibility values and target data <NUM> can be received by interface <NUM> and the eligibility value can be used to adjustable based on a computed incremental outcome likelihood (e.g., incrementality factor) that predicts the incrementality of a conversion. In particular, target data <NUM> can be specified in the form of a target score or other performance metric and the eligibility value is adjusted based on the incremental outcome likelihood determined by machine learning models of system <NUM>.

The adjusted eligibility value enables the digital content provider to earn the target score with efficient resource spend because the eligibility value is adjusted to reflect the extent to which displaying the digital component causes the given outcome to occur. Stated another way, this target incremental return on expenditure approach involves adjusting an eligibility value to discount, or reduce, bid amount for conversion outcomes that are not as incremental (e.g., discounting bid to account for minimal increase in likelihood of conversion from presenting the digital component <NUM> relative to the target/desired revenue return from occurrence the given outcome).

For example, modeling system <NUM> can adjust an eligibility value to achieve the desired target score based on probability values determined by models <NUM> and <NUM>. Logic <NUM> can include software instructions and other programmed code for executing scoring and value adjustment features of back-end server <NUM>. In some implementations, modeling system <NUM> uses logic <NUM> to adjust an eligibility value based on an outcome incrementality factor for digital component <NUM>. Modeling system <NUM> can use the incrementality factor to determine an increase or boost in a probability of a conversion that is an outcome of presenting a digital component <NUM> at client device <NUM>.

For example, prediction data of the models can indicate that there is a <NUM>% chance of a given conversion outcome occurring if no digital component <NUM> is presented at client device <NUM> and/or interacted with by the user. However, the prediction data can also indicate that there is a <NUM>% chance of the user converting if digital component <NUM> is presented at client device <NUM>. In the preceding example, it can be estimated that presenting digital component <NUM> at client device <NUM> contributes only a <NUM>% overall increase, or boost, in the likelihood that a given conversion outcome will occur assuming that digital component <NUM> is presented at client device <NUM>.

Referring to the third approach discussed above for adjusting an eligibility value, a digital content provider can use third-party interface <NUM> to input an eligibility value that is a cost per incremental conversion bid of $<NUM> (e.g., a target score). An example content provisioning network executing system <NUM> may adjust (e.g., discount or reduce) the eligibility value to a bid of $<NUM> for presenting digital component <NUM> at the client device <NUM>.

Here, the adjusted eligibility value reflects a more accurate estimation of the contribution to a conversion that is a direct outcome of presenting digital component <NUM> at client device <NUM> (e.g., a more incremental conversion). In particular, presenting digital component <NUM> at the client device <NUM> for display to a user increased the likelihood of conversion by only <NUM>% (i.e., <NUM>% with no digital component <NUM> versus <NUM>% with digital component <NUM>). So, the entered eligibility value of $<NUM> is adjusted to $<NUM> to indicate an eligibility value that is <NUM>% of $<NUM>. Hence, use of the described teachings can enable improved computing and commercial efficiencies by accurately aligning resource expenditures with the revenue return that results from users viewing and/or interacting with digital components <NUM>.

In yet another example, an end user is predicted to have a <NUM>% probability of interacting with a digital component <NUM> that is presented at the client device <NUM> for display to a user. Model <NUM> is used to determine that, after interacting with digital component <NUM>, the user is predicted to have a <NUM>% probability of converting on a publisher or digital content provider's webpage. In some implementations, back-end server <NUM> includes an electronic user value (eValue) model, e.g., included as part of models <NUM> and <NUM>, that estimates a pecuniary value of a conversion, or of multiple conversions if more than one conversion is estimated to occur. Assuming at least one conversion occurs, the eValue model may estimate the value of the conversion(s) as $<NUM>.

In contrast, model <NUM> is used to determine or predict that there is a <NUM>% likelihood that this user will convert within a particular time window (e.g., <NUM> days) assuming no digital component <NUM> is presented at the client device <NUM>, or irrespective of whether digital component <NUM> is presented at the client device <NUM>. Moreover, the eValue model estimates the value of the conversion(s) at $<NUM> even if no digital component <NUM> is presented at the client device <NUM>.

Now the expected return from this user, assuming no digital component <NUM> is presented at the client device <NUM>, is $<NUM> (<NUM>% chance of conversion × $<NUM> estimated conversion value). Alternatively, the expected return assuming digital component <NUM> is presented at the client device <NUM> is $<NUM>, derived from the computation of <NUM>% (likelihood of interacting with digital component <NUM>) × <NUM>% (likelihood of conversion after interacting with digital component <NUM> × $<NUM> (eValue)).

For the preceding example, the incremental return from presenting digital component <NUM> at the client device <NUM> is: $<NUM> (component presented) - $<NUM> (no component) = $<NUM>. A provider of digital content may enter a target data <NUM> that includes a desired/target incremental return on expenditures of <NUM>%. In other words, for every $<NUM> the digital content provider spends, or expends, to cause transmission of digital components to client device <NUM>, the provider would like to realize a revenue return of $<NUM> above revenue the provider otherwise would have received from this user. This revenue return is <NUM>% more than the content provider spends to cause transmission of the digital component <NUM> to influence the likelihood that a given outcome will occur.

For this example, system <NUM> can adjust an existing eligibility value, or generate an initial eligibility value, to include a pecuniary bid amount of $<NUM>. By setting the eligibility value, or bid amount, to $<NUM> for an estimated incremental return of $<NUM>, system <NUM> has satisfied the digital content provider's requirement of <NUM>% return on expenditures for causing transmission of digital components to client device <NUM>.

<FIG> is a flow chart of an example process <NUM> for adjusting an eligibility value for transmitting a digital component. Process <NUM> can be implemented using system <NUM> described above. Thus, descriptions of process <NUM> may reference one or more of the above-mentioned components, models, or computational devices of system <NUM>. In some implementations, described actions of process <NUM> are enabled by computing logic or software instructions executable by a processor and memory of an example electronic device, such as client device <NUM>, system <NUM>, or both.

At block <NUM> of the process <NUM>, back-end server <NUM> identifies an opportunity to transmit a digital component to client device <NUM>. For example, back-end server <NUM> can identify an opportunity to transmit a digital component to client device <NUM> based on the user data <NUM> about user interactions with a webpage displayed at client device <NUM>. In some implementations, the opportunity is identified when a user visits particular publisher web pages that are configured to display content to cause completion of a specified transaction.

At block <NUM>, back-end server <NUM> determines a first probability of a given outcome occurring following user interaction with the digital component assuming that the digital component is transmitted to the client device in response to the identified opportunity. For example, model <NUM> of server <NUM> can determine a first probability that the given outcome will occur following user interaction with the digital component assuming that the digital component is transmitted for presentation in a webpage responsive to the request.

At block <NUM>, back-end server <NUM> determines a second probability of the given outcome occurring assuming that the digital component is not transmitted to the client device in response to the identified opportunity. For example, model <NUM> of server <NUM> can determine a second probability that the given outcome will occur assuming that the digital component is not transmitted for presentation in the webpage.

At block <NUM>, the back-end server <NUM> generates an outcome incrementality factor for the digital component. In some implementations, generating the outcome incrementality factor for the digital component includes determining a ratio of the first probability relative to the second probability. In some instances, generating the outcome incrementality factor comprises determining an extent to which transmission of the digital component changes a probability that the user will perform an action that results in the given outcome.

At block <NUM>, the back-end server <NUM> triggers adjustment of an eligibility value that controls transmission of the digital component based on the outcome incrementality factor for the digital component. At block <NUM>, the back-end server <NUM> controls transmission of the digital component to the client device using the adjusted eligibility value.

<FIG> is a block diagram of computing devices <NUM>, <NUM> that may be used to implement the systems and methods described in this document, either as a client or as a server or plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, smartwatches, head-worn devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and/or claimed in this document.

In various different implementations, the storage device <NUM> may be a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations.

The high-speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations.

The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM>. The processor may also include separate analog and digital processors.

The display <NUM> may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. External interface <NUM> may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).

Expansion memory <NUM> may also be provided and connected to device <NUM> through expansion interface <NUM>, which may include, for example, a SIMM card interface. Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

The memory may include for example, flash memory and/or MRAM memory, as discussed below. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, or memory on processor <NUM>.

In addition, GPS receiver module <NUM> may provide additional wireless data to device <NUM>, which may be used as appropriate by applications running on device <NUM>.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs, computer hardware, firmware, software, and/or combinations thereof.

These computer programs, also known as programs, software, software applications or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any computer program product, apparatus and/or device, e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

As discussed above, systems and techniques described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component such as an application server, or that includes a front-end component such as a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication such as, a communication network.

Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs or features described herein may enable collection of user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, in some embodiments, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the present disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Also, although several applications of the payment systems and methods have been described, it should be recognized that numerous other applications are contemplated. Accordingly, other embodiments are within the scope of the following claims.

Claim 1:
A method, comprising:
identifying (<NUM>), by a back-end computing server (<NUM>), an opportunity for a digital component server (<NUM>) to transmit a digital component (132A, 132B) to a client device (<NUM>) for display in an electronic document in response to the digital component server receiving a request (<NUM>) from the client device, the digital component selected by the digital component server for presentation in the electronic document in response to receiving the request;
determining (<NUM>), by the back-end computing server, a first probability of a given outcome occurring following user interaction with the digital component assuming that the digital component is transmitted to the client device for presentation in the electronic document responsive to the request;
the method being characterized by:
determining (<NUM>), by the back-end computing server, a second probability of the given outcome occurring assuming that the digital component is not transmitted to the client device for presentation in the electronic document responsive to the request; and
triggering, by the back-end computing server, transmission of a response (<NUM>) to the request from the digital component server to the client device based on a difference between the first probability and the second probability, said response comprising the digital component,
wherein transmission of the digital component is prevented when the first probability does not exceed the second probability by at least a specified amount.