System and method for ranking, allocation and pricing of content recommendations

A server computer system receives campaign data from one or more content providers. The server computer system determines assignments of recommendation opportunities in documents to recommendations based on the campaign data. The server computer system ranks the recommendations based on the assignments, allocates the recommendations to the documents based on the rankings of the recommendation opportunities, and determines pricing for the recommendations based on the allocations.

TECHNICAL FIELD

The present disclosure relates to content recommendations and, more particularly, to a technique of ranking, allocating and pricing content recommendations.

BACKGROUND

In the digital advertising industry, the onus of revenue-generating activity is shifting from selling interruptive and irrelevant advertisements to providing integrated high-quality content that is interesting, helpful and trustworthy to readers. The digital advertisement industry has not kept pace with this paradigm shift, and digital solutions that natively support content marketing are lacking. Traditionally, content providers place bids to have their content placed as recommendations on various websites. In general, bidding platforms promise to allocate content to the highest bidder on an impression-by-impression basis. Some conventional bidding platforms use Generalized Second Prize (GSP) to position the buyer bidding the highest first in the bidding queue. The buyer (e.g., content provider) thereupon pays the price suggested by the second highest bid. Traditional bidding platforms do not offer means to determine balanced ranking, allocation and pricing of content recommendations.

DETAILED DESCRIPTION

A system and method for ranking, allocating and pricing recommendations is described, according to various implementations. Content providers may wish to have their content distributed as recommendations on various destination documents. A content recommendation or recommendation hereinafter refers to a representation of digital content. A recommendation can include one or more text, links, thumbnails, images, icons, etc. A document hereinafter refers to webpage, a mobile application graphical user interface (GUI), etc. For example, a web blogger, John Smith, may wish to have his web blog post of his analysis of a baseball game to be included as a recommendation for users that are accessing a publisher's (for e.g. SportsNews, Inc.) webpage describing the highlights of the same baseball game. An impression hereinafter refers to an instance of populating a destination document (e.g., webpage, mobile application GUI) with a content recommendation. For example, the web blog post of John Smith may be an impression if the web blog post is served as a recommendation on the SportsNews, Inc. webpage.

The content provider (e.g., web blogger John Smith) may use a third party recommendation service provider to serve his content to various destination documents. The content provider can specify to the third party recommendation server a budget and/or the cost-per-click (CPC) the content provider is willing to pay for the service. “Content provider” is hereinafter also referred to as “buyer.” The CPC is the amount the buyer is willing to pay each time a content recommendation is selected. For example, the content provider may have a maximum budget of $800 and may be willing to pay a maximum CPC of $1.

Implementations dynamically provide balanced estimation, ranking, allocation and pricing of content recommendations. Implementations enable a variety of content recommendations to be served as impressions by ranking the content recommendations, determining an optimal allocation of which content should be served as the impressions for various destinations documents (e.g., webpages, mobile applications user interfaces, etc.) based on the rankings, and by determining an optimal price a content provider should pay based on the allocations, while taking into account the parameters (e.g., budget, CPC) that are specified by the content provider. Implementations can dynamically determine the optimal ranking, allocation and pricing of recommendations and can dynamically adjust the optimal ranking, allocation and pricing based on real-time data.

FIG. 1illustrates an example system architecture100in which examples of the present disclosure can be implemented. The system architecture100can include one or more user devices160A-B, one or more content provider servers101,103,105, one or more destination servers (e.g., publishing servers120) and one or more third party recommendation servers140coupled to each other over one or more networks130.

Content providers may have recommendations (e.g., recommendations107,109,111) they wish to have displayed, for example, in a publisher document (e.g., document150,160). A document150,160may be for example, and is not limited to a website homepage, section front, webpage, mobile application user interface (UI), gaming application UI, television application UI, etc. For example, content provider web blogger John Smith may have multiple recommendations107of various web blog posts and may wish to have the recommendations107served in document150(e.g., SportsNews webpage) and/or document160(e.g., BaseballNews website). A recommendation107,109,111can be a representation of digital content. Digital content can include, for example, such as, and not limited to, an article, a web blog entry, a social media posting, a data feed, an advertisement, a document (e.g., webpage), digital video, digital movies, digital photos, digital music, social media updates, electronic books (ebooks), electronic magazines, digital newspapers, digital audio books, electronic journals, electronic comic books, software applications, etc. Each content provider can have multiple recommendations107,109,111.

A content recommendation host or “publisher” hereinafter refers to an entity and platform hosting destination documents (e.g., webpages, mobile application graphical user interfaces (GUIs), gaming application GUIs, television application GUIs, etc.) that include recommendations107,109,111of the content providers. For example, SportsNews, Inc. and BaseballNews, Inc. may be publishers. The publisher platforms can include publishing servers120. The publisher servers120can be application servers that provide one or more applications (e.g., web application, mobile application, desktop application, gaming console applications, television console application, etc.). The publishing servers120can be coupled to one or more data stores that store publisher data and/or recommendations of content providers.

The publishers' destination documents150,160can be rendered with the recommendations107,109,111of the content providers in user devices160A-B using for example, an application (e.g., web browser, mobile application, desktop application, gaming application, television application, etc.) on the user device160A-B.

A document150,160can be within a source147,157. The source147,157can be for example, and is not limited to, a publisher's website, a publisher's mobile application, publisher's gaming application, publisher's television application, etc. A destination document150,160can include publisher content (e.g., publisher content153,163). For example, the SportsNews website (e.g., source147) can include a webpage (e.g., document150) that contains an article about a baseball game (e.g., publisher content153).

The publisher destination documents150,160can be configured with one or more recommendation opportunities155,165, which can be populated with recommendations107,109,111. For example, a document150(e.g., SportsNews webpage) may have four recommendation opportunities155and a document160(e.g. BaseballNews mobile application GUI) may have three recommendation opportunities165. The recommendation opportunities155,165may be populated with recommendations107,109,111from one or more content providers' servers101,103,105.

The content providers may wish to publish their respective recommendations107,109,111in widespread locations (e.g., document150, document160). The content providers can use the service hosted by the third party recommendation service provider to have the recommendations107,109,111of the content providers served to populate the recommendation opportunities155,165in the publisher documents150,160. The publishers (e.g., SportsNews Inc., BaseballNews, Inc.) can use the service hosted by the third party recommendation service provider to receive recommendations107,109,111to populate the recommendation opportunities155,165in their respective publisher documents150,160.

The content providers may purchase campaigns from the third party recommendation service provider. The campaigns provide recommendations107,109,111from the content providers to one or more destination documents150,160in accordance with criteria selected by the content provider and parameters of the documents150,160set by the publishers. The destination documents150,160may have parameters defining the scope of which recommendations107,109,111may be included in the destination documents150,160. For example, the parameters can include, and are not limited to, category (e.g., news), sub-category (e.g., international news), country, platform, language, etc.

The content providers may specify parameters for the campaign, such as, and not limited to, a maximum budget (B) of what the content provider is willing to pay for the campaign and the maximum CPC that the content provider is willing to pay each time users (e.g., users161A,B) select a recommendation.

The third party recommendation service provider can host one or more third party recommendation servers140to provide the service to the content providers and publishers. The servers140can include an optimization module145to determine an optimal allocation of which content provider recommendations107,109,111should be served as impressions to populate the recommendation opportunities155,165in the documents150,160while taking into account the campaign parameters, the document parameters, and the likelihood that the impressions would be selected by users161A-B and provide high user-engagement while generating click-through revenue. The publishers (e.g., SportsNews Inc., BaseballNews, Inc.) may wish to include impressions in their respective documents150,160that are of sufficiently high quality while providing a substantial income stream in return for hosting the recommendations. The optimization module145can select the recommendations107,109,111to be served which meet the criteria of the content providers and publishers. The optimization module145can select the recommendations107,109,111that provide the most valued content while still providing substantial overall revenue and increasing user-engagement throughout the network.

The network of content providers can include, for example, several hundred thousand recommendations107,109,111that may be stored in one or more data stores that are coupled to the network130. The architecture100can include, for example, several hundred publisher documents150,160that are accessed by multiple users161A-B on various user devices160A-B.

For example, document150may have four recommendation opportunities155. When a particular user (e.g., user161A) accesses the particular document150, the optimization module145can determine which set of recommendations107,109,111should be served as impressions to populate the four recommendation opportunities155in the document150. As each user161A-B accesses a document150,160, the optimization module145can determine which set of recommendations107,109,111should be served for the particular user and document combination.

The click-through rate (CTR) is the estimated probability (calculated as a percentage) that a user161A-B will select the recommendation if the recommendation populates a recommendation opportunity in a destination document. The CTR can be determined from user activity data that is stored in one or more data stores that a coupled to the network130.

The data stores can be persistent storage units coupled to the network130. A persistent storage unit can be a local storage unit or a remote storage unit. Persistent storage units can be a magnetic storage unit, optical storage unit, solid state storage unit, electronic storage units (main memory), or similar storage unit. Persistent storage units can be a monolithic device or a distributed set of devices. A ‘set’, as used herein, refers to any positive whole number of items.

The networks130can include one or more local area networks (LAN), one or more wireless networks, one or more mobile communications networks, one or more wide area networks (WAN), such as the Internet, or similar communication systems, or combinations of such. The networks130can include one or more networking and/or computing devices such as wired and wireless devices. In one implementation, not all servers101,103,105,120,140, and/or user devices160A-B are directly connected to each other. The networks130may be separate networks.

The servers101,103,105,120,140can be hosted on one or more machines. The machines can include for example, and are not limited to, any data processing device, such as a desktop computer, a laptop computer, a mainframe computer, a personal digital assistant, a server computer, a handheld device or any other device configured to process data.

A user device160A-B can be a computing device such as a server computer, a desktop computer, a set-top box, a gaming console, a television, a portable computing device such as, and not limited to, mobile telephones, personal digital assistants (PDAs), portable media players, netbooks, laptop computers, an electronic book reader and the like. For example, user device160A may be a laptop computer and user device160B may be a tablet computer.

FIG. 2is a block diagram of an optimization module200, in accordance with various implementations. The optimization module200can be the same as the optimization module145inFIG. 1. The optimization module200can include a user selection probability estimation sub-module201, a real-time ranking, allocation and pricing sub-module203, a master plan builder sub-module205, and an inventory estimation sub-module207. Note that in alternative implementations, the functionality of one or more of the user selection probability estimation sub-module201, the real-time allocation, ranking and pricing sub-module203, the master plan builder sub-module205, and the inventory estimation sub-module207are combined or divided. Note that in alternative implementations, the functionality of the optimization module200can be separate modules.

The optimization module200can be coupled to one or more data stores210,220,230,250,260,270,280that store data. The data stores210,220,230,250,260,270,280can be persistent storage units. A persistent storage unit can be a local storage unit or a remote storage unit. Persistent storage units can be a magnetic storage unit, optical storage unit, solid-state storage unit, electronic storage units (main memory), or similar storage unit. Persistent storage units can be a monolithic device or a distributed set of devices. A ‘set’, as used herein, refers to any positive whole number of items.

The data stores210,220,230,250,260,270,280can store input and/or output data. The modules (e.g., user selection probability estimation sub-module201, the real-time ranking, allocation and pricing sub-module203, the master plan builder sub-module205, and the inventory estimation sub-module207) can be coupled to the one or more data stores210,220,230,250,260,270,280and can read from and write to the one or more data stores210,220,230,250,260,270,280.

Data store210can store recommendations213and document parameters215. The recommendations213can be provided by content providers. A recommendation213is a representation of digital content. A recommendation213can include, for example, and is not limited to, one or more text, links, thumbnails, images, icons, etc.

Data store220can store campaign data221. The campaign data221can be provided by multiple content providers. The campaign data221can include parameters, such as, and not limited to, identifiers of the content for use by the campaign, a total budget (B) for the campaign, the maximum cost per click (CPC) for the campaign, the desired average CPC for the campaign, and links to the recommendations for use by the campaign, etc. For example, the data store210may store one hundred recommendations (e.g., Recommendation-1to Recommendation-100) for Content-Provider-A. The campaign data221for Content-Provider-A may specify twenty recommendation identifiers out of the one hundred recommendations that should be used by Campaign-A, links to the twenty recommendations, a total budget B of $1,000 for Campaign-A, a maximum CPC of $1 for Campaign-A and a desired average CPC of 70 cents for Campaign A.

The inventory estimation sub-module207can determine the expected number of views for the various destination documents (e.g., documents150,160inFIG. 1). For example, the inventory estimation sub-module207may predict that Document-XYZ may have 1 million views per day. In another example, the inventory estimation sub-module207may predict that Document-DEF may have 10 million views per day. The inventory estimation sub-module207can use the user activity data281stored in a data store280that is coupled to the optimization module200, one or more prediction models, statistical analysis, etc. to determine the expected number of views for a document. The prediction models can be stored in data store280. The user activity data281can include historical data that can be used to determine the expected number of views for a destination document. The inventory estimation sub-module207can store the expected number of views (e.g., predicted views261) for the various documents in the data store260.

In one implementation, the inventory estimation sub-module207can predict the number of views for a destination document based on data for the subsections of a publisher network. The inventory estimation sub-module207can identify a subsection of the publisher network using publisher subsection data283. The publisher subsection data283can group documents of the various publishers based on one or more attributes, such as, and not limited to, category and sub-category. A category can include one or more sub-categories. An example of a category can include, and is not limited to news, and an example of a sub-category for news can include, and is not limited to, international news. The publisher subsection data283can assign the documents of the various publishers to one or more categories and/or sub-categories. The assignments can be based on user (e.g., system administrator, content provider administrator) input. In one implementation, the inventory estimation sub-module207can predict the number of views for a destination document based on an average number of views of documents in the same category and/or sub-category as the destination document. For example, Document-XYZ may be in the news category and the sports news sub-category. The average number of views for the documents in the subsection of sports news in the publisher network may be 500,0000 views per day, and the inventory estimation sub-module207may predict that Document-XYZ will have 500,0000 views per day.

The optimization module200may collect registration information, personal information, and/or demographic information pertaining to users and/or groups of users (e.g., age of user groups, geography (e.g., country, state, city) of user groups) and may store the information in one or more data stores (e.g., hard disks, memories, databases, etc.) (e.g., data store210,280). In situations in which the implementations discussed herein collect personal information and/or demographic about users, or may make use of personal information and/or demographic information, the user may be provided with an opportunity to control whether the optimization module200is allowed collect the information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content from the content sharing platform that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. Thus, the user may have control over how, when, and what type of information is collected about the user and used by the optimization module200.

The master plan builder sub-module205can generate an allocation plan551. The master plan builder sub-module205can use the number of predicted views261, the number of recommendation opportunities (e.g., recommendation opportunities155inFIG. 1) for each destination document (e.g., Document-XYZ150inFIG. 1) as specified in the document parameters215, other document parameters (e.g., language, country, device platform, application operating system platform, etc.) specified in the document parameters215, and the campaign data221to generate an allocation plan251. The master plan builder sub-module205can also use the historical data accumulated in the data stores230and240. The master plan builder sub-module205can store the allocation plan251in the data store250. The allocation plan251is a plan for the aggregate of the campaign data221from the content providers. The allocation plan251can indicate the desired number of impressions for the recommendations213for the various destination documents. An impression is an instance of populating a destination document (e.g., webpage, mobile application GUI) (e.g., Document-XYZ inFIG. 1) with a content recommendation213(e.g., recommendations107,109,11inFIG. 1). The allocation plan251can include quotas at the recommendation level. For example, Content-Provider-A may have Recommendation-1(e.g., online article) as part of Campaign-A. The allocation plan251may indicate that “15,000 impressions” on Document-XYZ should be allocated to Recommendation-1based on the document parameters215(e.g., language, country, device platform, application operating system platform, number of recommendation opportunities, etc.) for Document-XYZ. Implementations of creating an allocation plan are described in greater detail below in conjunction withFIG. 4andFIG. 5.

The user selection probability estimation sub-module201can determine selection probabilities231for recommendations213and can store the selection probabilities231in the data store230. The selection probability231is the likelihood that a user (e.g., user161A inFIG. 1) may select an impression of a recommendation213that populates a recommendation opportunity (e.g., opportunity155inFIG. 1) in a destination document (e.g., Document-XYZ150inFIG. 1) within the context, for example, of the user's current session with the destination document. For example, User-Jane-Doe may view destination Document-XYZ to read a new book review. The user selection probability estimation sub-module201can estimate how likely User-Jane-Doe will select a recommendation213if the recommendation213is served as an impression on Document-XYZ within the context of User-Jane-Doe reading the book review. For example, the user selection probability estimation sub-module201may determine that Recommendation-1has a selection probability231of 5%, Recommendation-2has a selection probability231of 2%, Recommendation-3has a selection probability231of 60%, etc.

The user selection probability estimation sub-module201can use the user parameters217for the particular user (e.g., User-Jane-Doe) stored in the data store210, prediction models, and statistical analysis to determine the selection probability231. The user parameters217can include, for example, and are not limited to, location information of the user, device information of the user, user preferences, user interests, etc. The user selection probability estimation sub-module201can also determine the selection probabilities231for impressions of recommendations that have not yet been served to destination documents.

The real-time ranking, allocation and pricing sub-module203can allocate the recommendations213to the destination documents based on a lottery. In one implementation, the real-time ranking, allocation and pricing sub-module203allocates the recommendations217to the destinations documents based on a lottery using the allocation plan251and the selection probabilities231.

The real-time ranking, allocation and pricing sub-module203can include a ranking component291, an allocation component293, and a pricing component295. The ranking component291can determine the grades241for the recommendations213in the allocation plan251and store the grades241in the data store240. In one implementation, the ranking component291determines the grades241based on the selection probabilities231and the maximum CPC a buyer (e.g., content provider) is willing to pay. For example, the grade241for a recommendation213in the allocation plan251may be calculated as the product of the selection probability231for the recommendation213multiplied by the CPC for the recommendation213. For example, Recommendation-1may be assigned a grade241of “5” and Recommendation-2may be assigned a grade241of ‘7’. In another example, the grades241may be calculated as the product of (selection probability)α×(CPC)β. In one implementation, α and β are values selected by the ranking component291, for example, based on configuration data that may be stored in a data store (e.g., data store240). In one implementation, the optimization module200can be coupled to another system that determines the grades241, which the real-time ranking, allocation and pricing sub-module203can access. The ranking component291can use alternative computations to determine the grades241.

The allocation component293can use the grades241calculated by the ranking component291to derive the lottery probabilities243for the recommendations213in the allocation plan251based on the proportions of the grades241. The allocation component293can process any set of grades241and is not limited to the grade calculations described above. The lottery probability243is the probability that a recommendation213in the allocation plan251may be selected for allocation to be served as an impression in a destination document. The lottery probabilities243can be stored in the data store240. In one implementation, the allocation component293calculates the lottery probability243for each of the recommendations213in the allocation plan251as a proportion of the grade241of the particular recommendation213to the total of the grades241of the recommendations213.

For brevity and simplicity, an example illustrating the calculation of the lottery probabilities243is described using two recommendations213in an allocation plan251. For example, Recommendation-1may be assigned a grade241of “5” and Recommendation-2may be assigned a grade241of ‘7’. The total of the grades241is “12”. The allocation component293may determine that Recommendation-1has a lottery probability243of 5/12, and that Recommendation-2has a lottery probability243of 7/12.

The allocation component293can determine the allocations245of the recommendations213based on the proportion of the grades (i.e., the lottery probabilities251). The allocations245can be stored as an allocation order in the data store240. For example, the allocation component293may select the recommendations213for allocation in order of lottery probability243. For example, the recommendation213having the highest lottery probability243may be selected first, etc. The allocation component293can position the first selected recommendation213in the most favorable position (e.g., top) in the allocation order (e.g., allocations245). The allocation component93can use the allocation plan251, together with the grades241, to determine the lottery probabilities243. Recommendations213with high allocation plan priority can be assigned a higher lottery probability243than the one warranted by its respective grade241. In one implementation, the allocation of recommendations213to positions on the document is performed by sequential lottery. For example, there may be two positions for recommendations, and Recommendation-A may have a lottery probability of 0.7, Recommendation-B may have a lottery probability of 0.2, and Recommendation-C may have a lottery probability of 0.1. The allocation component293can perform the lottery for position one, with the above lottery probabilities. For example, the lottery can select Recommendation A for position one. Then, the allocation component293can perform a second lottery for position two, which may select Recommendation-C (even though Recommendation-B has a higher lottery probability).

The pricing component295can determine the price (e.g., CPC) a buyer (e.g., content provider) should pay for a user selection of a recommendation213in the allocations245. A buyer may specify a maximum CPC for a campaign in the campaign data221. The price that is determined by the pricing component295may be less than or equal to the maximum CPC. The pricing component295can store the optimal pricing247in the data store240. The pricing component295can also take into account the allocated position for the recommendation.

The allocation component293can allocate each of the recommendations213in the allocations245to a specific position to create a ranked recommendation list271and can store the ranked recommendation list271in the data store270. The real-time ranking, allocation and pricing sub-module203can provide the ranked recommendation list271to a system (e.g., serving system) to serve the recommendations213to the servers hosting the destination documents, for example, based on the ranking order in the ranked recommendation list271. For each recommendation213in the allocations245, for example, the ranked recommendation list271can include, and is not limited to, a recommendation identifier, a rank identifier, a price, the associated content provider, etc.

FIG. 3is a flow diagram of an implementation of a method300for ranking, allocating and pricing recommendations. The method may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method300may be performed by the optimization module145hosted in a server140ofFIG. 1or by the optimization module200hosted inFIG. 2.

At block301, the server receives campaign data from one or more content providers. For example, the server receives campaign data for a campaign from Content-Provider-A and campaign data for a campaign from Content-Provider-B. The campaign data can include parameters, such as, and are not limited to, identifiers of the recommendations for use by the campaign, a total budget (B) for the campaign, the maximum CPC for the campaign, the desired average CPC for the campaign, links to the recommendations for use by the campaign, etc. For example, the server may store fifty recommendations provided by Content-Provider-A, and the campaign data for Content-Provider-A may specify that twenty of those recommendations (e.g., online articles, web blogs, social network posts, data feeds, etc.) should be used in the campaign for Content-Provider-A.

At block303, the server creates an allocation plan based on the campaign data from the content providers and the predicted number of views for the destination documents. The allocation plan can describe the optimal number of impressions for the recommendations that will be served on the various destination documents. For example, Content-Provider-A may have Recommendation-1(e.g., online article) as part of a campaign. The allocation plan may indicate that “15,000 impressions” on Document-XYZ should be allocated to Recommendation-1. Implementations of creating an allocation plan are described in greater detail below in conjunction withFIG. 4andFIG. 5.

At block305, the server determines selection probabilities based on user activity data. The selection probabilities represent the likelihood that a user will select a recommendation that is served as an impression in the destination document based on the context of the user's current session with the destination document. The server can determine the selection probabilities in real time. The allocation plan may indicate that “5,000 impressions” on Document-XYZ should be allocated to Recommendation-1and “10,000 impressions” on Document-XYZ should be allocated to Recommendation-2. The server can estimate how likely User-Jane-Doe will select each of the recommendations (e.g., Recommendation-1, Recommendation-2) in the allocation plan. For example, the server may determine that Recommendation-1has a selection probability231of 5% and that Recommendation-2has a selection probability231of 2%.

At block307, the server calculates the grades for each of the recommendations in the allocation plan. The calculated grades induce a natural ranking of the recommendations by their grade value. The server can use the selection probabilities and additional recommendation parameters such as the maximum CPC a buyer (e.g., content provider) is willing to pay, to determine grades for the recommendation in the allocation. In one implementation, the grade is calculated as the product of the selection probability for the recommendation multiplied by the maximum or desired CPC for the recommendation.

At block309, the server allocates the recommendations to the destination document and positions based on the allocation plan, grades and selection probabilities. The server can use the grades to determine lottery probabilities for the recommendations in the allocation plan. The lottery probability is the probability that a recommendation in the allocation plan may be selected for allocation to be served as an impression in a destination document. In one implementation, the server calculates the lottery probability for each of the recommendations in the allocation plan as a proportion of the grade of the particular recommendation to the total of the grades of the recommendations. In the preferred implementation, the server adjusts the proportion-based grade.

The server can use the lottery probabilities to allocate the recommendations to the destination documents. For example, the server may select the recommendations for allocation in order of lottery probability. For example, the server may allocate the recommendation having the highest lottery probability first and may position the recommendation that is allocated first in the most favorably position (e.g., top) in the allocation order.

At block311, the server determines pricing for the recommendations based on the allocations (e.g., allocation order), recommendations parameters and allocation plan. One implementation of a grade-adjusted allocation and pricing method that encourages agents to use their true value as their maximum CPC bid is described below in conjunction withFIG. 6.

FIG. 4is a flow diagram of an implementation of a method400for creating an allocation plan. The method may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method400may be performed by the optimization module145hosted in a server140ofFIG. 1or by the optimization module200hosted inFIG. 2.

At block401, the server determines the expected number of views for the destination documents. For example, the server may predict that Document-XYZ may have 5,000 views per day. In one implementation, the server uses user activity data, one or more prediction models, statistical analysis, etc. to determine the expected number of views for the destination document. In another example, the server may determine that Document-XYZ may have 1 million views per day. In another implementation, the server predicts the number of views for a destination document based on data for the subsections of a publisher network. For example, Document-XYZ may be in the news category and the sports news sub-category, and the data for the subsections of the publisher network may indicate there is an average number of views for the documents in the sports news subsection of 8,0000 views per day. The server may use the average of 8,0000 views per day as the predicted number of views per day for Document-XYZ.

At block403, the server identifies the number of recommendation opportunities per view for the destination documents. The publishers may specify the number of recommendation opportunities (e.g., recommendation opportunities155,165inFIG. 1) per view for the destination documents. For example, the publisher GlobalNews, Inc. may specify that Document-XYZ (e.g., GlobalNews webpage) pertains to Subsection-Y (e.g., sub-category international news) and has three recommendation opportunities per view. In another example, the publisher EntertainmentNews, Inc. may specify that Document-DEF (e.g., EntertainmentNews webpage) pertains to Subsection-W (e.g., sub-category entertainment news) and has five recommendation opportunities per view.

At block405, the server determines the total number of recommendation opportunities for each destination document. For example, the server may determine that the total number of recommendation opportunities for Document-XYZ for Subsection-Y is 15,000 recommendation opportunities, which is the product of 5,000 expected views per day multiplied by three recommendation opportunities per view. In another example, the server may determine that the total number of recommendation opportunities for Document-DEF for Subsection-W is 40,000 recommendation opportunities, which is the product of 8,000 expected views per day multiplied by five recommendation opportunities per view.

At block407, the server identifies the campaign parameters associated with each recommendation. For example, Recommendation-1may be from Content-Provider-A for Campaign-A. Content-Provider-A may have specified that Campaign-A has parameters that include a budget (B) of $1000 and a cost-per-click (CPC) of $1. In another example, Recommendation-2may be from Content-Provider-B for Campaign-B. Content-Provider-B may have specified that Campaign-B has parameters that include a budget of $800 and a cost-per-click (CPC) of $2.

At block409, the server determines a predicted CTR for the combinations of recommendations and destination documents. The predicted CTR represents the predicted rate that the recommendation will be selected at when served as an impression. The predicted CTR can be determined from user activity data that is stored in the data store. For example, the server may predict that upon Document-XYZ displaying Recommendation-1from Content-Provider-A the combination may result in a predicted CTR of 4%.

At block411, the server determines an optimal allocation of clicks and impressions for the recommendations for the combinations subject to the campaign parameters. A click hereinafter refers to a selection of an impression in a document. The selection may be made via an input device (e.g., touch screen, mouse, keyboard, motion sensor, remote control, voice, etc.). The server can use the budget B, CPC, CTR, and total number of recommendation opportunities for each combination (e.g., recommendation/document combination) to determine the optimal allocation of clicks. For example, Content-Provider-A may have agreed to pay the third party service provider a maximum budget B of $1000 and a maximum of $1 CPC. In another example, Content-Provider-B may have agreed to pay the third party service provider a maximum budget B of $800 and a maximum of $2 CPC.

The server can use linear programming to determine the optimal allocation of clicks and impressions for the recommendations based on the maximum budgets and maximum cost-per-clicks (CPCs). For example, let i be a destination document (e.g., Document-XYZ, Document-DEF) and j be a set of campaign parameters, which can include the budget (Bj) for the campaign and the cost per click (CPCj) for the campaign. The server can use the click through rate (CTRij), campaign parameters, and total number of recommendation opportunities for a destination document i to determine, Nij, the total number of impressions for a recommendation on destination document i, such that Nijdoes not exceed the budget Bjand the CPCj.
ΣijNijCPCjCTRij≤Bj(ConstraintC1)
ΣijNij≤Nj(ConstraintC2: inventory estimate)
max(ΣijNijCPCjCTRij) Subject to constraints   (Objective FunctionO1)

For example, the server may use linear programming optimizes O1subject to C1and C2to determine the optimal allocations of clicks and impressions (Nij) for each recommendation. The allocation plan can be the optimal number of impressions (Nij) for each recommendation.

FIG. 5illustrates an example of creating an allocation plan550, according to various implementations. The allocation plan550can specify the optimal number of impressions allocated to a recommendation (e.g., Recommendation-1, Recommendation, etc.). There can be multiple recommendations. For simplicity and brevity two recommendations511,513are shown. Portion560ofFIG. 5represents the inputs of the optimization module (e.g., optimization module145inFIG. 1, optimization module200inFIG. 2). Another portion570represents an intermediate solution in terms of clicks. Another portion580represents a translation of the intermediate solution to impression terms.

The server may use the campaign parameters (e.g., budgets, CPC, etc.) for the various campaigns to create the allocation plan550. For example, Content-Provider-A may specify parameters509for a campaign that includes a budget of $1000 and a CPC of $1. In another example, Content-Provider-B may specify parameters509for a campaign that includes a budget of $800 and a CPC of $2. The server can associate the respective parameters with the various recommendations511,515from the content providers.

The server may use the total number of recommendation opportunities for the various destination documents to create the allocation plan550. The total number of recommendation opportunities can be determined as the product of the number of expected views that is predicted for documents in a subsection of the publisher network multiplied by the number of recommendation opportunities on a destination document per view. For example, the server may determine that destination Document-XYZ503for Subsection-Y has 15,000 total recommendation opportunities501. For example, the server may predict that Subsection-Y has 5,000 expected views and that there are three recommendation opportunities per view for Document-XYZ503. In another example, the server may determine that destination Document-DEF505for Subsection-W has 40,000 total recommendation opportunities507. For example, the server may predict that Subsection-W has 8,000 expected views and that there are five recommendation opportunities per view for Document-DEF505.

The server can estimate the click-through-rates (CTRs)520for the various combinations of recommendations511,513and destination documents503,505. For example, the server may estimate that upon Document-XYZ503displaying Recommendation-1511from Content-Provider-A the document and recommendation combination may result in a CTR of 4%. In another example, the server may estimate that upon Document-XYZ503displaying Recommendation-2513from Content-Provider-B the document/recommendation combination may result in a CTR of 2%. In another example, the server may estimate that upon Document-DEF505displaying Recommendation-1511from Content-Provider-A the document/recommendation combination may result in a CTR of 0.5%. In another example, the server may estimate that upon Document-DEF505displaying Recommendation-2513from Content-Provider-B the combination may result in a CTR of 2%.

The server can determine an optimal number of clicks530and an optimal number of impressions550for the recommendations511,513for the various document/recommendation combinations. The allocation plan550is the optimal number of impressions550for the recommendations511,513for the various document/recommendation combinations. The allocation plan550can include quotas (e.g., “15,000 impressions”551, “0 impressions”553, “20,000 impressions”555, “20,000 impressions”557) at the recommendation level. The server can use linear programming, and for example, Objection Function1subject to Constraint1and Constraint2above, to determine the optimal number of clicks530and the optimal number of impressions550while taking into account the budgets, CPCs, CTRs, and total number of recommendation opportunities.

For example, the server may determine that the optimal number of clicks530includes “600 clicks” to Recommendation-1511for Document-XYZ503, “100 clicks” to Recommendation-1511for Document-DEF505, “0 clicks” to Recommendation-2513for Document-XYZ503, and “400 clicks” to Recommendation-2513for Document-DEF505. For example, the server may determine that if all of the 15,000 recommendation opportunities501are allocated as “15,000 impressions”551to Recommendation-1511, and there is a CTR of 4%, the number of clicks is (15,000)*(0.04)=“600 clicks”531. If all of the 15,000 recommendation opportunities501are allocated to Recommendation-1511, then “0 impressions”553for Document-XYZ503would be allocated to Recommendation-2515, resulting in “0 clicks”533for Recommendation-2513for Document-XYZ503. If Content-Provider-B has a budget B of $800 and a CPC of $2, then the number of clicks is $800/$2=“400 clicks”537, which may all be allocated to Recommendation-2513for Document-DEF505.

If “400 clicks”537are allocated to Recommendation-2513for Document-DEF505, the number of impressions allocated to Recommendation-2513for Document-DEF505is the number of clicks divided by the CTR, which is (400 clicks)/(0.02)=“20,000 impressions”557. If Document-DEF505has a total number of recommendation opportunities of “40,000 recommendation opportunities”507and “20,000 impressions”557are allocated to Recommendation-2513, then the remaining “20,000 impressions”555can be allocated to Recommendation-1511.

The allocation of the “20,000 impressions”555for Document-DEF505to Recommendation-1511can result in “100 clicks”535. If the CPC is $1 and there are “100 clicks”535, then Content-Provider-A should pay $100 (e.g., (20,000)*(0.5%)*($1)=$100) to the third party service provider. With the allocation plan550, Content-Provider-A is predicted to pay $100 for the 20,000 impression quota for Document-DEF505and $600 (e.g., (15,000)*(4%)*($1)=$600) for the 15.000 impression quota for Document-XYZ503, which is a total of $700 and less than the overall budget B of $1000 for Content-Provider-A. With the allocation plan550, Content-Provider-B is predicted to pay $0 for the “0 impressions”553quota for Document-XYZ503and $800 (e.g., (20,000)*(2%)*($2)=$600) for the 20,000 impression quota for Document-DEF505, which is a total of $800 and equal to the overall budget B of $800 for Content-Provider-B.

FIG. 6is a flow diagram of one implementation of a method600for determining the allocation and pricing for recommendations. The method may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method600may be performed by the optimization module145hosted in a server140ofFIG. 1or by the optimization module200hosted inFIG. 2, and especially module203ofFIG. 2.

The method600implements a grade-adjusted allocation and pricing mechanism that encourages agents to use their true value as their maximum CPC (Max-CPC) bid. Method600can encourage buyers (e.g., content providers) to specify a maximum CPC in the campaign parameters that is close to the buyer's actual value of a click. The server can adjust the grades that are assigned to the recommendations and can determine prices for the recommendations according to the actual demand reflected through the allocation list (e.g., allocation list generated in block307inFIG. 3).

At block601, the server identifies a grade for each recommendation and derives from it a lottery probability (e.g., assigned in block309inFIG. 3) for a recommendation. In one implementation, the server calculates the lottery probability (p) for the recommendation as a proportion of the grade of the particular recommendation to the total of the grades of the recommendations. For example, the grade (pi) for Recommendation-1may be

p1=712=0.58.
At block603, the server identifies the number of impressions (Ni) assigned to the recommendation for a particular destination document from an allocation plan. For example, the number of impressions (N1) for Recommendation-1for Document-DEF may be N1=20,000.

At block605, the server determines whether the expected number of impressions for the recommendation is more than requested by the plan, in which case the server would need to decrease the original lottery probability so the expected number of impressions matches the number of impressions (Ni) from the allocation plan. Let N be the total number of recommendation opportunities for the destination document. For example, for Document-DEF, N=40,000. The server can use Statement 1 and Statement 2 to determine whether to decrease the original grade.
If (pi×N)≥Ni, then determine decreased lottery probabilityp′i(Statement 1)
If (p1×N)<Ni, then do not decrease lottery probabilitypi(Statement 2)

If the server should decrease the original lottery probability, the server determines the decreased lottery probability at block607and assigns the decreased lottery probability to the recommendation at block609. For example, for Recommendation-1for Document-DEF, let:

The server may determine for Recommendation-1for Document-DEF that p1×N≥N1(e.g., 0.58≥0.50) and can determine the decreased lottery probability at block607. In one implementation, the server determines the decreased lottery probability to be:

For example, the server may determine the decreased lottery probability, p′1, for Recommendation-1for Document-DEF is

NiN=20⁢,⁢00040⁢,⁢000=0.50
and can assign the decreased lottery probability as pito Recommendation-1for Document-DEF at block609.

At block611, the server determines the prices for the recommendation based on the lottery probability. In one implementation, the server determines the price for Recommendation-i using Equation 1:

Price(i)=MaxCPCiif the lottery probability was not decreased (see stmt. 2)

For example, for Recommendation-1for Document-DEF, discussed above with p1=0.58 and p1′=0.5; let MaxCPC1=$1. The server may determine that the price for Recommendation-1is:

The computer system700may further include a network interface device704. The computer system700also may include a video display unit710(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device712(e.g., a keyboard), a cursor control device714(e.g., a mouse), and a signal generation device716(e.g., a speaker).

The data storage device714may include a computer-readable storage medium724on which is stored one or more sets of instructions722(e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions722may also reside, completely or at least partially, within the main memory704and/or within the processor702during execution thereof by the computer system700, the main memory704and the processor702also constituting computer-readable storage media. The instructions722may further be transmitted or received over a network720via the network interface device708.

Reference throughout this specification to “one implementation” or “an implementation” means that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation. Thus, the appearances of the phrase “in one implementation” or “in an implementation” in various places throughout this specification are not necessarily all referring to the same implementation. In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” Moreover, the words “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion.