Patent Publication Number: US-8533067-B1

Title: System for obtaining recommendations from multiple recommenders

Description:
CROSS-REFERENCED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 13/183,291, filed Jul. 14, 2011, entitled “SYSTEM FOR OBTAINING RECOMMENDATIONS FROM MULTIPLE RECOMMENDERS,” which is a divisional of U.S. patent application Ser. No. 12/190,501, filed Aug. 12, 2008, entitled “SYSTEM FOR OBTAINING RECOMMENDATIONS FROM MULTIPLE RECOMMENDERS,” both of which are hereby incorporated by reference in their entirety. U.S. patent application Ser. No. 12/190,501 was filed on the same day as U.S. patent application Ser. No. 12/190,471, entitled “SYSTEM FOR OBTAINING RECOMMENDATIONS FROM MULTIPLE RECOMMENDERS,” which is hereby incorporated by reference in its entirety. 
    
    
     BACKGROUND 
     Relatively large and sophisticated web sites commonly implement some form of personalization system to provide personalized content, such as personalized item recommendations, to their users. To illustrate, some personalization systems can monitor and record one or more types of item-related user activity such as item purchases, item viewing events, and/or item rentals, and analyze the collected data to detect and quantify associations between particular items. When a user accesses a particular item, such as a particular product in a catalog of an e-commerce site, or an article on a news site, an appropriate message may be displayed notifying the user of related items (e.g., “people who bought this item also bought . . . ,” or “people who viewed this item also viewed . . . ”). The personalization system may also generate personalized item recommendations that are based on a target user&#39;s purchase history, item viewing history, item ratings, and/or some other type of user data. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate embodiments of the inventions described herein and not to limit the scope thereof. 
         FIG. 1  illustrates an embodiment of a network environment suitable for providing recommendations to content sites; 
         FIG. 2  illustrates an embodiment of a process for responding to recommendation requests; 
         FIG. 3  illustrates an embodiment of a process for computing recommender weights; 
         FIG. 4  illustrates an embodiment of a process for selecting recommender weights; 
         FIG. 5  illustrates an embodiment of a process for compensating recommender providers; and 
         FIG. 6  illustrates an embodiment of a process for registering recommenders with a personalization network service. 
     
    
    
     DETAILED DESCRIPTION 
     Existing personalization systems suffer from several drawbacks. For instance, personalization systems tend to be expensive to implement and maintain. Some relatively sophisticated personalization systems use infrastructure components which, among other tasks, store customer behavior data, process the stored behavior data to detect the item associations, and store databases which relate items to one another. As a result, among other reasons, sophisticated personalization systems are typically available only to relatively large companies. In addition, no single recommendation algorithm may be suitable for all recommendation contexts, and it can be desirable to be able to use different recommendation algorithms for different contexts. Recommendation contexts can encompass the type of content site being browsed, the type(s) of items being recommended, the type or quantity of information available for a given user, and the like. 
     In certain embodiments, a computer system and service are provided that allow operators of interactive computer systems (e.g., content sites) to offload the tasks of generating and hosting recommendation software to a community of developers. In some embodiments, the service is implemented as a network service, such as a web service, that is accessible over a public network such as the Internet. The network service may obtain the recommendations from external recommenders provided by different business entities from the operator/provider of the network service. Advantageously, in certain embodiments the network service enables content sites to use different recommendation algorithms for different contexts. Additionally, the network service can determine, in an automated manner, which recommenders work best in particular recommendation contexts. 
     For purposes of illustration, the network service will be described in the context of content sites (e.g., web sites) that use the network service to generate or select content to present to users of the content sites. As will be recognized, however, content sites are merely one type of interactive computer system that can use the network service. Other examples include interactive television systems, online services networks, in-store kiosks, and systems that provide recommendations to users by email. The content sites typically host electronic catalogs or repositories of items that are available for purchase, rent, download, subscription, viewing, and/or some other form of consumption. Examples include retailer web sites, music and video download sites, online travel reservation sites, news sites, auction sites, shopping portals, dating sites, social networking sites and sites that provide a combination of these functions. The content sites may use the network service to perform a variety of services, including but not limited to the following: (a) storing and maintaining user-specific event histories, such as item purchase histories, item viewing histories, item rental histories, and/or search histories and (b) obtaining personalized item recommendations for users. 
     As used herein, the term “item” in addition to having its ordinary meaning, is used interchangeably to refer to an item itself (e.g., a particular product) and to its description or representation in an electronic catalog or data repository. As will be apparent from the context in which it is used, the term is also sometimes used herein to refer only to the item itself or only to its representation in an electronic catalog or data repository. Items that may be recommended to users may include, among others, products, web sites, advertisements, news articles, blogs, travel destinations, service providers, other users, and events. Additionally, as used herein, the terms “view,” “viewing event,” and the like, in addition to having their ordinary meaning, can include impressions (e.g., of recommendations), user selections, mouse clicks, and the like. 
       FIG. 1  illustrates an embodiment of a network environment  100  suitable for providing item recommendations. The network environment  100  includes a personalization network service (PNS)  120  that can be implemented with a server system that may include one or more physical servers or computing machines (not shown). In addition, the PNS  120  can be implemented as a network resource, such as a web service, web site, or the like. Advantageously, in certain embodiments, the PNS  120  can provide recommendations from external recommenders  180  to end users of one or more content sites  102 . 
     Two primary types of entities access and establish accounts with the PNS  120 : content site  102  operators and developers. The content site  102  operators operate web sites, and/or other types of interactive sites or systems, that use the PNS  120  to obtain recommendations to present to end users operating user systems  104 . The developers are individuals or organizations that develop the recommenders used to generate such recommendations. (The developers are also referred to herein as “recommender providers.”) Account data for registered content site operators and recommender providers may be stored in a data repository  175 . Different recommenders  180  may be provided by different developers, although a single developer may provide multiple recommenders  180 . As described below, the content site  102  operators may be charged a usage-based or other fee for using the PNS  120  to provide recommendations to their users. A portion of the resulting revenue may be paid to the developers, with the amount paid to a given developer optionally being based on the performance of its recommender(s)  180 . 
     The content sites  102  can communicate with the PNS  120  over a network  110   a . Each of the content sites  102  can include network resources such as web sites that respond to network requests received from browsers or other software on user systems  104  of end users. The user systems  104  may be computing devices such as desktop computers, web pads, personal digital assistants (PDAs), mobile phones, set-top television boxes, media players, laptop computers, tablets, desktop computers, electronic book readers, kiosks, and the like. The network  110   a  may be a network such as the Internet, a local or wide area network (LAN or WAN), or the like. 
     The content sites  102  may be operated or provided by different business entities from one another, and by entities other than the operator/provider of the PNS  120 . In general, the user systems  104  access electronic catalogs or repositories of items hosted by the content sites  102 . As user systems  104  perform specific types of item-related events (e.g., item purchases, item downloads, item rentals, item viewing events, item rating events, etc.), the content sites  102  can report these events to the PNS  120 . For example, the content sites  102  may report the events using application programming interface (API) calls to an API provided by a network service interface  130  of the PNS  120 . Thus, in certain embodiments, the content sites  102  can act as web service clients when communicating with the PNS  120 . 
     A client-side API module  106  of each content site  102  can implement the API calls to the network service interface  130 . In certain embodiments, the client-side API module  106  can include one or more software components that implement the API described in U.S. patent application Ser. No. 11/691,758, entitled “Service for Providing Item Recommendations,” filed Mar. 30, 2007 (the “758 application”), the disclosure of which is hereby incorporated by reference in its entirety. In particular, paragraphs [0065] through [0095] of the &#39;758 application, which disclose an example API, are specifically incorporated by reference herein. In other embodiments, the content sites  102  can use a non-API based mechanism for reporting events, such as a representational state transfer (REST) architecture. In addition, in other implementations, some or all of the content sites  102  may report the events directly to the external recommenders  180 , either in addition or as an alternative to reporting such events to the PNS  120 . 
     The PNS  120  may additionally or alternatively collect event data using other methods. For example, operators of the content sites  102  may be provided with widget code to add to their web pages. When a browser running on a user system  104  loads such a web page, the widget code may cause the browser to report the page-load event directly to the PNS  120 , as described in U.S. application Ser. No. 11/966,780, filed Dec. 28, 2007, the disclosure of which is hereby incorporated by reference. The event data reported/recorded in such scenarios may, for example, include any one or more of the following: the URL accessed, an identifier of an item represented on the accessed page, a cookie that identifies the user, and the type of event performed. 
     As another example, the operator of the PNS  120  may also operate an e-commerce site that hosts an electronic catalog of items. As described the &#39;780 application referenced above, events recorded by this e-commerce site (e.g., item purchases, item viewing events, etc.) then be used to supplement the event data reported by the content sites  102 . As yet another example, end users may be provided with a browser toolbar or plug-in that causes the user systems  104  to report various types of events directly to the PNS  120 . 
     The network service interface  130  can include one or more software modules for communicating events received from the content sites  102  to a data service  140 . The data service  140  can include one or more software components for persistently storing data descriptive of the events in a data repository  150 . The data service  140  may store the event data in association with an identifier of an end user and in association with an identifier of the content site  102  that reported the event. Each operator of a content site  102  can decide which events and event types to report to the PNS  120 . The content sites  102  can also provide bulk transfers of event data using an API of the network service interface  130 . 
     The network service interface  130  may also include a billing component  132  which can, in some embodiments, perform authentication, metering, and/or billing. According to some embodiments, for example, content sites  102  can be billed for requests and/or for the storage of user data. In addition, as will be described in greater detail below, the billing component  132  can cause operators or providers of external recommenders  180  to be compensated for purchases made, or other transaction conducted, in response to recommendations (see  FIG. 5 ). 
     The network service interface  130  can also respond to data requests from the content sites  102 . The data requests may include, for example, requests for lists of items that are related to a particular source item and requests for personalized recommendations for a particular user. (Both of these types of item content are referred to herein generally as recommendations.) According to some embodiments, the data requests are generated by the content sites  102  in response to page requests (e.g., HTTP requests) from user systems  104 . The PNS&#39;s  120  responses to these requests may be embodied in specific or general purpose languages, such as, among other languages, extensible markup language (XML). The requesting web site  102  typically parses the response, and uses the response to retrieve or generate HTML or other content for displaying the recommendations within the requested web page. For example, the requesting web site may translate an XML-based list of recommended items into an HTML sequence that displays descriptions of these items to the user systems  104 . 
     The network service interface  130  communicates the received data requests to a recommendations service  160 . In certain embodiments, the recommendations service  160  communicates with one or more external recommenders  180  over a network  110   b . The external recommenders  180  can be operated by separate business entities from each other and from the PNS  120 . The external recommenders  180  can be hosted on servers that are separate from the PNS  120 . For example, each provider/developer of an external recommender  180  may be responsible for determining how and where the respective recommender is to be hosted. Alternatively, some or all of the recommenders  180  can be hosted on one or more servers of the PNS  120 . Although they may be hosted on one or more servers of the PNS  120 , the recommenders  180  are still referred to herein as “external” to distinguish from recommenders that may be developed by an operator of the PNS  120 . The various external recommenders  180  can include different algorithms for generating recommendations. Competition between providers/developers of the external recommenders  180  can result in better recommendations being provided to the content sites  102 . 
     The depicted embodiment of the recommendations service  160  includes a registration module  162  that includes one or more software components for registering the external recommenders  180  with the recommendations service  160 . For example, the registration module  162  can allow a provider of an external recommender  180  to provide machine-readable descriptions of the external recommenders  180 . These descriptions may include network addresses, such as uniform resource indicators (URI), that identify the external recommenders  180  to the recommendations service  160 . The registration module  162  can store the machine-readable and human-readable descriptions of the recommenders in the data repository  175 . The registration module  162  may also collect information about the external recommenders  180  and make this information available to the content sites  102  using a web site or extranet of the PNS  120  provider. The operators of the content sites  102  can optionally select which external recommenders  180  to use to receive recommendations (see  FIG. 6 ). 
     The recommendation service  160  also includes a recommendations retriever  164 , which can include one or more software components for querying, invoking, or otherwise causing the external recommenders  180  to provide recommendations. For instance, in one embodiment the recommendations retriever  164  queries the external recommenders  180  for recommendations in response to a request from a content site  102 . Requests for recommendations from the recommendations retriever  164  may be web services calls, such as REST-based or SOAP-based calls. The requests for recommendations may include a user identifier corresponding to an end user to whom recommendations may be provided, a content site identifier to identify a content site  102 , a segment identifier to identify a segment or group of users (e.g., segmented by gender, age, interest, occupation, or the like), and/or one or more item identifiers. In alternative embodiments, the recommendations requests may come directly from the computing devices of the end users. For example, widget code embedded in a web page served by a content site  102  to a user system  104  system can request recommendations from the network service interface  130  over the network  110   a.    
     One or more of these identifiers may be created by either the content site  102  or the recommendation service  160 . For example, the recommendation service  160  may issue a cookie containing a user identifier to an end user&#39;s browser in response to determining that no user identifier has been created. Alternatively, the content site  102  can create the cookie or use another mechanism for creating user identifiers. In addition, the content sites  102  or recommendation service  160  can associate segment identifiers with various user identifiers. The segment identifiers may be associated with users of a single content site  102  or with users across multiple content sites  102  (e.g., all 50+ year-old males across all content sites  102 ). 
     In response to the recommendation requests, the external recommenders  180  may communicate with a data interface  170  in real-time to retrieve event data from the data repository  150 . The real-time transfer of event data can be signed and authenticated to protect user privacy. In addition, the external recommenders  180  can communicate with the data interface  170  to obtain the item data from the data repository  150 . This item data may include catalog data provided by the content sites  102  to the PNS  120 . In alternative embodiments, the external recommenders  180  may obtain at least some of the event and/or item data by performing bulk transfers from the data interface  170 . In one embodiment, the bulk transfer of event data can be performed over a secure FTP connection to preserve the privacy of customer data. The bulk transfers may be performed in an off-line process, for example, once daily. In alternative embodiments, the external recommenders  180  may communicate with the content sites  102  directly through the network  110  to receive event and/or item data. 
     The external recommenders  180  can analyze the event and/or item data to generate recommendations for the content sites  102 . Different external recommenders  180  may analyze different types of events, items, item attributes, and/or user profile data in order to generate recommendations. Different external recommenders  180  may also specialize in providing recommendations for different browsing contexts, web site categories, user segments, event types and/or types of items. The external recommenders  180  can return recommendations that include item identifiers and corresponding scores to indicate the relative strengths of the recommendations. The content sites  102  can cause the recommendations to be displayed on user systems  104 , such as by dynamically incorporating the recommendations into web pages requested by the user systems  104 . To increase brand exposure for recommender providers, the content sites  102  may also cause recommender providers&#39; logos to be displayed in proximity to the recommendations. 
     An optimization module  166  of the recommendation service  160  can optimize the results returned by the external recommenders  180  based on various criteria, which will be described below with respect to  FIG. 3 . In one embodiment, the optimization module  166  applies weights to the recommendations from each external recommender  180 . The weights can be used to selectively emphasize or deemphasize recommendations from various external recommenders  180 . In some implementations, the content sites  102  may be able to select which weights are applied by the optimization module  166 . The optimization module  166  may store the weights in the data repository  175 . Additionally, the optimization module  166  can store data on the external recommenders&#39;  180  performance in the data repository  175 . 
     In addition to the recommendations provided by the external recommenders  180 , the PNS  120  may be capable of using the event data to generate lists of the most popular items. Separate lists may be generated for different types of items, and for different types of events (e.g., views versus purchases). These lists may be made available to the content sites  102  as recommendations via an API. In addition, in some implementations the recommendations service  160  may also generate personalized recommendations in addition to those generated by the external recommenders  180 . Additionally or alternatively, the recommendations service  160  may generate item-to-item similarity mappings based on the event and item data and make these similarity mappings available to the external recommenders  180 . The external recommenders  180  can use the similarities as a basis for their own recommendations algorithms. Examples of processes that may be used to generate item-to-item similarity mappings, and to use such mappings to generate personalized recommendations, are described in U.S. Pat. No. 6,853,982, entitled “Content Personalization Based on Actions Performed During a Current Browsing Session.” 
     Advantageously, in certain embodiments the PNS  120  may be tailored for use with any suitable content site  102  (or other type of system) and is not limited to any specific type of content site  102 . Examples include, but are not limited to, electronic shopping sites, news sites, content aggregator sites, music download sites, redirectors, social networks, interactive television systems, email-based mass marketing systems, and search engines. The PNS  120  enables the content sites  102  to provide item recommendations and other forms of behavior-based content to their users. For example, an end user viewing a pair of shoes on a merchant&#39;s content site  102  might receive a recommendation for shoe laces or shoes with similar designs. Similarly, end users viewing a news site for information on a local festival might receive a recommendation for articles written about other local festivals. Likewise, end users viewing a retailer site might receive personalized product recommendations based on a history of the user&#39;s transactions or other types of events associated with the user (e.g., a list of the last N items viewed or the last N items purchased). These recommendations in turn may encourage end users to spend more time using the content sites  102  and conducting associated transactions. 
     Advantageously, in certain embodiments the PNS  120  allows content site  102  operators to offload the tasks of generating and hosting recommendation software to a community of developers. Moreover, by optimizing the recommendations from the external recommenders  180 , in certain embodiments the PNS  120  can improve the quality of the recommendations provided to the content sites  102 . 
     In addition to the embodiments shown, some or all of the recommenders  180  may be developed internally by the operator of the PNS  120  and hosted on the PNS&#39;s  120  servers. Advantageously, in certain embodiments, the optimization techniques described herein may be used to assign weights to these internal recommenders based on their performance. The internal recommenders may also be compared and weighted against the external recommenders  180 . In addition, the internal recommenders may also be implemented with certain of the techniques described in U.S. patent application Ser. No. 11/771,914, entitled “Recommendation System with Multiple Integrated Recommenders,” filed Jun. 29, 2007 (the “&#39;914 application”), the disclosure of which is hereby incorporated by reference in its entirety. In particular, paragraphs [0019] through [0047] of the &#39;914 application, which discuss various embodiments of multiple recommender systems, are specifically incorporated by reference herein. 
       FIG. 2  illustrates an embodiment of a recommendations process  200  for responding to recommendation requests. The recommendations process  200  can be implemented by the PNS  120  described above. In particular, the recommendations process  200  may be implemented by the recommendations service  160 . The recommendations process  200  can retrieve recommendations from a plurality of external recommenders in response to a request from a content site. 
     At block  202 , a request for recommendations is received from a content site. The request for recommendations may be received, for example, via an API call to a PNS. The content site may request the recommendations to be targeted to a particular user and/or item. The content site may further request the recommendations to be optimized for one or more given event types (e.g., purchases, ratings, views, or the like). 
     At block  204 , some or all of the external recommenders can be queried, for example, by the recommendations retriever  164 . In response, the recommenders can provide recommendations (e.g., item identifiers) along with raw scores for those items. The recommendations process  200  may normalize these scores, as described below; alternatively, each recommender can provide scores on a normalized scale (e.g., 0 to 1). 
     At decision block  206 , it is determined whether one or more results were received within an allotted time. The allotted time may be a timeout period or the like, which may be defined by a service-level agreement (SLA) with the content site or PNS. According to some embodiments, if the results are not received within the allotted time, other items may be substituted for the results. For example, popular, best-selling, or other pre-selected items may be returned as recommendations at block  208 . Thereafter, the recommendations process  200  ends. 
     If results are received within the allotted time, weights for the recommenders that returned results can be selected at block  210 , for example, from a database or lookup table. The weights may be used to selectively emphasize or deemphasize the results from individual recommenders. The weights can be pre-generated by the optimization module  166  based on which recommendations have been most effective for a given user, segment of users, item, event type, combinations of the same, and the like. Thus, a single recommender may have multiple weights. Processes for generating and selecting the weights are described in greater detail below with respect to  FIGS. 3 and 4 . 
     Raw scores returned from each recommender may be normalized at block  212 . Normalization of the scores can allow the recommendations from different recommenders to be compared. The scores may be normalized in certain embodiments using the techniques described in U.S. patent application Ser. No. 11/771,914, entitled “Recommendation System with Multiple Integrated Recommenders,” filed Jun. 29, 2007 (the “&#39;914 application”), referenced above. In particular, paragraphs [0060] through [0091] of the &#39;914 application, which discuss various embodiments of normalization techniques, are specifically incorporated by reference herein. As one example technique, the scores may be normalized as follows: 
                     Normalized   ⁢           ⁢     Score   i       =       (       Score   i     -     Score   min       )       (       Score   max     -     Score   min       )               (   1   )               
where Normalized Score i  represents a normalized score for an i th  recommendation provided by a given recommender, Score i  represents the raw score for the i th  recommendation, Score min  represents the minimum observed score from the recommender over a period of time (e.g., a window), and Score max  represents the maximum observed score from the recommender over the same period of time.
 
     At block  214 , the normalized scores are combined with the recommenders&#39; weights to produce overall scores. As a result, the normalized scores may be adjusted by the weights. One way that normalized scores may be combined with a weight for a given recommender is to multiply the normalized scores by the weight, e.g., as follows:
 
Overall Score i =(Normalized Score i )·(Weight recommender )  (2)
 
where Overall Score i  represents the overall score for an i th  recommendation provided by a given recommender and Weight recommender  represents a weight assigned to that recommender. Thus, a higher weight may emphasize scores from a recommender, while a lower weight may deemphasize scores. A process for computing recommender weights is described below with respect to  FIG. 3 . In addition, if multiple recommenders recommend the same item, the overall scores from each of the recommenders can be combined to produce a new overall score for that item.
 
     A subset of the recommendations may be selected based on the overall scores at block  216 . For example, a most highly-scored subset of the recommendations may be selected. At block  218 , the selected recommendations and the corresponding recommenders may be logged to keep track of which recommenders were responsible for which recommendations. At block  220 , the selected recommendations are returned to the content site, and the recommendations process  200  ends. 
       FIG. 3  illustrates an embodiment of an optimization process  300  for computing recommender weights. The optimization process  300  can be implemented by the PNS  120  described above. In particular, in certain embodiments the optimization process  300  may be implemented by the optimization module  166  of the recommendations service  160 . The optimization process  300  can generate weights that optimize recommendations based on the external recommenders&#39; performance. The optimization process  300  may run initially to assign weights to recommenders. The optimization process  300  may also run periodically to recompute or re-optimize the weights as more events are reported by content sites. For example, the weights may be recomputed daily. 
     The optimizer  166  can compute weights for a given recommender based on the context-specific performance of the recommender. This context-specific performance can include the recommender&#39;s performance with particular users, segments of users, content sites, and the like. The optimization process  300  can generate weights as specific to a user as possible, while maintaining statistical significance. As will be described below with respect to  FIG. 4 , when selecting weights for a particular content site, the optimizer  166  may select user-specific weights if available, then fall back to segment-specific weights, then site-specific weights (for content sites), and then global weights. If no global-level weights exist, then the recommender may be new or not received traffic yet, so a default weight can be assigned to the recommender. 
     At block  302 , events and a recommender log can be retrieved for a given time period. The events and recommender log may be retrieved from the data repository  150  by the optimization module  166 . The events can include user purchases, views of items, ratings of items, and the like, as described above. The events may also include time stamps, indicating when the events were stored in the data repository  150 . The recommender log can include information on recommendations provided to content sites and information on which recommenders were responsible for the recommendations (see  FIG. 2 ). In addition, the recommender log may include time stamps of when recommendations were provided. 
     For each recommender, at block  304  the number of events attributed to the recommender can be determined. In addition, the number of recommendation requests the recommender fulfilled can be determined. In certain embodiments, the events that are attributed to a given recommender include item-related events that were recorded by the PNS within a predetermined time from when the recommender provided the recommendations for the items. For example, if a purchase event was recorded for an item within an hour of a recommender recommending that item (e.g., as determined by the time stamps), the purchase event may be attributed to that recommender. The amount of time used to determine this attribution can vary in different implementations and may be on the order of seconds, minutes, hours, days, a month, or the like. According to some embodiments, a relatively larger time window may be used in conjunction with the application of an exponential decay model to the event and recommender log data, such that more recent activity is considered more relevant. In some implementations, the amount of time for determining attribution can include the period of time between the providing of two sets of recommendations for a given user. 
     The number of events attributed to a given recommender can be separated by event type. For example, ten item purchases and five item views may be attributed to a single recommender. Alternatively, the events attributed to a recommender can be an aggregate of events regardless of event type. 
     Where multiple recommenders each recommend an item that is the subject of an event, attribution for the event may be apportioned among the recommenders based on their weights. To illustrate, a first recommender might have a weight of 2 and a second recommender might have a weight of 0.5. To distribute credit for one event, the first and second recommenders may be considered to have contributed to 0.8 and 0.2 events, respectively. In another embodiment, attribution may be apportioned among recommenders based on overall recommendation scores provided by those recommenders. For instance, two recommenders may provide the same recommendation with different overall scores of 0.9 and 0.6, respectively. Based on these scores, attribution for an event might be apportioned 60% to one recommender and 40% to the other. 
     At block  306 , global weights can be computed (or recomputed) for each recommender. As will be described in further detail below with respect to  FIG. 4 , the global weights for each recommender may be used if more specific weights are not available, such as weights specific to a user, segment, or content site. In one embodiment, the global weights can be computed as follows: 
                     Global   ⁢           ⁢     Weight     recommender   ,           ⁢   type         =       Attributed   ⁢           ⁢     Events   type         Requests   ⁢           ⁢     Served   recommender                 (   3   )               
where Global Weight recommender, type  represents a recommender&#39;s global weight for a given event type, Attributed Events type  represents events from multiple content sites attributed to the recommender for a given event type during the period of time selected above at block  302 , and Requests Served recommender  represents the number of requests served by the recommender to content sites over that period of time. The global weights can be stored in a database (e.g., the data repository  150 ). The database may be a flat-file database.
 
     A single recommender may have a separate global weight for each event type. Having separate global weights for different event types can allow content sites to specify recommender optimizations based on event type. For example, a content site may wish to receive recommendations optimized based on user views rather than user purchases. Alternatively, each recommender may have a single global weight based on all events attributed to the recommender. 
     At decision block  308 , it is determined whether enough requests have been served by a recommender for a given content site. For example, it can be determined whether a statistically significant number of requests were served by the recommender to the content site. The number of requests considered to be statistically significant can be determined heuristically or experimentally. For example, it might be determined that 10,000 requests are statistically significant. If the number of requests is considered enough to be statistically significant, site-specific weights for each recommender used by the content site can be computed at block  310 . The site-specific weights may be computed using an equation similar to equation (3) above, with Attributed Events type  representing events attributed to the recommender for a given event type and for the specific content site during the selected period of time. The site-specific weights can be stored in a database (e.g., the data repository  150 ). 
     If not enough requests were served for the content site to justify generating site-specific weights, it is determined at decision block  324  whether there are additional content sites. If not, the optimization process  300  ends. Otherwise, the optimization process  300  returns to blocks  308  and  310  to compute site-specific weights for another content site. 
     The optimization process  300  proceeds from block  310  to decision block  312 . At decision block  312  it is determined whether enough requests have been served by recommenders for a selected segment of users. As above, the number of request that are considered enough can be a number of statistically significant requests. If enough requests have been served, segment-specific weights for each recommender that generated recommendations for the segment may be computed at block  314 . The segment-specific weights may be computed using an equation similar to equation (3) above, with Attributed Events type  representing events from the segment attributed to the recommender for a given event type during a selected period of time. The segment-specific weights can be stored in a database (e.g., the data repository  150 ). 
     If not enough requests were served for the segment, it is determined at decision block  320  to whether there are additional segments. If not, the optimization process  300  proceeds to decision block  324 . Otherwise the optimization process  300  loops back to blocks  312  and  314  to compute segment-specific weights for another segment. 
     The optimization process  300  proceeds from block  314  to decision block  316 . At decision block  316  it is determined whether enough requests have been served for a given user. As above, the number of requests that are considered enough can be a number of statistically significant requests. If so, user-specific weights for each recommender that generated recommendations for the user may be computed at block  318 . The user-specific weights may be computed using an equation similar to equation (3) above, with Attributed Events type  representing events of the user attributed to the recommender for a given event type during a selected period of time. The user-specific weights can be stored in a database (e.g., the data repository  150 ). 
     If not enough requests were served for the user, it can be determined at decision block  320  whether there are additional users. If not, the optimization process  300  proceeds to decision block  322 . Otherwise the optimization process  300  loops back to blocks  316  and  318  to compute user-specific weights for another user. Thus, the optimization process  300  may continue until it is determined that no additional users, segments, or content sites are to be processed. 
     Although not shown, if a recommender has served little or no requests (e.g., the recommender is new or has little traffic), the recommender may be assigned a default weight for each event type. The default weight may be determined experimentally so as to allow recommendations from the recommender to be exposed to content sites. As one example, a default weight could be 0.1 (e.g., on a scale of 0 to 1). Alternatively, certain techniques for giving advertising messages exposure on web pages can be applied to give new recommenders exposure to content sites, examples of which are described in U.S. patent application Ser. No. 11/758,932 entitled “Real-time Adaptive Probabilistic Selection of Messages,” filed Jun. 6, 2007, the disclosure of which is hereby incorporated by reference in its entirety. 
     The optimization techniques described above optimize recommendations based on implicit user feedback, such as purchases, views, and the like. In addition, in some embodiments recommendations can also be optimized based on explicit user feedback about recommendations. Voting-based user feedback, for instance, can be provided by end users to the optimization module  166 . The voting-based feedback may be in the form of 1-5 (or any number of) star rankings, helpful/not helpful rankings, a “not interested” indicator, thumbs up/down rankings, or the like, which may be displayed next to each recommendation. The rankings may also be normalized for each user (e.g., by each user&#39;s average ranking), so as to more readily compare rankings amongst users. The user rankings provided for a given recommender can be averaged and combined with the weights described above. For instance, an average ranking can be multiplied with the weights assigned to a recommender to produce new weights. In addition, the form of feedback used (implicit, explicit, or a combination of both) can also vary for a given user, segment, or content site. The type(s) of feedback used can be selected empirically based on resulting recommender performance. 
       FIG. 4  illustrates an embodiment of a weight selection process  400  for selecting recommender weights. The weight selection process  400  can be implemented by the PNS  120  described above. In particular, in certain embodiments the weight selection process  400  may be implemented by the optimization module  166  of the recommendations service  160 . The weight selection process  400  can select the weights generated by the optimization process  300  in response to the recommendations service  160  receiving a request for recommendations. In certain embodiments, the weight selection process  400  illustrates how the optimizer  166  can return the best recommender weights it can provide for the data available. 
     At block  401 , a request for a recommender weights may be received. At decision block  402 , it can be determined whether recommender weights are available for a user of a given content site. If so, user-specific recommender weights may be returned at block  404 . Thereafter, the weight selection process  400  ends. If at decision block  402 , weights are not available for a user, the weight selection process  400  proceeds to decision block  406 . 
     At decision block  406 , it is determined whether recommender weights are available for segment of users. If so, segment specific recommender weights are returned at block  408 . Thereafter, the weight selection process  400  ends. If at decision block  406 , weights are not available for segment of users, the weight selection process  400  proceeds to decision block  410 . 
     At decision block  410 , it is determined whether recommender weights are available for the content site. If so, at block  412  site-specific recommender weights are returned. The weight selection process  400  then ends. If at decision block  410 , recommender weights are not available for the content site, the weight selection process  400  proceeds to decision block  414 . 
     At decision block  414 , it is determined whether global recommender weights are available. If so, global recommender weights are returned at block  416 . Thereafter, the weight selection process  400  ends. If, at decision block  414 , it is determined that global recommender weights are not available, default weights for new recommenders are returned at block  418 . Thereafter, the weight selection process  400  ends. 
       FIG. 5  illustrates an embodiment of a compensation process  500  for compensating recommender providers. The compensation process  500  can be implemented by the PNS  120  described above. In particular, in certain embodiments the compensation process  500  may be implemented by the billing component  132  of the network service interface  130 . The compensation process  500  can compensate recommender providers based, or based in part, on the performance of their recommenders. 
     At block  502 , a user-generated event is recorded. The user-generated event may be a purchase or viewing event, as described above. At decision block  504 , it is determined whether the event is attributable to a recommendation. The event may be attributable to a recommendation, for instance, if the recommendation and the event occurred within a predetermined timeframe, a number of mouse clicks of a user, or the like. If the event is not attributable to a recommendation, the compensation process  500  ends. 
     Otherwise, at block  506 , the content site may be charged a fee. In the case of a purchase event, the fee may be, for example, a percentage of the purchase price, a fixed fee, and/or a fixed fee that is dependent on the purchase price, among other possibilities. At block  508 , a portion of the fee may be distributed to the recommender providers based on the performance of their recommenders. For example, the PNS  120  can credit accounts of the recommender providers. The performance of a recommender may be based on one or more factors, such as a number of events attributed to a recommender, a number of requests served by a recommender, and the like. In one embodiment, the weight of a recommender reflects the performance of the recommender, and the portion of the fee may be distributed according to recommender weights. The weights used to determine the fee distribution may be the weights used when the recommendation was provided to the content site or the most recent weights used to provide recommendations, if different. The portion of the fee may be a portion of the percentage of the purchase price, a fixed fee, or the like. 
     In various alternative embodiments, the PNS  120  can charge a flat rate per recommendation and distribute a portion of the proceeds to the recommender providers based on the amount of recommendations they provided and their weights. In another model, recommenders may be free by default, but certain recommender providers can charge for their recommendations. The PNS  120  might receive a portion of any fees charged by those recommenders. 
     In another embodiment, content sites could opt to share their user event data with new recommender providers they have not yet subscribed to. Recommender providers may incent content sites by providing a percentage of the revenue earned by the recommenders that consumed the content site&#39;s data. In another model, recommendations are free to content sites, and the recommender providers and the PNS monetize the user data collected by using the user data for targeting advertisements online. 
     Moreover, in another embodiment, content sites are provided with HTML-based widgets instead of a network services-based list of recommendations. These widgets can contain advertisements, so the content site owner may not be charged for the recommendations. Ad revenue can be shared between the operator of the PNS and the recommender providers based on their global weights. 
       FIG. 6  depicts how a registered recommender provider  602  and a registered operator  606  of a content site may interact with the PNS  120 . The depicted interactions may, for example, occur via a web site, extranet, and/or web service interface of the PNS. At block  610 , the recommender provider  602  may develop a recommender. At block  612 , the recommender provider  602  may submit a description of the recommender to the PNS  120  by providing a machine-readable description of the recommender (e.g., a list of input and output parameters) along with a human-readable description of the recommender. 
     The PNS  120  may register the recommender at  614  using, e.g., the registration module  162 . When registering the recommender, the PNS  120  may store information pertaining to the recommender, such as a URI for locating the recommender, in computer storage. In addition, the PNS  120  may publish a human-readable description of the recommender at block  616  for viewing by registered operators of content sites  102 . By registering recommenders, the PNS  120  can plug recommenders into a recommendations service dynamically. Similarly, the PNS  120  can remove recommenders dynamically (e.g., by removing their registration). 
     The published description may be provided in a catalog of recommenders that can be browsed by the content site  606 . The content site  606  can access the recommender description and select the recommender at block  618 . The content sites&#39;  606  selection of the recommender at block  618  can cause the PNS  120  to register the recommender for the content site at block  620 . Thus, the content site  606  can choose which recommenders provide recommendations to the content site  606 . 
     Additionally, although not shown, a list of recommenders and/or recommender providers  602  can be made accessible to the recommender providers  602 . The list of recommender providers  602  may be ranked based on the success of their recommender algorithms to foster competition among recommender providers  602 . Moreover, the PNS  120  can send performance reports to the recommender providers  602 . The performance reports can include information on attributed events, weights, and so forth. 
     The various modules of the systems described herein can be implemented as software applications, hardware and/or software modules, or components on one or more computers, such as servers. While the various modules are illustrated separately, they may share some or all of the same underlying logic or code. In addition, each of the processes, components, and algorithms described above may also be embodied in, and fully automated by, modules executed by one or more computers or computer processors. The modules may be stored on any type of computer-readable medium or computer storage device. The processes and algorithms may also be implemented partially or wholly in application-specific circuitry. The results of the disclosed processes and process steps may be stored, persistently or otherwise, in any type of computer storage. In one embodiment, the modules may be configured to execute on one or more processors, including sub-processors. In addition, the modules may comprise, but are not limited to, any of the following: software or hardware components such as software object-oriented software components, class components and task components, processes methods, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, variables, combinations of the same, and the like. 
     For example, with reference to  FIG. 1 , components  130 ,  132 ,  140 ,  160 ,  162 ,  164 ,  166 ,  170  of the PNS  120  may each be implemented in whole or in part in program modules that run on appropriate computer hardware resources, such as a server system that comprises one or more physical servers. The hardware resources used to implement the PNS may be co-located or geographically distributed. The data repository  150  may be implemented as one or more databases, flat file systems, or any other type of computer data repository. 
     The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate. For example, described blocks or states may be performed in an order other than that specifically disclosed, or multiple blocks or states may be combined in a single block or state. 
     Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. 
     While certain embodiments of the inventions disclosed herein have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions disclosed herein. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions disclosed herein. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of certain of the inventions disclosed herein.