Abstract:
A system of content/query feature based data structure retrieval of predicted values is provided. The system can create a data structure having a plurality of rows corresponding to individual content/query features and a plurality of columns corresponding to individual predicted values. The processors can obtain a set of session features associated with a selection by a computing device in response to a query, and a set of content/query features associated with the selection of the content item. The processors can retrieve, from the data structure, a set of predicted values for each of the set of content/query features. The processors can generate, for each of the set of content/query features, a set of aggregate predicted values for each of the set of content/query features, and can include the set of aggregate predicted values in the data structure.

Description:
BACKGROUND 
     Field of the Invention 
       [0001]    Implementations described herein relate generally to measuring and predicting values associated with an online resource retrieved by a computing device following computing device activity in a computer network environment. 
       Description of Related Art 
       [0002]    Computer networked content delivery systems can deliver content items to computing devices over the computer network. Activation of a content item by a computing device can cause the computing device to retrieve an online resource from a server via the computer network. The content delivery system may seek to measure or predict rates of activation of the delivered content items; however, the rate of activation for a delivered content item may not yield useful data about the corresponding online resource. 
       SUMMARY 
       [0003]    According to one aspect, a method may include determining quality values associated with multiple selections of an advertisement, each of the quality values estimating the likelihood that the advertisement is a good advertisement. The method may further include aggregating the quality values and using the aggregated quality values to predict a future likelihood that the advertisement is good. 
         [0004]    According to another aspect, a method may include providing one or more advertisements to users in response to search queries and logging user behavior associated with user selection of the one or more advertisements. The method may further include logging features associated with selected ones of the one or more advertisements, or associated with the search queries and using a statistical model and the logged user behavior to estimate quality scores associated with the selected advertisements. The method may also include aggregating the estimated quality scores and predicting the quality of an advertisement of the one or more advertisements using the aggregated quality scores. 
         [0005]    According to a further aspect, a method may include receiving a search query from a user and providing a group of advertisements to the user based on the search query. The method may further include receiving, from the user, an indication of a selection of an advertisement from the group of advertisements and logging features associated with the search query or with the selected advertisement. The method may also include retrieving past quality scores from memory using the logged features and predicting a future quality of the selected advertisement based on the retrieved past quality scores. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]    The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the invention and, together with the description, explain the invention. In the drawings, 
           [0007]      FIGS. 1 and 2  are exemplary diagrams of an overview of an implementation in which observed user behavior and known quality ratings associated with a set of advertisements are used to construct a statistical model that can be used for predicting advertisement quality; 
           [0008]      FIG. 3  is an exemplary diagram of a network in which systems and methods consistent with the principles of the invention may be implemented; 
           [0009]      FIG. 4  is an exemplary diagram of a client or server of  FIG. 3  according to an implementation consistent with the principles of the invention; 
           [0010]      FIG. 5  is a flowchart of an exemplary process for constructing a model of user behavior associated with the selections of multiple on-line advertisements according to an implementation consistent with the principles of the invention; 
           [0011]      FIGS. 6-13  illustrate various exemplary session features, corresponding to observed or logged user actions, that may be used for constructing a statistical model for predicting advertisement quality; 
           [0012]      FIG. 14  is a flowchart of an exemplary process for determining predictive values relating to the quality of an advertisement according to an implementation consistent with the principles of the invention; 
           [0013]      FIG. 15  is a diagram that graphically illustrates the exemplary process of  FIG. 14  consistent with an aspect of the invention; 
           [0014]      FIG. 16  is a diagram of an exemplary data structure for storing the predictive values determined in  FIG. 14 ; 
           [0015]      FIG. 17  is a flowchart of an exemplary process for predicting the quality of advertisements according to an implementation consistent with the principles of the invention; 
           [0016]      FIG. 18  is a diagram that graphically illustrates the exemplary process of  FIG. 17  consistent with an aspect of the invention; 
           [0017]      FIG. 19  is a flowchart of an exemplary process for predicting the quality of advertisements according to an implementation consistent with the principles of the invention; and 
           [0018]      FIG. 20  is a diagram that graphically illustrates the exemplary process of  FIG. 19  consistent with an aspect of the invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    The following detailed description of the invention refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Also, the following detailed description does not limit the invention. 
         [0020]    Systems and methods consistent with aspects of the invention may use multiple observations of user behavior (e.g., real-time observations or observations from recorded user logs) associated with user selection of on-line advertisements to more accurately estimate advertisement quality as compared to conventional determinations of quality based solely on CTR. Quality ratings associated with known rated advertisements, and corresponding measured observed user behavior associated with selections (e.g., “clicks”) of those known rated advertisements, may be used to construct a statistical model. The statistical model may subsequently be used to estimate qualities associated with unrated advertisements based on observed user behavior associated with selections of the unrated advertisements. 
         [0021]    A “document,” as the term is used herein, is to be broadly interpreted to include any machine-readable and machine-storable work product. A document may include, for example, an e-mail, a web page or site, a business listing, a file, a combination of files, one or more files with embedded links to other files, a news group posting, a blog, an on-line advertisement, etc. Documents often include textual information and may include embedded information (such as meta information, images, hyperlinks, etc.) and/or embedded instructions (such as Javascript, etc.). A “link,” as the term is used herein, is to be broadly interpreted to include any reference to/from a document from/to another document or another part of the same document. 
       Overview 
       [0022]      FIGS. 1 and 2  illustrate an exemplary overview of an implementation in which a statistical model, and observed user behavior associated with selection of advertisements is used to estimate predictive values that are further aggregated to provide a future prediction of advertisement quality. The future predictions of ad quality may be used in filtering, ranking or promoting advertisements. 
         [0023]    As shown in  FIG. 1 , each one of multiple rated advertisements  100 - 1  through  100 -N (collectively referred to herein as ad  100 ) may be associated with a corresponding document  105 - 1  through  105 -N (collectively referred to herein as document  105 ). Each document  105  may include a set of search results resulting from a search executed by a search engine based on a search query provided by a user and may further include one or more advertisements in addition to a rated ad  100 . Each advertisement  100  may be associated with ratings data  120  provided by human raters who have rated a quality of each rated advertisement  100 . Each advertisement  100  may advertise various products or services. 
         [0024]    In response to receipt of an advertisement  100 , the receiving user may, based on the “creative” displayed on the advertisement, select  110  the advertisement (e.g., “click” on the displayed advertisement using, for example, a mouse). After ad selection  110 , an ad landing document  115  may be provided to the selecting user by a server hosting the advertisement using a link embedded in ad  100 . The ad landing document  115  may provide details of the product(s) and/or service(s) advertised in the corresponding advertisement  100 . 
         [0025]    Before, during and/or after each ad selection  110  by a user, session features  125  associated with each ad selection  110  during a “session” may be measured in real-time or logged in memory or on disk. A session may include a grouping of user actions that occur without a break of longer than a specified period of time (e.g., a group of user actions that occur without a break of longer than three hours). 
         [0026]    The measured session features  125  can include any type of observed user behavior or actions. For example, session features  125  may include a duration of the ad selection  110  (e.g., a duration of the “click” upon the ad  100 ), the number of selections of other advertisements before and/or after a given ad selection, the number of selections of search results before and/or after a given ad selection, the number of selections on other types of results (e.g., images, news, products, etc.) before and/or after a given ad selection, a number of document views (e.g., page views) before and/or after a given ad selection (e.g., page views of search results before and/or after the ad selection), the number of search queries before and/or after a given ad selection, the number of queries associated with a user session that show advertisements, the number of repeat selections on a same given advertisement, or an indication of whether a given ad selection was the last selection in a session, the last ad selection in a session, the last selection for a given search query, or the last ad selection for a given search query. Other types of observed user behavior associated with ad selection, not described above, may be used consistent with aspects of the invention. 
         [0027]    Using the measured session features  125  and ad ratings data  120 , associated with each ad selection  110  of a corresponding rated advertisement  100 , a statistical model  130  may be constructed (as further described below). The statistical model may include a probability model derived using statistical techniques. Such techniques may include, for example, logistic regression, regression trees, boosted stumps, or any other statistical modeling technique. Statistical model  130  may provide a predictive value that estimates the likelihood that a given advertisement  100  is good given measured session features associated with a user selection of the advertisement  100  (e.g., P(good ad|ad selection)=f g (session features)). 
         [0028]    Subsequent to construction of statistical model  130 , ad qualities of unrated advertisements selected by one or more users may be estimated. An unrated ad  135 , associated with a document  140  and hosted by a server in a network, may be provided to an accessing user. Session features  155  associated with user selection  145  of unrated ad  135  may be measured or logged in memory or on disk, and the measurements may be provided as inputs into statistical model  130 . Statistical model  130  may determine a likelihood that unrated ad  135  is a good ad, given the measured session features, and may generate a predictive value  160  for unrated ad  135 . 
         [0029]    Ad/query features  165 , associated with the selection of unrated ad  135 , may also be observed and logged. Ad/query features  165  may include different features associated with the ad  135  or the advertiser that hosted or generated the ad, or features associated with a query issued by a user that resulted in display of the ad  135 . For example, ad/query features  165  may include an identifier associated with the advertiser of ad  135  (e.g., a visible uniform resource locator (URL) of the advertiser), a keyword that the ad  135  targets, words in the query issued by the user that ad  135  did not target, and/or a word in the query issued by the user that ad  135  did not target but which is similar to a word targeted by ad  135 . Other types of ad or query features, not described above, may be used consistent with principles of the invention. The estimated predictive value  160  may be stored in a data structure  170  according to the associated ad/query features  165 , as described in further detail below. 
         [0030]    Though  FIG. 1  depicts the estimation of a predictive quality value associated with a single unrated ad  135 , predictive values  160  may be estimated for each unrated ad  135  selected by one or more users over a span of time to produce multiple ad predictive values  160 , with each predictive value  160  being associated with one or more ad/query feature(s)  165 . The multiple ad predictive values  160  may be aggregated in data structure  170  to produce aggregated predictive values  200 , as shown in  FIG. 2 . Aggregation of predictive values is described below with respect to  FIG. 14 . As further shown in  FIG. 2 , odds may be estimated  210  for each ad/query feature in data structure  170 . The estimated odds may predict a quality of an advertisement given a specific ad/query feature. Further exemplary details of odds estimation is described below with respect to  FIGS. 17 and 18 . The estimated odds for each ad/query feature may be stored in data structure  170 . 
         [0031]    Ad/query features associated with the selection of an advertisement  220  may then be obtained  220 . When a user selects an advertisement from a document (e.g., a search result document), ad/query features associated with that selection may be noted. Estimated odds for each of the ad/query features obtained with respect to the selection of the advertisement may be retrieved  230  from data structure  170 . An overall ad quality may then be predicted  240  using the retrieved estimated odds for each ad/query feature associated with the ad selection. Further exemplary details of the prediction of an overall ad quality is described below with respect to  FIG. 19 . 
       Exemplary Network Configuration 
       [0032]      FIG. 3  is an exemplary diagram of a network  300  in which systems and methods consistent with the principles of the invention may be implemented. Network  300  may include multiple clients  310  connected to one or more servers  320 - 330  via a network  340 . Two clients  310  and two servers  320 - 330  have been illustrated as connected to network  340  for simplicity. In practice, there may be more or fewer clients and servers. Also, in some instances, a client may perform a function of a server and a server may perform a function of a client. 
         [0033]    Clients  310  may include client entities. An entity may be defined as a device, such as a personal computer, a wireless telephone, a personal digital assistant (PDA), a lap top, or another type of computation or communication device, a thread or process running on one of these devices, and/or an object executable by one of these devices. One or more users may be associated with each client  310 . Servers  320  and  330  may include server entities that access, fetch, aggregate, process, search, and/or maintain documents in a manner consistent with the principles of the invention. Clients  310  and servers  320  and  330  may connect to network  340  via wired, wireless, and/or optical connections. 
         [0034]    In an implementation consistent with the principles of the invention, server  320  may include a search engine system  325  usable by users at clients  310 . Server  320  may implement a data aggregation service by crawling a corpus of documents (e.g., web documents), indexing the documents, and storing information associated with the documents in a repository of documents. The data aggregation service may be implemented in other ways, such as by agreement with the operator(s) of data server(s)  330  to distribute their hosted documents via the data aggregation service. In some implementations, server  320  may host advertisements (e.g., creatives, ad landing documents) that can be provided to users at clients  310 . Search engine system  325  may execute a query, received from a user at a client  310 , on the corpus of documents stored in the repository of documents, and may provide a set of search results to the user that are relevant to the executed query. In addition to the set of search results, server  320  may provide one or more advertising creatives, associated with results of the executed search, to the user at client  310 . 
         [0035]    Server(s)  330  may store or maintain documents that may be crawled by server  320 . Such documents may include data related to published news stories, products, images, user groups, geographic areas, or any other type of data. For example, server(s)  330  may store or maintain news stories from any type of news source, such as, for example, the Washington Post, the New York Times, Time magazine, or Newsweek. As another example, server(s)  330  may store or maintain data related to specific products, such as product data provided by one or more product manufacturers. As yet another example, server(s)  330  may store or maintain data related to other types of web documents, such as pages of web sites. Server(s)  330  may further host advertisements, such as ad creatives and ad landing documents. 
         [0036]    Network  340  may include one or more networks of any type, including a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network, such as the Public Switched Telephone Network (PSTN) or a Public Land Mobile Network (PLMN), an intranet, the Internet, a memory device, or a combination of networks. The PLMN(s) may further include a packet-switched sub-network, such as, for example, General Packet Radio Service (GPRS), Cellular Digital Packet Data (CDPD), or Mobile IP sub-network. 
         [0037]    While servers  320 - 330  are shown as separate entities, it may be possible for one of servers  320 - 330  to perform one or more of the functions of the other one of servers  320 - 330 . For example, it may be possible that servers  320  and  330  are implemented as a single server. It may also be possible for a single one of servers  320  and  330  to be implemented as two or more separate (and possibly distributed) devices. 
       Exemplary Client/server Architecture 
       [0038]      FIG. 4  is an exemplary diagram of a client or server entity (hereinafter called “client/server entity”), which may correspond to one or more of clients  310  and/or servers  320 - 330 , according to an implementation consistent with the principles of the invention. The client/server entity may include a bus  410 , a processor  420 , a main memory  430 , a read only memory (ROM)  440 , a storage device  450 , an input device  460 , an output device  470 , and a communication interface  480 . Bus  410  may include a path that permits communication among the elements of the client/server entity. 
         [0039]    Processor  420  may include a processor, microprocessor, or processing logic that may interpret and execute instructions. Main memory  430  may include a random access memory (RAM) or another type of dynamic storage device that may store information and instructions for execution by processor  420 . ROM  440  may include a ROM device or another type of static storage device that may store static information and instructions for use by processor  420 . Storage device  450  may include a magnetic and/or optical recording medium and its corresponding drive. 
         [0040]    Input device  460  may include a mechanism that permits an operator to input information to the client/server entity, such as a keyboard, a mouse, a pen, voice recognition and/or biometric mechanisms, etc. Output device  470  may include a mechanism that outputs information to the operator, including a display, a printer, a speaker, etc. Communication interface  480  may include any transceiver-like mechanism that enables the client/server entity to communicate with other devices and/or systems. For example, communication interface  480  may include mechanisms for communicating with another device or system via a network, such as network  340 . 
         [0041]    The client/server entity, consistent with the principles of the invention, may perform certain operations or processes, as will be described in detail below. The client/server entity may perform these operations in response to processor  420  executing software instructions contained in a computer-readable medium, such as memory  430 . A computer-readable medium may be defined as a physical or logical memory device and/or carrier wave. 
         [0042]    The software instructions may be read into memory  430  from another computer-readable medium, such as data storage device  450 , or from another device via communication interface  480 . The software instructions contained in memory  430  may cause processor  420  to perform operations or processes that will be described later. Alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, implementations consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software. 
       Exemplary Process for Constructing a Statistical Model of User Behavior Associated with Ad Selections 
       [0043]      FIG. 5  is a flowchart of an exemplary process for constructing a statistical model of user behavior associated with the selections of multiple on-line advertisements. As one skilled in the art will appreciate, the process exemplified by  FIG. 5  can be implemented in software and stored on a computer-readable memory, such as main memory  430 , ROM  440 , or storage device  450  of server  320 , server  330  or a client  310 , as appropriate. 
         [0044]    The exemplary process may begin with obtaining ratings data associated with rated advertisements (block  500 ). The ratings data may include human generated data that rates the quality of each of the rated ads (e.g., one way of rating an ad is to rate how relevant is the ad relative to the query issued). Session features associated with each selection of a rated advertisement may then be obtained (block  510 ). The session features may be obtained in real-time by observing actual user behavior during a given user session, that occurred before, during and after the presentation of each ad impression to a user, or may be obtained from recorded logs of session features (i.e., user behavior and actions) that were stored in a data structure before, during and/or after the presentation of each ad impression to a user. The obtained session features  125  can include any type of observed user behavior. Each of the session features  125  may correspond to an indirect measurement of user satisfaction with a given advertisement. Certain ones of the session features  125  may be factors in determining how different users have different values for other ones of the session features  125  (e.g., users with dial-up connections may have longer ad selection durations than users who have high speed Internet connections). 
         [0045]    Session features  125  may include, but are not limited to, a duration of an ad selection (e.g., a duration of the “click” upon the advertisement), a number of selections of other advertisements before and/or after a given ad selection, a number of selections of search results before and/or after a given ad selection, a number of selections of other results before and/or after a given ad selection, a number of document views (e.g., page views) before and/or after a given ad selection, a number of search queries before and/or after a given ad selection, a number of search queries associated with a user session that show advertisements, a number of repeat selections on a same given advertisement, or an indication of whether a given ad selection was the last selection in a session, the last ad selection in a session, a last selection for a given search query, or the last ad selection for a given search query.  FIGS. 6-13  below depict various exemplary types of user behavior, consistent with aspects of the invention, that may be measured as session features. 
         [0046]      FIG. 6  illustrates the measurement of a duration of an ad selection as a session feature  600 . As shown in  FIG. 6 , an ad  605 , that is associated with a document  610 , may be provided to a user. In response to receipt of ad  605 , the user may select  615  ad  605 , and an ad landing document  620  may be provided to the user. A duration  625  of the ad selection (e.g., the period of time from selection of the advertisement until the user&#39;s next action, such as clicking on another ad, entering a new query, etc.) may be measured as a session feature  600 . 
         [0047]      FIG. 7  illustrates the measurement of a number of other ad selections before and/or after a particular ad selection as a session feature  700 . Given a particular selection  705  of an ad N  710 , and provision of an ad landing document  715  in response to the ad selection  705 , a number of one or more previous ad selections  720  of ads N−x  725 , corresponding to provisions of previous ad landing documents  730 , may be measured. Additionally, or alternatively, given a particular selection  705  of an ad N  710 , a number of one or more subsequent ad selections  735  of ads N+x  740 , corresponding to provisions of subsequent ad landing documents  745 , may be measured. The number of other ad selections before and/or after a particular ad selection may be measured as a session feature  700 . 
         [0048]      FIG. 8  illustrates the measurement of a number of search result selections before and/or after a particular ad selection as a session feature  800 . Given a particular selection  805  of an ad N  810 , and provision of an ad landing document  815  in response to the ad selection  805 , a number of search result documents  820  viewed by the user before the ad selection  805  may be measured as a session feature  800 . The search result documents may be provided to the user based on the execution of a search using a search query issued by the user. Additionally, or alternatively, a number of search result documents  825  viewed by the user after the ad selection  805  may be measured as a session feature  800 . 
         [0049]      FIG. 9  illustrates the measurement of a number of documents viewed by a user before and/or after a particular ad selection as a session feature  900 . Given a particular selection  905  of an ad  910 , and provision of an ad landing document  915  in response to the ad selection  905 , a number of documents  920  viewed by a user (e.g., page views) before the ad selection  905  may be measured as a session feature  900 . Additionally, or alternatively, a number of documents  925  viewed by a user (e.g., page views) after the ad selection  905  may be measured as a session feature  900 . 
         [0050]      FIG. 10  illustrates the measurement of a number of search queries issued by a user before and/or after a particular ad selection as a session feature  1000 . Given a particular selection  1005  of an ad  1010 , and provision of an ad landing document  1015  in response to the ad selection  1005 , a number of search queries  1020  issued by a user before the ad selection  1005  may be measured as a session feature  1000 . Additionally, or alternatively, a number of search queries  1025  issued by a user after the ad selection  1005  may be measured as a session feature  1000 . 
         [0051]      FIG. 11  illustrates the measurement of a number of search queries, in a session that includes a particular ad selection, that results in the display of an advertisement as a session feature  1100 . Given a session that includes a particular ad selection, a number of search queries  1105  may be measured that result in the display of a corresponding ad  1110 - 1  through  1110 -N. The number of search queries may be measured as a session feature  1100 . The number of search queries  1105  resulting in the display of an advertisement may indicate the commercial nature of a given user session. 
         [0052]      FIG. 12  illustrates the measurement of a number of repeat selections of the same advertisement by a user as a session feature  1200 . As shown in  FIG. 12 , an ad  1205 , that may be associated with multiple documents  1210 - 1  through  1210 -N, may be provided to a user one or more times. In response to each receipt of the ad  1205 , the user may select  1215  ad  1205 , and an ad landing document  1220  may be provided to the user for each of the repeated user selections. The number of repeat selections of the same advertisement by the user may be measured as a session feature  1200 . 
         [0053]      FIG. 13  illustrates the determination of whether an ad selection is the last ad selection for a given search query, or whether the ad selection is the last ad selection for a user session as a session feature  1300 . As shown in  FIG. 13 , a user may issue a search query  1305  during a given session  1310 , and one or more ads  1315  may be provided to the user subsequent to issuance of search query  1305 . In response to each receipt of the ad(s)  1315 , the user may select  1320  ad  1315 , and an ad landing document  1325  may be provided to the user. A determination may be made whether the ad selection  1320  is the last ad selection for search query  1305 . Thus, if multiple ads were selected by the user that issued search query  1305 , then only the last ad selection for search query  1305  may be identified. A determination may also be made whether the ad selection  1320  was the last ad selection for session  1310 . Therefore, if multiple ad selections have been made by the user during a given session, then only the last ad selection for the session may be identified. 
         [0054]    Other types of user behavior, not shown in  FIGS. 6-13 , may be used as session features consistent with principles of the invention. The following lists numerous examples of other exemplary session features: 
         [0055]    1) instead of an ad selection duration, a ratio of a given ad selection duration relative to an average ad selection duration for a given user may be used as a session feature. 
         [0056]    2) a ratio of a given ad selection duration relative to all selections (e.g., search result selections or ad selections); 
         [0057]    3) how many times a user selects a given ad in a given session. 
         [0058]    4) a duration of time, from an ad result selection, until the user issues another search query. This may include time spent on other pages (reached via a search result click or ad click) subsequent to a given ad click. 
         [0059]    5) a ratio of the time, from a given ad result selection until the user issues another search query, as compared to all other times from ad result selections until the user issued another search query. 
         [0060]    6) time spent, given an ad result selection, on viewing other results for the search query, but not on the given ad result. 
         [0061]    7) a ratio of the time spent in 6) above (i.e., the time spent on other results rather than the click duration) to an average of the time spent in 6) across all queries. 
         [0062]    8) how many searches (i.e., a unique issued search query) that occur in a given session prior to a given search result or ad selection; 
         [0063]    9) how many searches that occur in a given session after a given search result or ad selection. 
         [0064]    10) rather than searches, how many result page views that occur for a given search query before a given selection. This can be computed within the query (i.e., just for a unique query), or for the entire session; 
         [0065]    11) rather than searches, how many search result page views that occur for a given search query after this selection. This can be computed within the query (i.e., just for the unique query), or for the entire session; 
         [0066]    12) the total number of page views that occur in the session; 
         [0067]    13) the number of page views in the session that show ads; 
         [0068]    14) the ratio of the number of page views in the session that show ads to the total number of page views that occur in the session; 
         [0069]    15) total number of ad impressions shown in the session; 
         [0070]    16) average number of ads shown per query that shows ads, another measure of the commerciality of the session; 
         [0071]    17) query scan time—how long from when the user sees the results of a query to when the user does something else (click on an ad, search result, next page, new query, etc.); 
         [0072]    18) ratio between a given query scan time and all other query scan times; 
         [0073]    19) total number of selections (e.g., clicks) that occurred on a given search. These selections include all types of selections (e.g., search, onebox, ads) rather than just ad selections; 
         [0074]    20) total number of selections that occurred on a search before a given ad selection; 
         [0075]    21) total number of selections that occurred on a search after a given ad selection; 
         [0076]    22) total number of ad selections that occurred on a search. May need to be normalized by the number of ads on the page; 
         [0077]    23) total number of ad selections that occurred on a search before a given ad selection; 
         [0078]    24) total number of ad selections that occurred on a search after a given ad selection; 
         [0079]    25) total number of ad selections, that occurred on a search, whose ad positions on a document were located above a position of a given ad on the document; 
         [0080]    26) total number of ad selections, that occurred on a search, whose ad positions on a document were located below a position of a given ad on the document; 
         [0081]    27) total number of ad selections that occurred on a search that are not on a given ad; 
         [0082]    28) total number of search result selections that occurred on a search; 
         [0083]    29) total number of search selections that occurred on a search before a given ad selection; 
         [0084]    30) total number of search result selections that occurred on a search after a given ad selection; 
         [0085]    31) total number of search result selections of a long duration that occurred in the session; 
         [0086]    32) total number of search result selections of a short duration that occurred in the session; 
         [0087]    33) total number of search result selections that are last that occurred in the session. A given user may end a session by clicking on a search result, with no subsequent actions, or the user may end a session in some other fashion (e.g., ad result click, issuing a query and not clicking, etc.); 
         [0088]    34) total number of non-search result and non-ad selections that occurred on a search; 
         [0089]    35) an indication of whether there was a conversion from this ad selection; 
         [0090]    36) an indication of the connection speed of the user (e.g., dialup, cable, DSL); 
         [0091]    37) an indication of what country the user is located in. Different cultures might lead to users reacting differently to the same ad or having different cultural reactions or staying on sites differently; 
         [0092]    38) an indication of what region of the world (e.g., APAC==asia pacific) that the user is located in; 
         [0093]    39) was the keyword for a given ad an exact match to the search query (i.e., has all of the same terms as the query) or is missing one word, more than one word, or has rewrite terms. Often, the quality of an ad can vary (the more exact the match, the higher the quality) and keyword matching can be a reasonable way to segment ads, and predict whether an ad is good or bad separately for different match types. 
         [0094]    40) an indication of an estimated click through rate (CTR) for a given ad; 
         [0095]    41) what cost per click (CPC) did the advertiser pay for a given ad selection? The likelihood that an ad is good may depend on how much the advertiser paid (more is higher quality); 
         [0096]    42) what CPC was the advertiser willing to pay? In ad auctioning, advertiser bidding may be used to set ad ranking and the ad/advertiser ranked lower than a given ad/advertiser sets the price that is actually paid by the next higher ranked ad/advertiser; 
         [0097]    43) effective CPC * predicted CTR; or 
         [0098]    44) bid CPC * predicted CTR. 
         [0000]    The above describes numerous examples of session features that may be used for the statistical model. However, one skilled in the art will recognize that other session features may be used, alternatively, or in conjunction with any of the above-described session features. 
         [0099]    Returning to  FIG. 5 , a statistical model may then be derived that determines the probability that each selected ad is a good quality ad given the measured session features associated with the ad selection (block  520 ). An existing statistical technique, such as, for example, logistic regression may be used to derive the statistical model consistent with principles of the invention. Regression involves finding a function that relates an outcome variable (dependent variable y) to one or more predictors (independent variables x 1 , x 2 , etc.). Simple linear regression assumes a function of the form: 
         [0000]        y=c   0   +c   1   *x   1   +c   2   *x   2 + . . .   Eqn. (1)
 
         [0000]    and finds the values of c 0 , c 1 , c 2 , etc. (c 0  is called the “intercept” or “constant term”). In the context of the present invention, each predictor variable x 1 , x 2 , x 3 , etc. corresponds to a different session feature measured during ad selection. Logistic regression is a variation of ordinary regression, useful when the observed outcome is restricted to two values, which usually represent the occurrence or non-occurrence of some outcome event, (usually coded as 1 or 0, respectively), such as a good advertisement or a bad advertisement in the context of the present invention. 
         [0100]    Logistic regression produces a formula that predicts the probability of the occurrence as a function of the independent predictor variables. Logistic regression fits a special s-shaped curve by taking the linear regression (Eqn. (1) above), which could produce any y-value between minus infinity and plus infinity, and transforming it with the function: 
         [0000]        P =exp( y )/(1+exp( y ))   Eqn. (2)
 
         [0000]    which produces P-values between 0 (as y approaches minus infinity) and 1 (as y approaches plus infinity). Substituting Eqn. (1) into Eqn. (2), the probability of a good advertisement, thus, becomes the following: 
         [0000]    
       
         
           
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         
                           good 
                            
                           
                               
                           
                            
                           ad 
                         
                          
                         
                           ad 
                            
                           
                               
                           
                            
                           selection 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         f 
                         g 
                       
                       ( 
                       
                         
                           session 
                            
                           
                               
                           
                            
                           features 
                            
                           
                               
                           
                            
                           
                             x 
                             1 
                           
                         
                         , 
                         
                           x 
                           2 
                         
                         , 
                         
                           
                             x 
                             3 
                           
                            
                           
                               
                           
                            
                           … 
                         
                       
                        
                       
                           
                       
                       ) 
                     
                     = 
                     
                       
                         e 
                         
                           ( 
                           
                             
                               c 
                               
                                 g 
                                  
                                 
                                     
                                 
                                  
                                 0 
                               
                             
                             + 
                             
                               
                                 c 
                                 
                                   g 
                                    
                                   
                                       
                                   
                                    
                                   1 
                                 
                               
                               * 
                               
                                 x 
                                 1 
                               
                             
                             + 
                             
                               
                                 c 
                                 
                                   g 
                                    
                                   
                                       
                                   
                                    
                                   2 
                                 
                               
                               * 
                               
                                 x 
                                 2 
                               
                             
                             + 
                             
                                 
                             
                              
                             … 
                           
                            
                           
                               
                           
                           ) 
                         
                       
                       
                         1 
                         + 
                         
                           e 
                           
                             ( 
                             
                               
                                 c 
                                 
                                   g 
                                    
                                   
                                       
                                   
                                    
                                   0 
                                 
                               
                               + 
                               
                                 
                                   c 
                                   
                                     g 
                                      
                                     
                                         
                                     
                                      
                                     1 
                                   
                                 
                                 * 
                                 
                                   x 
                                   1 
                                 
                               
                               + 
                               
                                 
                                   c 
                                   
                                     g 
                                      
                                     
                                         
                                     
                                      
                                     2 
                                   
                                 
                                 * 
                                 
                                   x 
                                   2 
                                 
                               
                               + 
                               
                                   
                               
                                
                               … 
                             
                              
                             
                                 
                             
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Eqn 
                   . 
                   
                       
                   
                    
                   
                     ( 
                     3 
                     ) 
                   
                 
               
             
           
         
       
     
         [0000]    where c g0  is the constant of the equation, and c gn  is the coefficient of the session feature predictor variable x n . The probability of a bad advertisement may, similarly, be determined by the following: 
         [0000]    
       
         
           
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         
                           bad 
                            
                           
                               
                           
                            
                           ad 
                         
                          
                         
                           ad 
                            
                           
                               
                           
                            
                           selection 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         f 
                         b 
                       
                       ( 
                       
                         
                           session 
                            
                           
                               
                           
                            
                           features 
                            
                           
                               
                           
                            
                           
                             x 
                             1 
                           
                         
                         , 
                         
                           x 
                           2 
                         
                         , 
                         
                           
                             x 
                             3 
                           
                            
                           
                               
                           
                            
                           … 
                         
                       
                        
                       
                           
                       
                       ) 
                     
                     = 
                     
                       ( 
                       
                         
                           e 
                           
                             ( 
                             
                               
                                 c 
                                 
                                   b 
                                    
                                   
                                       
                                   
                                    
                                   0 
                                 
                               
                               + 
                               
                                 
                                   c 
                                   
                                     b 
                                      
                                     
                                         
                                     
                                      
                                     1 
                                   
                                 
                                 * 
                                 
                                   x 
                                   1 
                                 
                               
                               + 
                               
                                 
                                   c 
                                   
                                     b 
                                      
                                     
                                         
                                     
                                      
                                     2 
                                   
                                 
                                 * 
                                 
                                   x 
                                   2 
                                 
                               
                               + 
                               
                                   
                               
                                
                               … 
                             
                              
                             
                                 
                             
                             ) 
                           
                         
                         
                           1 
                           + 
                           
                             e 
                             
                               ( 
                               
                                 
                                   c 
                                   
                                     b 
                                      
                                     
                                         
                                     
                                      
                                     0 
                                   
                                 
                                 + 
                                 
                                   
                                     c 
                                     
                                       b 
                                        
                                       
                                           
                                       
                                        
                                       1 
                                     
                                   
                                   * 
                                   
                                     x 
                                     1 
                                   
                                 
                                 + 
                                 
                                   
                                     c 
                                     
                                       b 
                                        
                                       
                                           
                                       
                                        
                                       2 
                                     
                                   
                                   * 
                                   
                                     x 
                                     2 
                                   
                                 
                                 + 
                                 
                                     
                                 
                                  
                                 … 
                               
                                
                               
                                   
                               
                               ) 
                             
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   Eqn 
                   . 
                   
                       
                   
                    
                   
                     ( 
                     4 
                     ) 
                   
                 
               
             
           
         
       
     
         [0000]    where c b0  is the constant of the equation, and c bn  is the coefficient of the session feature predictor variables x n . 
         [0101]    A fit of the statistical model may be tested to determine which session features are correlated with good or bad quality advertisements. If a logistic regression technique is used to determine the statistical model, the goal of logistic regression is to correctly predict the outcome for individual cases using the most parsimonious model. To accomplish this goal, a model is created that includes all predictor variables (e.g., session features) that are useful in predicting the outcome of the dependent y variable. To construct the statistical model, logistic regression can test the fit of the model after each coefficient (c n ) is added or deleted, called stepwise regression. For example, backward stepwise regression may be used, where model construction begins with a full or saturated model and predictor variables, and their coefficients, are eliminated from the model in an iterative process. The fit of the model is tested after the elimination of each variable to ensure that the model still adequately fits the data. When no more predictor variables can be eliminated from the model, the model construction has been completed. The predictor variables that are left in the model, each corresponding to a measured session feature, identify the session features that are correlated with good or bad advertisements. Logistic regression, thus, can provide knowledge of the relationships and strengths among the different predictor variables. The process by which coefficients, and their corresponding predictor variables, are tested for significance for inclusion or elimination from the model may involve several different known techniques. Such techniques may include the Wald test, the Likelihood-Ratio test, or the Hosmer-Lemshow Goodness of Fit test. These coefficient testing techniques are known in the art and are not further described here. In other implementations, existing techniques for cross validation and independent training may be used instead of techniques of classical estimation and testing of regression coefficients, as described above. 
         [0102]    Other existing statistical techniques, instead of, or in addition to logistic regression, may be used to derive a statistical model consistent with principles of the invention. For example, a “stumps” model, using “boosting” techniques may be used to derive the statistical model. As one skilled in the art will recognize, “boosting” is a machine learning technique for building a statistical model by successively improving an otherwise weak statistical model. The basic idea is to repeatedly apply the same algorithm to an entire training data set, but differentially weight the training data at each stage. The weights are such that cases that are well-fit by the model through stage k receive relatively small weights at stage k+1, while cases that are ill-fit by the model through stage k receive relatively large weights at stage k+1. 
         [0103]    Stumps are a weak statistical model that can be applied at each stage. A stump is a 2-leaf classification tree consisting of a root node and a binary rule that splits the cases into two mutually exclusive subsets (i.e., the leaf nodes). A rule could take the form “ClickDuration&lt;120 sec” and all cases with ClickDuration satisfying the rule go into one leaf node and those not satisfying the rule go into the other leaf node. Another rule could take the form “AdSelection was the last ad selection” and all cases with AdSelection satisfying the rule go into one leaf node and those not satisfying the rule go into the other leaf node. 
         [0104]    Various algorithms can be used to fit the “boosted stump” model including, for example, gradient-based methods. Such algorithms may proceed as follows: given a set of weights, among all possible binary decision rules derived from session features that partition the cases into two leaves, choose that one which minimizes the (weighted) loss function associated with the algorithm. Some examples of loss functions are “Bernoulli loss” corresponding to a maximum likelihood method, and “exponential loss” corresponding to the well-known ADABoost method. After choosing the best binary decision rule at this stage, the weights may be recomputed and the process may be repeated whereby the best binary rule is chosen which minimizes the new (weighted) loss function. This process may be repeated many times (e.g., several hundred to several thousand) and a resampling technique (such as cross-validation) may be used to define a stopping rule in order to prevent over-fitting. 
         [0105]    Boosted stumps have been shown to approximate additive logistic regression models whereby each feature makes an additive nonlinear contribution (on the logistic scale) to the fitted model. The sequence of stumps define the relationship between session features and the probability that an ad is rated “good”. The sequence can be expressed by the statistical model: 
         [0000]    
       
         
           
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         
                           good 
                            
                           
                               
                           
                            
                           ad 
                         
                          
                         
                           session 
                            
                           
                               
                           
                            
                           feature 
                            
                           
                               
                           
                            
                           x 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       e 
                       
                         ( 
                         
                           
                             c 
                             0 
                           
                           + 
                           
                             
                               c 
                               1 
                             
                             * 
                             B 
                              
                             
                                 
                             
                              
                             1 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           + 
                           
                             
                               c 
                               2 
                             
                             * 
                             B 
                              
                             
                                 
                             
                              
                             2 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           + 
                           
                               
                           
                            
                           … 
                         
                          
                         
                             
                         
                         ) 
                       
                     
                     
                       1 
                       + 
                       
                         e 
                         
                           ( 
                           
                             
                               c 
                               0 
                             
                             + 
                             
                               
                                 c 
                                 1 
                               
                               * 
                               B 
                                
                               
                                   
                               
                                
                               1 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             + 
                             
                               
                                 c 
                                 2 
                               
                               * 
                               B 
                                
                               
                                   
                               
                                
                               2 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             + 
                             
                                 
                             
                              
                             … 
                           
                            
                           
                               
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   Eqn 
                   . 
                   
                       
                   
                    
                   
                     ( 
                     5 
                     ) 
                   
                 
               
             
           
         
       
     
         [0000]    where Bk(x)=1 if session feature x satisfies the kth binary rule, or Bk(x)=0 if session feature x does not satisfy the kth binary rule. The coefficients c k , k=1, . . . , are a by-product of the algorithm and relate to the odds of a good ad at the kth binary rule. In practice, given session feature x, each binary rule can be evaluated and the corresponding coefficients accumulated to get the predicted probability of a good ad. A statistical model, similar to Eqn. (5) above, may similarly be derived that defines the relationship between session features and the probability that an ad is rated “bad.” 
         [0106]    Though logistic regression and boosted stumps have been described above as exemplary techniques for constructing a statistical model, one skilled in the art will recognize that other existing statistical techniques, such as, for example, regression trees may be used to derive the statistical model consistent with principles of the invention. 
       Exemplary Process for Determining Predictive Values Related to Ad Quality 
       [0107]      FIG. 14  is a flowchart of an exemplary process for determining predictive values relating to the quality of an advertisement according to an implementation consistent with the principles of the invention. As one skilled in the art will appreciate, the process exemplified by  FIG. 14  can be implemented in software and stored on a computer-readable memory, such as main memory  430 , ROM  440 , or storage device  450  of servers  320  or  330  or client  310 , as appropriate. 
         [0108]    The exemplary process may begin with the receipt of a search query (block  1400 ). A user may issue the search query to server  320  for execution by search engine system  325 . A set of ads that match the received search query may be obtained by search engine system  325  (block  1405 ). Search engine system  325  may execute a search, based on the received search query, to ascertain the set of ads, and other documents, that match the search query. Search engine system  325  may provide the set of ads, and a list of the other documents, to the user that issued the search query. 
         [0109]    Session features associated with the selection of an ad from the set of ads may be obtained (block  1410 ). The session features may be measured in real-time during user ad selection or may be obtained from logs of recorded user behavior associated with ad selection. As shown in  FIG. 15 , a user may select  1500  an ad  1505  associated with a document  1510  (e.g., a document containing search results and relevant ads). An ad landing document  1515  may be provided to the user in response to selection of the ad  1505 . As shown in  FIG. 15 , session features  1520  associated with the selection  1500  of ad  1505  may be measured. The measured session features may include any type of user behavior associated with the selection of an advertisement, such as those described above with respect to block  510  ( FIG. 5 ). 
         [0110]    The statistical model, derived in block  520  above, and the obtained session features may be used to determine predictive values  1530  that the ad is a good ad and/or a bad ad (block  1415 ). The predictive values may include a probability value (e.g., derived using Eqn. (3) or (5) above) that indicate the probability of a good ad given session features associated with user selection of that ad. The predictive values may also include a probability value (Eqn. (4) above) that indicates the probability of a bad ad given measured session features associated with user selection of that ad. Therefore, session feature values may be input into Eqn. (3), (4) and/or (5) to obtain a predictive value(s) that the selected ad is good or bad. For example, values for session features x 1 , x 2 , x 3  and x 4  may be input into Eqn. (3) to obtain a probability value for P(good ad|session features x 1 , x 2 , x 3 , x 4 ). As shown in  FIG. 15 , the measured session features  1520  may be input into statistical model  130  and statistical model  130  may output predictive values  1530  for the ad  1505 . 
         [0111]    Ad/query features associated with the selection of the advertisement may be obtained (block  1420 ). As shown in  FIG. 15 , the ad/query features  1535  may be obtained in association with selection  1500  of the ad  1505 . The ad/query features  1535  may include an identifier associated with the advertiser of ad  1505  (e.g., a visible uniform resource locator (URL) of the advertiser), a keyword that ad  1505  targets, words in the search query issued by the user that ad  1505  did not target, and/or a word in the search query issued by the user that the advertisement did not target but which is similar to a word targeted by advertisement  1505 . Other types of ad or query features, not described above, may be used consistent with principles of the invention. For example, any of the above-described ad/query features observed in combination (e.g., a pairing of two ad/query features) may be used as a single ad query/feature. 
         [0112]    For each obtained ad/query feature (i.e., obtained in block  1420  above), the determined predictive values may be summed with stored values that correspond to the ad/query feature (block  1425 ). The determined predictive values may be summed with values stored in a data structure, such as, for example, data structure  1600  shown in  FIG. 16 . As shown in  FIG. 16 , data structure  1600  may include multiple ad/query features  1610 - 1  through  1610 -N, with a “total number of ad selections”  1620 , a total “good” predictive value  1630  and a total “bad” predictive value  1640  being associated with each ad/query feature  1610 . Each predictive value determined in block  1405  can be summed with a current value stored in entries  1630  or  1640  that corresponds to each ad/query feature  1610  that is further associated with the advertisement and query at issue. As an example, assume that an ad for “1800flowers.com” is provided to a user in response to the search query “flowers for mother&#39;s day.” The session features associated with the selection of the ad return a probability P(good ad|ad selection) of 0.9. Three ad/query features are associated with the ad and query: the query length (the number of terms in the query), the visible URL of the ad, and the number of words that are in the query, but not in the keyword that&#39;s associated with the ad. For each of the three ad/query features, a corresponding “total number of ad selections” value in entry  1620  is incremented by one, and 0.9 is added to each value stored in the total good predictive value  1630  that corresponds to each of the ad/query features. 
         [0113]    As shown in  FIG. 15 , each of the determined predictive values  1530  may be summed with a current value in data structure  1600 . Blocks  1400  through  1425  may be selectively repeated for each selection of an ad, by one or more users, to populate data structure  1600  with numerous summed predictive values that are associated with one or more ad/query features. 
       Exemplary Odds Estimation Process 
       [0114]      FIGS. 17 and 18  are flowcharts of an exemplary process for estimating odds of good or bad qualities associated with advertisements using the total predictive values  1630  or  1640  determined in block  1425  of  FIG. 14 . As one skilled in the art will appreciate, the process exemplified by  FIGS. 17 and 18  can be implemented in software and stored on a computer-readable memory, such as main memory  430 , ROM  440 , or storage device  450  of servers  320  or  330  or client  310 , as appropriate. 
         [0115]    The estimated odds that a given advertisement is good or bad is a function of prior odds that the given advertisement was good or bad, and one or more model parameters associated with ad/query features associated with selection of the given advertisement. The model parameters may be calculated using an iterative process that attempts to solve for the parameter values that produce the best fit of the predicted odds of a good or bad advertisement to the actual historical data used for training. 
         [0116]    The model parameters associated with each ad/query feature may consist of a single parameter, such as a multiplier on the probability or odds of a good advertisement or bad advertisement. Alternatively, each ad/query feature may have several model parameters associated with it that may affect the predicted probability of a good or bad advertisement in more complex ways. 
         [0117]    In the following description, various odds and probabilities are used. The odds of an event occurring and the probability of an event occurring are related by the expression: probability=odds/(odds+1). For example, if the odds of an event occurring are 1/2 (i.e., the odds are “1:2” as it is often written), the corresponding probability of the event occurring is 1/3. According to this convention, odds and probabilities may be considered interchangeable. It is convenient to express calculations in terms of odds rather than probabilities because odds may take on any non-negative value, whereas probabilities must lie between 0 and 1. However, it should be understood that the following implementation may be performed using probabilities exclusively, or using some other similar representation such as log(odds), with only minimal changes to the description below. 
         [0118]      FIG. 17  is a flow diagram illustrating one implementation of a prediction model for generating an estimation of the odds that a given advertisement is good or bad based on ad/query features associated with selection of the advertisement. In accordance with one implementation of the principles of the invention, the odds of a good or bad ad may be calculated by multiplying the prior odds (q 0 ) of a good ad or bad ad by a model parameter (m i ) associated with each ad/query feature (k i ), henceforth referred to as an odds multiplier. Such a solution may be expressed as: 
         [0000]        q=q   0   .m   1   .m   2   .m   3    . . . m   m . 
         [0119]    In essence, the odds multiplier m for each ad/query feature k may be a statistical representation of the predictive power of this ad/query feature in determining whether or not an advertisement is good or bad. 
         [0120]    In one implementation consistent with principles of the invention, the model parameters described above may be continually modified to reflect the relative influence of each ad/query feature k on the estimated odds that an advertisement is good or bad. Such a modification may be performed by comparing the average predicted odds that advertisements with this query/ad feature are good or bad, disregarding the given ad/query feature, to an estimate of the historical quality of advertisements with this ad/query feature. In this manner, the relative value of the analyzed ad/query feature k may be identified and refined. 
         [0121]    Turning specifically to  FIG. 17 , for each selected ad/query feature (k i ), an average self-excluding probability (P i ) may be initially calculated or identified (act  1700 ). In one implementation, the self-excluding probability (P i ) is a value representative of the relevance of the selected ad/query feature and may measure the resulting odds that an advertisement is good or bad when the selected ad/query feature&#39;s model parameter (m i ) is removed from the estimated odds calculation. For ad/query feature  3 , for example, this may be expressed as: 
         [0000]        P   3n +(( q   o   .m   1   .m   2   .m   3    . . . m   n )/ m   3 /((( q   o   .m   1   .m   2   .m   3  . . . m n )/ m   3 +1). 
         [0122]    In one embodiment, the self-excluding probability for each ad/query feature may be maintained as a moving average, to ensure that the identified self-excluding probability converges more quickly following identification of a model parameter for each selected ad/query feature. Such a moving average may be expressed as: 
         [0000]        P   in (avg)=α P   i(n−1) (avg)+(1−α) P   in ,
 
         [0000]    where α is a statistically defined variable very close to 1 (e.g., 0.999) used to control the half-life of the moving average. As shown in the above expression, the value of P, for the current number of ad selections (n) (e.g., a current value for “total number of ad selections”  1620  for ad/query feature k i ) is weighted and averaged by the value of P, as determined at the previous ad selection (e.g., n−1). 
         [0123]    Next, the average self-excluding probability (P i (avg)), may be compared to historical information relating to the number of advertisement selections observed and the odds of a good or bad advertisement observed for the observed selections (act  1710 ). The model parameter m, associated with the selected ad/query feature k i  may then be generated or modified based on the comparison of act  1710  (act  1720 ) (as further described below with respect to blocks  1820  and  1830  of  FIG. 18 ). 
         [0124]      FIG. 18  is a flow diagram illustrating one exemplary implementation of blocks  1710 - 1720  of  FIG. 17 . Initially, a confidence interval relating to the odds of a good ad or bad ad may be determined (act  1800 ). Using a confidence interval technique enables more accurate and stable estimates when ad/query features k having lesser amounts of historical data are used. In one implementation, the confidence interval includes a lower value L i  and an upper value U i  and is based on the number of ad selections (n i ) (e.g., a current value in “total number of ad selections”  1620  in data structure  1600  for ad/query feature k i ) and total goodness/badness (j i ) observed for the selected ad/query feature (e.g., a current total “good” predictive value  1630  or total “bad” predictive value  1640  in data structure  1600  for ad/query feature k i ). For example, the confidence interval may be an 80% confidence interval [L i ,U i ] calculated in a conventional manner based on the number of ad selections (e.g., a current value in “total number of ad selections”  1620  in data structure  1600  for ad/query feature k i ) and total goodness or badness observed (e.g., a current total “good” predictive value  1630  or total “bad” predictive value  1640  in data structure  1600  for ad/query feature k i ). Following confidence interval calculation, it may then be determined whether the average self-excluding probability (P i (avg)) falls within the interval (act  1810 ). If so, it may be determined that the selected ad/query feature (k i ) has no effect on the odds of a good ad or bad ad and its model parameter (m 1 ) may be set to 1, effectively removing it from the estimated odds calculation (act  1820 ). However, if it is determined that P i (avg) falls outside of the confidence interval, then the model parameter (m i ) for the selected ad/query feature k i  may be set to the minimum adjustment necessary to bring the average self-excluding probability (P i (avg)) into the confidence interval (act  1830 ). This calculation may be expressed mathematically as: 
         [0000]        m   i   =[L   i (1− P   i (avg))]/[ P   i (avg)(1− L   i )]
 
         [0125]    Returning now to  FIG. 17 , once the model parameter m i  for the selected ad/query feature k i  is calculated, it may be determined whether additional ad/query features (e.g., of ad/query features  1610 - 1  through  1610 -N of  FIG. 16 ) remain to be processed (i.e., whether k i &lt;k m , where m equals the total number of ad/query features in data structure  1600 ) (act  1730 ). If additional ad/query features remain to be processed, the counter variable i may be incremented (act  1740 ) and the process may return to act  1700  to process the next ad/query feature k i . Once model parameters for all ad/query features have been calculated or modified, the odds of a good ad or bad ad may be estimated using the equation q=q 0 .m 1 .m 2 .m 3  . . . m m  (act  1750 ). The estimated odds of a good ad (e.g., ODDS(good ad|ad query feature)) may be stored in a “good” ad odds entry  1650  of data structure  1600  that corresponds to the ad/query feature  1610 . The estimated odds of a bad ad (e.g., ODDS(bad ad|ad/query feature)) may be stored in “bad” ad odds entry  1660  of data structure  1600  that corresponds to the ad/query feature  1610 . 
         [0126]    In one implementation consistent with principles of the invention, the odds prediction model may be trained by processing log data as it arrives and accumulating the statistics mentioned above (e.g., ad selections, total goodness or badness, self-including probabilities, etc.). As additional ad selections occur, the confidence intervals associated with each ad/query feature may shrink and the parameter estimates may become more accurate. In an additional implementation, training may be accelerated by reprocessing old log data. When reprocessing log data, the estimated odds of a good ad or bad ad may be recalculated using the latest parameter or odds multiplier values. This allows the prediction model to converge more quickly. 
       Exemplary Ad Quality Prediction Process 
       [0127]      FIG. 19  is a flowchart of an exemplary process for predicting the quality of advertisements according to an implementation consistent with the principles of the invention. As one skilled in the art will appreciate, the process exemplified by  FIG. 19  can be implemented in software and stored on a computer-readable memory, such as main memory  430 , ROM  440 , or storage device  450  of servers  320  or  330  or client  310 , as appropriate. 
         [0128]    The exemplary process may begin with the receipt of a search query from a user (block  1900 ). The user may issue the search query to server  320  for execution by search engine system  325 . A set of ads that match the received search query may be obtained by search engine system  325  (block  1910 ). Search engine system  325  may execute a search, based on the received search query, to ascertain the set of ads, and other documents, that match the search query. For each ad of the set of ads, every ad/query feature that corresponds to the received search query and the ad may be determined (block  1920 ). The ad/query features for each search query and ad pair may include an identifier associated with the advertiser (e.g., a visible uniform resource locator (URL) of the advertiser), a keyword that the ad targets, words in the search query issued by the user that ad did not target, and/or a word in the search query issued by the user that the advertisement did not target but which is similar to a word targeted by the advertisement. Other types of ad or query features, not described above, may be used consistent with principles of the invention. For example, any of the above-described ad/query features observed in combination (e.g., a pairing of two ad/query features) may be used as a single ad query/feature. 
         [0129]    For each ad of the set of ads, stored ODDS i  (e.g., ODDS (good ad|ad/query feature)  1650 , ODDS (bad ad|ad/query feature)  1660 ), for every one of the determined ad/query features i, may be retrieved from data structure  1600  (block  1930 ). As shown in  FIG. 20 , data structure  1600  may be indexed with ad/query features  2000  that correspond to the search query and the ad to retrieve one or more ODDS i    2010  associated with each ad/query feature. For example, as shown in  FIG. 16 , a “good” ad odds value  1650  corresponding to each ad/query feature  1610  may be retrieved. As another example, as shown in  FIG. 16 , a “bad” ad odds value  1660  corresponding to each ad/query feature  1610  may be retrieved. 
         [0130]    For each ad of the set of ads, the retrieved ODDS i  for each ad/query feature i may be multiplied together (block  1940 ) to produce a total ODDS value (ODDS t ): 
         [0000]        ODDS   t   =ODDS   1   *ODDS   2   *ODDS   3 * . . .   Eqn. (6)
 
         [0000]    For example, the “good” ad odds values  1650  for each ad/query feature may be multiplied together to produce a total good ad odds value ODDS t   _   GOOD AD . As another example, the “bad” ad odds values  1660  for each ad/query feature may be multiplied together to produce a total bad ad odds value ODDS t   _   BAD AD . As shown in  FIG. 20 , the ODDS  2010  retrieved from data structure  1600  may be multiplied together to produce a total odds value ODDS t    2020 . 
         [0131]    For each ad of the set of ads, a quality parameter that may include a probability that the ad is good (P GOOD AD ) and/or that the ad is bad (P BAD AD ) may be determined (block  1950 ): 
         [0000]        P   GOOD AD   =ODDS   t   _   GOOD AD /(1+ ODDS   t   _   GOOD AD )   Eqn. (7)
 
         [0000]        P   BAD AD   =ODDS   t   _   BAD AD /(1+ ODDS   t   _   BAD AD )   Eqn. (8)
 
         [0000]    As shown in  FIG. 20 , the total odds value ODDS t    1820 , and equations (7) or (8) may be used to derive a quality parameter (P)  2030 . The derived quality parameter P may subsequently be used, for example, to filter, rank and/or promote advertisements as described in co-pending U.S. application Ser. No. ______ (Attorney Docket No. 0026-0158), entitled “Using Estimated Ad Qualities for Ad Filtering, Ranking and Promotion,” filed on a same date herewith, and incorporated by reference herein in its entirety. 
       Conclusion 
       [0132]    The foregoing description of preferred embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings, or may be acquired from practice of the invention. For example, while series of acts have been described with regard to  FIGS. 5, 14, and 17-19 , the order of the acts may be modified in other implementations consistent with the principles of the invention. Further, non-dependent acts may be performed in parallel. 
         [0133]    In addition to the session features described above, conversion tracking may optionally be used in some implementations to derive a direct calibration between predictive values and user satisfaction. A conversion occurs when a selection of an advertisement leads directly to user behavior (e.g., a user purchase) that the advertiser deems valuable. An advertiser, or a service that hosts the advertisement for the advertiser, may track whether a conversion occurs for each ad selection. For example, if a user selects an advertiser&#39;s ad, and then makes an on-line purchase of a product shown on the ad landing document that is provided to the user in response to selection of the ad, then the advertiser, or service that hosts the ad, may note the conversion for that ad selection. The conversion tracking data may be associated with the identified ad selections. A statistical technique, such as, for example, logistic regression, regression trees, boosted stumps, etc., may be used to derive a direct calibration between predictive values and user happiness as measured by conversion. 
         [0134]    It will be apparent to one of ordinary skill in the art that aspects of the invention, as described above, may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement aspects consistent with the principles of the invention is not limiting of the invention. Thus, the operation and behavior of the aspects have been described without reference to the specific software code, it being understood that one of ordinary skill in the art would be able to design software and control hardware to implement the aspects based on the description herein. 
         [0135]    No element, act, or instruction used in the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.