Using resource load times in ranking search results

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for using resource load times in ranking search results. In one aspect, a method includes receiving a search query from a particular user device; receiving, for each of a plurality of resources responsive to the search query, a respective first score; accessing load time data that specifies, for each of the plurality of resources, a load time measure for the resource; and adjusting the first score for each of the plurality of resources based on the load time measure for the resource to generate a second score for each of the plurality of resources.

BACKGROUND

This specification relates to search systems.

The Internet provides access to a wide variety of resources, examples of which include video or audio files, web pages for particular subjects, book articles, and news articles. A search engine can identify resources in response to a user query that includes one or more search terms or phrases. The search engine ranks the resources based on their relevance to the query and importance and provides search results that link to the identified resources.

The resources referred to by the search results may take different amounts of time to load in users' web browsers. For example, for any particular resource, the size of the resource, the number of images the resource includes or references, the web server that serves the resource, and the particular network connection can impact the amount of time the resource takes to load in a user's web browser. Given two resources that are of similar relevance to a search query, a typical user may prefer to visit the resource having the shorter load time.

SUMMARY

This specification describes technologies relating to using resource load times in ranking search results.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving a search query from a particular user device; receiving, for each of a plurality of resources responsive to the search query, a respective first score; accessing load time data that specifies, for each of the plurality of resources, a load time measure for the resource; and adjusting the first score for each of the plurality of resources based on the load time measure for the resource to generate a second score for each of the plurality of resources. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more of the following features. Providing search results to the particular user device, each search result corresponding to one of the resources, for presentation in an order according to the second scores of the resources corresponding to the search results. Making further adjustments to the second scores before providing the search results to the particular user device. For each resource, the load time measure for the resource comprises a statistical measure of a sample of load times for the resource obtained from a plurality of user devices. The statistical measure is a central tendency of the sample.

Each of the plurality of user devices shares at least one attribute with the particular user device and wherein the at least one shared attribute comprises a location of the particular user device. The location of the particular user device is a country from which the search query originates. Each of the plurality of user devices shares at least one attribute with the particular user device and wherein the at least one shared attribute comprises an agent type of the particular user device.

Identifying one or more of the plurality of user devices that share at least one attribute with the particular user device, where the at least one shared attribute comprises an agent type of the particular user device or a location of the particular user device. Computing a new load time measure using only load times from the identified one or more user devices. Adjusting the first score for each of the plurality of resources comprises for each of the resources, computing a multiplier factor for the resource based on the load time measure for the resource; and, for each of the resources, modifying the first score of the resource by the multiplier factor to generate the second score.

Computing the multiplier factor for a particular resource comprises determining whether the load time measure for a particular resource exceeds a first threshold value; and if the load time measure for the particular resource exceeds the first threshold value, setting the multiplier factor for the particular resource to a first value. Computing the multiplier factor for a particular resource further comprises if the load time measure for the particular resource does not exceed the first threshold value, determining whether the load time measure for the particular resource exceeds a second threshold value; and if the load time measure for the particular resource exceeds the second threshold value, setting the multiplier factor for the particular resource to a second value.

In general, another aspect of the subject matter described in this specification can be embodied in methods that include the actions of collecting load time data from each of a plurality of user devices and each of a plurality of resources, the load time data specifying, for each of the user devices and for each of the plurality of resources, a time the resource took to load on the user device, and, for each resource, aggregating the load time data for the resource to generate a load time measure for the resource, the load time measure being a statistical measure of the time the resource took to load on the plurality of user devices. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more of the following features. The load time data for the user devices is collected from one or more of a web browser, a web browser add-on, or monitoring software associated with the particular user device. The load time measure is a central tendency of load times from the plurality of user devices. The operations further comprising identifying one or more of the plurality of user devices having a specific attribute, the one or more of the plurality of user devices being less than all of the plurality of user devices; and generating a new load time measure for each of the resources, the load time measure being generated using only load time data from the one or more identified user devices. The specific attribute is an agent type. The specific attribute is a location of the user device. The operations further comprising providing the load time measure for each resource to a search engine.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. A search result for a resource having a short load time relative to resources having longer load times can be promoted in a presentation order, and search results for the resources having longer load times can be demoted. The demotion can, in some situations, result in a search result for a resource having a short load time being presented earlier in the order than a search result for a similarly-relevant resource having a longer load time. Thus, for two resources that, apart from load times, appear to equally satisfy a user's informational needs, the user will often select the resource that will likely load the most quickly of the two, resulting in a better user experience.

DETAILED DESCRIPTION

FIG. 1is a block diagram of an example environment100in which a load time data system120can be used. A computer network102, e.g., a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof, connects web sites104, user devices106, and a search engine110. The environment100may include many thousands web sites104and user devices106.

A user device106is an electronic device that is under control of a user and is capable of requesting and receiving resources over the network102. Example user devices106include personal computers, mobile communication devices, and other devices that can send and receive data over the network102. A user device106typically includes a user application, e.g., a web agent (browser) or other communication software, to facilitate the sending and receiving of data over the network102.

A web site104is one or more resources105associated with a domain name, and one or more servers host each web site. Respective publishers and/or advertisers, i.e., entities that manage and/or own the web sites104, maintain the web sites104.

A resource105is any data that can be provided by the web site104over the network102and that is associated with a resource address. Resources include HTML pages, word processing documents, portable document format (PDF) documents, images, video, and feed sources, to name just a few. The resources can include content, such as words, phrases, pictures, and so on, and may include embedded information (such as meta information and hyperlinks) and/or embedded instructions (such as JavaScript scripts).

To facilitate searching of these resources, the search engine110crawls the web sites104and indexes the resources105that the web sites104host. The indexed and, optionally, cached copies of the resources are stored in a resource index112.

The user devices106submit search queries109(Q1, Q2 . . . Qn) to the search engine110. In response, the search engine110uses the resource index112to identify resources that are relevant to the queries. The search engine110identifies the resources in the form of search results111and returns the search results111to the user devices106in search results pages ({S1}, {S2} . . . {Sn}).

A search result111is data generated by the search engine110that identifies a resource that satisfies a particular search query, and includes a resource locator for the resource. An example search result111can include a web page title, a snippet of text extracted from the web page, and the URL of the web page. The search results are ranked (i.e., placed in an order) according to respective scores determined for each of the resources identified by the search results. The scores generally relate to both the quality and the relevance of the corresponding resource. The search results109are provided to the user device according to the ranking.

The user devices106receive the search results pages and render the pages for presentation to users. In response to the user selecting a search result at a user device106, the user device106requests the resource identified by the resource locator included in the selected search result. The web site104hosting the resource receives the request for the resource from the user device106and provides the resource to the requesting user device106.

In some implementations, the load time of each particular resource is an additional signal that the search engine110uses when evaluating a resource for a search query. As used herein, the term “load time” refers to the time that elapses between an initial request for a resource and at least the time that data to completely render the resource is provided to a device that requested the resource. The load time can vary, depending on the particular metrics used. For example, the time that elapses between the time that a user device106requests a resource and the time that the resource is presented in a web browser on the user device can be defined as the load time. Alternatively, the time that elapses between the time that a web server104hosting a resource receives a request for the resource and the time that that the resource is presented in a web browser on the user device can be defined as the load time.

The load time can vary depending on one or more of the user device106, the resource105that is being requested, the web server104that is serving the resource, and the network connections. In particular, the load time for a resource may depend on the amount of content included in the resource (e.g., the number of images or embedded instructions included in the resource), the web server(s) hosting the resource, the current network connectivity of the user device requesting the resource, etc. For example, a resource that includes multiple embedded videos may have a longer load time than a resource that does not include any images or embedded videos. Furthermore, a resource hosted by a web server in France may load faster on a user device in France than on a user device in the United States.

The search engine110can use data that specifies the load times for resources in various search-related operations. One example operation is reordering search results from a first order to a second order. For example, if two search results are adjacent to each other in the first order (e.g., their respective underlying resources are initially scored such that the search results are adjacent in an order), and the resource referenced by the second search result has a significantly shorter load time relative to the resource referenced by the first search result, the search engine110will adjust the search results to a second order so that the second search result is before the first search result in the order. This can improve the user experience, as it is likely that a user would prefer to access the resource that loads with the least amount of delay.

The search engine110includes, or is in data communication with, a load time data system120that collects and aggregates data about the load times of resources105on user devices106over the network102. In some implementations, in order to protect user privacy, the load time data is anonymized to remove any user-identifying information so that the resources provided to user devices and their load times on the user devices cannot be associated with a particular user or user identifier. In some implementations, the load time data system120includes a load time data collector122and a load time data aggregator124. The partitioning of the functionality of load time data system120between the load time data collector122and the load time data aggregator124is illustrative only. Additional partitioning of the functions described below can be implemented, or, alternatively, the load time data system120can be a single software engine that performs all the functions described below.

In some implementations, the load time data collector122receives load time data from user devices106over the network102. For example, for each resource accessed by user device106using a web browser, the web browser can determine the time the resource took to load on the user device. The web browser can then send the load time data to load time data system120, where it is collected by the load time data collector122. Alternatively, a browser add-on can observe the time a resource takes to load in the web browser and send it to load time data system120. This browser add-on can be a browser toolbar that adds additional functionality to the browser interface. In some implementations, the browser or add-on will monitor load times and send them to the load time data system120only if the user opts-in (i.e, agrees to report their load times). In some implementations, network monitoring software associated with a user device106monitors communications between the user device106and the network102. The network monitoring software may be installed on user device106or on another device in the network102. As part of monitoring network traffic, the network monitoring software can record load times for resources on the user device106and send the recorded load times to the load time data collector122. In some implementations, the user devices106store the load time data in a local store and send the stored data to the load time data system120at pre-determined intervals. Alternatively, the user devices106may send the load time data to the load time data system120immediately after a resource has finished loading on the user device. The collector122receives load time data from user devices106over the network102.

For each resource105, the load time data collector122receives load time data for the resource from multiple user devices106. In some implementations, because the load time is dependent on user device-specific factors like the agent used to request and display the resource and the network connectivity of the user device, the load time data collector122also derives or receives data identifying attributes of each user device providing load time data. The identified attributes can include the agent type (e.g., web browser version) and the location (e.g., the country) of the user device. In some implementations, the load time data collector122only collects load times from certain types of devices. For example, load times may not be collected from mobile devices because of the high latency of all requests for resources on such devices. In some implementations, the load time data collector122anonymizes the identified attributes of the user devices for privacy protection. This ensures that attributes of a particular user device are not associated with or able to identify a particular user or their navigational history.

In other implementations in which the load time is measure relative to a web server only, an agent (or other software) can be installed at each publisher104, and can monitor the time required to serve a requested resource in response to a request. The time can be included as a resource tag or query parameter, and can be reported by the requesting user device to the load time data collector122. Alternatively, virtual machines implemented as part of the load time data system can be used to request the resources.

Load time data aggregator124aggregates the load time data collected by load time collector122. Load time data aggregator124can generate a load time measure for a resource105that specifies the time the resource takes to load on user devices106. In some implementations, the load time measure is a statistical measure (e.g., a central tendency) of the load times of the resource collected by the load time data collector122. Depending on the implementation, the central tendency may be a mean, trimmed mean, Winsorized mean, median, median of sample medians, etc.

In some implementations, the load time data aggregator124may generate multiple load time measures for a resource105based on attributes of the user devices providing the load time data. In addition to, or in place of, the load time measure that is generated using all of the collected page load time data for a resource, the load time data aggregator124can generate one or more load time measures using only data from user devices that share at least one attribute. In one example, the load time data aggregator124can generate a first load time measure for a particular resource using only load times from user devices in the United States, a second load time measure using load times from user devices in France, a third load time measure using load times from user devices in Germany, etc. Alternatively, the load time data aggregator124can generate separate load time measures for each identified agent type; each identified network connection, and so on. The identified network connection can be a type of network connection used by a user device to access the resource. In some implementations, generating load time measures based on multiple attributes (e.g., a load time measure using only load time data from devices that use a first agent type and are in the United States) is also done.

In some implementations, the load time data aggregator124determines whether sufficient load time data has been collected for a load time measure to be statistically significant before generating the load time measure. For example, load time data aggregator124can determine whether the number of user devices reporting load time data for the resource exceeds a specified threshold value before generating the load time measure. Some resources105may not have enough traffic from particular locations or types of devices for load time measures derived from devices sharing a particular attribute to be meaningful. For example, resources in Chinese may not have enough visits from user devices located in France to generate a meaningful load time measure using solely devices from France, or a newly launched website may not have sufficient load time data associated with its resources.

Once the load time measures for a resource are generated, data including the load time measures is associated with the resource locator for the resource in the resource index112as load time data114. Accordingly, by accessing the resource index112, the search engine110can access the load time measures for the resource referenced by a particular resource locator.

In some implementations, load times for resources associated with the same domain name (e.g., www.example.com/resource1.html, www.example.com/resource2.html, . . . www.example.com/resourceN.html) can all be assigned the same load time measure based on the load time statistics of the resources, and the load times for the resources are thus resolved to a domain level. Other ways of attributing load times to resources can also be used.

FIG. 2is a block diagram illustrating a process flow of a search results adjusting engine128. The load time measures of resources referenced in search results responsive to a query are used to compute multiplier factors for the search results. The search results adjusting engine128adjusts the order of the search results based on these multiplier factors, as described below.

At query time, the search results adjusting engine128receives search results responsive to the query. Each search result includes a resource locator referencing a resource and is associated with an initial score for the result. For each search result including a resource locator associated with load time data in the resource index112, the search results adjusting engine128computes a multiplier factor for the search result. Each multiplier factor is a measure of the effect the load time of the resource will have on the initial score of the search result and is dependent on a load time measure for the resource referred to by the search result. The search results adjusting engine128applies the multiplier factor for each search result to the initial score for the search result (i.e., multiplies the initial score for the search result by the multiplier factor for the search result) to generate a load-adjusted score. The search results can then be ranked according to their load-adjusted scores. Alternatively, an additive factor for each result can be determined based on the load time measures. The additive factors can then be added to each initial score to generate the load-adjusted scores.

As described above, attributes of the user device submitting the query can affect the load times of resources on the user device. Thus, in some implementations, the multiplier factor is based in part on device attributes of the user device106submitting the query (i.e., the current user device). The device attributes can include an agent type of the agent used to submit the query and the country of the current user device. The device attributes can also include the type of network connection currently being used by the user device. These device attributes are derived from or provided by the current user device. Alternatively, the device attributes can be obtained from account data associated with the user of the current user device. The device attributes of the current device can also be anonymized so that they cannot identify or be associated with a user of the current user device. In some implementations, once the results adjusting engine128obtains the desired device attributes of the current user device, it can access a load time measure compiled using only load times from user devices sharing at least one of the desired attributes with the user device submitting the query. Alternatively, the search results adjusting engine128can access general load time data for a resource and determine which of the user devices used to compile the load time data share at least one desired attribute with the current user device. The search results adjusting engine128can then generate a new load time measure using only load times from the identified user devices.

In some implementations, the search results adjusting engine128compares the load time measures for search results to threshold values to generate the multiplier factor. Specifically, the search results adjusting engine128can compare the load time measure to a first threshold value. If the load time measure exceeds the first threshold value, the search results adjusting engine128sets the multiplier factor to a first demotion value. In one implementation, if the load time measure does not exceed the first threshold value, the search results adjusting engine128compares the load time measure to a second threshold value. If the load time measure exceeds the second value, the multiplier factor is set to a second demotion value. If the load time measure does not exceed either threshold value, the multiplier factor is set to equal 1. More thresholds and corresponding demotion values can be used for greater granularity. Alternatively, continuous functions that asymptotically approach a maximum or minimum demotion value can be used.

The threshold values can be determined using a sample of all collected page load times. For example, the first threshold value can be any value in a range of values corresponding to the 96thpercentile of all collected page load times to the 99thpercentile of all collected page load times. The second threshold value is lower than the first threshold value and can be any value in a range of values corresponding to the 85thpercentile of all collected page load times to the 95thpercentile of all collected page load times. Alternatively, other ranges can be used. The demotion values can be adjusted in accordance with the desired effect on search result scores, with the first demotion value being lower than the second demotion value to further demote resources with extremely low load times.

Alternatively, the search results adjusting engine128can promote search results with shorter load times by comparing the load time measures for resources to a different threshold value (e.g., any value in the range of the 5thpercentile of all collected page load time values to the 15thpercentile of all collected page load time values), and setting the multiplier factor equal to a promotion value if the load time measure is below the threshold value.

In some implementations, particular search results are not subject to being promoted or demoted (i.e., their multiplier factor is set to 1) no matter their load time measure. For instance, search results whose load time measure is based on too few data points (e.g., the load time measure is based on less than 1000 reported load times), search results that have relevancy scores exceeding a certain threshold value, search results corresponding to a resource that is a home page of a web site, and search results for navigational resources can all have their multiplier factor set to 1 regardless of their load time measure. Similarly, load times can be ignored for certain queries, such as queries that are determined to be directed to obtaining particular information (e.g., navigational queries, news queries, weather queries, etc.).

Additionally, the search results adjusting engine128can apply the same multiplier factor to groups of related resources. For example, all resources from the same host may be demoted or promoted by the same factor in order to preserve the relative order of results from the host. Furthermore, all resources included in the same web site, having the same domain, or sharing an IP address can be assigned the same multiplier value. Various techniques may be used to determine which multiplier value to apply, including demoting or promoting each group of related resources based on the smallest or largest load time measure of any resource in the group. In some implementations, an average multiplier value for the group of related resources can be computed based on the individual multiplier values for each related resource.

Other implementations of the systems and techniques for using resource load times in ranking search results are possible. In some implementations, a load time for a particular resource on a particular user device can be predicted using a model that has as its input information about the resource, e.g., size of the resource, web server hosting the resource, etc., and, optionally, attributes of the user device. The model can be developed using conventional machine learning techniques, e.g., using a data set where the output and input are known to train the model. The predicted load time can then be used in place of the load time measure to generate the multiplier factor. Alternatively, the model can use the load time measure for the resource as an input to generate the predicted load time.

Once the load-adjusted scores for each of the search results have been generated by search results adjusting engine128, the search engine110reorders the search results according to the load-adjusted scores of the resources corresponding to the search results. The search engine110can provide the search results to a user device106. Alternatively, further adjustments based on other factors can be made to the load-adjusted scores before the load-adjusted scores are used to order the search results for presentation on the user device.

An example adjustment is illustrated inFIG. 2. As shown inFIG. 2, a query provided from a user device in the United States (us) and submitted using an agent having an agent ID of agent1, is received by the search engine110and the search engine110provides search results202ranked according to a first order. Some of the resources referenced by the search results202are associated with a load time measure (www.example 1.com has a load time measure of 15 seconds, www.example2.com has a load time measure of 5 seconds, and www.example3.com has a load time measure of 10 seconds). In some implementations, the load time measure is generated using only load times from user devices in the United States and/or using the agent having the agent type agent1. The search results adjusting engine uses the load time measures to generate a multiplier factor for each search result. The multiplier factor is applied to an initial score used to generate the first ranking to obtain a load-adjusted score for each search result. The load-adjusted scores are then ranked according to a second order204based on the second scores.

In the example adjustment illustrated inFIG. 2, a first threshold value is set at 12 seconds and a second threshold value is set at 9 seconds. Because the load time measure for www.example1.com (15 s) exceeds the first threshold, the initial score for the search result referencing www.example1.com is adjusted by a first demotion value. The load time measure for www.example2.com (5 s) does not exceed either threshold value, and thus the initial score for that search result remains unchanged. The load time measure for www.example3.com (10 s) exceeds only the second threshold value and the initial score for that search result is adjusted by a second demotion value. In the ranking based on the load-adjusted scores, the search result www.example2.com is promoted to the position previously occupied by the search result www.example1.com (e.g., position n) and the search results www.example3.com and www.example1.com are demoted to positions n+6 and n+20, respectively, because of the applied demotion values.

FIG. 3is a flow diagram of an example process300for generating a load time measure for a particular resource. The load time data system120ofFIG. 1can be used to perform the process300.

The load time data system120collects load time data from user devices, the load time data specifying, for each of the user devices, the time a particular resource took to load on the user device (302). For example, the load time data system120can collect the load time data system120over a network102from web browsers, web browser add-ons or network monitoring software associated with the user devices106. The load time data system120can receive user device attributes along with the load time data.

The load time data system120aggregates the received load time data to generate a load time measure for the particular resource (304), the load time measure being a statistical measure of the time the particular resource took to load on the user device. For example, the load time measure can be a central tendency, e.g., a mean, of the collected load times for the particular resource. In some implementations, the load time data system120identifies attributes of the user devices providing the load time data and generates multiple load time measures for the resource, each load time measure using only load times from user devices sharing at least one attribute. In some implementations, the load time data system120determines whether to compute a load time measure for the particular resource based on whether sufficient load time data exists for the resource (e.g., whether the number of user devices providing load times for the particular resource exceeds a threshold value).

FIG. 4is a flow diagram of an example process400for adjusting an order of search results based on resource load times. The search results adjusting engine128ofFIG. 1can be used to perform the process400.

The search results adjusting engine128receives a search query from a particular user device (402). In some implementations, the search results adjusting engine128also receives or obtains attributes of the particular user device.

The search results adjusting engine128also receives initial scores for resources, each initial score corresponding to a corresponding resource and being a measure of relevance of the corresponding resource to the search query (404). For example, the search results adjusting engine128can receive the initial scores from the search engine110.

The search results adjusting engine128accesses load time data that specifies, for each of the resources, a load time measure for the resource that measures the time the resource takes to load on a user device (406). For example, the load time measure can be a central tendency of load times of the resource from a sample of user devices. In some implementations, the load time measure is compiled using only load time data from user devices that share at least one attribute (e.g., location or agent type) with the particular user device. Alternatively, the search results adjusting engine128can generate a new load time measure that uses only data from user devices sharing at least one attribute with the particular user device as described above. The load times can be on a per-resource basis, or on a per-domain basis.

The search results adjusting engine128adjusts the initial score for each resource based on the load time measure for the resource to generate a load-adjusted score for each resource (408). For example, the search results adjusting engine128can compute a multiplier factor for each resource. The multiplier factor can be computed based on a comparison of the load time measure for the resource to one or more pre-defined threshold values. The multiplier factor is then applied to the initial score for the resource to generate the load-adjusted score.

FIG. 5is a flow diagram of an example process500for adjusting an initial score of a resource based on the load time measure for the resource. The search results adjusting engine128ofFIG. 1can be used to perform the process500.

The search results adjusting engine128receives a first score for the resource, the initial score being a measure of relevance of the resource to a search query (502). The search results adjusting engine128accesses load time data that specifies a load time measure for the resource that measures the time the resource takes to load on user devices (504).

The search results adjusting engine128determines whether the load time measure exceeds a first threshold value (506), and if it does, a first demotion value is applied to the initial score for the resource (508). If the load time measure does not exceed the first threshold value, the results adjusting engine128determines whether the load time measure exceeds a second threshold value (510). If the load time measure exceeds the second threshold value, a second demotion value is applied to the initial score (512). Otherwise, no change is made to the initial score for the resource (514).