Providing system configuration information to a search engine

Providing a search engine with system configuration information. The system configuration information pertains to a system having a web server that provides content. For example, the content may be web pages associated with a web site, and the system may include hardware and software used to provide the content of the web site to end users. More particularly, the system can include one or more computer systems, web server software, application server software, and application programs that facilitate providing content. A search engine requests system configuration information from the web server. In response to the request, the web server provides system configuration information to the search engine. The search engine can use the system configuration information to reference portions of the content in an index. The index can be used to respond to a search query that involves content served by the web server.

FIELD OF THE INVENTION

The present invention relates to computer systems, and more particularly to providing a search engine with system configuration information to facilitate indexing content associated with the system.

BACKGROUND

As is well-known, search engines return search results that match a search query submitted by a user. The corpus of documents that search engines search can be extraordinarily large, and in some cases, almost unbounded. For example, some search engines search, or at least attempt to search, the entire “World Wide Web.” To facilitate the search, search engines typically build an index of the content of the corpus. The content may contain web pages, and the search engine may add Uniform Resource Identifiers (URLs) of the web pages to the index. However, due to the nature of the World Wide Web, the search engine must first locate a web page in order to add a reference to the web page to the index.

One common technique to locate web pages is to “crawl the web.” Crawling the web means to start with one web page, and follow “hyperlinks” in that web page to discover other web pages. The starting web page might have been provided to the search engine as a web page of interest. For example, the provider of content associated with the web page might provide the search engine with a URL of the web page such that the search engine will, hopefully, include that web page, and also those that it links to, in the index.

However, relying only on crawling the web to discover web pages has limitations. For example, some web pages might never be discovered. Further, content of web pages is subject to change; therefore, web pages should be re-indexed at some point in time. A possible solution to this problem is for the content provider to provide the search engine with a list of URLs rather than a single URL. The content provider might also provide the search engine with information about a particular web page itself, such as how frequently the content on the web page changes. Such information could benefit the search engine in that it would know how often it should re-examine the content of the web page to update the index.

Another technique that may be used by some search engines is to apply algorithms to discover additional information about the content of the web pages. For example, some web pages might be identified by multiple URLs. It would not benefit a user to provide the user with search results identifying all the URLs, as that would, in effect, be providing the user with redundant web pages. If the search engine can discover, through analysis of the content of two web pages, that they are substantially identical, the search engine can discard one web page. As another example, a web page is often divided into a region with content of interest to a search, such as a news article, and content that may not be of interest, such as an ad banner. The search engine could provide better search results if the search engine ignores the ad banner portion of the web page. Thus, the search engine might algorithmically predict which portions of the web page are of interest to a search.

However, a problem with the search engine applying algorithms to attempt to learn more information about the content of web pages is that the algorithms may fail to correctly characterize the content of the web pages. For example, the web page might be constructed such that it is difficult to determine what portion is not relevant to a search. Moreover, applying algorithms may be limited to the content on the web page itself.

Therefore, limitations exist with respect to how well a search engine can provide search results due to the limited amount and accuracy of information that search engines have with respect to the content in the corpus to be searched.

DETAILED DESCRIPTION

Overview

Techniques are disclosed herein to provide a search engine with system configuration information. The system configuration information pertains to a system having one or more web servers that provide content. For example, the content may be web pages associated with a web site, and the system may include hardware and software used to provide the content of the web site to end users. More particularly, the system can include one or more computer systems, web server software, application server software, and application programs that facilitate providing content. An example of an application program to facilitate providing content is a program that is used to develop “blog” web pages. The system configuration information might pertain to any of the aforementioned system components. The search engine might use the system configuration information to index content, such as web pages, that is provided by the system.

The following steps are performed, in accordance with one embodiment. A search engine requests system configuration information from a web server. In response to the request, the web server provides system configuration information to the search engine. The search engine can use the system configuration information to help determine how to reference portions of the content in an index. The index can be used to respond to a search query that involves content served by the web server.

The following example is provided to illustrate one possible use of system configuration information to index content associated with the system. A particular web site might have different configurations for web pages in different sections of the web site. More particularity, blog web pages might be configured differently than news web pages. The system configuration information may describe the configuration of each type of web page, as well as what section of the web site the web pages may be found. The section of the web site might be identified by the leading portion of the URL. The search engine can make use of this information when indexing the web pages. For example, the search engine might add, to the index, keywords extracted from a first portion of a blog web page and ignore another portion. Thus, when responding to a search query, the web page is only returned if search terms match keywords in the first portion of the blog web page. Many other examples are provided herein of providing system configuration information to a search engine such that the search engine is able to reference, in an index, content associated with the system.

Example Host Machine for Providing System Configuration Information to Search Engine

FIG. 1is an example host machine100for providing system configuration information120, in accordance with an embodiment of the present invention. The host machine100has one or more web servers106executing thereon. The web servers106serve content110to clients. A web server106may include an application server to facilitate accessing the various application programs108. The host machine100is one of possibly many host machines100that provide the infrastructure (e.g., hardware and software) to implement a web site. As used herein, the term “web site” refers to a collection of web pages or other documents. Thus, the content110may comprise one or more web pages. However, the content110is not limited to web pages.

As used herein, the term “web server” refers to a software process, which runs on a host machine100. In one embodiment, the web servers106deliver the content110to clients in accordance with an HTTP protocol. One or more web servers106may run on a single host machine100. An example of a web server106is an Apache Web Server, which is provided by the Apache software foundation. Another example of a web server106is the Sun Java System Web Server, which is commercially available from SUN Microsystems of Santa Clara, Calif.

The term web server106, as used throughout this description, includes “web proxy servers.” A web proxy server is a web server106that provides indirect access to content110. For example, if a web server106is functioning as a web proxy server, it might provide content110to a client by accessing content110from another host machine100. Once accessed, the web server106might cache the content110, wherein the cached content may be provided to satisfy future client requests. An example of a web proxy server is the Java System Web Proxy Server, which is commercially available from Sun Microsystems.

The example host machine100has application programs108that facilitate providing the content110. An example of an application program108is a blog program that allows blog entries to be developed. Another example of an application program108is an e-commerce program that allows an e-commerce portion of a web site to be developed and maintained.

The example host machine100has a content console120for submitting content110to the host machine100. This allows a content publisher to provide the content of web pages, for example. The example host machine100has an administration console120, which may be used for providing configuration information to the host machine100. As an example, a content publisher might provide a template for web pages. Once the host machine100has the content, and optionally a template for the web page, software (e.g., web server106and/or application program108) on the host machine100formats the content110into web pages.

The system configuration information120pertains to any of the hardware or software associated with the host machine100. In this embodiment, the web servers106have configuration logic112that is used to provide web server configuration information120ato a client, such as a search engine. In this embodiment, the application programs108have application configuration information120b, which may be stored in a file. In one embodiment, the configuration logic112in a web server106obtains the application configuration information120band forwards it on to the client. However, an application program108or another component of the host machine100may also have logic that provides system configuration information120to a client.

The example host machine100has a hardware platform102on which various software processes run. Examples of hardware platforms102are a Sun Fire™ server and a Sun Blade™ Modular System, which are commercially available from SUN Microsystems of Santa Clara, Calif. The hardware platform102has an operating system104executing thereon. Examples of operating systems include a Solaris™ Operating System, Linux™ Operating System, and Microsoft™ Operating System. The system configuration information120may pertain to the operating system104and/or the hardware platform102.

Overview of System that Provides Configuration Information and Search Engine

FIG. 2illustrates a system202that provides system configuration information120to a search engine208, in accordance with an embodiment of the present invention. The system configuration information120may include configuration information pertaining to any hardware or software in the system202that is used to facilitate providing content110provided by the system202.

The system202has several host machines100, each of which may be running one or more web servers106. The host machines100may reside in vastly different physical locations. For example, host machines100may be on different continents. The system202has content110associated with it that is provided to clients, such as end user211and search engine208. The end user211may be a personal computer running a web browser. The content110associated with the system202may be web pages, or other documents, that collectively form a web site.

The web server106has an HTTP channel225, which it uses to communicate with end users311and the search engine208. For example, the web server106transfers web pages on the HTTP channel225to clients in accordance with an HTTP protocol. Furthermore, the system configuration information120may be transferred to the search engine208on the HTTP channel225, as will be discussed more fully below. A web server106may also have a control channel215, which it may use to communicate system configuration information120to the search engine208, as discussed more fully below.

The search engine208is communicatively coupled to one or more web servers106via network204. Thus, the search engine208is able to crawl the web site associated with system202to add references to web pages to the index205. The search engine208applies the system configuration information120to reference the web pages in the index205, in one embodiment.

Network204may be the Internet, although this is not required. The search engine208is able to perform a search of the World Wide Web, in one embodiment.

Process Flow for Proving System Configuration Information120to Search Engine

FIG. 3is a flowchart illustrating a process300for requesting and applying system configuration information120, in accordance with an embodiment of the present invention. Process300will be discussed in connection with the example host machine100ofFIG. 1and the example system202ofFIG. 2, although process300is not so limited. In step302, the search engine208sends a request to a web server106for system configuration information120pertaining to a system202that includes the web server106.

In step304, the web server106provides the system configuration information120to the search engine208. Several example protocols are discussed below for how the web server106might provide the system configuration information120to the search engine208. However, process300is not limited to the example protocols. In step306, the search engine208receives and stores the system configuration information120.

In step308, the search engine208applies the system configuration information120to reference portions of the content110in an index. For example, the system configuration information120could be applied when the search engine208crawls the web site associated with the system202to build the index205. As a particular example, web pages from the system202are referenced in the index205.

Example Protocols for Providing System Configuration Information

The system configuration information120can be transferred to the search engine208in a variety of ways. The following example protocols are for illustration and are not exhaustive. Note that some types of system configuration information120may be more suitable to transfer with one protocol than another.

Control Channel

In one embodiment, the web server106and the search engine208establish a control channel215on which the web server106provides the search engine208the system configuration information120. When the search engine208contacts the web server106associated with a particular web site, the search engine208identifies itself to the web server106and indicates that it wishes to obtain system configuration information120. The web server106may inform the search engine208that the web server106has another port on which the search engine208may connect. As an example, the web server106tells the search engine208that it supports the other port by including information in the HTTP headers of the response that is sent on the HTTP channel225. The control channel215allows communication by means other than a web browser, which may speed up the communication process. For example, the search engine208and web server106do not need to communicate with the normal web browser request/response mechanism when using the control channel215.

Web Crawling Robots (Spiders)

The web server106may have a directive file that defines information such as where a spider is permitted to crawl. For example, the directive file tells the spider not to even attempt to crawl a particular directory because that directory is behind a login wall, such that the spider cannot get the content anyway. In one embodiment, the system configuration information120is used to automatically update the directive files.

HTTP Headers

In one embodiment, the web server106adds system configuration information120in HTTP headers. For example, when returning a web page to the search engine208, the web server106adds system configuration information120to the HTTP headers of the HTTP response. The system configuration information120could be relevant to the particular web page sent in the response, a section of the web site of which the web page is a part, or the entire system202.

Structured Document

Rather than returning a normal HTML document, the web server106may return a document in a format that is easier for the search engine208to consume. For example, instead of returning an HTML document for a web page, the web server106returns the same web page in XML format. This technique may be suitable for conveying some system configuration information120, such as the template of a web page, although not necessarily all types of configuration information120is transferred with this technique.

Example System Configuration Information

For the purposes of illustration the following examples of system configuration information120are provided. The examples of system configuration information120are divided into web server configuration information and application program information for purposes of illustration. The system configuration information120can pertain to other than web server and application program.

Web Server Configuration Information

Preferred Canonical URL

One type of system configuration information120is a preferred canonical form of a URL for a particular web page, as stated by the web server106. The particular web page may be accessible by multiple URLs. However, one of the URLs may be preferred for one or more reasons. Without this preferred canonical URL, the search engine208could apply an algorithm to predict a preferred URL, such as predicting that the shortest URL is the preferred URL. However, this prediction may be inaccurate. Therefore, by knowing the preferred URL, as provided by the web server106, the search engine208can leave other URLs out of the index205, or mark those URLs in the index205in a way to indicate that those URLs are not preferred. Thus, the search engine208may be able to return better search results by using the preferred form of the URL in the search results.

Other URL Information

Other URL information such as whether URLs are case sensitive may be included in the system configuration information120. In the event that the case sensitivity of URLs change for a particular web site, or portion thereof, the time at which the change in this configuration occurred can be provided to the search engine.

Directory or Section-level Information

Often a web site is divided into different sections, such as a blog section, an e-commerce section, etc. The different sections are sometimes demarcated by a directory structure. For example, a blog area might have a URL that starts “acme.com/blog,” whereas a sales section might have a URL that starts “acme.com/sales.” The search engine208may benefit from having system configuration information120that is specific to a particular section. Therefore, such directory or section level information may be included in the system configuration information120.

Session Information

The system configuration information120may include information related to session tracking. By providing the search engine208with session tracking information used by the system202, the search engine208is better able to crawl the web site when building the index205. Examples of session tracking information include, but are not limited to, URL parameters and session cookies.

URL parameters are information added to a URL to help track sessions. For example, a system202managing an e-commerce web site might add a URL parameter to the URL with the parameter staying with the URL as the user migrates from one web page to the next to track the user's session. Note that if a first user were to obtain a URL with a URL parameter identifying the session for another user, then the session information is not valid for the first user.

It is possible that a search engine208might obtain a URL that has a URL parameter. For example, if a user embedded a URL that has a session tracking URL parameter into a particular web page, then the search engine208, when crawling that particular web page, should not attempt to use the URL with the session tracking URL parameter. If the web server106informs the search engine208what the URL parameter looks like, the search engine208can avoid using that URL, or at least re-format the URL by removing the session tracking URL parameter.

Cookies may also be used for session tracking. For example, a web server106may send a session cookie to a client web browser to track a session. The session cookie has a name and a session identifier. The web server106provides the search engine208with the name of the cookie, such that the search engine208can easily identify the cookie in the future. Once identified, the cookie may simply be ignored by the search engine208.

Application Identifier

The system configuration information120may describe what application program108is used on one or more sections of the web site. For example, if a web/application server106is running multiple application programs108on the same host machine100, the web/application server106could return an Application Identifier in use for a given path in the directory, in order to help the search engine208differentiate between the different types of content110being served. For example, the application identifier can help the search engine208distinguishing an e-commerce section of the web site from a blog section of the web site. Once the search engine208knows the application program, the search engine208may be able to deduce that different web pages in the section of the web site corresponding to a particular application program have similar properties, such as web page templates.

Hosts Served

The system configuration information120may describe what hosts are served by the web server106. For example, if the same web server106might host both a “www” and “non-www” version of a web site, the web server106could report this in the system configuration information120. Thus, the search engine208does not have to resort to heuristics or webmaster input to learn this information. Note that the different versions of the web site could have URLs that are very different from each other.

Contact Information

The system configuration information120may include contact information for reporting problems with the web server106. For example, if the search engine208is not able to crawl the web site, an administrator of the web site can be contacted.

Error Page Information

The system configuration information120may describe how the web server106responds to a request when a web page is not found or some other error occurs. For example, if the web server106is configured to return an actual web page even when a non-existent web page is requested, the web server106could inform the search engine208of this fact. Such information could be used by the search engine208to improve “soft-404detection” algorithms used by the search engine208. A soft-404error refers to the web server106returning a web page even thought the requested web page does not exist.

Duplicate Content Guidance

The system configuration information120may provide guidance on duplicate content110on the web site. For example, the web server106could inform the search engine208that the paths “/blog” and “/journal” correspond to the same content110, and should be treated as duplicate branches.

Spider Information

The system configuration information120may include rules that apply to spiders. As previously discussed, a file can tell a spider not to even attempt to crawl a particular directory because that directory is behind a login wall, such that the spider cannot get the content anyway.

Application Program Configuration Information

Templates

The system configuration information120may include templates that are applicable to a particular level of a directory. For example, web pages in a blog section of the web site can have a different format from web pages in a shopping section. A web server106provides either the templates or the location of the templates to the search engine208, in one embodiment. The web server106obtains the templates from an application program108, in one embodiment.

In one embodiment, the web server106provides this template information by providing a location of a template. For example, the template location is specified in the HTTP headers of an HTTP response to a request for the web page having that template. The following are two examples of how to describe the template location, although there are many other ways of describing a location.

Alternatively, the template could be transferred directly to the search engine208in response to a request for the web page. For example, the template could be transferred in the control channel215, and the web page might be transferred in the HTTP channel225.

Language

The system configuration information120may describe the language used on one or more sections of the web site. Some system202manage a web site in such a way as to divide the content110into a section presented substantially in a first language, another section in a second language, etc. The search engine208may be able to discover the language being used on its own; however, being provided the language of each section can help the search engine208when processing web pages to add them to the index205. The web server106provides this information by identifying the language used for a given sub-path of the directory, in one embodiment. The web server106obtains the language information from an application program108, in one embodiment.

Bulk Change Information

The system configuration information120may describe bulk changes made to on one or more sections of the web site. For example, an application program108can tell a web-crawling robot that formally was at “acme.com/x” is not at “acme.com/y.” Other examples of bulk change information are a deletion or addition of an entire section of a web site.

The system configuration information120may describe what microformats are used in a particular web page. Microformats are ways of annotating a web page to indicate a semantic meaning associated with content110on the web page. For example, a microformat can be used to indicate that the term “San Jose” is a city. A microformat could be an elemental microformat, which may be as simple as an attribute attached to an existing attribute. A microformat can also be a compound microformat, which can be an organized collection of data. Examples of compound microformats include hCalendar, hCard, and hReview.

If the search engine208, or more particularly, a web crawler is aware of the particular microformats being used in a web page, then the web crawler can perform element extraction from the web page. However, there are many possible different microformats, and if the search engine208has to consider all of the possible microformats that might be used on a web page the process of analyzing the web page could be inefficient.

By the web server106informing the search engine208what microformats are in use in a particular web page or collection of web pages, the search engine208can efficiently look for content110that is described by those microformats. For example, the search engine208is aware that the web page uses an hCalender microformat somewhere on the web page, then the search engine208can scan the web page for content110that uses the hCalender microformat and process that portion of the web page accordingly. Thus, the web server106could inform the search engine208that web pages in a certain section of the web site support the hCalendar, hCard, or hReview formats. The web server106obtains the microformat information from an application program108, in one embodiment.

Implicit System Configuration Information

System configuration information120can be implicit, based on information provided by the web server106. For example, in one embodiment, the web server106returns only content110from a specific portion of a web page. The web server106might strip off template headers and footers prior to sending the web page. By knowing the content110of the specific portion, the search engine208is able to deduce the configuration of the web page by comparing the returned content110with a whole version of the web page. Thus, the information provided by the web server106is not explicitly system configuration information120; however, such information may be deduced by the search engine208from the content110.

Hardware Overview