Dynamically identifying and removing potential stopwords from a local search query

Dynamic identification and removal of potential stopwords from a search query is provided. A search query comprising plural terms is received. A possible stopword among the terms that appears in a predefined list of possible stopwords is identified. Query splits are determined based on one or more query patterns, each query split comprising a location portion using a first subset of the plural terms and a subject portion using a second subset. Either the location portion or the subject portion of the query splits is modified, by removing the possible stopword from that portion. The search engine is queried with the location portion and the subject portion for each of the query splits, and scored search results are obtained for each of the query splits. The search results returned for at least one of the query splits is selected to return as a response to the search query.

BACKGROUND

The present disclosure generally relates to search systems, and, in particular, to dynamically identifying and removing potential stopwords from a local search query.

Internet search engines aim to identify resources (e.g., web pages, images, text documents, multimedia context) that are relevant to a user's needs, and to present information about the resources in a manner that is most useful to the user. Internet search engines return a set of search results in response to a user-submitted query.

For mapping search systems, the query must typically be interpreted with some geographic context. In particular, a query can be interpreted based on a search subject (e.g., a {what}) and a location (e.g., a {where}). To improve the interpretation, stopwords (i.e., words that are not particularly helpful in identifying a query subject or location) can be removed from the query.

SUMMARY

The disclosed subject matter relates to a machine-implemented method for dynamically identifying and removing potential stopwords from a search query. The method comprises receiving a search query comprising plural terms, identifying a possible stopword among the plural terms that appears in a predefined list of possible stopwords, and determining one or more query splits by matching the search query to one or more query patterns, each query split comprising at least one of a location portion that uses one or more of the plural terms or a subject portion that uses one or more of the plural terms. The method further comprises modifying at least one of the location portion or the subject portion of each of the one or more query splits by removing the possible stopword from that portion, querying the search engine with the location portion and the subject portion for each of the one or more query splits, and respectively obtaining scored search results for each of the one or more query splits, and selecting the search results returned for at least one of the one or more query splits to return as a response to the search query.

The disclosed subject matter further relates to a system for dynamically identifying and removing potential stopwords from a search query. The system comprises one or more processors, and a machine-readable medium comprising instructions stored therein, which when executed by the processors, cause the processors to perform operations comprising receiving a search query comprising plural terms, and identifying a possible stopword among the plural terms that appears in a predefined list of possible stopwords. The operations further comprise determining one or more query splits by matching the search query to one or more query patterns, each query split comprising at least one of a location portion that uses one or more of the plural terms or a subject portion that uses one or more of the plural terms, determining whether the possible stopword is more closely associated with the location portion or the subject portion of the plurality of query splits, and removing the possible stopword from either the location portion or the subject portion of the plurality of query splits, based on the determination of whether the possible stopword is more closely associated with the location portion or the subject portion. In addition, the operations comprise querying the search engine with the location portion and the subject portion for each of the one or more query splits, and respectively obtaining scored search results for each of the one or more query splits, and selecting the search results returned for at least one of the one or more query splits to return as a response to the search query.

The disclosed subject matter also relates to a machine-readable medium comprising instructions stored therein, which when executed by a system, cause the system to perform operations comprising receiving a search query comprising plural terms, identifying a possible stopword among the plural terms that appears in a predefined list of possible stopwords, and determining one or more query splits by matching the search query to one or more query patterns, each query split comprising at least one of a location portion that uses one or more of the plural terms or a subject portion that uses one or more of the plural terms. The operations further comprise determining whether the possible stopword is more closely associated with the location portion or the subject portion of the plurality of query splits, and removing the possible stopword from either the location portion or the subject portion of the plurality of query splits, based on the determination of whether the possible stopword is more closely associated with the location portion or the subject portion. In addition, the operations comprise querying the search engine with the location portion and the subject portion for each of the one or more query splits, and respectively obtaining scored search results for each of the one or more query splits, and selecting the search results returned for at least one of the one or more query splits to return as a response to the search query, wherein the selected search results are the search results with the highest score.

DETAILED DESCRIPTION

The disclosed subject matter dynamically identifies and removes possible stopwords from a search query by rewriting the query based on known query patterns, removing possible stop words from some of the rewritten query patterns, and issuing multiple queries against the backend search server based on the rewritten query patterns. Per the disclosed subject matter, a search query comprising plural terms is received. A term of the plural terms appearing within a predefined list of stopwords is identified as a possible stopword. The plural terms are split into multiple query splits according to one or more query patterns. Each query split uses some of the plural terms in the search query as a location value (e.g., {where}) and some of the plural terms in the search query as a subject value (e.g., {what}). Either the location value or the subject value for one or more of the query splits is modified by removing the term identified as a possible stop word from the value. The search engine is queried with the location value and the subject value for each query split, and scored search results are obtained for each split. A query split is selected based on the returned scores. In some cases, the selected query split may not include the possible stop word, indicating dynamic identification and removal of that possible stop word. In some cases, the selected query split may include the possible stopword, indicating dynamic rejection of the possible stop word as a stopword.

FIG. 1illustrates an example search system. More particularly, system100can provide search results relevant to submitted queries (e.g., queries submitted over an internet, an intranet, or another client and server environment). The search system100is an example of an information retrieval system in which the systems, components, and techniques described herein can be implemented.

A user102can interact with the search system100through a client device104. For example, the client104can be a computer coupled to the search system100through a local area network (LAN) or wide area network (WAN), e.g., the Internet. In some implementations, the search system100and the client device104can be one machine. For example, a user can install a desktop search application on the client device104. The client device104will generally include a random access memory (RAM)106and a processor108.

A user102can submit a query110to a search engine130within the search system100. When the user102submits the query110, the query110is transmitted to the search system100(e.g., through a network112). The search system100can be implemented, for example, as one or more computer programs running on one or more computers in one or more locations that are coupled to each other through a network. The search system100includes one or more index databases122and a search engine130. The search system100responds to the query110by generating search results128, which are transmitted to the client device104in a form that can be presented to the user102(e.g., as a search results web page to be displayed in a web browser running on the client device104).

When the query110is received by the search engine130, the search engine130identifies resources that match the query110. The search engine130will generally include an indexing engine120and a ranking engine152. The indexing engine120indexes resources (e.g., web pages, images, or news articles on the Internet) found in one or more corpora (e.g., collections or repositories of content) in the one or more index databases122that store the index information. The ranking engine152(or other software) ranks the resources that are found to match the query110. The indexing and ranking of the resources can be performed using conventional techniques.

In some implementations, the one or more index databases122store different types of resources that can be searched separately using part or all of the query110. For example, one or more of the index databases122can store index information associated with geographic resources, local resources, or other types of specifically indexed resources. The search engine130can submit the query or a query component to particular indices depending the interpretation of the query, e.g., using a query pattern as described below. The search engine130can transmit the search results128through the network to the client device104for presentation to the user102.

FIG. 2illustrates an example process by which a search is performed based on a search query. For convenience, and by way of non-limiting example, the search process200will be described with respect to a system (e.g., a system including one or more computing devices) that performs the search process200.

At202, the system receives an input query. For example, the query can be received from a user through a user interface associated with a search system (e.g., the search system100). In some implementations, the query is received in a user interface associated with a particular set of resources. In particular, the query can be received in a user interface of a mapping application that identifies geographic locations and other information associated with geographic locations based on user input.

For example, an input query of “san francisco” can result in a map of San Francisco being displayed. Similarly, a query of “city hall san francisco” can result in information associated with San Francisco's city hall being displayed relative to a map of San Francisco or to a relevant portion of San Francisco. The information can include, for example, the address of city hall, a phone number for city hall, a position locator of city hall on the displayed map, images, reviews, links to web pages, and other responsive information. In another example, multiple results can be displayed relative to a displayed map. For example, the query “restaurants near opera san francisco” can provide multiple mapped results for particular restaurants within a specified proximity of the opera house in San Francisco.

At204, the system matches the received query to a query pattern. A query pattern provides one or more predefined rules for interpreting a received query. In some implementations, a number of query patterns are defined manually. In some implementations, query patterns are dynamically generated by testing various ways of splitting queries into candidate queries, running the candidate queries to obtain scored search results, and scoring the patterns of the candidate queries based on the scores received for their associated search results. A particular score, or confidence, can be required to select a query pattern from the candidate query patterns.

A query pattern can include, for example, stopwords, a connector, a confidence value, and a query split strategy. Stopwords, e.g., “a” and “the” are words that do not provide information that assists in identifying search results or search locations (e.g., words that do not provide information that influences the results or locations). The stopwords included in a query pattern can be removed from a received query and ignored when matching the received query to the query pattern. In a mapping search system, certain phrases such as “map of” and “location of,” can be removed from a received query as stopwords since all results in such a system are maps or locations.

One solution for identifying and removing stopwords from a received query is to simply compare those terms within the query to a predefined list of known stopwords. However, this approach may be inaccurate since some words can be meaningless stopwords in one query (e.g., “visiting” in “visiting new york”) but meaningful subject matter terms in another query (e.g., “visiting” in “visiting nurse new york”). Thus, as explained in further detail below, the subject disclosure provides a system and method for dynamically identifying and removing possible stopwords from a search query.

As noted above, a query pattern can be associated with a particular confidence value. The confidence indicates a likelihood that a given query that matches a query pattern is correctly interpreted according to the query pattern. For example, a query that matches the query pattern of the form {what} near {where} (e.g., “pizza near new york”) can have a higher confidence than a query that matches a query pattern in the form {what} in {where} (e.g., “pizza in New York”) because query interpretations for queries that match the (what) near {where} query pattern produce fewer false positives than query interpretations for queries that match the {what} in (where) query pattern.

Query patterns can be represented in a protocol buffer (“protobuffer” hereinafter). A protobuffer is a language-neutral, platform-neutral, extensible way of serializing structured data. For example, the query pattern (what) near {where} can be described in a protobuffer as follows:

The query pattern describes a query where the term “near” is used as a connector, and separates the {what} and {where} portions of the query. The “location_first: false” flag indicates that when the connector term “near” occurs at the beginning of a query, this pattern does not apply. As a result, this query pattern describes a query of the form {what} near {where}, where the (what) term cannot be empty since the connector “near” cannot be the first term in the query. As a result, the pattern is matched to queries such as “pizza near times square” and “burgers near 4th & king streets.” When the pattern is matched, the pattern can be split at the connector into corresponding {what} and {where} query components.

Other types of query patterns can be generated and/or provided. A prefix type query pattern has the form stopwords+{query}. In this example, {query} represents any valid query without the initial stopwords. When a received query is matched to a prefix type query pattern, the stopwords are removed, and the remaining {query} is further matched to one or more other query patterns, as described in greater detail below. Examples of queries in this form include “map of {query},” “location of {query},” and “where is {query}.” The dynamic determination and removal of stopwords will be described in greater detail below.

A suffix type query pattern has the form: {query}+stopwords. Similar to the prefix type query pattern, when a received query is matched to a suffix type query pattern, the stopwords are removed and the remaining {query} is further matched to one or more other query patterns. Examples of queries in this form include “{query} street map,” and “{query} map.” Furthermore, a prefix_suffix type query pattern has the form: stopwords+{query}+stopwords. Again, stopwords are removed, leaving the remaining query portion for additional processing. An example query in this form is “map of texas cities.”

Combinations of query patterns can also be generated. For example, a prefix_connector type query pattern has the form: {subject}+connector+{location} OR stopwords+{location}+connector+{subject}. For a query matching this type of query pattern, the stopwords are removed and the query split according to the connector, e.g., into the {subject} component and the {location} component. An example query in this form is “map of {subject} near {location}.”

Upon receiving a query and removing its stopwords (as further described below), the system tokenizes the query such that every term in the query is assigned to a token, and every token represents a single term or phrase in the query. The query can then be represented by a vector of tokens that can then be matched against the stored query patterns. For example, the query “pizza near new york” can be tokenized into the vector <pizza> <near> <new york> which can then be matched to the query pattern {what} {connector} {where} for the connector “near”.

At206, the system splits the query into query components according to the matching query pattern. In some implementations, a query can be split more than once depending on the matched query patterns. For example, a query that matches the query pattern {what} near {where} can be split into query components corresponding to {what} and {where}. Each of these components can be used to separately search specialized indexes (e.g., the {where} component can be used to search a geocoding index). Thus, the query “pizza near times square” can be split into a “pizza” component and a “times square” component. In another example, the query “map of restaurants near times square” can be first split according to the prefix type query pattern to remove the “map of” stopwords. The remaining query portion “restaurants near times square” can be subsequently split into query components “restaurants” and “times square” since it matches the {what} near {where} connector type query pattern.

Received queries that match a query pattern are split according to the split type defined by the matching query pattern. For example, a query that matches a query pattern with a “split_exact” query split type is split according to the identified query pattern. A query matching a query pattern with an “all” type of query split can be split in any or all possible ways.

A query that matches a query pattern with a “local” query split type is split so that it has a nonempty {what} portion. The split can have an empty {where} portion, though the {where} portion need not be empty. For example, if the query “phone number of city hall” matches a query pattern having a “local” query split type, the query can be split to have a {what} portion of “phone number” and a {where} portion of “city hall”, or to have a {what} portion of “phone number of city hall” and no {where} portion (e.g., the location can be implicit in the current map or user location information).

A query that matches a query pattern having a “geocode” query split type is split so that it has an empty {what} portion. For example, the query can simply identify a location, e.g., by name or address. Thus, the query, “401 van ness avenue, san francisco” can match a query pattern having a “geocode” query split type, and can be split so that it has an empty {what} portion and a non-empty {where} portion (i.e., 401 van ness avenue, san francisco). A geocode index can be searched for the precise location (i.e. geographical coordinates) corresponding to the {where} portion of the split query. This index can also identify what is located at or nearby that address, e.g., San Francisco City Hall, without a specific {what} query component.

At208, the system uses the components of the split query to search one or more corresponding indices. In some implementations, the query can be split more than once based on the applicable matching query patterns, and each portion of the split query can be used to search a corresponding index related to a particular corpus of documents. For example, when a query is split into a {what} portion and a {where} portion, the system can use the {where} portion to search a geocoding index, and the {what} portion to search a local search index. In some implementations, the search system first searches the geocoding index to identify geographical coordinates corresponding to the {where} component of the split query, and subsequently searches a local search index to identify documents that match the {what}component of the split query that have been geocoded to the {where} component. For example, given the query “pizza near times square,” split into a {what} component of “pizza” and a {where} component of “times square,” the system first searches a geocode index to identify the geolocation of“times square,” and then searches a local search index to identify documents that include the term “pizza” and that have been geocoded to a location that is near “times square”.

At210, the system presents one or more search results responsive to the received search query. The results can be ranked, (e.g., based on how well the results match the search query), and only a specified number of top ranked results can be presented. For example, presentation of the results for the search “pizza near times square” can include displaying the location of a plurality of pizza restaurants on a map of Manhattan where the displayed pizza restaurants are located near Times Square.

As noted above, stopwords can be removed from a received query prior to matching the query to a query pattern. This can be done, for example, by comparing the terms within a query against a predefined list of known stopwords. For example, the term “visiting” in the input query “visiting new york” may appear within a predefined list of known stopwords. Thus, “visiting” can be removed from the input query, resulting in a modified query of “New York”. The resulting query can match a query pattern with a geocode type query split, and be split into a {where} portion of “new york” and an empty {what} portion.

However, this approach can be improved upon since some stopwords can be meaningless in one query (e.g., “visiting” in “visiting new york”), but meaningful in another (e.g., “visiting” in “visiting nurse new york”). Thus, instead relying on a list of words as definite stopwords, the list can be considered a list of possible stopwords. In this way, potential stopwords can be dynamically identified and removed as actual stopwords from some search queries, but identified and left in place as meaningful query terms in other search queries.

For example, a search query comprising plural terms can be received. A term of the plural terms appearing in a predefined list of stopwords can be identified as a possible stopword. The query can be matched to one or more query patterns as described above, and the terms of the query split into one or more components according to the splitting rules of the matching query patterns. Each query split can use some or all of the plural terms in the search query as a location value (e.g., {where}) and some or all of the plural terms in the search query as a subject value (e.g., {what}). Either the location value, the subject value, or both, for one or more of the query splits can be modified by removing the term identified as a possible stop word from the value. The search engine can be queried with the location value and the subject value (which may be modified) for each query split, and scored search results can be obtained for each split. A query split can be selected based on the returned scores. In some cases, the selected query split may not include the possible stop word, indicating dynamic identification and removal of that possible stop word. In some cases, the selected query split may include the possible stopword, indicating dynamic rejection of the possible stop word as a stopword.

Two examples will now be provided based on the above description. In Examples 1 and 2, the input search queries of “visiting new york” and “visiting nurse new york” are described, and the term “visiting” is included in a list of possible stopwords.

Query of “Visiting New York”

The query “visiting new york” can be split in a number of ways as indicated by matching predefined query patterns, where each query split comprises a location value (e.g. {where}) and a subject value (e.g., {what}) as described above. Example query splits can include:

A stopword removal rule can specify removing a potential stopword from either the location value, the subject value or both for the query splits. In this case, the rule can specify removing the possible stopword “visiting” from the location value only, since “visiting” is likely to be a stop word for a location query, but may not be a stop word for a subject query. Thus, the term “visiting” can be removed from the {where} value of the identified splits, resulting in the following modified query splits:

A search engine can then be queried with the location value and the subject value for each query split to obtain search results for each query split. In this example, because {new york} is a well-known location, but {visiting} is not a popular search term, split 1 will typically have better search results than splits 2 and 3, and the search results for split 1 can be returned in response to the query. In this way, the possible stopword “visiting” is dynamically recognized and removed from the search query “visiting new york.”

Query of “Visiting Nurse New York”

The query “visiting nurse new york” can similarly be split according to one or more matching query patterns into a location portion and a search portion, e.g., as indicated below:

The possible stopword “visiting” can be identified, and a stopword removal rule can specify that the term “visiting” be removed only from the {where} values of the query splits, as follows:

A search engine can then be queried with the location value and the subject value for each query split to obtain search results for each query split. In this example, since “new york” is a well known location, and “visiting nurse” is a well known search subject, split 2 will yield high quality search results, while splits 1 and 3 will not. Thus, the possible stop word “visiting” will be dynamically retained as a meaningful search term, and used to provide search results to the user in response to the query “visiting nurse new york.”

FIG. 3illustrates an example process by which a potential stopword is dynamically identified and removed from a search query. Following start block302, a search query comprising plural terms is received at step304. At step306, a possible stopword among the plural terms that appears in a predefined list of possible stopwords is identified. The predefined list of possible stopwords can include connectors, which appear between and connect search query terms.

At step308, one or more query splits are determined based on one or more matching query patterns. Each query split includes a location portion that may be empty or may use some or all of the plural terms of the search query, and a subject portion that may be empty or that may use some or all of the plural terms of the search query.

At step310, a stopword removal rule can be used to remove the possible stopword from either the location portion, the subject portion or both portions of each of the one or more query splits. In this regard, a determination can be made whether the possible stopword is more closely associated with the location portion or the subject portion of the plurality of query splits. The possible stopword can be removed from the location portion of the one or more splits when it is more closely associated with the subject portion. On the other hand, the possible stopword can be removed from the subject portion of the one or more splits when it is more closely associated with the location portion.

At step312, the search engine can be queried with the location portion and the subject portion for each of the one or more query splits, and scored search results can be respectively obtained for each of the one or more query splits.

At step314, the search results returned for at least one of the one or more query splits is selected to return as a response to the search query. The selected search results can be the search results with the highest score. The selected search results can correspond to a query split in which the possible stopword was removed, or to a query split in which the possible stopword was not removed. The process then ends at end block316.

FIG. 4conceptually illustrates an example electronic system with which some implementations of the subject technology are implemented. Electronic system400can be a computer, phone, PDA, or any other sort of electronic device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system400includes a bus408, processing unit(s)412, a system memory404, a read-only memory (ROM)410, a permanent storage device402, an input device interface414, an output device interface406, and a network interface416.

Bus408collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of electronic system400. For instance, bus408communicatively connects processing unit(s)412with ROM410, system memory404, and permanent storage device402.

From these various memory units, processing unit(s)412retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The processing unit(s) can be a single processor or a multi-core processor in different implementations.

ROM410stores static data and instructions that are needed by processing unit(s)412and other modules of the electronic system. Permanent storage device402, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system400is off. Some implementations of the subject disclosure use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device402.

Other implementations use a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) as permanent storage device402. Like permanent storage device402, system memory404is a read-and-write memory device. However, unlike storage device402, system memory404is a volatile read-and-write memory, such a random access memory. System memory404stores some of the instructions and data that the processor needs at runtime. In some implementations, the processes of the subject disclosure are stored in system memory404, permanent storage device402, and/or ROM410. For example, the various memory units include instructions for processing a search query in accordance with some implementations. From these various memory units, processing unit(s)412retrieves instructions to execute and data to process in order to execute the processes of some implementations.

Bus408also connects to input and output device interfaces414and406. Input device interface414enables the user to communicate information and select commands to the electronic system. Input devices used with input device interface414include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). Output device interfaces406enables, for example, the display of images generated by the electronic system400. Output devices used with output device interface406include, for example, printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some implementations include devices such as a touchscreen that functions as both input and output devices.

Finally, as shown inFIG. 4, bus408also couples electronic system400to a network (not shown) through a network interface416. In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system400can be used in conjunction with the subject disclosure.

All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.