Abstract:
An information retrieval system includes computing hardware, a database operatively coupled with the computing hardware and including documents for delivery to a user in response to a user search as well as a non-transitory, computer-readable memory component operatively coupled with the computing hardware. The non-transitory, computer-readable memory component further includes a receiving module configured to, upon actuation by the computing hardware, receive a search query and a corresponding relevancy expectation score from the user; a search module configured to, upon actuation by the computing hardware, search the database against the search query provided by the user and output an original list of results each of which exhibits a numerical score; and a post-processing module configured to, upon actuation by the computing hardware, perform post-processing of the numerical scores of the results of the original list.

Description:
TECHNICAL FIELD  
       [0001]    The present invention relates generally to information retrieval systems, and more specifically, to a system and method for adjusting search result set size based on user provided expectation. 
       BACKGROUND  
       [0002]    The process of information retrieval involves two parties. On one hand there are producers of the information to be stored by the information retrieval system. The producers either actively publish the information to the system or let the information system select the information from the producer&#39;s source system (e.g. internet search engines work that way). On the other hand there are consumers of the information stored in the information retrieval system. Examples of published and stored information types, include, but are not limited to, social media messages, digitized documents (e.g. scanned and converted with optical character recognition), electronic message documents (e-mails), images, HTML pages, binary documents such as office documents, and text documents. 
         [0003]    Information retrieval systems work by extracting information out of a source document and storing the corresponding document representation for later retrieval. The representation of a document is composed in parts of meta-data associated with the source document (data describing the source document) and data from the actual document content. Examples of meta data attributes, include, but are not limited to, document author name, document author social media platform identifier (e.g. twitter handle), document creation date, document modification date, length of document in number of bytes, length of document in number of characters, length of document in number of words, document domain (e.g. dietary publication in the field of medicine), and document content attributes (e.g. age group studied in the dietary publication). 
         [0004]    Consumers of the system formulate queries that are formal statements of information needed. Queries are formulated by using arbitrary text keywords and, or, specific parts from the document meta-data. Examples of query terms, include, but are not limited to, individual text keywords, text phrases, meta-data attribute filters, and meta-data attribute range filters. The information retrieval system evaluates each query and locates matching document representations, with varying levels of relevance. The corresponding documents are returned to the user. The information retrieval system may also calculate the level of relevancy of each document representation and sort the returned documents according to the respective relevancy level. 
         [0005]    One reason for the broad adaption of information retrieval systems is the large amount of digital information available. Technologies such as the public Internet, electronic messaging systems (e-mail), social media networks, and mobile devices allow producers to publish information more easily and thus more frequently. All resulting in an exponential digital information growth. On the consumer side information overload is a wide spread problem. There is too much information to be processed, thus consumers need tools to efficiently navigate through the information available and find the sub-set of information that satisfies their information need. 
         [0006]    However, a typical information retrieval system may overwhelm the user with the result set returned for any given query. The result set may include a large number of results, making it difficult and time consuming for the user to comprehend the relevant results. The user may limit the number of search results by narrowing down the search query. However, the narrowed search query may not yield some results, which would otherwise be relevant. 
         [0007]    Attempts have been made to improve the relevancy of the results of a search query based on relative expertise between a searcher (consumer) and the creator(s) and/or contributor(s) (producers) of a document. Personalized rankings of search results have also been provided. However, neither of these techniques has been proven to be very efficient in reducing the size of the result set and increasing the probability of relevant results, without narrowing down the search query. 
         [0008]    There is a need for an information retrieval system that enables a user to limit the number of search results without narrowing down the search query. 
       SUMMARY  
       [0009]    The present disclosure seeks to provide an information retrieval system and a method of operating the information retrieval system for processing a search query. 
         [0010]    In one aspect, embodiments of the present disclosure provide an information retrieval system that includes computing hardware operable to execute one or more software products recorded on machine-readable data storage media. The computing hardware is operable to receive from a user device, a search query and a corresponding relevancy expectation score, and search a database against the search query and generate a first list of results, wherein each result of the first list has a numeric score associated therewith. The computing hardware is further operable to transform the relevancy expectation score with a polynomial function, wherein the order of the polynomial function depends on the number of results in the first list, to (i) determine a volume regulation function by fitting a natural language specific distribution to the numeric scores of the first list, and (ii) to estimate a cut-off score from a transformed relevancy expectation score and total number of results in the first list, using the determined volume regulation function. The computing hardware is furthermore operable to generate a filtered list based on the cut-off score, wherein the filtered list includes one or more results of the first list that have a numeric score greater than or equal to the cut-off score. 
         [0011]    In an embodiment of the present invention, the computing hardware is further operable to cap the cut-off score using a pre-calculated minimal score. 
         [0012]    In an embodiment of the present invention, the computing hardware is further operable to use a fall-back method to estimate the result set size based on user expectation if the volume regulation function cannot be determined. 
         [0013]    In an embodiment of the present invention, the computing hardware is further operable to sort one or more results of the filtered list according to respective timestamps. 
         [0014]    In an embodiment of the present invention, the computing hardware is further operable to use a sub-set of the first list for computing the cut-off score, wherein the sub-set includes relevant results. 
         [0015]    The user device is implemented using at least one of: a personal computer, a laptop, a personal digital assistant, a tablet, a phablet, a webpad, a wearable computer, a mobile telephone, a wrist-worn computer, and a smart phone. 
         [0016]    In another aspect, embodiments of the present disclosure provide a computer implemented method of varying result set size during information retrieval based upon user expectations, which method includes the actions of receiving a search query and a corresponding relevancy expectation score from a user device searching a database against the received search query and generating a first list of one or more results, wherein each result of the first list has a numeric score associated therewith. 
         [0017]    In an embodiment of the present disclosure, the method further includes 
         [0018]    transforming the relevancy expectation score with a polynomial function, wherein the order of the polynomial function depends on the number of results in the first list; fitting a natural language specific distribution to the numeric scores of the one or more results of the first list to determine a volume regulation function; using the determined volume regulation function to estimate a cut-off score from the transformed relevancy expectation score and the total number of results in the first list; and generating a filtered list including results of the first list having a numeric score greater than or equal to the cut-off score. 
         [0019]    In an embodiment of the present disclosure, the method also further includes applying a fall-back method to calculate a result set size based on a relevancy expectation value and the original result set size. 
         [0020]    In an embodiment of the present disclosure, the method also further includes capping the cut-off score using a pre-calculated minimal score. 
         [0021]    In an embodiment of the present disclosure, the method also further includes computing the cut-off score with a sub-set of the first list, wherein the sub-set includes one or more relevant results. 
         [0022]    In an embodiment of the present disclosure, the method also further includes sorting one or more results of the filtered list according to respective timestamps. 
         [0023]    In an embodiment of the present disclosure, receiving a search query and a corresponding relevancy expectation score from a user device may include receiving the search query from at least one of: a personal computer, a laptop, a personal digital assistant, a tablet, a phablet, a webpad, a wearable computer, a mobile telephone, a wrist-worn computer, and a smart phone. 
         [0024]    In yet another aspect, embodiments of the present disclosure provide an information retrieval system including computing hardware, a database operatively coupled with the computing hardware and including documents for delivery to a user in response to a user search and a non-transitory, computer-readable memory component operatively coupled with the computing hardware. The memory component further includes a receiving module configured to, upon actuation by the computing hardware, receive a search query and a corresponding relevancy expectation score from the user; a search module configured to, upon actuation by the computing hardware, search the database against the search query provided by the user and output an original list of results each of which exhibits a numerical score; and a post-processing module configured to, upon actuation by the computing hardware, perform post-processing of the numerical scores of the results of the original list. 
         [0025]    In an embodiment of the present disclosure, the post-processing module is further configured to, upon actuation by the computing hardware, transform a relevancy expectation score, determine a volume regulation function and estimate a cut-off score. 
         [0026]    In an embodiment of the present disclosure, the information retrieval system a further includes a result set filtering module configured to, upon actuation by the computing hardware, filter the results of the original list based on the cut-off score. 
         [0027]    In an embodiment of the present disclosure, the information retrieval system further includes a result set sorting module configured to, upon actuation by the computing hardware, sort the filtered results based on respective timestamps. 
         [0028]    In an embodiment of the present disclosure, the information retrieval system further includes an output module configured to, upon actuation by the computing hardware, transmit the sorted list of results to the user. 
         [0029]    Further details of aspects, objects, and advantages of the invention are described below in the detailed description, figures, and claims. Both the foregoing general description and the following detailed description are exemplary and explanatory, and are not intended to be limiting as to the scope of the invention. 
     
    
     
       BRIEF DESCRIPTION OF THE FIGURES  
         [0030]    The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, example constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers. 
           [0031]    Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein: 
           [0032]      FIG. 1  is a schematic illustration of an environment, wherein various embodiments of the present invention can be practiced; 
           [0033]      FIG. 2  is a schematic illustration of an information retrieval system, in accordance with an embodiment of the present disclosure; 
           [0034]      FIG. 3  is a schematic illustration of processing a search query by the information retrieval system, in accordance with an embodiment of the present disclosure; 
           [0035]      FIG. 4  is a schematic illustration of post processing of the document scores by the post-processing module, in accordance with an embodiment of the present disclosure; 
           [0036]      FIG. 5  illustrates an example determination of a volume regulation function by the post-processing module, in accordance with an embodiment of the present disclosure; 
           [0037]      FIG. 6  illustrates an example of an original score list as received by the post-processing module, in accordance with an embodiment of the present disclosure; 
           [0038]      FIG. 7  illustrates an example of an estimated score function determined by the post-processing module, in accordance with an embodiment of the present disclosure; 
           [0039]      FIG. 8  illustrates filtering of the results by the result set filtering module, in accordance with an embodiment of the present disclosure; and 
           [0040]      FIG. 9  schematically illustrates a list of search results generated by the information retrieval system, in accordance with an embodiment of the present disclosure. 
       
    
    
     DETAILED DESCRIPTION  
       [0041]    The following detailed description illustrates embodiments of the present disclosure and manners by which they can be implemented. Although the best mode of carrying out the present disclosure has been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible. 
         [0042]    Embodiments of the present disclosure provide an information retrieval system that includes computing hardware operable to execute one or more software products recorded on machine-readable data storage media. The computing hardware is operable to receive from a user device, a search query and a corresponding relevancy expectation score, and search a database against the search query and generate a first list of results, wherein each result of the first list has a numeric score associated therewith. The computing hardware is further operable to transform the relevancy expectation score with a polynomial function, wherein the order of the polynomial function depends on the number of results in the first list, to determine a volume regulation function by fitting a natural language specific distribution to the numeric scores of the first list, and to estimate a cut-off score from a transformed relevancy expectation score and total number of results in the first list, using the determined volume regulation function. The computing hardware is furthermore operable to generate a filtered list based on the cut-off score, wherein the filtered list includes one or more results of the first list that have a numeric score greater than or equal to the cut-off score. 
         [0043]    Embodiments of the present disclosure substantially eliminate, or at least partially address, problems in the prior art and enable the user to limit the number of search results without narrowing down the search query. 
         [0044]    Referring now to the drawings, particularly by their reference numbers,  FIG. 1  is an illustration of an environment  100 , wherein various embodiments of the present invention can be practiced. The environment  100  includes an information retrieval system  102 , at least one database  104 , and one or more user devices  106   a  and  106   b,  wherein the information retrieval system  102 , the database  104 , and the one or more user devices  106   a  and  106   b  are communicatively coupled to each other via a communication network  107 . Examples of the communication network  107  include, but are not limited to, Internet, Intranet, MAN, LAN, and WAN. 
         [0045]    The information retrieval system  102  generally includes one or more servers that serve data to the user devices  106   a  and  106   b,  in the form of webpages that may include redirects and other Hypertext mark-up language (HTML) information such as text, figures, videos, images, and scripts. The user devices  106   a  and  106   b  are generally representative of user computing devices such as personal computers, laptops, personal digital assistants, tablets, phablets, webpads, wearable computers, mobile telephones, wrist-worn computers, smart phones or any other devices capable of displaying a web-based interface or using a application programming interface (API) for communicating with the information retrieval system  102 . 
         [0046]    The database  104  stores a plurality of documents, images, videos, figures, scripts pertaining to a variety of topics such as finance, literature, science, healthcare, legal, etc. In an example, the database  104  is the World Wide Web. In an embodiment, the information retrieval system  102  includes a search engine that is configured to receive and process one or more queries against the database  104 . In another embodiment, the information retrieval system  102  executes a web page on a display of the user device  106   a,  that includes a search section for inputting one or more search queries, and a display section for displaying one or more search results based on the one or more search queries. 
         [0047]      FIG. 1  is merely an example. One of ordinary skill in the art would recognize many variations, alternatives, and modifications of embodiments herein. 
         [0048]      FIG. 2  is a schematic illustration of the information retrieval system  102 , in accordance with an embodiment of the present disclosure. The information retrieval system  102  includes a receiving module  202 , a search module  204 , a post-processing module  206 , a result set filtering module  208 , a result set sorting module  210 , and an output module  212 . 
         [0049]    The receiving module  202  is configured to receive a search query and a corresponding relevancy expectation score from a user. In an exemplary embodiment, the relevancy expectation score is a numerical value ranging from 1.0 (highest relevancy expectation score) to 0.0 (lowest relevancy expectation score). The relevancy expectation score for a search query indicates the user&#39;s expectation on the relevancy of the results from the search query. If the relevancy expectation score is high, the user expects a small number of results from the search query and all the results may be relevant. If the relevancy expectation score is low, the user expects a large number of results from the query, out of which few results may be relevant. The low relevancy expectation score indicates large number of irrelevant search results in a result set, and thus, a more noisy result set. Therefore, with the relevancy expectation score, the user may indicate how noisy the result set should be. 
         [0050]    The search module  204  is configured to search the database  104  against the search query provided by the user, and output an original list of results, where each result has a respective numerical score, and the results may be sorted according to respective numerical scores. As an example, numerical scores are based on term frequencies (tf) and inverse document frequencies (idf). The search module  204  orders the result set by relevance in decreasing order. In an embodiment, the user may input a text query for searching one or more documents in the database  104  and the search module  204  may search an internal document index and return one or more documents arranged in decreasing order of relevance according to the respective numerical scores. 
         [0051]    The post-processing module  206  is configured to perform post-processing of the numerical scores of the results of the original list, wherein the post-processing includes relevancy expectation score transformation, determining a volume regulation function, and cut-off score estimation. 
         [0052]    The result set filtering module  208  filters the results of the original list based on the cut-off score, i.e., the result set filtering module  208  generates those results of the original list that have a numerical score greater than or equal to the cut-off score. 
         [0053]    In an embodiment, the result set filtering module  208  calculates a fall-back method to determine the optimal number of results n to return if less than M results are returned in the original result list of length N. This fall-back method acts as a quality safe-guard to prevent an undesired cut-off score estimation due to over-fitting of the linear regression solution. This situation arises if less than M results are returned in the original result list. The fall-back method is calculated as a quadratic function of the relevancy expectation score r e  and returns a number of items n. Moreover a minimum number of items to return n min  is provided to the method. 
         [0000]        n=n   min +( N−n   min )*(1 −r   e ) 2    (1)
 
         [0054]    The final number of results is calculated using: 
         [0000]        n =max( n   min , ceil( n ))   (2)
 
       Example I  
       [0055]    Total number of results N in the original result list=15 
         [0056]    Minimal required number of results for volume regulation function M=50 
         [0057]    User relevancy expectation score r e =0.2 
         [0058]    Minimal required number of results to return i min =5 
         [0059]    Number of results to return i=max(5, ceil(5+(15−5)*(1−0.2) 2 ))=12 
         [0060]    The result set sorting module  210  sorts the filtered results based on respective timestamps, for providing the user with a more natural way to consume results, as humans tend to favour recent results over old ones. 
         [0061]    The output module  212  displays the sorted list of results on a display of the user device  106   a.    
         [0062]      FIG. 2  is merely an example. One of ordinary skill in the art would recognize many variations, alternatives, and modifications of embodiments herein. 
         [0063]      FIG. 3  is a schematic illustration of processing a search query by the information retrieval system  102 , in accordance with an embodiment of the present disclosure. At step  302 , the receiving module  202  receives a search query from a user device  106   a.  In an embodiment, the user inputs the search query in a search field on a web page displayed on the user device  106   a.  At step  306 , the search module  204  searches an internal document index, of which database  104  is an example, and obtains an original list of results/documents and their respective scores. At step  310 , the post-processing module  206  performs post-processing of the document scores based on a relevancy expectation score and total number of results obtained from the user search query. 
         [0064]    The post-processing module  206  is configured to perform post-processing of the numerical scores of the results contained in the original list of length N in accordance with an action  310 , wherein the post-processing includes transforming relevancy expectation score r e , determining a volume regulation function, and estimating a cut-off score. The relevancy expectation score is transformed with the polynomial function 
         [0000]        f ( r   e   , N )=(1 −r   e ) m    (3)
 
         [0065]    where the exponent m is 
         [0000]      max(1, floor(log 10 (N)))   (4)
 
         [0066]    The resulting order of the polynomial function depends on the total number of results N obtained in the original list. In an example, the order is increased for each order of magnitude of the total number of results (linear transformation for up to 100, quadratic transformation for up to 1000, etc.). This is similar to a logarithmic potentiometer which is used to control the volume on audio equipment. The resulting behaviour feels more natural as human perception of audio volume is not linear but logarithmic. 
         [0067]    The post-processing module  206  determines a volume regulation function by fitting a natural language specific distribution to the numerical scores of the original list. To control the quality of the volume regulation function, only the sub-set of the first M results of the result set is used for this process. In an embodiment, when the search module  204  returns a list of documents against a text query, a natural language specific distribution such as a Zipf distribution is used to model the volume regulation function. The Zipf distribution estimates the numerical score of a document at a rank r i  using the equation 
         [0000]      score( r   i )=α* r   i   β   (5)
 
         [0068]    where α and β are constants and are obtained by solving a regression problem using machine learning techniques. 
         [0069]    In an embodiment, the post-processing module  206  solves the regression problem by transforming equation (5) with a logarithmic function. This transformation allows the usage of linear regression to estimate the parameters α and β of equation (5). The general linear regression function to solve is 
         [0000]        y′=B*X+A    (6)
 
         [0070]    The logarithmic transformation of equation (5) is further simplified to 
         [0000]      ln(score( r   i ))=β*ln( r   i )+ln(α)   (7)
 
         [0071]    and the first M ranks and relevancy scores are used as input variables to solve the linear regression. 
         [0072]    The post-processing module  206  estimates a cut-off score from a transformed relevancy expectation score and total number of results in the original list, using the transformed relevancy expectation function of equation (3) and (4) and the volume regulation function of equation (5). The cut-off score may be estimated based on the following equation: 
         [0000]        r   i =α*( N*f ( r   e   , N )) β   (8)
 
       Example I  
       [0073]    Total number of results N in the original result list=120 
         [0074]    Polynomial function order m=max(1, floor(log 10 (120)))=2 
         [0075]    User relevancy expectation score r e =0.2 
         [0076]    Transformed user relevancy expectation score (1−r e ) m =0.8 2 =0.64 
         [0077]    Cut off score r c ≈α*(120*0.8 2 ) β . 
       Example II  
       [0078]    Total number of results N in the original result list=1200 
         [0079]    Polynomial function order m=max(1, floor(log 10 (1200)))=3 
         [0080]    User relevancy expectation score r e =0.9 
         [0081]    Transformed user relevancy expectation score (1−r e ) m =0.1 3 =0.001 
         [0082]    Cut off score r c ≈α*(1200*0.1 3 ) β . 
         [0083]    In an embodiment, the post-processing module  206  optionally defines a pre-calculated minimal score r m  to cap the cut-off score r c  using the following equation: 
         [0000]        r   c ′=min( r   c   , r   m )   (9)
 
         [0084]    The minimal score r m  can be estimated by using the original relevancy score of a reference document judged by the user as a valuable document for this information need. 
       Example I  
       [0085]    Total number of results N in the original result list=120 
         [0086]    Polynomial function order m=max(1, floor(log 10 (120)))=2 
         [0087]    User relevancy expectation score r e =0.2 
         [0088]    Transformed user relevancy expectation score (1−r e ) m =0.8 2 =0.64 
         [0089]    Pre-calculated minimal score r m =0.05 
         [0090]    Cut off score r c ′=min(α*(120*0.2 2 ) β , 0.05) 
         [0091]    As a result, the post-processing module  206  does not let the cut off score exceed the pre-calculated minimal score. 
         [0092]    At step  312 , the result set filtering module  208  filters the original results list based on a cut-off score, i.e., the result set filtering module  208  generates only those results of the original list that have a score greater than or equal to the cut-off score. At step  314 , the result set sorting module  210  sorts the filtered results based on respective timestamps ( FIG. 9 ). At step  316 , the output module  212  transmits the sorted list of results to the user device  106   a,  and at step  318 , the user device  106   a  receives and displays the sorted list of results on a respective display. 
         [0093]      FIG. 4  is a schematic illustration of post-processing of the document scores by the post-processing module  206 , in accordance with an embodiment of the present disclosure. At step  402 , the post-processing module  206  receives and transforms a relevancy expectation score with a polynomial function; at step  406 , selects a sub-set of the original result list that includes relevant results; at step  410 , interpolates scores of the sub-set of the original result list; and at step  412 , calculates a cut-off score using the equations (4) and (5). 
         [0094]      FIG. 5  is an exemplary determination of a volume regulation function by the post-processing module. A first table  510  lists the relevancy scores for each of the N resulting documents as calculated by the search system. With a logarithmic transformation both the ranks and the scores are transformed (table  520 ) as a preparation for the linear regression fitting applied on the first M results of the list. The resulting parameters  530  (α and β) from the regression are then used to calculate the cut-off score based on the relevancy expectation score. 
         [0095]      FIG. 6  is an exemplary illustration of an original score list graph as received by the post-processing module. The original score list graph shows the distribution of the original relevancy score as calculated by the search system over the resulting ranks. 
         [0096]      FIG. 7  is an exemplary illustration of an estimated score function determined by the post-processing module. The estimated score graph demonstrates the solution of the linear regression for the result set from  FIG. 6 . 
         [0097]      FIG. 8  is an illustration of filtering of the results by the result set filtering module  208 , in accordance with an embodiment of the present disclosure. At step  802 , the result set filtering module  208  extracts a numerical score of a document/result of the original result list and at step  806 , it checks whether the numerical score is greater than or equal to the cut-off score. At step  808 , the result set filtering module  208  includes/retains the document/result when the numerical score is greater than or equal to the cut-off score. Alternatively, the result set filtering module  208  ignores the document at step  810  when the numerical score is less than the cut-off score. The result set filtering module  208  repeats the steps  802  to  810  for the remaining documents of the original result list. 
         [0098]      FIG. 8  is merely an example. One of ordinary skill in the art would recognize many variations, alternatives, and modifications of embodiments herein. 
         [0099]      FIG. 9  is a schematic illustration of an original result list  902  and first through third filtered results lists  904 ,  906  and  908  in accordance with an embodiment of the present disclosure. The original result list  902 , which is an example of the original result list referred to above, is generated based on the search query input by the user and includes seven documents arranged in order of decreasing relevance. Each document of the original list  902  has a score and timestamp associated therewith. A first filtered list  904  is obtained by filtering the original list  902  using a cut-off score of 0.75 and arranging in order of increasing timestamp. A second filtered list  906  is obtained by filtering the original list  902  using a cut-off score of 0.30 and arranging in order of increasing timestamp. A third filtered list  908  is obtained by filtering the original list  902  using a cut-off score of 0.00 and arranging in order of increasing timestamp. 
         [0100]    Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as “including”, “comprising”, “incorporating”, “consisting of”, “have”, “is” used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural.