Patent ID: 7526470

Claim:
A method for measuring the relevance of results in a search session, the search session including a set of actions, the method comprising: executing instructions stored in memory, the instructions comprising: logging a plurality of user actions in an event log; determining a start of the search session by scanning the event log to identify an initial query; determining an end of the search session by identifying an expiration of a timeout before an action is committed by a user; for each result in the search session, identifying at least one user behavior in response to the result, the user behavior identified by determining whether the user has performed each of ignoring the result, rejecting the result, exploring the result, and accepting the result; estimating a level of satisfaction with respect to the results in the search session based on the user behavior, whereby the level of satisfaction is estimated using three satisfaction tiers comprising dissatisfaction, satisfaction, and partial satisfaction, and whereby ignoring the result is estimated as dissatisfaction with the result, rejecting the result is estimated as dissatisfaction with the result, accepting the result is estimated as satisfaction with the result, and exploring the result is estimated as partial satisfaction with the result; using the estimated level of satisfaction with respect to the results in the search session to determine an electronic search system's ability to provide relevant search results that correlate to intents of user queries; observing symptoms based on the at least one user behavior and the estimated level of satisfaction, the symptoms indicating a need for improvements within the electronic search system; deriving diagnostics based on the observed symptoms, the diagnostics being conditions that prevent the electronic search system from performing optimally, wherein the diagnostics comprise at least one member of a group comprising content quantity, poor content quality, intent determination, result ranking, result precision, and result description, wherein deriving diagnostics comprises: deriving the diagnostic content quantity when symptoms re-query present, no result, no accept, and re-query successful are observed; and deriving the diagnostic poor content quality when symptom lower partial-satisfaction is observed; and generating a performance report for the electronic search system based on the diagnostics and the estimated level of satisfaction.