Patent Document ID: 9436729
Application ID: 14328809

Base Claim:
1. An information retrieval system evaluation method, comprising: Obtaining, by an evaluation electronic device, behavior data sample reported by an information retrieval system within a pre-determined period; obtaining, by an evaluation electronic device, a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample; invoking, by an evaluation electronic device, the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and computing, by an evaluation electronic device, an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, where the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword; the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.

---

Claim 2:
2. The information retrieval system evaluation method according to claim 1 , wherein the evaluation indicator of the information retrieval system is computed according to the following formula, by an evaluation electronic device: F = ( β 2 + 1 ) ⁢ PR β 2 ⁢ P + R wherein, F is the evaluation indicator, R is an arithmetic mean value of recall rates corresponding to all evaluation keywords, P is an arithmetic mean value of correctness percentages corresponding to all evaluation keywords, and β is a pre-determined weight coefficient.