Package: elasticsearch-ir-evaluator
Elasticsearch IR EvaluatorOverviewelasticsearch-ir-evaluatoris a Python package designed for easily calculating a range of information retrieval (IR) accuracy metrics using Elasticsearch and datasets. This tool is ideal for users who need to assess the effectiveness of search queries in Elasticsearch. It supports the following key IR metrics:PrecisionRecallMean Reciprocal Rank (MRR)Mean Average Precision (MAP)Cumulative Gain (CG)Normalized Discounted Cumulative Gain (nDCG)False Positive Rate (FPR)Binary Preference (BPref)These metrics provide a comprehensive assessment of search performance, catering to various aspects of IR system evaluation. The tool's flexibility allows users to select specific metrics according to their evaluation needs.InstallationTo installelasticsearch-ir-evaluator, use pip:pipinstallelasticsearch-ir-evaluatorPrerequisitesElasticsearch version 8.11 or higher running on your system.Python 3.8 or higher.Complete Usage ProcessThe following steps will guide you through usingelasticsearch-ir-evaluatorto calculate search accuracy metrics.
For more detailed and practical examples, please refer to theexamplesdirectory in this repository.Step 1: Set Up Elasticsearch ClientConfigure your Elasticsearch client with the appropriate credentials:fromelasticsearchimportElasticsearches_client=Elasticsearch(hosts="https://your-elasticsearch-host",basic_auth=("your-username","your-password"),verify_certs=True,ssl_show_warn=True,)Step 2: Create and Index the CorpusCreate and index a new corpus. You can customize index settings and text field configurations, including analyzers:fromelasticsearch_ir_evaluatorimportElasticsearchIrEvaluator,Document# Initialize the ElasticsearchIrEvaluatorevaluator=ElasticsearchIrEvaluator(es_client)# Specify your documentsdocuments=[Document(id="doc1",title="Title 1",text="Text of document 1"),Document(id="doc2",title="Title 2",text="Text of document 2"),# ... more documents]# Set custom index settings and text field configurationsindex_settings={"number_of_shards":1,"number_of_replicas":0}text_field_config={"analyzer":"standard"}evaluator.set_index_settings(index_settings)evaluator.set_text_field_config(text_field_config)# Create a new index or set an existing oneevaluator.set_index_name("your_index_name")# Index documents with an optional ingest pipelineevaluator.index(documents,pipeline="your_optional_pipeline")Step 3: Set a Custom Search TemplateCustomize the search query template for Elasticsearch. Use{{question}}for the question text and{{vector}}for the vector value in QandA:search_template={"match":{"text":"{{question}}"}}evaluator.set_search_template(search_template)Step 4: Calculate Accuracy MetricsUse.calculate()to compute all possible metrics based on the structure of the provided dataset:# Load QA pairs for evaluationqa_pairs=[QandA(question="What is Elasticsearch?",answers=["doc1"]),# ... more QA pairs]# Calculate all metricsresults=evaluator.calculate(qa_pairs)# Output resultsprint(result.model_dump_json(indent=4))This step involves a comprehensive evaluation of search performance using the provided question-answer pairs. The.calculate()method computes all metrics that can be derived from the dataset's structure.Licenseelasticsearch-ir-evaluatoris available under the MIT License.