Package: benchmarks-runner
Translator Benchmarks RunnerThis repository provides a set of benchmarks as well as the code to send the queries and evaluate the returned results of a benchmark.benchmarks-runnercontains the code to query targets and evaluate results.benchmarks-runner.configcontains the data sets, query templates, targets, and benchmark definitions necessary to run a benchmark. Seeconfig/README.mdfor details about targets and benchmarks.UsageRunning a benchmark is a two-step process:Execute the queries of a benchmark and store the scored results.Evaluate the scored results against the set of ground-truth relevant results.Installationof thebenchmarks-runnerpackage provides access to the functions and command-line interface necessary to run benchmarks.CLIThe command-line interface is the easiest way to run a benchmark.benchmarks_fetchFetches (un)scored results given the name of a benchmark (specified inconfig/benchmarks.json), target (specified inconfig/targets.json), and a directory to store results.By default,benchmarks_fetchfetches scored results using 5 concurrent requests. Runbenchmarks_fetch --helpfor more details.benchmarks_scoreScores results given the name of a benchmark (specified inconfig/benchmarks.json), target (specified inconfig/targets.json), a directory containing unscored results, and a directory to store scored results.By default,benchmarks_scoreuses 5 concurrent requests. Runbenchmarks_score --helpfor more details.benchmarks_evalEvaluates a set of scored results given the name of a benchmark (specified inconfig/benchmarks.json) and a directory containing scored results.By default, the evaluation considers the top 20 results of each query, and plots are not generated. Runbenchmarks_eval --helpfor more details.FunctionsThe CLI functionality is also available by importing functions from thebenchmarkspackage.frombenchmarks.requestimportfetch_results,score_resultsfrombenchmarks.evalimportevaluate_results# Fetch unscored resultsfetch_results('benchmark_name','target_name','unscored_results_dir',scored=False)# Score unscored resultsscore_results('unscored_results_dir','target_name','results_dir')# Evaluate scored resultsevaluate_results('benchmark_name','results_dir OR results_dict')See the documentation of each function for more information.InstallationInstall the repository as an editable package usingpip.pip install -e .UIThese benchmarks come with a frontend for viewing the scored results.InstallationRequires python 3.9.Create a python virtual environment:python3.9 -m venv benchmark_venvActivate your environment:. ./benchmark_venv/bin/activateInstall dependencies:pip install -r requirements.txtStart the frontend server:python server.pyOpen in your browserBenchmark RunnerThe benchmarks can be installed from pypi and used as part of the Translator-wide automated testing.pip install benchmarks-runnerTo run benchmarks:importasynciofrombenchmarks_runnerimportrun_benchmarksoutput=asyncio.run(run_benchmarks(<benchmark>,<target>))where benchmark is the name of a benchmark that is specified in config/benchmarks.json, and a target that is specified in config/targets.jsonSample OutputBenchmark: GTRx
Results Directory: /tmp/tmpaf10m9_q/GTRx/bte/2023-11-10_13-03-11
                        k=1     k=5     k=10    k=20
Precision @ k           0.0000  0.0500  0.0250  0.0125
Recall @ k              0.0000  0.2500  0.2500  0.2500
mAP @ k                 0.0000  0.0833  0.0833  0.0833
Top-k Accuracy          0.0000  0.2500  0.2500  0.2500
Mean Reciprocal Rank    0.08333333333333333