Package: orbis-eval
Orbis quickstartOrbis is a versatile framework for performing NEL evaluation analyses. It supports standard metrics such as precision, recall and F1-score and visualizes gold standard and annotator results in the context of the annotated document. Color coding the entities allows experts to quickly identify correct and incorrect annotations and the corresponding links to the KB that are also provided by Orbis. Due to the modular pipeline architecture used by Orbis different stages in the evaluation process can be easily modified, replaced or added.Results of our first Orbis based drill-down analyses efforts were presented at the SEMANTiCS 2018 Conference in ViennaOdoni, Kuntschik, Braşoveanu, & Weichselbraun, 2018.PrerequisitesTo be able to develop and run Orbis you will need the following installed and
configured on your system:Python 3.7Python Setup ToolsA Linux or Mac OS (Windows is untested)InstallTo use Orbis, download and install it from PyPI:$python3-mpipinstall-Uorbis-eval['all']--userThere are more extras options available but we recommend you use the all option. Only use the other options if you really know what you are doing.- all: Install all extras for Orbis. Recommended option
    - all_plugins: Install only all plugins for Orbis.
    - all_addons: Install only all addons for Orbis.
    - aggregation: Install only all aggregation plugins for Orbis.
    - evaluation: Install only all evaluation plugins for Orbis.
    - metrics: Install only all metrics plugins for Orbis.
    - scoring: Install only all scoring plugins for Orbis.
    - storage: Install only all storage plugins for Orbis.
    - "plugin or addon name": Install only the specified addon or plugin named.Alternatively Orbis can be install by cloning the Repo and installing it manually. Plugins and addons must be installed seperatly.$gitclonehttps://github.com/orbis-eval/Orbis.git$cdOrbis$python3setup.pyinstall--user# or$pythonsetup.pyinstall--userDepending on your system and if you have Python 2 and Python 3 installed you either need to usepython3(like on Ubuntu) or maybe justpython.Test runTo get a first impression of orbis and for setting up the user folder runorbis-eval -t. You will be requested to set an orbis user folder. This folder will contain the evaluation run queue, the logs, the corpora and monocle data, the output and the documentation. Default location will be~/orbis-evalin the user's home folder. An alternative location can be specified.Runningorbis-eval -twill run the test files located in~/orbis-eval/queue/tests. These test configs are short evaluation runs for different annotators (AIDA, Babelfly, Recognyze and Spotlight). It is possible to just take one of these YAML files as template, copy them to the folder~/orbis-eval/queue/activatedand modify them to your own needs.The results of the test runs as HTML can be found in your user orbis folder, e.g.~/orbis-eval/output/html_pagesOrbis AddonsTo run an Orbis addon Orbis provides a CLI that can be accessed by runningorbis-addonsororbis-eval --run-addon. The menu will guide you to the addons and the addons mostly provide an own menu.RunAfter installation Orbis can be executed by runningorbis-eval. The Orbis help can be called by using-h(orbis-eval -h). Runningorbis-evalexecutes all yaml config files in the folder~/orbis-eval/queue/activated.
Before you can run an evaluation, please install the corpus you're referencing in the yaml config files using the repoman addonorbis-addons.Configure evaluation runsOrbis uses yaml files to configure the evaluation runs. These config files are located in the queue folder in the Orbis user directory~/orbis-eval/queue/activated.A YAML configuration file is divided into the stages of the pipeline:aggregation:service:name:aidalocation:webinput:data_set:name:rss1lenses:-3.5-entity_list_en.txt-14dec-0130pmmappings:-redirects-v2.json-15dec-1121amfilters:-us_states_list_en-txt-12_jan_28-0913amevaluation:name:binary_classification_evaluationscoring:name:nel_scorercondition:overlapentities:-Person-Organization-Placeignore_empty:Falsemetrics:name:binary_classification_metricsstorage:-cache_webservice_resultsAggregationThe aggregation stage of orbis collects all the data needed for an evaluation run. This includes corpus, quering the annotator and mappings, lenses and filters used by monocle. The aggregation settings specify what service, dataset and what lenses, mappings and filters should be used.aggregation:service:name:aidalocation:webinput:data_set:name:rss1lenses:-3.5-entity_list_en.txt-14dec-0130pmmappings:-redirects-v2.json-15dec-1121amfilters:-us_states_list_en-txt-12_jan_28-0913amThe service section of the yaml config specifies the name of the web service (annotation service). This should be the same (written the same) as the webservice plugin minus theorbis_plugin_aggregation_prefix.Location specifies where the annotations should come from. If it's set to web, then the aggregation plugin will attemt to query the webservice. If location is set to local, then the local cache (located in~/orbis-eval/data/corpora/{corpus_name}/copmuted/{annotator_name}/) will be used assuming there is a cache to be used.
If there is no cache, run the evaluation in web mode and add- cache_webservice_resultsto the storage section to build a cache.aggregation:service:name:aidalocation:webThe input section defines what corpus should be used (in the example rss1). The corpora name should be written the same as the corpus folder located in~/orbis-eval/data/corpora/.
Orbis will locate from there on automatically the corpus texts and the gold standard.input:data_set:name:rss1lenses:-3.5 -entity_list_en.txt-14dec-0130pmmappings:-redirects-v2.json-15dec-1121amfilters:-us_states_list_en-txt-12_jan_28-0913amIf needed, the lenses, mappings and filters can also be specified in the input section. These should be located in~/orbis-eval/data/[filters|lenses|mappings]and should be specified in the section without the file ending.EvaluationThe evaluator stage evaluates the the annotator results against the gold standard. The evaluation section defines what kind of evaluation should be used. The evaluator should have the same name as the evaluation plugin minus theorbis_plugin_evaluation_prefix.evaluation:name:binary_classification_evaluationScoringThe scoring stage scores the evaluation according to specified conditions. These conditions are preset in the scorer and can be specified in the scoring section as well as what entity types should be scored. If no entity type is defined, all are scored. If one or more entity types are defined, then only those will be scored. Additionallyignore_emptycan be set to define if the scorer should ignore empty annotation results or not.
The scorer should have the same name as the scoring plugin minus theorbis_plugin_scoring_prefix.scoring:name:nel_scorercondition:overlapentities:-Person-Organization-Placeignore_empty:FalseCurrently available conditions are:- simple:
    - same url
    - same entity type
    - same surface form

  - strict:
    - same url
    - same entity type
    - same surface form
    - same start
    - same end

  - overlap:
    - same url
    - same entity type
    - overlapMetricsThe metrics stage calculates the metrics to analyze the evaluation. The metric should have the same name as the metrics plugin minus theorbis_plugin_metrics_prefix.metrics:name:binary_classification_metricsStorageThe storage stage defines what kind of output orbis should create. As allways, the storage should have the same name as the storage plugin minus theorbis_plugin_storage_prefix.storage:-cache_webservice_results-csv_result_list-html_pagesMultiple storage options can be chosen and the ones in the example above are the recomended (at the moment working) possibilities.Orbis addons can be called directly by appending the Addon name the orbis-addon command:orbis-addon repomanDatasetsFor NER/NEL tasks, evaluation datasets need to be in the NIF format. If this is not the case, feel free to use the following converter packages:nifconverter- Pythonpynif- PythonNIF-lib- Java - the original NIF libraryLocal Development (Pycharm)Create a new project folder.mkdirOrbisClone orbis-eval in the newly created folder.cdOrbis
gitclonehttps://github.com/orbis-eval/orbis_eval.gitOpen orbis-eval as a new project in PycharmFile->OpenExecute the script clone_plugins.shcdorbis_eval
./clone_plugins.shAttach all downloaded plugin/addon to the project in PycharmFile->OpenFor every additional plugin/addon of your orbis-eval run-configuration file:Clone the repository into the folder created in step 1.Attach the project to your orbis-eval project.In Pycharm, go to File->Settings->Projects Dependencies. Select all plugins/addons as dependencies of orbis-eval. For every plugin/addon select orbis-eval as dependency.In Pycharm, go to File->Settings->Projects Interpreter. Create a new Python interpreter within the project folder created in step 1 (You can use an existing intrerpreter as well). Make the interpreter available for all projects.Verify that all projects use this newly created interpreter.Install all dependencies of orbis-eval and additional plugins/addons (each project contains a requirements.txt with all dependencies).Add a Pycharm run configuration pointing to the main file of orbis-eval (orbis-eval/orbis-eval/main.py).If necessary, run repoman to create your gold-documents (Pycharm run configuration: orbis-eval/orbis-eval/interfaces/addons/main.py).
Note: If you don't have any gold-documents yet, you can also use the sample files located in queue/tests in order to check that your installation is working (use the run configuration created in step 10 with -t as parameter)Run orbis-eval with the run configuration created in step 10.
Note: The first execution will create an orbis-eval folder on the location of your choice. This folder contains all files to run an evaluation. Within orbis-eval/queue/ create a folder "activated". Create a configuration file in this folder.