Package: qualle
QualleThis is an implementation of the Qualle framework as proposed in the paper
[1] and accompanying source code.The framework allows to train a model which can be used to predict
the quality of the result of applying a multi-label classification (MLC)
method on a document. In this implementation, only therecallis predicted for a document, but in principle
any document-level quality estimation (such as the prediction of precision)
can be implemented analogously.Qualle provides a command-line interface to train
and evaluate models. In addition, a REST webservice for predicting
the recall of a MLC result is provided.RequirementsPython>= 3.8is required.InstallationChoose one of these installation methods:With pipQualle is available onPyPI. You can install Qualle using pip:pip install qualleThis will install a command line tool calledqualle. You can callqualle -hto see the help message which will
display the available modes and options.Note that it is generally recommended to use avirtual environmentto avoid
conflicting behaviour with the system package manager.From sourceYou also have the option to checkout the repository and install the packages from source. You needpoetryto perform the task:# call inside the project directorypoetryinstall--withoutciDockerYou can also use a Docker Image from theContainer Registry of Github:docker pull ghcr.io/zbw/qualleAlternatively, you can use the Dockerfile included in this project to build a Docker image yourself. E.g.:docker build -t qualle .By default, a container built from this image launches a REST interface listening on0.0.0.0:8000You need to pass the model file (see below the section REST interface) per bind mount or volume to the docker container.
Beyond that, you need to specify the location of the model file with an
environment variable namedMODEL_FILE:docker run --rm -it --env MODEL_FILE=/model -v /path/to/model:/model -p 8000:8000 ghcr.io/zbw/qualleGunicornis used as HTTP Server. You can use the environment variableGUNICORN_CMD_ARGSto customize
Gunicorn settings, such as the number of worker processes to use:docker run --rm -it --env MODEL_FILE=/model --env GUNICORN_CMD_ARGS="--workers 4" -v /path/to/model:/model -p 8000:8000 ghcr.io/zbw/qualleYou can also use the Docker image to train or evaluate by using the Qualle command line tool:dockerrun--rm-it-v\/path/to/train_data_file:/train_data_file/path/to/model:/modelghcr.io/zbw/qualle\qualletrain/train_data_file/modelThe Qualle command line tool is not available for the release 0.1.0 and 0.1.1. For these releases,
you need to call the python modulequalle.maininstead:dockerrun--rm-it-v\/path/to/train_data_file:/train_data_file/path/to/model:/modelghcr.io/zbw/qualle:0.1.1\python-mqualle.maintrain/train_data_file/modelUsageInput dataIn order to train a model, evaluate a model or predict the quality of an MLC result
you have to provide data.This can be a tabular-separated file (tsv) in the format (tabular is marked with\t)document-content\tpredicted_labels_with_scores\ttrue_labelswheredocument-contentis a string describing the content of the document
(more precisely: the string on which the MLC method is trained), e.g. the titlepredicted_labels_with_scoresis a comma-separated list of pairspredicted_label:confidence-score(this is basically the output of the MLC method)true_labelsis a comma-separated list of true labels (ground truth)Note that you can omit thetrue_labelssection if you only want to predict the
quality of the MLC result.For example, a row in the data file could look like this:Optimal investment policy of the regulated firm\tConcept0:0.5,Concept1:1\tConcept0,Concept3For those who use an MLC method via the toolkitAnniffor automated subject indexing:
You can alternatively specify afull-text document corpuscombined with the result of the Annif index method (tested with Annif version 0.59) applied on the corpus.
This is a  folder with three files per document:doc.annif: result of Annif index methoddoc.tsv: ground truthdoc.txt: document contentAs above, you may omit thedoc.tsvif you just want to
predict the quality of the MLC result.TrainTo train a model, use thetrainmode, e.g.:qualle train /path/to/train_data_file /path/to/output/modelIt is also possible to use label calibration (comparison of predicted vs actual labels) using the subthesauri of a thesaurus (such as theSTW)
as categories (please read the paper for more explanations). Consult the help (see above) for the required options.EvaluateYou must provide test data and the path to a trained model in order to evaluate that model. Metrics
such as theexplained variationare printed out, describing the quality
of the recall prediction (please consult the paper for more information).REST interfaceTo perform the prediction on a MLC result, a REST interface can be started.uvicornis used as HTTP server. You can also use any
ASGI server implementation and create the ASGI app directly with the methodqualle.interface.rest.create_app. You need to provide the environment variable
MODEL_FILE with the path to the model (seequalle.interface.config.RESTSettings).The REST endpoint expects a HTTP POST with the result of a MLC for a list of documents
as body. The format is JSON as specified inqualle/openapi.json. You can also use
the Swagger UI accessible athttp://address_of_server/docsto play around a bit.ContributeContributions via pull requests are welcome. Please create an issue beforehand
to explain and discuss the reasons for the respective contribution.qualle code should follow theBlack style.
The Black tool is included as a development dependency; you can runblack .in the project root to autoformat code.References[1]Toepfer, Martin, and Christin Seifert. "Content-based quality estimation for automatic subject indexing of short texts under precision and recall constraints." International Conference on Theory and Practice of Digital Libraries. Springer, Cham, 2018., DOI 10.1007/978-3-030-00066-0_1Context informationThis code was created as part of the subject indexing automatization effort atZBW - Leibniz Information Centre for Economics. Seeour homepagefor more information, publications, and contact details.