Package: openkiwi
Open-Source Machine Translation Quality Estimation in PyTorchQuality estimation (QE) is one of the missing pieces of machine translation: its goal is to evaluate a translation system’s quality without access to reference translations. We presentOpenKiwi, a Pytorch-based open-source framework that implements the best QE systems from WMT 2015-18 shared tasks, making it easy to experiment with these models under the same framework. Using OpenKiwi and a stacked combination of these models we have achieved state-of-the-art results on word-level QE on the WMT 2018 English-German dataset.NewsAn experimental demonstration interface called OpenKiwi Tasting has been released onGitHuband can be checked out inStreamlit Share.A new major version (2.0.0) of OpenKiwi has been released. Introducing HuggingFace Transformers support and adoption of Pytorch-lightning.
For a condensed view of changed, check thechangelogFollowing our nomination in early July, we are happy to announce we won theBest Demo Paper at ACL 2019! Congratulations to the whole team and huge thanks for supporters and issue reporters.Check out thepublished paper.We have released the OpenKiwitutorialwe presented at MT Marathon 2019.FeaturesFramework for training QE models and using pre-trained models for evaluating MT.Supports both word and sentence-level (HTER or z-score) Quality estimation.Implementation of five QE systems in Pytorch: NuQE [2,3], predictor-estimator [4,5], BERT-Estimator [6], XLM-Estimator [6] and XLMR-EstimatorOlder systems only supported in versions <=2.0.0: QUETCH [1], APE-QE [3] and a stacked ensemble with a linear system [2,3].Easy to use API. Import it as a package in other projects or run from the command line.Easy to track and reproduce experiments via yaml configuration files.Based on Pytorch-Lightning making the code easier to scale, use and keep up-do-date with engineering advances.Implemented using HuggingFace Transformers library to allow easy access to state-of-the-art pre-trained models.Quick InstallationTo install OpenKiwi as a package, simply runpipinstallopenkiwiYou can nowimportkiwiinside your project or run in the command linekiwiOptionally, if you'd like to take advantage of ourMLflowintegration, simply install it in the same virtualenv as OpenKiwi:pipinstallopenkiwi[mlflow]Getting StartedDetailed usage examples and instructions can be found in theFull Documentation.ContributingWe welcome contributions to improve OpenKiwi.
Please refer toCONTRIBUTING.mdfor quick instructions or tocontributing instructionsfor more detailed instructions on how to set up your development environment.LicenseOpenKiwi is Affero GPL licensed. You can see the details of this license inLICENSE.CitationIf you use OpenKiwi, please cite the following paper:OpenKiwi: An Open Source Framework for Quality Estimation.@inproceedings{openkiwi,
    author = {Fábio Kepler and
              Jonay Trénous and
              Marcos Treviso and
              Miguel Vera and
              André F. T. Martins},
    title  = {Open{K}iwi: An Open Source Framework for Quality Estimation},
    year   = {2019},
    booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics--System Demonstrations},
    pages  = {117--122},
    month  = {July},
    address = {Florence, Italy},
    url    = {https://www.aclweb.org/anthology/P19-3020},
    organization = {Association for Computational Linguistics},
}References[1]Kreutzer et al. (2015): QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation[2]Martins et al. (2016): Unbabel's Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task[3]Martins et al. (2017): Pushing the Limits of Translation Quality Estimation[4]Kim et al. (2017): Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation[5]Wang et al. (2018): Alibaba Submission for WMT18 Quality Estimation Task[6]Kepler et al. (2019): Unbabel’s Participation in the WMT19 Translation Quality Estimation Shared Task