Package: opuscleaner
OpusCleanerOpusCleaner is a machine translation/language model data cleaner and training scheduler. The Training scheduler has moved toOpusTrainer.CleanerThe cleaner bit takes care of downloading and cleaning multiple different datasets and preparing them for translation.opuscleaner-clean--parallel4data/train-parts/dataset.filter.json|gzip-c>clean.gzInstallation for cleaningIf you just want to use OpusCleaner for cleaning, you can install it from PyPI, and then run itpip3installopuscleaner
opuscleaner-serverserveThen you can go tohttp://127.0.0.1:8000/to show the interface.You can also install and run OpusCleaner on a remote machine, and useSSH local forwarding(e.g.ssh -L 8000:localhost:8000 you@remote.machine) to access the interface on your local machine.Dependencies(Mainly listed as shortcuts to documentation)FastAPIas the base for the backend part.Pydanticfor conversion of untyped JSON to typed objects. And because FastAPI automatically supports it and gives you useful error messages if you mess up things.Vuefor frontendScreenshotsList and categorize the datasets you are going to use for training.Download more datasets right from the interface.Filter each individual dataset, showing you the results immediately.Compare the dataset at different stages of filtering to see what the impact is of each filter.Using your own dataOpusCleaner scans for datasets and finds them automatically if they're in the right format. When you download OPUS data, it will get converted to this format, and there's nothing stopping you from adding your own in the same format.By default, it scans for files matchingdata/train-parts/*.*.gzand will derive which files make up a dataset from the filenames:name.en.gzandname.de.gzwill be a dataset calledname. The files are your standard moses format: a single sentence per line, and each Nth line in the first file will match with the Nth line of the second file.When in doubt, just download one of the OPUS datasets through OpusCleaner, and replicate the format for your own dataset.If you want to use another path, you can use theDATA_PATHenvironment variable to change it, e.g. runDATA_PATH="./my-datasets/*.*.gz" opuscleaner-server.Pathsdata/train-partsis scanned for datasets. You can change this by setting theDATA_PATHenvironment variable, the default isdata/train-parts/*.*.gz.filtersshould contain filter json files. You can change theFILTER_PATHenvironment variable, the default is<PYTHON_PACKAGE>/filters/*.json.Installation for developmentcdfrontend
npmclean-install
npmrunbuildcd..

python3-mvenv.env
bash--init-file.env/bin/activate
pipinstall-e.Finally you can runopuscleaner-serveras normal. The--reloadoption will cause it to restart when any of the python files change.opuscleaner-serverserve--reloadThen go tohttp://127.0.0.1:8000/for the "interface" orhttp://127.0.0.1:8000/docsfor the API.Frontend developmentIf you're doing frontend development, try also running:cdfrontend
npmrundevThen go tohttp://127.0.0.1:5173/for the "interface".This will put vite in hot-reloading mode for easier Javascript dev. All API requests will be proxied to the python server running in 8000, which is why you need to run both at the same time.FiltersIf you want to use LASER, you will also need to download its assets:python-mlaserembeddingsdownload-modelsPackagingRunnpm buildin thefrontend/directory first, and then runhatch build .in the project directory to build the wheel and source distribution.To push a new release to Pypi from Github, tag a commit with avX.Y.Zversion number (including thevprefix). Then publish a release on Github. This should trigger a workflow that pushes a sdist + wheel to pypi.AcknowledgementsThis project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]