Package: medspellchecker
MedSpellCheckerFast and effective tool for correcting spelling errors in Russian medical texts.
The tool takes the raw medical text and returns the corrected text in lemmatized form.This project is in its final stages.
If you have found a bug or would like to improve something, please create an issue or pull request.How to installThe project is written in Python 3.9 and works on Python version 3.9 or higher.From thepip repositorypipinstallmedspellcheckerOr from the sourcesInstall requirementspipinstall-rrequirements.txtInstall the packagepipinstall.Note!MedSpellChecker has theeditdistpypackage as a dependency.
To install it you will needMicrosoft Visual C++ 14(for
Windows)
orbuild-essentialpackage (for Linux).How to useAn example of using the `medspellchecker' is shown below.frommedspellchecker.tool.medspellcheckerimportMedSpellcheckerfrommedspellchecker.tool.distilbert_candidate_rankerimportRuDistilBertCandidateRankercandidate_ranker=RuDistilBertCandidateRanker()spellchecker=MedSpellchecker(candidate_ranker)fixed_text=spellchecker.fix_text("У больногодиагностирован инфркт и тубркулез")print(fixed_text)# -> "У больной диагностировать инфаркт и тубркулез"The first two lines import the main class of the package.Lines 3 and 4 then import one of the three available classes to rank the edit candidates.Line 6 creates an instance of the candidate ranking class based on the fine-tuned DistilBert model.Line 7 creates an instance of the class to correct spelling errors.On line 8, thefix_textmethod is called, which takes the raw text and returns the corrected text.Finally line 10 prints the corrected result, which looks like "У больной диагностировать инфаркт и тубркулез".In this way, the package can be used to correct spelling errors in Russian medical texts in a few lines.Tips for using the toolInitiate and create MedSpellchecker class once. And then use it many times to correct Russian medical texts.The tool must be used before all the preprocessing steps. This way the various preprocessing steps cannot affect the
result of the tool correction.Nevertheless, the input raw text must be cleared of all sorts of tags, technical symbols and other trash. The input
text should be plain text without any structure.Before using the tool on the whole dataset, test how it works on a small chunk to choose the best available candidate
ranker.If the available candidate rankers do not perform well enough, consider fine-tuning the BERT-based model on your
dataset and using the candidate ranker on the fine-tuned model. The quality of the corrections should improve.InternalsMedSpellCheckeruses the SymDel algorithm to speed up the generation of correction candidates,
and a fine-tuned BERT-based machine learning model to rank candidates and select the best fit.This architecture allows each component to be developed almost independently and
the correction process to be implemented flexibly.Spellchecker Manager- responsible for coordinating other components and implementing high-level logic.PreprocessorandPostProcessor- responsible for splitting the incoming text and assembling the result.Dictionary- contains a dictionary of correct words, which allows to check the correct word or not.Edit Distance Index- allows to optimize and speed up the calculation of the editing distance required to generate
candicates for fixing an incorrect word.Error Model- responsible for generating candidates for fixing incorrect words.Language Model- based on the fine-tuned BERT-based model, ranks candidates for fixing and selects the most
suitable word for correction.Supported Candidate RankersCandidate rankers rank the candidates to replace the incorrect word and choose the most appropriate one.The tool contains several rankers based on different BERT-based models.
However, the list of supported rankers can easily be extended with rankers based on any other approach and models.
It is sufficient to implement theAbstractCandidateRankerinterface.Available rankers out of the box:RuRobertaCandidateRankerbased
onDmitryPogrebnoy/MedRuRobertaLargemodelRuDistilBertCandidateRankerbased
onDmitryPogrebnoy/MedDistilBertBaseRuCasedmodelRuBertTiny2CandidateRankerbased
onDmitryPogrebnoy/MedRuBertTiny2modelRuBioRobertCandidateRankerbased onalexyalunin/RuBioRoBERTamodelRuBioBertCandidateRankerbased onalexyalunin/RuBioBERTmodelMore informationThis project is part of master's thesis. The current state is the result of the first year of work.
More details aboutMedSpellCHeckeryou can find in the text of theterm report.