Package: contextualSpellCheck
spellCheckContextual word checker for better suggestionsTypes of spelling mistakesIt is essential to understand that identifying whether a candidate is a spelling error is a big task.Spelling errors are broadly classified as non- word errors (NWE) and real word errors (RWE). If the misspelt string is a valid word in the language, then it is called an RWE, else it is an NWE.--Monojit Choudhury et. al. (2007)This package currently focuses on Out of Vocabulary (OOV) word or non-word error (NWE) correction using BERT model. The idea of using BERT was to use the context when correcting OOV. To improve this package, I would like to extend the functionality to identify RWE, optimising the package, and improving the documentation.InstallThe package can be installed usingpip. You would require python 3.6+pipinstallcontextualSpellCheckUsageNote:For use in other languages checkexamplesfolder.How to load the package in spacy pipeline>>>importcontextualSpellCheck>>>importspacy>>>nlp=spacy.load("en_core_web_sm")>>>>>>## We require NER to identify if a token is a PERSON>>>## also require parser because we use `Token.sent` for context>>>nlp.pipe_names['tok2vec','tagger','parser','ner','attribute_ruler','lemmatizer']>>>contextualSpellCheck.add_to_pipe(nlp)>>>nlp.pipe_names['tok2vec','tagger','parser','ner','attribute_ruler','lemmatizer','contextual spellchecker']>>>>>>doc=nlp('Income was $9.4 milion compared to the prior year of $2.7 milion.')>>>doc._.outcome_spellCheck'Income was $9.4 million compared to the prior year of $2.7 million.'Or you can add to spaCy pipeline manually!>>>importspacy>>>importcontextualSpellCheck>>>>>>nlp=spacy.load("en_core_web_sm")>>>nlp.pipe_names['tok2vec','tagger','parser','ner','attribute_ruler','lemmatizer']>>># You can pass the optional parameters to the contextualSpellCheck>>># eg. pass max edit distance use config={"max_edit_dist": 3}>>>nlp.add_pipe("contextual spellchecker")<contextualSpellCheck.contextualSpellCheck.ContextualSpellCheckobjectat0x1049f82b0>>>>nlp.pipe_names['tok2vec','tagger','parser','ner','attribute_ruler','lemmatizer','contextual spellchecker']>>>>>>doc=nlp("Income was $9.4 milion compared to the prior year of $2.7 milion.")>>>print(doc._.performed_spellCheck)True>>>print(doc._.outcome_spellCheck)Incomewas$9.4millioncomparedtotheprioryearof$2.7million.After addingcontextual spellcheckerin the pipeline, you use the pipeline normally. The spell check suggestions and other data can be accessed usingextensions.Using the pipeline>>>doc=nlp(u'Income was $9.4 milion compared to the prior year of $2.7 milion.')>>>>>># Doc Extention>>>print(doc._.contextual_spellCheck)True>>>print(doc._.performed_spellCheck)True>>>print(doc._.suggestions_spellCheck){milion:'million',milion:'million'}>>>print(doc._.outcome_spellCheck)Incomewas$9.4millioncomparedtotheprioryearof$2.7million.>>>print(doc._.score_spellCheck){milion:[('million',0.59422),('billion',0.24349),(',',0.08809),('trillion',0.01835),('Million',0.00826),('%',0.00672),('##M',0.00591),('annually',0.0038),('##B',0.00205),('USD',0.00113)],milion:[('billion',0.65934),('million',0.26185),('trillion',0.05391),('##M',0.0051),('Million',0.00425),('##B',0.00268),('USD',0.00153),('##b',0.00077),('millions',0.00059),('%',0.00041)]}>>>>>># Token Extention>>>print(doc[4]._.get_require_spellCheck)True>>>print(doc[4]._.get_suggestion_spellCheck)'million'>>>print(doc[4]._.score_spellCheck)[('million',0.59422),('billion',0.24349),(',',0.08809),('trillion',0.01835),('Million',0.00826),('%',0.00672),('##M',0.00591),('annually',0.0038),('##B',0.00205),('USD',0.00113)]>>>>>># Span Extention>>>print(doc[2:6]._.get_has_spellCheck)True>>>print(doc[2:6]._.score_spellCheck){$:[],9.4:[],milion:[('million',0.59422),('billion',0.24349),(',',0.08809),('trillion',0.01835),('Million',0.00826),('%',0.00672),('##M',0.00591),('annually',0.0038),('##B',0.00205),('USD',0.00113)],compared:[]}ExtensionsTo make the usage easy,contextual spellcheckerprovides custom spacy extensions which your code can consume. This makes it easier for the user to get the desired data. contextualSpellCheck provides extensions on thedoc,spanandtokenlevel. The below tables summarise the extensions.spaCy.Doclevel extensionsExtensionTypeDescriptionDefaultdoc._.contextual_spellCheckBooleanTo check whether contextualSpellCheck is added as extensionTruedoc._.performed_spellCheckBooleanTo check whether contextualSpellCheck identified any misspells and performed correctionFalsedoc._.suggestions_spellCheck{Spacy.Token:str}if corrections are performed, it returns the mapping of misspell token (spaCy.Token) with suggested word(str){}doc._.outcome_spellCheckstrcorrected sentence(str) as output""doc._.score_spellCheck{Spacy.Token:List(str,float)}if corrections are identified, it returns the mapping of misspell token (spaCy.Token) with suggested words(str) and probability of that correctionNonespaCy.Spanlevel extensionsExtensionTypeDescriptionDefaultspan._.get_has_spellCheckBooleanTo check whether contextualSpellCheck identified any misspells and performed correction in this spanFalsespan._.score_spellCheck{Spacy.Token:List(str,float)}if corrections are identified, it returns the mapping of misspell token (spaCy.Token) with suggested words(str) and probability of that correction for tokens in thisspan{spaCy.Token: []}spaCy.Tokenlevel extensionsExtensionTypeDescriptionDefaulttoken._.get_require_spellCheckBooleanTo check whether contextualSpellCheck identified any misspells and performed correction on thistokenFalsetoken._.get_suggestion_spellCheckstrif corrections are performed, it returns the suggested word(str)""token._.score_spellCheck[(str,float)]if corrections are identified, it returns suggested words(str) and probability(float) of that correction[]APIAt present, there is a simple GET API to get you started. You can run the app in your local and play with it.Query: You can use the endpointhttp://127.0.0.1:5000/?query=YOUR-QUERYNote: Your browser can handle the text encodingGET Request: http://localhost:5000/?query=Income%20was%20$9.4%20milion%20compared%20to%20the%20prior%20year%20of%20$2.7%20milion.Response:{"success":true,"input":"Income was $9.4 milion compared to the prior year of $2.7 milion.","corrected":"Income was $9.4 milion compared to the prior year of $2.7 milion.","suggestion_score":{"milion":[["million",0.59422],["billion",0.24349],...],"milion:1":[["billion",0.65934],["million",0.26185],...]}}Task Listuse cython for part of the code to improve performance (#39)Improve metric for candidate selection (#40)Add examples for other langauges (#41)Update the logic of misspell identification (OOV) (#44)better candidate generation (solved by#44?)add metric by testing on datasetsImprove documentationImprove logging in codeAdd support for Real Word Error (RWE) (Big Task)add multi mask out capabilityCompleted Taskspecify maximum edit distance forcandidateRankingallow user to specify bert modelInclude transformers deTokenizer to get better suggestionsdependency version in setup.py (#38)Support and contributionIf you like the project, please ⭑ the project and show your support! Also, if you feel, the current behaviour is not as expected, please feel free to raise anissue. If you can help with any of the above tasks, please open aPRwith necessary changes to documentation and tests.CiteIf you are using contextualSpellCheck in your academic work, please consider citing the library using the below BibTex entry:@misc{Goel_Contextual_Spell_Check_2021,author={Goel, Rajat},doi={10.5281/zenodo.4642379},month={3},title={{Contextual Spell Check}},url={https://github.com/R1j1t/contextualSpellCheck},year={2021}}ReferenceBelow are some of the projects/work I referred to while developing this packageExplosion AI.Architecture. May 2020. url:https://spacy.io/api.Monojit Choudhury et al. “How difficult is it to develop a perfect spell-checker? A cross-linguistic analysis through complex network approach”. In:arXiv preprint physics/0703198(2007).Jacob Devlin et al. BERT: Pre-training of Deep Bidirectional Transform-ers for Language Understanding. 2019. arXiv:1810.04805 [cs.CL].Hugging  Face.Fast Coreference Resolution in spaCy with Neural Net-works. May 2020. url:https://github.com/huggingface/neuralcoref.Ines.Chapter 3: Processing Pipelines. May 20202. url:https://course.spacy.io/en/chapter3.Eric Mays, Fred J Damerau, and Robert L Mercer. “Context based spellingcorrection”. In:Information Processing & Management27.5 (1991), pp. 517–522.Peter Norvig. How to Write a Spelling Corrector. May 2020. url:http://norvig.com/spell-correct.html.Yifu  Sun  and  Haoming  Jiang.Contextual Text Denoising with MaskedLanguage Models. 2019. arXiv:1910.14080 [cs.CL].Thomas  Wolf  et  al.  “Transformers:  State-of-the-Art  Natural  LanguageProcessing”. In:Proceedings of the 2020 Conference on Empirical Methodsin Natural Language Processing: System Demonstrations. Online: Associ-ation for Computational Linguistics, Oct. 2020, pp. 38–45. url:https://www.aclweb.org/anthology/2020.emnlp-demos.6.