Package: natas
NATASThis library has methods for processing historical English corpora, especially for studying neologisms. The first functionalities  relate to normalization of historical spelling and OCR post-correction. This library is maintained byMika Hämäläinen.The normalization methods use Spacy for lemmatization, but they are not based on Spacy (regardless of whatever some ignorant people say online).InstallationNote: It is highly recommended to use a virtual environment because of the strict version requirements for dependencies. The library has been tested with Python 3.6pip3 install natas
python3 -m natas.download
python3 -m spacy download en_core_web_mdHistorical normalizationFor a list of non-modern spelling variants, the tool can produce an ordered list of the candidate normalizations. The candidates are ordered based on the prediction score of the NMT model.import natas
natas.normalize_words(["seacreat", "wiþe"])
>> [['secret', 'secrete'], ['with', 'withe', 'wide', 'white', 'way']]Possible keyword arguments are n_best=10, dictionary=None, all_candidates=True, correct_spelling_cache=True, return_scores=False.n_bestsets the number of candidates the NMT will outputdictionarysets a custom dictionary to be used to filter the NMT output (see more in the next section)all_candidates, if False, the method will return only the topmost normalization candidate (this will improve the speed of the method)correct_spelling_cache, used only when checking if a candidate word is correctly spelled. Set this to False if you are testing with multipledictionaries.return_scores, if True, returns the model's predictions scores together with the normalization candidates. For example [['secret', -1.0969021320343018], ['secrete', -4.121032238006592]]OCR post correctionYou can use our pretrained model for OCR post correction by doing the followingimport natas
natas.ocr_correct_words(["paft", "friendlhip"])
>> [['past', 'pall', 'part', 'part'], ['friendship']]This will return a list of possible correction candidates in the order of probability according to the NMT model. The same parameters can be used as for historical text normalization.Training your own OCR error correction modelYou can extract the parallel data for the OCR model if you have an access to a word embeddings model on your OCR data, a list of known correctly spelled words and a vocabulary of the language.from natas import ocr_builder
from natas.normalize import wiktionary
from gensim.models import Word2Vec

model = Word2Vec.load("/path/to/your_model.w2v")
seed_words = set(["logic", "logical"]) #list of correctly spelled words you want to find matching OCR errors for
dictionary = wiktionary #Lemmas of the English Wiktionary, you will need to change this if working with any other language
lemmatize = True #Uses Spacy with English model, use natas.set_spacy(nlp) for other models and languages

results = ocr_builder.extract_parallel(seed_words, model, dictionary=dictionary, lemmatize=lemmatize, use_freq=False)
>> {"logic": {
    "fyle": 5, 
    "ityle": 5, 
    "lofophy": 5, 
    "logick": 1
}, 
"logical": {
    "lofophy": 5, 
    "matical": 3, 
    "phical": 3, 
    "praaical": 4, 
    "pracical": 4, 
    "pratical": 4
}}The code results in a dictionary of correctly spelled English words (fromseed_words) and their mapping to semantically similar non-correctly spelled words (not indictionary). Each non-correct word has aLevenshtein distancecalculated with the correctly spelled word. In our paper, we used 3 as the maximum edit distance.Use the dictionary to make parallel data files for OpenNMT on a character level. This means splitting the words into letters, such asl o g i c k->l o g i c. Seetheir documentation on how to train the model.Check if a word is correctly spelledYou can check whether a word is correctly spelled easilyimport natas
natas.is_correctly_spelled("cat")
natas.is_correctly_spelled("ca7")
>> True
>> FalseThis will compare the word with Wiktionary lemmas with and without Spacy lemmatization. The normalization method depends on this step. By default,natasuses Spacy'sen_core_web_mdmodel. To change this model, do the followingimport natas, spacy
nlp = spacy.load('en')
natas.set_spacy(nlp)If you want to replace the Wiktionary dictionary with another one, it can be passed as a keyword argument. Usesetinstead oflistfor a faster look-up. Notice that the models operate on lowercased words.import natas
my_dictionary= set(["hat", "rat"])
natas.is_correctly_spelled("cat", dictionary=my_dictionary)
natas.normalize_words(["ratte"], dictionary=my_dictionary)By default, caching is enabled. If you want to use the method with multiple different parameters, you will need to setcache=False.import natas
natas.is_correctly_spelled("cat") #The word is looked up and the result cached
natas.is_correctly_spelled("cat") #The result will be served from the cache
natas.is_correctly_spelled("cat", cache=False) #The word will be looked up againBusiness solutionsNon-standard historical or OCRed data can be a mess to deal with when you want to squeeze all the juice out of your corpora. Let us help!Rootroo offers consulting related to non-standard data. We have a strong academic background in the state-of-the-art AI solutions for every NLP need. Just contact us, we won't bite.CiteIf you use the library, please cite one of the following publications depending on whether you used it for normalization or OCR correction.NormalizationMika Hämäläinen, Tanja Säily, Jack Rueter, Jörg Tiedemann, and Eetu Mäkelä. 2019.Revisiting NMT for Normalization of Early English Letters. InProceedings of the 3rd Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature.@inproceedings{hamalainen-etal-2019-revisiting,
title = "Revisiting {NMT} for Normalization of Early {E}nglish Letters",
author = {H{\"a}m{\"a}l{\"a}inen, Mika  and
  S{\"a}ily, Tanja  and
  Rueter, Jack  and
  Tiedemann, J{\"o}rg  and
  M{\"a}kel{\"a}, Eetu},
booktitle = "Proceedings of the 3rd Joint {SIGHUM} Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature",
month = jun,
year = "2019",
address = "Minneapolis, USA",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/W19-2509",
doi = "10.18653/v1/W19-2509",
pages = "71--75",
abstract = "This paper studies the use of NMT (neural machine translation) as a normalization method for an early English letter corpus. The corpus has previously been normalized so that only less frequent deviant forms are left out without normalization. This paper discusses different methods for improving the normalization of these deviant forms by using different approaches. Adding features to the training data is found to be unhelpful, but using a lexicographical resource to filter the top candidates produced by the NMT model together with lemmatization improves results.",
}OCR correctionMika Hämäläinen, and Simon Hengchen. 2019.From the Paft to the Fiiture: a Fully Automatic NMT and Word Embeddings Method for OCR Post-Correction. Inthe Proceedings of Recent Advances in Natural Language Processing.@inproceedings{hamalainen-hengchen-2019-paft,
title = "From the Paft to the Fiiture: a Fully Automatic {NMT} and Word Embeddings Method for {OCR} Post-Correction",
author = {H{\"a}m{\"a}l{\"a}inen, Mika  and
  Hengchen, Simon},
booktitle = "Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)",
month = sep,
year = "2019",
address = "Varna, Bulgaria",
publisher = "INCOMA Ltd.",
url = "https://www.aclweb.org/anthology/R19-1051",
doi = "10.26615/978-954-452-056-4_051",
pages = "431--436",
abstract = "A great deal of historical corpora suffer from errors introduced by the OCR (optical character recognition) methods used in the digitization process. Correcting these errors manually is a time-consuming process and a great part of the automatic approaches have been relying on rules or supervised machine learning. We present a fully automatic unsupervised way of extracting parallel data for training a character-based sequence-to-sequence NMT (neural machine translation) model to conduct OCR error correction.",
}