Package: ocrd-keraslm
ocrd_keraslmcharacter-level language modelling using KerasIntroductionThis is a tool for statisticallanguage modelling(predicting text from context) with recurrent neural networks. It models probabilities not on the word level but thecharacter levelso as to allow open vocabulary processing (avoiding morphology, historic orthography and word segmentation problems). It manages a vocabulary of mapped characters, which can be easily extended by training on more text. Above that, unmapped characters are treated with underspecification.In addition to character sequences, (meta-data) context variables can be configured as extra input.ArchitectureThe model consists of:an input layer: characters are represented as indexes from the vocabulary mapping, in windows of a numberlengthof characters,a character embedding layer: window sequences are converted into dense vectors by looking up the indexes in an embedding weight matrix,a context embedding layer: context variables are converted into dense vectors by looking up the indexes in an embedding weight matrix,character and context vector sequences are concatenated,a numberdepthof hidden layers: each with a numberwidthof hidden recurrent units ofLSTM cells(Long Short-term Memory) connected on top of each other,an output layer derived from the transposed character embedding matrix (weight tying): hidden activations are projected linearly to vectors of dimensionality equal to the character vocabulary size, then softmax is applied returning a probability for each possible value of the next character, respectively.The model is trained by feeding windows of text in index representation to the input layer, calculating output and comparing it to the same text shifted backward by 1 character, and represented as unit vectors ("one-hot coding") as target. The loss is calculated as the (unweighted) cross-entropy between target and output. Backpropagation yields error gradients for each layer, which is used to iteratively update the weights (stochastic gradient descent).This is implemented inKeraswithTensorflowas backend. It automatically uses a fast CUDA-optimized LSTM implementation (Nividia GPU and Tensorflow installation with GPU support, see below), both in learning and in prediction phase, if available.Modes of operationNotably, this model (by default) runsstatefully, i.e. by implicitly passing hidden state from one window (batch of samples) to the next. That way, the context available for predictions can be arbitrarily long (abovelength, e.g. the complete document up to that point), or short (belowlength, e.g. at the start of a text). (However, this is a passive perspective abovelength, because errors are never back-propagated any further in time during gradient-descent training.) This is favourable to stateless mode because all characters can be output in parallel, and no partial windows need to be presented during training (which slows down).Besides stateful mode, the model can also be runincrementally, i.e. by explicitly passing hidden state from the caller. That way, multiple alternative hypotheses can be processed together. This is used for generation (sampling from the model) and alternative decoding (finding the best path through a sequence of alternatives).Context conditioningEvery text has meta-data like time, author, text type, genre, production features (e.g. print vs typewriter vs digital born rich text, OCR version), language, structural element (e.g. title vs heading vs paragraph vs footer vs marginalia), font family (e.g. Antiqua vs Fraktura) and font shape (e.g. bold vs letter-spaced vs italic vs normal) etc.This information (however noisy) can be very useful to facilitate stochastic modelling, since language has an extreme diversity and complexity. To that end, models can be conditioned on extra inputs here, termedcontext variables. The model learns to represent these high-dimensional discrete values as low-dimensional continuous vectors (embeddings), also entering the recurrent hidden layers (as a form of simple additive adaptation).UnderspecificationIndex zero is reserved for unmapped characters (unseen contexts). During training, its embedding vector is regularised to occupy a center position of all mapped characters (all other contexts), and the hidden layers get to see it every now and then by random degradation. At runtime, therefore, some unknown character (some unknown context) represented as zero does not disturb follow-up predictions too much.InstallationRequired Ubuntu packages:Python (pythonorpython3)pip (python-piporpython3-pip)virtualenv (python-virtualenvorpython3-virtualenv)Create and activate a virtualenv as usual.If you need a custom version ofkerasortensorflow(likeGPU support), install them viapipnow.To install Python dependencies and this module, then do:makedepsinstallWhich is the equivalent of:pipinstall-rrequirements.txt
pipinstall-e.Useful environment variables are:TF_CPP_MIN_LOG_LEVEL(set to1to suppress most of Tensorflow's messagesCUDA_VISIBLE_DEVICES(set empty to force CPU even in a GPU installation)UsageThis packages has two user interfaces:command line interfacekeraslm-rateTo be used with string arguments and plain-text files.Usage:keraslm-rate[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:traintrainalanguagemodeltestgetoverallperplexityfromlanguagemodelapplygetindividualprobabilitiesfromlanguagemodelgeneratesamplecharactersfromlanguagemodelprint-charsetPrintthemappedcharactersprune-charsetDeleteonecharacterfrommappingplot-char-embeddings-similarityPaintaheatmapofcharacterembeddingsplot-context-embeddings-similarityPaintaheatmapofcontextembeddingsplot-context-embeddings-projectionPainta2-dPCAprojectionofcontextembeddingsExamples:keraslm-ratetrain--width64--depth4--length256--modelmodel_dta_64_4_256.h5dta_komplett_2017-09-01/txt/*.tcf.txt
keraslm-rategenerate-mmodel_dta_64_4_256.h5--number6"für die Wiſſen"keraslm-rateapply-mmodel_dta_64_4_256.h5"so schädlich ist es Borkickheile zu pflanzen"keraslm-ratetest-mmodel_dta_64_4_256.h5dta_komplett_2017-09-01/txt/grimm_*.tcf.txtOCR-D processorinterfaceocrd-keraslm-rateTo be used withPageXMLdocuments in anOCR-Dannotation workflow. Input could be anything with a textual annotation (TextEquivon the giventextequiv_level). The LM rater could be used for both quality control (without alternative decoding, using only each first indexTextEquiv) and part of post-correction (withalternative_decoding=True, finding the best path amongTextEquivindexes)."tools":{"ocrd-keraslm-rate":{"executable":"ocrd-keraslm-rate","categories":["Text recognition and optimization"],"steps":["recognition/text-recognition"],"description":"Rate elements of the text with a character-level LSTM language model in Keras","input_file_grp":["OCR-D-OCR-TESS","OCR-D-OCR-KRAK","OCR-D-OCR-OCRO","OCR-D-OCR-CALA","OCR-D-OCR-ANY","OCR-D-COR-CIS","OCR-D-COR-ASV"],"output_file_grp":["OCR-D-COR-LM"],"parameters":{"model_file":{"type":"string","format":"uri","content-type":"application/x-hdf;subtype=bag","description":"path of h5py weight/config file for model trained with keraslm","required":true,"cacheable":true},"textequiv_level":{"type":"string","enum":["region","line","word","glyph"],"default":"glyph","description":"PAGE XML hierarchy level to evaluate TextEquiv sequences on"},"alternative_decoding":{"type":"boolean","description":"whether to process all TextEquiv alternatives, finding the best path via beam search, and delete each non-best alternative","default":true},"beam_width":{"type":"number","format":"integer","description":"maximum number of best partial paths to consider during search with alternative_decoding","default":100}}}}Examples:makedeps-test# installs ocrd_tesserocrmaketest/assets# downloads GT, imports PageXML, builds workspacesocrdworkspaceclone-atest/assets/kant_aufklaerung_1784/mets.xmlws1cdws1
ocrd-tesserocr-segment-region-IOCR-D-IMG-OOCR-D-SEG-BLOCK
ocrd-tesserocr-segment-line-IOCR-D-SEG-BLOCK-OOCR-D-SEG-LINE
ocrd-tesserocr-recognize-IOCR-D-SEG-LINE-OOCR-D-OCR-TESS-WORD-p'{ "textequiv_level" : "word", "model" : "Fraktur" }'ocrd-tesserocr-recognize-IOCR-D-SEG-LINE-OOCR-D-OCR-TESS-GLYPH-p'{ "textequiv_level" : "glyph", "model" : "deu-frak" }'# get confidences and perplexity:ocrd-keraslm-rate-IOCR-D-OCR-TESS-WORD-OOCR-D-OCR-LM-WORD-p'{ "model_file": "model_dta_64_4_256.h5", "textequiv_level": "word", "alternative_decoding": false }'# also get best path:ocrd-keraslm-rate-IOCR-D-OCR-TESS-GLYPH-OOCR-D-OCR-LM-GLYPH-p'{ "model_file": "model_dta_64_4_256.h5", "textequiv_level": "glyph", "alternative_decoding": true, "beam_width": 10 }'Testingmakedeps-testtestWhich is the equivalent of:pipinstall-rrequirements_test.txttest-etest/assets||test/prepare_gt.bashtest/assetstest-fmodel_dta_test.h5||keraslm-ratetrain-mmodel_dta_test.h5test/assets/*.txt
keraslm-ratetest-mmodel_dta_test.h5test/assets/*.txt
python-mpytesttest$(PYTEST_ARGS)SetPYTEST_ARGS="-s --verbose"to see log output (-s) and individual test results (--verbose).