Evaluating State‐of‐the‐Art Handwritten Text Recognition (HTR) Engines; with Large Language Models (LLMs) for Historical Document Digitisation CRediT (Contributor Roles Taxonomy) Conceptualization: AR; GL; CAR Data curation: AR; CAR; REPBUBLIC! Formal Analysis: GL; CAR Methodology: GL Software: GL; PS; AR; CAR Writing – original draft: GL; CAR Writing – review & editing: all. Conclusions: ‐ All used engines can handle challenges in the given data (small trainingset/difficult handwriting/abbreviations/stenography) and can be trained to improve performance. ‐ HTR+ outperforms Pylaia in Glagolitic, whereas Pylaia is better on Shorthand. A reason could be that HTR+ suffers from a too large subsampling of the image, so that the transcription does not fit into the probability matrix. ‐ There is a tradeoff between inference time and quality: HTR+ and Pylaia are very fast, whereas IDA and especially TrOCR are slower. Also the number of trainable parameters varies from 3M (HTR+, Pylaia) to 558M (TrOCR). ‐ While HTR+, Pylaia and IDA need good surrounding polygons, TrOCR is pre‐trained and fine‐ tuned on unmasked text lines and does not suffer from defective surrounding polygons. ‐ LMs always improve the three models. This cannot be applied for TrOCR because it always incorporates a LM. ‐ HTR+, Pylaia and IDA are trained from scratch. For IDA, it took about 48 hours on one GPU for the Republic dataset to reach the performance. Training Pylaia only takes 9 hours 22 minutes. ‐ TrOCR reaches very good performance even without fine‐tuning the Vision Transformer. Moreover, fine‐tuning on only several thousand lines is fast (e.,g., it took about 10h to fine‐ tune on the roughly 40k lines from the Republic dataset). ‐ The considerable CER improvement when analyzing the Glagolitic dataset and comparing base and alnum can be explained by the fact that the most difficult task for the models is the expansion of abbreviations containing parentheses “(” and “)”. Since they are filtered out in alnum, the CER goes down. Since TrOCR has a model structure that can connect multiple characters to one optical symbol, the TrOCR improvement is smaller. ‐ Easy training of Pylaia using Transkribus (no programming skills needed), TrOCR is freely available with programming skills, IDA closed source of a company. TrOCR and IDA are trained manually by IT experts, but no hyperparameter optimization has been applied. Methods: Engines: The engines can be separated into three technologies: ‐ Convolutional+LSTM architecture (approx. 3M trainable weights): HTR+, Pylaia. ‐ Convolutional+Conformer architecture (approx. 46M trainable weights): IDA. ‐ Vision‐Transformer+LLM‐Transformer architecture (approx. 558M trainable weights): TrOCR. Since TrOCR uses a Transformer‐Decoder, it directly generates a token/character sequence. Being pre‐trained on hundreds of millions of (synthetic) line images containing English text, it incorporates a strong English language model (LM). All engines except TrOCR are trained using CTC.1 This results in a probability matrix which enables the opportunity to apply an additional/external LM. The LM is trained on pure text, which in our case is the transcription in the training set. This mainly improves the word error rate, but also the character error rate (CER). Comparison: To compare the different HTR engines, we decided to mainly focus on the CER for now. In general for each text line we calculate the Levenshtein‐Distance between hypothesis and ground truth and sum up over all lines. Dividing these errors by the total number of all ground truth characters, results in the CER. Due to the different approaches of the HTR engines, they can have different strengths and weaknesses regarding the characters they can recognize. We decided to evaluate the CER by filtering the ground truth and hypothesis characters to find out these strengths and weaknesses. For simplification we filter by Unicode character categories2. As general normalization we map “