# Investigating Models for the Transcription of Mathematical Formulas in Images
## 5 Results
### Comparing the Combination of MER and NLP Models with Vision-Language Models  
Figure 13 presents the TERs of the combinations of our MER models and our NLP models in comparison with the TERs of our VL models. Our goal was to discover which combination leads to the lowest TER for the complete pipeline of transcribing mathematical formulas in images. In the case of the combination of MER and NLP models, an NLP model receives as input the erroneous output of an MER model. The TER of the MER model _LaTex-OCR_ is 23.45% (_Variant 1_). The TER of the MER model _TrOCR-Base_ is 27.71% (_Variant 2_). The combinations of _LaTex-OCR_ and the investigated NLP models are colored blue, while the combinations of _TrOCR-Base_ and the investigated NLP models are colored in purple. In the case of the VL models, there are no combinations with other models and no intermediate steps that provide erroneous input.  
Figure 12: Results of the VL models.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **TER [\%]** & **TER [\%]** & **\(\Delta\) Improvement [\%]** \\  & **Pre-Trained** & **Fine-Tuned** & **Relative** \\ \hline TrOCR-Base & 86.08 & **42.09** & +104.51 \\ BLIP-Base & 83.06 & 52.18 & +59.18 \\ SWIN-GPT-2 & 91.12 & 72.91 & +24.98 \\ GIT-Large & 82.29 & 80.96 & +1.64 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Impact of fine-tuning the VL models.  
We see in Figure 13 that the best six combinations of MER and NLP models achieve TERs that are close to each other (36.14-40.97%). The combination of _LaTeX-OCR_ and _BART-Base_ results in the lowest TER (36.14%), followed by the combination of _LaTeX-OCR_ and _T5-Base_ (36.91%). However, our t-test demonstrates a non-statistically significant performance difference between the combination of _LaTeX-OCR_ and _BART-Base_ (M = 36.14, SD = 19.32) and the combination of _LaTeX-OCR_ and _T5-Base_ (M = 36.91, SD = 19.45), where t(80) = 0.36 and \(p\) = 0.72 > 0.05. The third best system is the combination of _LaTeX-OCR_ and _FLAN-T5-Base_ (38.10%). This is followed by the systems that combine _BART-Base_ (39.29%), _T5-Base_ (40.06%), and _FLAN-T5-Base_ (40.97%) with _TrOCR_. When combining the MER models with the _ChatGPT_ versions, the combination of _LaTeX-OCR_ and _ChatGPT_ version _text-davnice-003_ (_ChatGPT_) performs best, with a TER of 44.40%.  
The best VL model is _TrOCR-Base_, achieving a TER of 42.09%, which is 16.46% relatively higher than the best system--the combination of _LaTeX-OCR_ and _BART-Base_. Our _t_-test demonstrates a statistically significant performance difference between the combination of _LaTeX-OCR_ and _BART-Base_ (M = 36.14, SD = 19.35) and _TrOCR-Base_ (M = 42.09, SD = 22.00), where t(80) = 2.10 and p = 0.04 < 0.05. However, _TrOCR-Base_ outperforms seven combinations of MER and NLP models. This demonstrates that a VL model is able to achieve a decent result in the transcription of mathematical formulas in images. The second best VL model is _BLIP-Base_, with a 52.18% TER. _SWIN-GPT-2_ achieves a TER of 72.91% and _GIT-Large_ of 80.96%.  
Comparing the runtimes of the best combination of MER model and NLP model to the runtimes of the best VL model shows that the combination of _LaTeX-OCR_ (consisting of 25 M parameters) and _BART-Base_ (consisting of 239 M parameters) is on average almost four times faster than _TrOCR-Base_ (consisting of 385 M parameters): For transcribing one formula, the combination of _LaTeX-OCR_ and _BART-Base_ takes only 1.66 s in our Google Colab Pro environment--_LaTeX-OCR_ takes 1.15 s and _BART-Base_ 0.51 s. In contrast, _TrOCR-Base_ needs 6.53 s. This shows that, to use our systems for real-time speech synthesis, the transcriptions have to be generated in advance in order not to create a delay.