|
--- |
|
license: apache-2.0 |
|
language: |
|
- sr |
|
metrics: |
|
- accuracy |
|
- wer |
|
library_name: transformers |
|
tags: |
|
- legal |
|
--- |
|
|
|
# BERTić-COMtext-SR-legal-MSD-ekavica |
|
|
|
**BERTić-COMtext-SR-legal-MSD-ekavica** is a variant of the [BERTić](https://huggingface.co/classla/bcms-bertic) model, fine-tuned on the task of morphosyntactic (MSD) tag prediction in Serbian legal texts written in the Ekavian pronunciation. |
|
The model was fine-tuned for 15 epochs on the Ekavian variant of the [COMtext.SR.legal](https://github.com/ICEF-NLP/COMtext.SR) dataset. |
|
|
|
# Benchmarking |
|
|
|
This model was evaluated on the tasks of MSD prediction and lemmatization of Serbian legal texts. |
|
Lemmatization was performed using the predicted MSD tags and the [srLex](http://hdl.handle.net/11356/1233) inflectional lexicon. |
|
|
|
Accuracy and Word Error Rate were used as evaluation metrics. |
|
|
|
This model was compared to: |
|
- The [CLASSLA](http://pypi.org/project/classla/) library |
|
- A variant of [BERTić](https://huggingface.co/classla/bcms-bertic) fine-tuned for MSD prediction using the [SETimes.SR 2.0](http://hdl.handle.net/11356/1843) corpus of newswire texts |
|
- [SrBERTa](http://huggingface.co/nemanjaPetrovic/SrBERTa), a model specially trained on Serbian legal texts |
|
|
|
All large language models were fine-tuned for 15 epochs. |
|
CLASSLA and BERTić-SETimes were directly tested on the entire COMtext.SR.legal.ekavica corpus. |
|
BERTić-COMtext-SR-legal-MSD-ekavica and SrBERTa were fine-tuned and evaluated on the COMtext.SR.legal.ekavica corpus using 10-fold CV. |
|
|
|
The code and data to run these experiments is available on the [COMtext.SR GitHub repository](https://github.com/ICEF-NLP/COMtext.SR). |
|
|
|
## Results |
|
|
|
| Model | MSD ACC | MSD WER | Lemma ACC | Lemma WER | |
|
| --------------------------------------------------------- | -------- | ---------- | --------- | ---------- | |
|
| CLASSLA-SR (gold tokens) | 0.9144 | 0.0856 | 0.9432 | 0.0568 | |
|
| *CLASSLA-SR (CLASSLA tokenizer)* | / | *0.0983* | / | *0.0739* | |
|
| BERTić-SETimes (gold tokens) | 0.9231 | 0.0768 | 0.9649 | 0.0351 | |
|
| *BERTić-SETimes.SR (CLASSLA tokenizer)* | / | *0.0884* | / | *0.0542* | |
|
| BERTić-COMtext-SR-legal-MSD-ekavica (gold tokens) |**0.9674**| **0.0326** |**0.9666** | **0.0334** | |
|
| *BERTić-COMtext-SR-legal-MSD-ekavica (CLASSLA tokenizer)* | / |***0.0447***| / |***0.0526***| |
|
| SrBERTa (gold tokens) | 0.9288 | 0.0712 | 0.9391 | 0.0609 | |
|
| *SrBERTa (CLASSLA tokenizer)* | / | *0.0851* | / | *0.0819* | |
