---
pipeline_tag: translation
datasets:
- ascolda/ru_en_Crystallography_and_Spectroscopy
language:
- ru
- en
metrics:
- bleu
tags:
- chemistry
---
# nllb-200-distilled-600M_ru_en_finetuned_crystallography

This model is a fine-tuned version of [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) trained on the [ascolda/ru_en_Crystallography_and_Spectroscopy](https://huggingface.co/datasets/ascolda/ru_en_Crystallography_and_Spectroscopy) dataset. 
It achieves the following results on the evaluation set:
- Loss: 0.5602
- Bleu: 56.5855

## Model description

The finetuned model yieled better performance on the machine translation task of domain-specific scientific articles related to the Crystallography and Spectroscopy domain.

## Metrics used to describe the fine-tuning effect

Below is the comparison of the translation quality metrics for the original NLLB model and my finetuned version. Evaluation is focused on: (1) general translation quality, (2) quality of translation of specific
terminology, and (3) uniformity of translation of domain-specific terms in different contexts. 

(1) The general translation quality was evaluated using the Bleu metric.  

(2) Term Success Rate. In the terminology success rate we compared the machine-translated terms with their dictionary equivalents by checking for the presence of the reference terminology translation in the output by the regular expression match.

(3) Term Consistency. This metric looks at whether technical terms are translated uniformly across the entire text corpus in different contexts. We aim for high consistency,
measured by the low occurrence of multiple translations for the same term within the evaluation dataset. 

| Model                                                          | BLEU    | Term Success Rate   | Term Consistency | 
|:--------------------------------------------------------------:|:-------:|:-------------------:|:----------------:|
| nllb-200-distilled-600M                                        | 38.19   | 0.246               | 0.199            | 
| nllb-200-distilled-600M_ru_en_finetuned_crystallography        | 56.59   | 0.573               | 0.740            |