Create README.md

README.md

@@ -0,0 +1,41 @@
+---
+license: mit
+language:
+- en
+- fr
+- es
+- pt
+- de
+---
+# Model Card for NoRef-ER
+
+Referenceless Error Metric for Automatic Speech Recognition  
+via Contrastive Fine-Tuning of mMiniLMv2 without References
+
+# Model Details
+
+## How to use
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("aixplain/NoRef-ER")
+model = AutoModel.from_pretrained("aixplain/NoRef-ER")
+
+tokens = tokenizer([
+    "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced.",
+    "In Italy, pizzas serves in formal settings, such as at an restaurant, is presented unslicing."
+], padding=True, return_tensors="pt")
+scores = model.infer(**tokens)
+```
+
+## Model Description
+
+This work presents a novel multi-language referenceless quality metric for automatic speech recognition (ASR). The metric is based on a language model (LM) trained with contrastive-learning and without references indicating the quality. Instead, the known order of quality in-between increasing compression levels of the same ASR model is used for self-supervision. All unique pair combinations are extracted from the outputs of ASR models in multiple compression levels to compile a dataset for model training and validation. The LM is part of a siamese network architecture (with shared weights) for giving pair-wise ranking decisions considering the ASR output quality. The referenceless metric achieves 77% validation accuracy in this pair-wise ranking task and can generalize for quality comparisons in-between different ASR models. When experimented on a blind test dataset consisting of outputs of top commercial ASR engines, it has been observed that the referenceless metric has a 36% correlation with word-error-rate (WER) ranks of them across samples, and can outperform the best engine's WER by 7-8% via selecting among alternative hypotheses. The referenceless metric is compared against the perplexity metric from various state-of-art pre-trained LM(s) and obtained superior performance in all experiments. The referenceless metric allows comparing the performance of different ASR models on a speech dataset that lacks ground-truth references. It also enables obtaining an ensemble of ASR models that can outperform any individual model in the ensemble. Finally, it can be used to prioritize hypotheses for referencing (via post-editing) or human-evaluation processes within ASR model improvement lifecycle in production, and for A/B testing different versions of an ASR model (such as previous and current) on a referenceless production data stream.
+
+
+
+- **Developed by:** aiXplain AI Lab
+- **Language(s) (NLP):** English, French, Spanish, Portuguese, German
+- **License:** MIT License
+- **Finetuned from model [optional]:** nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large