Improve README
Browse files- Quality.svg +570 -0
- README.md +195 -40
Quality.svg
ADDED
|
README.md
CHANGED
|
@@ -1,60 +1,215 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
tags:
|
| 5 |
-
-
|
| 6 |
-
|
| 7 |
-
- name: multilingual-e5-small-aligned-quality-20241214-new
|
| 8 |
-
results: []
|
| 9 |
---
|
| 10 |
|
| 11 |
-
|
| 12 |
-
should probably proofread and complete it, then remove this comment. -->
|
| 13 |
|
| 14 |
-
|
| 15 |
|
| 16 |
-
|
| 17 |
-
It achieves the following results on the evaluation set:
|
| 18 |
-
- Loss: 0.1958
|
| 19 |
-
- Mse: 0.1958
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
-
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
##
|
| 30 |
|
| 31 |
-
|
|
|
|
|
|
|
| 32 |
|
| 33 |
-
|
| 34 |
|
| 35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
- lr_scheduler_type: linear
|
| 44 |
-
- num_epochs: 3.0
|
| 45 |
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|:-------------:|:-----:|:-----:|:---------------:|:------:|
|
| 50 |
-
| 0.2436 | 1.0 | 7813 | 0.2296 | 0.2296 |
|
| 51 |
-
| 0.1927 | 2.0 | 15626 | 0.2079 | 0.2079 |
|
| 52 |
-
| 0.1615 | 3.0 | 23439 | 0.1958 | 0.1958 |
|
| 53 |
|
|
|
|
| 54 |
|
| 55 |
-
|
|
|
|
| 56 |
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- multilingual
|
| 5 |
+
- af
|
| 6 |
+
- am
|
| 7 |
+
- ar
|
| 8 |
+
- as
|
| 9 |
+
- az
|
| 10 |
+
- be
|
| 11 |
+
- bg
|
| 12 |
+
- bn
|
| 13 |
+
- br
|
| 14 |
+
- bs
|
| 15 |
+
- ca
|
| 16 |
+
- cs
|
| 17 |
+
- cy
|
| 18 |
+
- da
|
| 19 |
+
- de
|
| 20 |
+
- el
|
| 21 |
+
- en
|
| 22 |
+
- eo
|
| 23 |
+
- es
|
| 24 |
+
- et
|
| 25 |
+
- eu
|
| 26 |
+
- fa
|
| 27 |
+
- fi
|
| 28 |
+
- fr
|
| 29 |
+
- fy
|
| 30 |
+
- ga
|
| 31 |
+
- gd
|
| 32 |
+
- gl
|
| 33 |
+
- gu
|
| 34 |
+
- ha
|
| 35 |
+
- he
|
| 36 |
+
- hi
|
| 37 |
+
- hr
|
| 38 |
+
- hu
|
| 39 |
+
- hy
|
| 40 |
+
- id
|
| 41 |
+
- is
|
| 42 |
+
- it
|
| 43 |
+
- ja
|
| 44 |
+
- jv
|
| 45 |
+
- ka
|
| 46 |
+
- kk
|
| 47 |
+
- km
|
| 48 |
+
- kn
|
| 49 |
+
- ko
|
| 50 |
+
- ku
|
| 51 |
+
- ky
|
| 52 |
+
- la
|
| 53 |
+
- lo
|
| 54 |
+
- lt
|
| 55 |
+
- lv
|
| 56 |
+
- mg
|
| 57 |
+
- mk
|
| 58 |
+
- ml
|
| 59 |
+
- mn
|
| 60 |
+
- mr
|
| 61 |
+
- ms
|
| 62 |
+
- my
|
| 63 |
+
- ne
|
| 64 |
+
- nl
|
| 65 |
+
- 'no'
|
| 66 |
+
- om
|
| 67 |
+
- or
|
| 68 |
+
- pa
|
| 69 |
+
- pl
|
| 70 |
+
- ps
|
| 71 |
+
- pt
|
| 72 |
+
- ro
|
| 73 |
+
- ru
|
| 74 |
+
- sa
|
| 75 |
+
- sd
|
| 76 |
+
- si
|
| 77 |
+
- sk
|
| 78 |
+
- sl
|
| 79 |
+
- so
|
| 80 |
+
- sq
|
| 81 |
+
- sr
|
| 82 |
+
- su
|
| 83 |
+
- sv
|
| 84 |
+
- sw
|
| 85 |
+
- ta
|
| 86 |
+
- te
|
| 87 |
+
- th
|
| 88 |
+
- tl
|
| 89 |
+
- tr
|
| 90 |
+
- ug
|
| 91 |
+
- uk
|
| 92 |
+
- ur
|
| 93 |
+
- uz
|
| 94 |
+
- vi
|
| 95 |
+
- xh
|
| 96 |
+
- yi
|
| 97 |
+
- zh
|
| 98 |
+
datasets:
|
| 99 |
+
- agentlans/en-translations
|
| 100 |
+
base_model:
|
| 101 |
+
- agentlans/multilingual-e5-small-aligned
|
| 102 |
+
pipeline_tag: text-classification
|
| 103 |
tags:
|
| 104 |
+
- multilingual
|
| 105 |
+
- quality-assessment
|
|
|
|
|
|
|
| 106 |
---
|
| 107 |
|
| 108 |
+
# multilingual-e5-small-aligned-quality
|
|
|
|
| 109 |
|
| 110 |
+
This model is a fine-tuned version of [agentlans/multilingual-e5-small-aligned](https://huggingface.co/agentlans/multilingual-e5-small-aligned) designed for assessing text quality across multiple languages.
|
| 111 |
|
| 112 |
+
## Key Features
|
|
|
|
|
|
|
|
|
|
| 113 |
|
| 114 |
+
- Multilingual support
|
| 115 |
+
- Quality assessment for text
|
| 116 |
+
- Based on E5 small model architecture
|
| 117 |
|
| 118 |
+
## Intended Uses & Limitations
|
| 119 |
|
| 120 |
+
This model is intended for:
|
| 121 |
+
- Assessing the quality of multilingual text
|
| 122 |
+
- Filtering multilingual content
|
| 123 |
+
- Comparative analysis of corpus text quality across different languages
|
| 124 |
|
| 125 |
+
Limitations:
|
| 126 |
+
- Performance may vary for languages not well-represented in the training data
|
| 127 |
+
- Should not be used as the sole criterion for quality assessment
|
| 128 |
|
| 129 |
+
## Usage Example
|
| 130 |
|
| 131 |
+
```python
|
| 132 |
+
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 133 |
+
import torch
|
| 134 |
|
| 135 |
+
model_name = "agentlans/multilingual-e5-small-aligned-quality"
|
| 136 |
|
| 137 |
+
# Initialize tokenizer and model
|
| 138 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 139 |
+
model = AutoModelForSequenceClassification.from_pretrained(model_name)
|
| 140 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 141 |
+
model = model.to(device)
|
| 142 |
|
| 143 |
+
def quality(text):
|
| 144 |
+
"""Assess the quality of the input text."""
|
| 145 |
+
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True).to(device)
|
| 146 |
+
with torch.no_grad():
|
| 147 |
+
logits = model(**inputs).logits.squeeze().cpu()
|
| 148 |
+
return logits.tolist()
|
|
|
|
|
|
|
| 149 |
|
| 150 |
+
# Example usage
|
| 151 |
+
score = quality("Your text here.")
|
| 152 |
+
print(f"Quality score: {score}")
|
| 153 |
+
```
|
| 154 |
|
| 155 |
+
## Performance Results
|
|
|
|
|
|
|
|
|
|
|
|
|
| 156 |
|
| 157 |
+
The model was evaluated on a diverse set of multilingual text samples:
|
| 158 |
|
| 159 |
+
- 10 English text samples of varying quality were translated into Arabic, Chinese, French, Russian, and Spanish.
|
| 160 |
+
- The model demonstrated consistent quality assessment across different languages for the same text.
|
| 161 |
|
| 162 |
+
<details>
|
| 163 |
+
<summary>The following table presents the 10 original texts along with their translations:</summary>
|
| 164 |
+
|Text |English |French |Spanish |Chinese |Russian |Arabic |
|
| 165 |
+
|:----|:-------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------|
|
| 166 |
+
|A |Discover the secret to eternal youth with our revolutionary skincare product! |Découvrez le secret de la jeunesse éternelle avec notre produit de soin révolutionnaire ! |¡Descubre el secreto de la eterna juventud con nuestro revolucionario producto de cuidado de la piel! |使用我们革命性的护肤产品发现永葆青春的秘密！ |Откройте для себя секрет вечной молодости с нашим революционным средством по уходу за кожей! |اكتشف سر الشباب الأبدي مع منتجنا الثوري للعناية بالبشرة! |
|
| 167 |
+
|B |Get rich quick with our foolproof investment strategy - no experience needed! |Devenez riche rapidement grâce à notre stratégie d’investissement infaillible – aucune expérience n’est requise ! |Hazte rico rápidamente con nuestra estrategia de inversión infalible: ¡no necesitas experiencia! |利用我们万无一失的投资策略快速致富 - 无需经验！ |Быстро разбогатейте с нашей надежной инвестиционной стратегией — опыт не требуется! |احصل على الثراء السريع مع استراتيجية الاستثمار الموثوقة لدينا - لا حاجة للخبرة! |
|
| 168 |
+
|C |Earn money from home by participating in online surveys - sign up today! |Gagnez de l'argent depuis chez vous en participant à des sondages en ligne - inscrivez-vous dès aujourd'hui ! |Gana dinero desde casa participando en encuestas online: ¡regístrate hoy! |通过参与在线调查在家赚钱 - 今天就注册！ |Зарабатывайте деньги из дома, участвуя в онлайн-опросах — зарегистрируйтесь сегодня! |اكسب المال من المنزل عن طريق المشاركة في الاستطلاعات عبر الإنترنت - سجل اليوم! |
|
| 169 |
+
|D |Congratulations! You've won a $1,000 gift card! Click here to claim your prize! |Félicitations ! Vous avez gagné une carte-cadeau de 1 000 $ ! Cliquez ici pour réclamer votre prix ! |¡Felicitaciones! ¡Ganaste una tarjeta de regalo de $1,000! ¡Haz clic aquí para reclamar tu premio! |恭喜！您赢了一张价值 1,000 美元的礼品卡！单击此处领取您的奖品！ |Поздравляем! Вы выиграли подарочную карту на $1000! Нажмите здесь, чтобы получить свой приз! |مبروك! لقد فزت ببطاقة هدايا بقيمة 1000 دولار! انقر هنا للحصول على جائزتك! |
|
| 170 |
+
|E |Act now! Limited time offer on miracle weight loss pills! |Agissez maintenant ! Offre à durée limitée sur les pilules amaigrissantes miracles ! |¡Actúe ahora! ¡Oferta por tiempo limitado en píldoras milagrosas para bajar de peso! |立即行动！神奇减肥药限时优惠！ |Действуйте сейчас! Ограниченное по времени предложение на чудодейственные таблетки для похудения! |تصرف الآن! عرض لفترة محدودة على حبوب إنقاص الوزن المعجزة! |
|
| 171 |
+
|F |Your computer is infected! Click here for a free scan and fix your issues now! |Votre ordinateur est infecté ! Cliquez ici pour une analyse gratuite et corrigez vos problèmes dès maintenant ! |¡Su computadora está infectada! Haga clic aquí para obtener un análisis gratuito y solucionar sus problemas ahora. |您的计算机已感染病毒！点击此处进行免费扫描并立即修复您的问题！ |Ваш компьютер заражен! Нажмите здесь для бесплатного сканирования и устранения проблем прямо сейчас! |جهاز الكمبيوتر الخاص بك مصاب! انقر هنا لإجراء فحص مجاني وإصلاح المشكلات التي تواجهك الآن! |
|
| 172 |
+
|G |Unlock the secrets of the universe with our exclusive online astronomy course! |Découvrez les secrets de l'univers avec notre cours d'astronomie en ligne exclusif ! |¡Descubre los secretos del universo con nuestro exclusivo curso de astronomía online! |通过我们独家的在线天文学课程揭开宇宙的秘密！ |Откройте тайны Вселенной с нашим эксклюзивным онлайн-курсом астрономии! |اكتشف أسرار الكون مع دورتنا الفلكية الحصرية عبر الإنترنت! |
|
| 173 |
+
|H |The Eiffel Tower can be 15 cm taller during the summer due to thermal expansion. |La tour Eiffel peut être plus haute de 15 cm en été en raison de la dilatation thermique. |La Torre Eiffel puede ser 15 cm más alta durante el verano debido a la expansión térmica. |由于热膨胀，埃菲尔铁塔在夏季可能会高出 15 厘米。 |Летом Эйфелева башня может стать на 15 см выше из-за теплового расширения. |يمكن أن يزيد ارتفاع برج إيفل بمقدار 15 سم خلال فصل الصيف بسبب التمدد الحراري. |
|
| 174 |
+
|I |Did you know? The average person spends 6 years of their life dreaming. |Le saviez-vous ? En moyenne, une personne passe 6 ans de sa vie à rêver. |¿Sabías que una persona promedio pasa 6 años de su vida soñando? |你知道吗？每个人一生中平均有 6 年的时间在做梦。 |Знаете ли вы? В среднем человек тратит 6 лет своей жизни на мечты. |هل تعلم؟ يقضي الشخص العادي 6 سنوات من حياته في الأحلام. |
|
| 175 |
+
|J |Did you know that honey never spoils? Archaeologists have found pots of honey in ancient Egyptian tombs that are over 3,000 years old and still edible. |Saviez-vous que le miel ne périme jamais ? Des archéologues ont découvert dans d'anciennes tombes égyptiennes des pots de miel datant de plus de 3 000 ans et toujours comestibles. |¿Sabías que la miel nunca se estropea? Los arqueólogos han encontrado tarros de miel en tumbas del antiguo Egipto que tienen más de 3000 años y aún son comestibles. |你知道蜂蜜是不会变质的吗？考古学家在古埃及墓穴中发现了已有 3000 多年历史的蜂蜜罐，至今仍可食用。 |Знаете ли вы, что мед никогда не портится? Археологи нашли в древнеегипетских гробницах горшки с медом, которым более 3000 лет, и которые до сих пор съедобны. |هل تعلم أن العسل لا يفسد أبدًا؟ لقد عثر علماء الآثار على أواني عسل في مقابر مصرية قديمة يزيد عمرها عن 3000 عام ولا تزال صالحة للأكل. |
|
| 176 |
+
|
| 177 |
+
</details>
|
| 178 |
+
|
| 179 |
+
<img src="Quality.svg" alt="Scatterplot of predicted quality scores grouped by text sample and language" width="100%"/>
|
| 180 |
+
|
| 181 |
+
## Training Data
|
| 182 |
+
|
| 183 |
+
The model was trained on the [Multilingual Parallel Sentences dataset](https://huggingface.co/datasets/agentlans/en-translations), which includes:
|
| 184 |
+
|
| 185 |
+
- Parallel sentences in English and various other languages
|
| 186 |
+
- Semantic similarity scores calculated using LaBSE
|
| 187 |
+
- Additional quality metrics
|
| 188 |
+
- Sources: JW300, Europarl, TED Talks, OPUS-100, Tatoeba, Global Voices, and News Commentary
|
| 189 |
+
|
| 190 |
+
## Training Procedure
|
| 191 |
+
|
| 192 |
+
### Hyperparameters
|
| 193 |
+
|
| 194 |
+
- Learning rate: 5e-05
|
| 195 |
+
- Train batch size: 128
|
| 196 |
+
- Eval batch size: 8
|
| 197 |
+
- Seed: 42
|
| 198 |
+
- Optimizer: AdamW (betas=(0.9,0.999), epsilon=1e-08)
|
| 199 |
+
- Learning rate scheduler: Linear
|
| 200 |
+
- Number of epochs: 3.0
|
| 201 |
+
|
| 202 |
+
### Training Results
|
| 203 |
+
|
| 204 |
+
| Epoch | Training Loss | Validation Loss | MSE |
|
| 205 |
+
|-------|---------------|-----------------|--------|
|
| 206 |
+
| 1.0 | 0.2436 | 0.2296 | 0.2296 |
|
| 207 |
+
| 2.0 | 0.1927 | 0.2079 | 0.2079 |
|
| 208 |
+
| 3.0 | 0.1615 | 0.1958 | 0.1958 |
|
| 209 |
+
|
| 210 |
+
## Framework Versions
|
| 211 |
+
|
| 212 |
+
- Transformers: 4.46.3
|
| 213 |
+
- PyTorch: 2.5.1+cu124
|
| 214 |
+
- Datasets: 3.1.0
|
| 215 |
+
- Tokenizers: 0.20.3
|