|
--- |
|
license: apache-2.0 |
|
language: |
|
- multilingual |
|
- af |
|
- am |
|
- ar |
|
- as |
|
- az |
|
- be |
|
- bg |
|
- bn |
|
- br |
|
- bs |
|
- ca |
|
- cs |
|
- cy |
|
- da |
|
- de |
|
- el |
|
- en |
|
- eo |
|
- es |
|
- et |
|
- eu |
|
- fa |
|
- fi |
|
- fr |
|
- fy |
|
- ga |
|
- gd |
|
- gl |
|
- gu |
|
- ha |
|
- he |
|
- hi |
|
- hr |
|
- hu |
|
- hy |
|
- id |
|
- is |
|
- it |
|
- ja |
|
- jv |
|
- ka |
|
- kk |
|
- km |
|
- kn |
|
- ko |
|
- ku |
|
- ky |
|
- la |
|
- lo |
|
- lt |
|
- lv |
|
- mg |
|
- mk |
|
- ml |
|
- mn |
|
- mr |
|
- ms |
|
- my |
|
- ne |
|
- nl |
|
- 'no' |
|
- om |
|
- or |
|
- pa |
|
- pl |
|
- ps |
|
- pt |
|
- ro |
|
- ru |
|
- sa |
|
- sd |
|
- si |
|
- sk |
|
- sl |
|
- so |
|
- sq |
|
- sr |
|
- su |
|
- sv |
|
- sw |
|
- ta |
|
- te |
|
- th |
|
- tl |
|
- tr |
|
- ug |
|
- uk |
|
- ur |
|
- uz |
|
- vi |
|
- xh |
|
- yi |
|
- zh |
|
tags: |
|
- text-classification |
|
- register |
|
- web-register |
|
- genre |
|
--- |
|
# Web register classification (multilingual model) |
|
|
|
A multilingual web register classifier, fine-tuned from [XLM-RoBERTa-large](https://huggingface.co/FacebookAI/xlm-roberta-large). |
|
The model is trained with the multilingual CORE corpora across five languages (English, Finnish, French, Swedish, Turkish) to classify documents based on the [CORE taxonomy](https://turkunlp.org/register-annotation-docs/). |
|
It can predict labels for the 100 languages covered by XLM-RoBERTa-large. The model achieves state-of-the-art performance in classifying web registers for the trained languages and has strong transfer performance (see Evaluation below). |
|
It is designed to support the development of open language models and for linguists analyzing register variation. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
- **Developed by:** TurkuNLP |
|
- **Funded by:** The Research Council of Finland, Emil Aaltonen Foundation, University of Turku |
|
- **Shared by:** TurkuNLP |
|
- **Model type:** Language model |
|
- **Language(s) (NLP):** English, Finnish, French, Swedish, Turkish |
|
- **License:** apache-2.0 |
|
- **Finetuned from model:** FacebookAI/xlm-roberta-large |
|
|
|
### Model Sources |
|
|
|
- **Repository:** https://github.com/TurkuNLP/pytorch-registerlabeling |
|
- **Paper:** Coming soon! |
|
|
|
## Register labels and their abbreviations |
|
|
|
Below is a list of the register labels predicted by the model. Note that some labels are hierarchical; when a sublabel is predicted, its parent label is also predicted. |
|
For a more detailed description of the label scheme, see [here](https://turkunlp.org/register-annotation-docs/). |
|
|
|
The main labels are uppercase. To only include these main labels in the predictions, simply slice the model's output to keep only the uppercase labels. |
|
|
|
- **MT:** Machine translated or generated |
|
- **LY:** Lyrical |
|
- **SP:** Spoken |
|
- **it:** Interview |
|
- **ID:** Interactive discussion |
|
- **NA:** Narrative |
|
- **ne:** News report |
|
- **sr:** Sports report |
|
- **nb:** Narrative blog |
|
- **HI:** How-to or instructions |
|
- **re:** Recipe |
|
- **IN:** Informational description |
|
- **en:** Encyclopedia article |
|
- **ra:** Research article |
|
- **dtp:** Description of a thing or person |
|
- **fi:** Frequently asked questions |
|
- **lt:** Legal terms and conditions |
|
- **OP:** Opinion |
|
- **rv:** Review |
|
- **ob:** Opinion blog |
|
- **rs:** Denominational religious blog or sermon |
|
- **av:** Advice |
|
- **IP:** Informational persuasion |
|
- **ds:** Description with intent to sell |
|
- **ed:** News & opinion blog or editorial |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
|
|
model_id = "TurkuNLP/multilingual-web-register-classification" |
|
|
|
# Load model and tokenizer |
|
model = AutoModelForSequenceClassification.from_pretrained(model_id).to(device) |
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
# Text to be categorized |
|
text = "A text to be categorized" |
|
|
|
# Tokenize text |
|
inputs = tokenizer([text], return_tensors="pt", padding=True, truncation=True, max_length=512).to(device) |
|
|
|
with torch.no_grad(): |
|
outputs = model(**inputs) |
|
|
|
# Apply sigmoid to the logits to get probabilities |
|
probabilities = torch.sigmoid(outputs.logits).squeeze() |
|
|
|
# Determine a threshold for predicting labels |
|
threshold = 0.5 |
|
predicted_label_indices = (probabilities > threshold).nonzero(as_tuple=True)[0] |
|
|
|
# Extract readable labels using id2label |
|
id2label = model.config.id2label |
|
predicted_labels = [id2label[idx.item()] for idx in predicted_label_indices] |
|
|
|
print("Predicted labels:", predicted_labels) |
|
|
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The model was trained using the Multilingual CORE Corpora, which will be published soon. |
|
|
|
### Training Procedure |
|
|
|
#### Training Hyperparameters |
|
|
|
- **Batch size:** 8 |
|
- **Epochs:** 21 |
|
- **Learning rate:** 0.00005 |
|
- **Precision:** bfloat16 (non-mixed precision) |
|
- **TF32:** Enabled |
|
- **Seed:** 42 |
|
- **Max Size:** 512 tokens |
|
|
|
#### Inference time |
|
|
|
Average inference time (across 1000 iterations), using a single NVIDIA A100 GPU and a batch size of one is **17 ms** for a single example. Wirh bigger batches, inference can be considerably faster. |
|
|
|
## Evaluation |
|
|
|
Micro-averaged F1 scores and optimized prediction thresholds for the five training languages (test set): |
|
|
|
| Language | F1 (All labels) | F1 (Main labels) | Threshold | |
|
| -------- | --------------- | ---------------- | ----------| |
|
| English | 0.72 | 0.75 | 0.40 | |
|
| Finnish | 0.79 | 0.82 | 0.45 | |
|
| French | 0.75 | 0.78 | 0.45 | |
|
| Swedish | 0.81 | 0.82 | 0.45 | |
|
| Turkish | 0.77 | 0.78 | 0.45 | |
|
|
|
Micro-averaged F1 scores and optimized prediction thresholds for additional languages (zero-shot): |
|
|
|
|
|
| Language | F1 (All labels) | F1 (Main labels) | Threshold | |
|
| ---------- | --------------- | ---------------- | ----------| |
|
| Arabic | 0.63 | 0.66 | 0.40 | |
|
| Catalan | 0.62 | 0.63 | 0.50 | |
|
| Spanish | 0.62 | 0.67 | 0.65 | |
|
| Persian | 0.71 | 0.70 | 0.35 | |
|
| Hindi | 0.77 | 0.78 | 0.40 | |
|
| Indonesian | 0.60 | 0.61 | 0.30 | |
|
| Japanese | 0.53 | 0.64 | 0.35 | |
|
| Norwegian | 0.65 | 0.70 | 0.65 | |
|
| Portuguese | 0.67 | 0.68 | 0.40 | |
|
| Urdu | 0.81 | 0.83 | 0.35 | |
|
| Chinese | 0.67 | 0.70 | 0.40 | |
|
|
|
## Technical Specifications |
|
|
|
### Compute Infrastructure |
|
|
|
- Mahti supercomputer (CSC - IT Center for Science, Finland) |
|
- 1 x NVIDIA A100-SXM4-40GB |
|
|
|
#### Software |
|
|
|
- torch 2.2.1 |
|
- transformers 4.39.3 |
|
|
|
## Citation |
|
|
|
The citation for this work will be available soon. In the meantime, please refer to earlier related work for citation: |
|
|
|
```bibtex |
|
@article{Laippala.etal2022, |
|
title = {Register Identification from the Unrestricted Open {{Web}} Using the {{Corpus}} of {{Online Registers}} of {{English}}}, |
|
author = {Laippala, Veronika and R{\"o}nnqvist, Samuel and Oinonen, Miika and Kyr{\"o}l{\"a}inen, Aki-Juhani and Salmela, Anna and Biber, Douglas and Egbert, Jesse and Pyysalo, Sampo}, |
|
year = {2022}, |
|
journal = {Language Resources and Evaluation}, |
|
issn = {1574-0218}, |
|
doi = {10.1007/s10579-022-09624-1}, |
|
url = {https://doi.org/10.1007/s10579-022-09624-1}, |
|
} |
|
|
|
@article{Skantsi_Laippala_2023, |
|
title = {Analyzing the unrestricted web: The finnish corpus of online registers}, |
|
doi = {10.1017/S0332586523000021}, |
|
journal = {Nordic Journal of Linguistics}, |
|
author = {Skantsi, Valtteri and Laippala, Veronika}, |
|
year = {2023}, |
|
pages = {1–31} |
|
} |
|
``` |
|
|
|
## Model Card Contact |
|
|
|
Erik Henriksson, Hugging Face username: erikhenriksson |
