|
--- |
|
language: en |
|
tags: |
|
- exbert |
|
|
|
license: cc-by-nc-4.0 |
|
--- |
|
|
|
# xlm-mlm-en-2048 |
|
|
|
# Table of Contents |
|
|
|
1. [Model Details](#model-details) |
|
2. [Uses](#uses) |
|
3. [Bias, Risks, and Limitations](#bias-risks-and-limitations) |
|
4. [Training](#training) |
|
5. [Evaluation](#evaluation) |
|
6. [Environmental Impact](#environmental-impact) |
|
7. [Citation](#citation) |
|
8. [Model Card Authors](#model-card-authors) |
|
9. [How To Get Started With the Model](#how-to-get-started-with-the-model) |
|
|
|
|
|
# Model Details |
|
|
|
The XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau. It’s a transformer pretrained with either a causal language modeling (CLM) objective (next token prediction), a masked language modeling (MLM) objective (BERT-like), or |
|
a Translation Language Modeling (TLM) object (extension of BERT’s MLM to multiple language inputs). This model is trained with a masked language modeling objective on English text. |
|
|
|
## Model Description |
|
|
|
- **Developed by:** Researchers affiliated with Facebook AI, see [associated paper](https://arxiv.org/abs/1901.07291) and [GitHub Repo](https://github.com/facebookresearch/XLM) |
|
- **Model type:** Language model |
|
- **Language(s) (NLP):** English |
|
- **License:** CC-BY-NC-4.0 |
|
- **Related Models:** Other [XLM models](https://huggingface.co/models?sort=downloads&search=xlm) |
|
- **Resources for more information:** |
|
- [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau (2019) |
|
- [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/pdf/1911.02116.pdf) by Conneau et al. (2020) |
|
- [GitHub Repo](https://github.com/facebookresearch/XLM) |
|
- [Hugging Face XLM docs](https://huggingface.co/docs/transformers/model_doc/xlm) |
|
|
|
# Uses |
|
|
|
## Direct Use |
|
|
|
The model is a language model. The model can be used for masked language modeling. |
|
|
|
## Downstream Use |
|
|
|
To learn more about this task and potential downstream uses, see the Hugging Face [fill mask docs](https://huggingface.co/tasks/fill-mask) and the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs. Also see the [associated paper](https://arxiv.org/abs/1901.07291). |
|
|
|
## Out-of-Scope Use |
|
|
|
The model should not be used to intentionally create hostile or alienating environments for people. |
|
|
|
# Bias, Risks, and Limitations |
|
|
|
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). |
|
|
|
## Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
|
# Training |
|
|
|
More information needed. See the [associated GitHub Repo](https://github.com/facebookresearch/XLM). |
|
|
|
# Evaluation |
|
|
|
More information needed. See the [associated GitHub Repo](https://github.com/facebookresearch/XLM). |
|
|
|
# Environmental Impact |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** More information needed |
|
- **Hours used:** More information needed |
|
- **Cloud Provider:** More information needed |
|
- **Compute Region:** More information needed |
|
- **Carbon Emitted:** More information needed |
|
|
|
# Citation |
|
|
|
**BibTeX:** |
|
|
|
```bibtex |
|
@article{lample2019cross, |
|
title={Cross-lingual language model pretraining}, |
|
author={Lample, Guillaume and Conneau, Alexis}, |
|
journal={arXiv preprint arXiv:1901.07291}, |
|
year={2019} |
|
} |
|
``` |
|
|
|
**APA:** |
|
- Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. |
|
|
|
# Model Card Authors |
|
|
|
This model card was written by the team at Hugging Face. |
|
|
|
# How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. See the [Hugging Face XLM docs](https://huggingface.co/docs/transformers/model_doc/xlm) for more examples. |
|
|
|
```python |
|
from transformers import XLMTokenizer, XLMModel |
|
import torch |
|
|
|
tokenizer = XLMTokenizer.from_pretrained("xlm-mlm-en-2048") |
|
model = XLMModel.from_pretrained("xlm-mlm-en-2048") |
|
|
|
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") |
|
outputs = model(**inputs) |
|
|
|
last_hidden_states = outputs.last_hidden_state |
|
``` |
|
|
|
<a href="https://huggingface.co/exbert/?model=xlm-mlm-en-2048"> |
|
<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png"> |
|
</a> |
|
|
