Create README.md

README.md

@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text2text-generation
+---
+# MANTa-LM (small)
+
+Pretrained MANTa-LM architecture as introduced in the paper [MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling](https://aclanthology.org/2022.findings-emnlp.207.pdf).
+<center><img src="https://github.com/NathanGodey/nathangodey.github.io/raw/main/img/posts/full_difftok_schema.png"  width="600"></center>
+
+## Model Details
+
+### Model Description
+
+The MANTa tokenizer aims at mimicking the combination of a subword tokenizer and an embedding matrix in a classical language model in a differentiable way.
+This trainable tokenizer is thus added as the first layer of an encoder-decoder model and trained using the language modeling objective.
+
+Our results show that MANTa-LM only slightly degrades the performance of a T5 equivalent on the GLUE benchmark while being **much more robust** to artificial and user-generated noise. 
+
+
+### Model Sources
+
+- **Paper:** [MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling](https://aclanthology.org/2022.findings-emnlp.207.pdf) (EMNLP 2022 Findings)
+
+## Uses
+
+### Direct Use
+
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
+manta_model = AutoModelForSeq2SeqLM.from_pretrained("almanach/manta-lm-small", trust_remote_code=True)
+
+tokens = tokenizer("The name of the capital of France is <extra_id_0> and it is a very big city.", return_tensors="pt")
+output = manta_model.generate(**tokens, decoder_start_token_id=0)
+
+print(tokenizer.batch_decode(output))
+```
+
+### Recommendations
+
+We recommend using a smaller learning rate for the tokenizer module during fine-tuning (byte embeddings, frontier predictor, pooler).
+
+
+## Training Details
+
+### Training Data
+
+This model was trained on the C4 dataset.
+
+### Training Procedure 
+
+The training objective is the same as ByT5, but most hyperparameters are taken from T5.
+
+
+## Citation
+
+**BibTeX:**
+
+```
+@inproceedings{godey-etal-2022-manta,
+    title = "{MANT}a: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling",
+    author = "Godey, Nathan  and
+      Castagn{\'e}, Roman  and
+      de la Clergerie, {\'E}ric  and
+      Sagot, Beno{\^\i}t",
+    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
+    month = dec,
+    year = "2022",
+    address = "Abu Dhabi, United Arab Emirates",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2022.findings-emnlp.207",
+    pages = "2859--2870",
+}
+
+```
+
+## Model Card Authors
+
+[Nathan Godey](https://nathangodey.github.io/)
+[Roman Castagné](https://romancast.github.io/)
+