MADLAD-400 MT Models
Author: noise-field
Pull request: no
Labels: name: New model

### Model description

From the paper:
> We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2309.04662
Github (with checkpoint links): https://github.com/google-research/google-research/tree/master/madlad_400

Comments:
@noise-field  Can you pls share the link of model(MADLAD-400) implementation?
I couldn't find it yet.
It would be appreciable.
Thanks

@yugaljain1999 as far as I understand, these are t5x (FLAX) models configured using `.gin` files (https://github.com/google/gin-config), e.g. for 3B model: https://console.cloud.google.com/storage/browser/_details/madlad-400-checkpoints/checkpoints/3b-mt/3b-mt.gin

Okay
@noise-field But how we can load these models and get translated outputs?
Can you share colab notebook or something to show how we can run these models?
Thanks

You should be able to use the conversion scripts in `transformers` like [this one](https://github.com/huggingface/transformers/blob/ad08137e473e00702fc3088a119da7026e1cb025/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py#L16)

There is a WIP notebook for this but output is nonsensical: https://colab.research.google.com/drive/1rZ2NRyl2zwmg0sQ2Wi-uZZF48iVYulTC

Was implemented in https://github.com/huggingface/candle/pull/1285 by @jbochi 

Thanks for tagging me, @noise-field .

I have published the safetensors weights in this [collection](https://huggingface.co/collections/jbochi/madlad-400-65491e6a78726cac9a4b84b7).

All machine translation models work with `T5ForConditionalGeneration`. However, the [language model](https://huggingface.co/jbochi/madlad400-8b-lm) is a decoder-only T5, which is currently not supported:

- [open issue](https://github.com/huggingface/transformers/issues/26647) 
- [Some observations](https://huggingface.co/jbochi/madlad400-3b-mt/discussions/2#654b6d28eeb563c6c86a3536)

Hey @jbochi would you be down to transfer the models you converted to the google organization? I think we can also open a PR to transformers similar #21929 to mention it in the doc, create a collection on the hub and not forget to mention that you converted the checkpoints! 
Regarding the T5Decoder only model, might make sense to add it in the PR to support these models. If the modelling changes are not too involved could make sense to have it in T5 (Like smae code as T5Encoder). 

WDYT? 

Hey @ArthurZucker . Sure, I can transfer the models if they are willing to maintain it. Opening the PR also sounds good to me. I think both things will help people find the model.

I'll open the PR shortly.

Regarding the decoder only model, I gave it a shot [here](https://huggingface.co/jbochi/madlad400-8b-lm/tree/main/decoder_only_t5).

It's hard to debug why it doesn't quite work without running the original code and comparing the output tensors.

Some things I changed that are different from T5:
- [Parallel Layers](https://paperswithcode.com/method/parallel-layers)
- [Multi-Query Attention](https://paperswithcode.com/method/multi-query-attention)

Edit:

A few additional differences:

- RoPE (`use_rotary_embedding=True`)
- MLP has swish instead of gelu activation
- bidrectional attention in the decoder
- eos token = `3` (`\n`)

I believe it is exactly the same architecture as [PaLM 8B](https://arxiv.org/pdf/2204.02311v5.pdf)


Parallelism is now supported with `accelerate` rather than in transformers! 

Sorry, but by [Parallel Layers](https://paperswithcode.com/method/parallel-layers) here, I don't mean model parallelism.

It's a variation of the transformer architecture. Instead of applying the feedforward dense layer after attention, the attention layer and the FF layer are called in parallel and added up. 

Here's their implementation: https://github.com/google/flaxformer/blob/ea17eb012a1d340ddff017b7a534c2162aaec34c/flaxformer/architectures/t5/t5_architecture.py#L534-L578

I opened the PR. Please take a look :)

Great thanks 😉 

@jbochi When I was trying MADLAD-400 base version, it's inference speed with batch_size=1 is much slower in both original and quantized version. So is there any way we can lower the latency?

And is it possible to run MADLAD-400 inference in batch size>1 because in my 16GB GPU memory , that 11 GB model throws following error in case of batch_size>1, not with batch_size=1

`[RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle)` with GPU only](https://stackoverflow.com/questions/66600362/runtimeerror-cuda-error-cublas-status-execution-failed-when-calling-cublassge)`

So may I know should we run MADLAD-3B model on high cost Nvidia GPUs like A30 to run it with batch_size>1 and hence reduce inference latency?

Your response would be really appreciable on this.

Thanks

Hey @yugaljain1999 . I'm not an expert, but I believe quantization should allow larger batch sizes, but will not help with the speed.

Loading the model in bfloat16 may help you run larger batch sizes. I just tried this on a Google Colab with a V100 with 16GB.

```
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("jbochi/madlad400-8b-lm")

model = T5ForConditionalGeneration.from_pretrained(
    "jbochi/madlad400-3b-mt", device_map="auto", torch_dtype=torch.bfloat16)

text = ["<2pt> I love pizza", "<2es> I love pizza", "<2it> I love pizza"]
input_ids = tokenizer(text, padding=True, return_tensors="pt").input_ids

%%time
outputs = model.generate(
    input_ids=input_ids.to('cuda')
)
for i in range(len(outputs)):
    print(tokenizer.decode(outputs[i], skip_special_tokens=True))
```

This prints:

> Eu amo pizza
> Me encanta la pizza
> Amo la pizza
> CPU times: user 4.18 s, sys: 1.19 s, total: 5.36 s
> Wall time: 7.03 s

It only used 6.5GB at peak.


@jbochi  Thanks for your response. Just one query, do we need to have same length of each input tensor in batch of input tensors?

Each input sentence can have a different number of tokens. You just need to
pass padding=True to the tokenizer, and it will append trailing padding
tokens where needed.

On Wed, Nov 29, 2023, 4:58 AM Yugal Jain ***@***.***> wrote:

> @jbochi <https://github.com/jbochi> Thanks for your response. Just one
> query, do we need to have same length of each input tensor in batch of
> input tensors?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/26696#issuecomment-1831573254>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACHO2F5JLV6CGQZYVRML3LYG4BMNAVCNFSM6AAAAAA5Y7RYEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZRGU3TGMRVGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@jbochi Thanks for your response. Btw do we need to set max_new_tokens in generate() method to 512 or 1024, or is it not necessary to set this parameter?

That is only useful if you have long inputs and don't want to translate the
whole thing.

On Thu, Dec 7, 2023, 2:09 AM Yugal Jain ***@***.***> wrote:

> @jbochi <https://github.com/jbochi> Thanks for your response. Btw do we
> need to set max_new_tokens in generate() method to 512 or 1024, or is it
> not necessary to set this parameter?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/26696#issuecomment-1844790765>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACHO2A7PGUHETHM4SI4MMDYIFTTLAVCNFSM6AAAAAA5Y7RYEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBUG44TANZWGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


>... Regarding the T5Decoder only model, might make sense to add it in the PR to support these models. If the modelling changes are not too involved could make sense to have it in T5 (Like smae code as T5Encoder).
> 


Hey @ArthurZucker, 
just saw that T5 decoder-only support has not yet been officially added to HF's T5 model? 
This May, I implemented a T5 `pseudo' decoder-only model based on the original HF T5 implementation; it seemed to run fine. You can find it here: https://github.com/jnaeg/T5PseudoDecOnly
As you can see, the changes to the original model are quite minimal; the architecture and its building blocks are very similar to what's already there. So maybe this might be a worthwhile addition to the standard model?

Disclaimer: I haven't touched it since May :joy: