noamaanMulla-03/transformers-issues huggingface dataset:
Tabert
Author: flozi00
Pull request: no
Labels: name: New model

name: Feature request

# 🌟 New model addition

## Model description
a pre-trained language model for learning joint representations of natural language utterances and (semi-)structured tables for semantic parsing. TaBERT is pre-trained on a massive corpus of 26M Web tables and their associated natural language context, and could be used as a drop-in replacement of a semantic parsers original encoder to compute representations for utterances and table schemas (columns).
<!-- Important information -->

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/facebookresearch/TaBERT
* [X] the model weights are available: (give details)
https://github.com/facebookresearch/TaBERT
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
In their readme they say that the implementation to this project is still WIP, but they said this on release.
It should be really easy to add it here while they are already using this library in their implementation.
Does the huggingface team has more information about this, if the community can open an PR for it or waiting for the original authors ?

Looking forward to this new model.

Hello all!

I looked at the list of models in the transformers site and TaBERT is still not listed. Does anyone know when it is going to be ready?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Not a stale, still cant see TaBert on HuggingFace List
![image](https://user-images.githubusercontent.com/22402962/107852163-f43c2500-6e30-11eb-8a7d-d4864d204a8f.png)
What does the "
 token" mean in Longformer's global_attention_mask?
Author: Anthonyive
Pull request: no

This might be a stupid question, but I couldn't find an answer. The documentation says "For example, for classification, the \<s\> token should be given global attention.". I've also checked the original [longformer paper](https://arxiv.org/pdf/2004.05150.pdf), but "\<s\> token" was only mentioned once. Can someone tell me what does it mean? Thanks for any help!

Comments:
Ok, I got it. That means [CLS] token.
Conditional generation with T5
Author: ShivanshuPurohit
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- t5: @patrickvonplaten, @patil-suraj

Library:

- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik

 -->

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] the official example scripts: (give details below)
* Generating conditional text from T5
```
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained('t5-3b')
model = T5ForConditionalGeneration.from_pretrained('t5-3b')
input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
labels = tokenizer('<extra_id_0> cute dog <extra_id_1> the <extra_id_2> 
', return_tensors='pt').input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits
input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
outputs = model.generate(input_ids)
```

## To reproduce

Steps to reproduce the behavior:

1. Run the code above

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
To see the generated text. Rather, the model outputs a torch Tensor like so
`tensor([[   0,  363,   19,    8, 1784,   13, 1473,   58,    1]])`
How do I get words out of it rather than a tensor?

Comments:
You can decode them back to a string using `T5Tokenizer`, like so:

`tokenizer.decode(outputs.squeeze().tolist(), skip_special_tokens=True)`

Btw, for a really good guide on the different generation strategies of models like T5, see this blog post: https://huggingface.co/blog/how-to-generate

This post was really helpful, thanks!
How to train an MBart model from scratch for a new language pair? 
Author: vineetha-thomas
Pull request: no

I want to train an MBART model from scratch, for a new language pair, unsupervised translation. I have monolingual data from both languages. Specifically, how do I prepare the data for the same? 

Currently I start with a code as follows

_tokenizer = MBartTokenizer.from_pretrained('./tokenizer_de_hsb.model') //My own tokenizer trained with google sentencepiece
batch = tokenizer.prepare_seq2seq_batch(src_texts=src_txts, src_lang="en_XX", 
                                                                tgt_texts=tgt_txts, tgt_lang="ro_RO",
                                                                 return_tensors="pt")   //The src and tgt language codes are dummy here. 
config = MBartConfig()
model = MBartModel(config)
model(input_ids=batch['input_ids'], decoder_input_ids=batch['labels']) # forward pass
model.save_pretrained('./trained_model')_

Following are the doubts I have.
- For pre-training mbart, what should input_ids and decoder_input_id in the forward pass be? Is there a function that generates the input with the masked tokens?
- Is the approach to combine src and tgt language data and train once on the combined data?
- Is there a sample code for this?



Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Back Translation
Author: chaituValKanO
Pull request: no

# 🚀 Feature request
I want to perform Back translation as a text data augmentation technique using TensorFlow

I want to augment data using translation techniques. I want to perform the below operation English ---> French ---> English. so that resulting english statement might be a new one. These sentences can be part of test data and will help to perform Behavioral Testing of NLP models.

## Motivation

I want to perform above operation using tensorflow. Currently there is no implementation of OPUS models for <any source text> to English. All the models are available in pytorch and not in tensorflow. This is really irritating. 

## Your contribution

I am not sure, if I can. But you can assign me simple tasks


Comments:
Hi @chaituValKanO 

Please use the [forum](https://discuss.huggingface.co/) to ask such questions. Issues are for bugs, feature requests etc. 

And to answer your question you could use the `MarianMT` models for this purpose.  Here's a nice blog-post about that
https://amitness.com/back-translation/.

Will close this issue. Thanks!

Hello Suraj,

But the issue is that there are no Pre trained models for MarianMT under
Tensorflow framework where target is English language.

Nevertheless, I will check with people in forum.
Thanks and regards,
Chaitanya Kanth.

On Fri, 12 Feb 2021 at 5:46 PM, Suraj Patil <notifications@github.com>
wrote:

> Closed #10138 <https://github.com/huggingface/transformers/issues/10138>.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10138#event-4324522055>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB6VJCAIPJNM36CXVKH37QDS6UL2XANCNFSM4XOTUU4A>
> .
>
-- 
Sent from my iphone


Marian is available in TF as well, you'll just need to pass `from_pt=True` to `from_pretrained` when loading the TF modle


Thanks Suraj 😁

On Fri, Feb 12, 2021 at 6:22 PM Suraj Patil <notifications@github.com>
wrote:

> Marian is available in TF as well, you'll just need to pass from_pt=True
> to from_pretrained when loading the TF modle
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10138#issuecomment-778177918>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB6VJCAC53RGMVY5OCVVYRLS6UQCXANCNFSM4XOTUU4A>
> .
>
Why is the attention_mask added to the attn_weights instead of multiplying/masking?
Author: fomalhautb
Pull request: no

https://github.com/huggingface/transformers/blob/8fae93ca1972c39d19c8cf3d3c6a3dd2530cc59a/src/transformers/models/bart/modeling_bart.py#L219-L227

As far as I understand, attention_mask is to prevent the model peek into the future or padded positions, shouldn't the weights in these positions be masked out? What does this addition do?

Comments:
Hello, Maybe this comment can help you out https://github.com/huggingface/transformers/issues/1935#issuecomment-561305086!

@LysandreJik Oh, I got it now, thank you!
Problem with evaluation_strategy
Author: mtortoli
Pull request: no

Hi everyone!
I have a problem (i think is a bug but i'm not sure) with the parameter "evaluation_strategy" in TFTrainingArguments.
I created a script for finetuning a transfomers model, based on the example "run_tf_text_classification.py" file.

In "TFTrainingArguments" i put the parameter "evaluation_strategy="epoch"", to see how the eval_loss change after  each epoch.

Unfortunately, the eval_loss is not printed after each epoch, but if a change from "epoch" to "steps", actually the eval_loss is printed after each steps.

Comments:
`evaluation_strategy` is not an argument fully implemented in `TFTrainer`, it only supports "steps". (The PyTorch counterpart supports all the possibilities.)
To evaluate every epoch, the best is to use the native Keras fit method.

Thanks a lot!!
Fix datasets set_format
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes a problem in `Trainer` when user provide a dataset using the new functionality in the upcoming v2 of `datasets` `set_transform` (see [here](https://github.com/huggingface/datasets/issues/1867) for more details). This is a hotfix that is not perfect and we will need to take some time to make this column ignoring more general (probably after batch creation) but I will look deeper into this at the end of next week.

Comments:
cant install from source
Author: prathameshk
Pull request: no

## Environment info
transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 3.3.1
- Platform: Linux-4.19.0-13-cloud-amd64-x86_64-with-debian-10.7
- Python version: 3.7.8
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

## To reproduce
pip install git+https://github.com/huggingface/transformers

Collecting git+https://github.com/huggingface/transformers
  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-BEpAl9
  Installing build dependencies ... done
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-BEpAl9/setup.py", line 192
        entries = "\n".join([f'    "{k}": "{v}",' for k, v in deps.items()])
                                                ^
    SyntaxError: invalid syntax




Comments:
Hi,

maybe `pip` is not connected to a proper Python version (f-strings work in >= 3.6). To make sure that the right version gets called, you can execute the following command:
```
python -m pip install git+https://github.com/huggingface/transformers
```

This is why it's a good idea to use a virtual env.

Let me know if this helps.

this was issue with the wrong pip version
[RAG] fix tokenizer
Author: patil-suraj
Pull request: yes

# What does this PR do?

- Introduce `as_target_tokenizer` context manager in `RagTokenizer` to later update the docs when `prepare_seq2seq_batch` is depricated.
- `RagTokenizer.prepare_seq2seq_batch` calls `super().prepare_seq2seq_batch`, but it does not inherit from `PreTrainedTokenizer`. Fix the method temporarily using the context manager.

Comments:
Add mBART-50
Author: patil-suraj
Pull request: yes

# What does this PR do?

This is the second part of splitting #9811

This PR adds the mBART-50 models.
- Add `MBart50Tokenizer` and `MBart50TokenizerFast`. A new tokenizer is needed because it adds extra languages and the encoding format is different than `MBartTokenizer`. The difference is that for `mbart-50` both source and target language text begin with the `<language token>`,  whereas for `mbart-cc25` `<language_token>` is used as suffix token.

- The new tokenizers use `src_lang` as a `getter` and `setter` property. This is needed because for many-to-many translation models whenever we change the `src_lang` we need to set special tokens for that language. The `src_lang.setter` calls `set_src_lang_special_tokens` method whenever a we set a new `src_lang` to handle this.

-  A new model class is not necessary as mBART-50 is similar to our existing mBART-25 model, the only difference being `relu` activation instead of `gelu` and emb size of 250054 instead of 250027

All model checkpoints are uploaded on hub  https://huggingface.co/models?filter=mbart-50

Comments:
> As a follow-up, is mBART aligned with mBART-50? We should have the same setter there. It would make a nice first issue I believe, once this PR is merged and provides a good model.

Yes. Here the setter was necessary because of the many to many models. But yes mBART can also be used for multilingual fine-tuning so the tokenizers should also be aligned.

Hi, I want to use MBart-Large-50 for finetuning, but I get the error:
`File "/home/michael/anaconda3/envs/paraphrases/lib/python3.9/site-packages/transformers/models/mbart/tokenization_mbart.py", line 199, in set_src_lang_special_tokens
    self.cur_lang_code = self.lang_code_to_id[src_lang]
KeyError: None
`
It was working with the previous version of the model. Is it correct, that this PR adresses this issue that MBartTokenizer is used and not MBart50Tokenizer?

Hey @MichaelJanz 

For mBART-50 you should use the `MBart50Tokenizer`. Also when fine-tuning make sure that you either pass or set the `src_lang` and `tgt_lang` attributes

Hi @patil-suraj and thanks for answering!
I am using the script under examples/seq2seq run_seq2seq.py, which has no reference to the `MBart50Tokenizer`, but I think it should have. `src_lang` and `tgt_lang` are set. I suspect that the base class of `MBart50Tokenizer` is used, which is simply `MBartTokenizer`. Will the script work when this commit is merged, or are there further changes neccesary, or am I executing the script wrong?

`MBart50Tokenizer` does not inherit from `MBartTokenizer` and for now, the script does not support mBART-50, but you could easily modify the script for mBART-50. I think the only necessary change is to use the correct tokenizer class. This PR will be merged today, please open an issue if you have more questions after the PR is merged. Happy to answer :)

Thanks for your help!
If I can get it to work, I will create a PR. Thanks for your great work :)

Thanks a lot, everyone! Merging!
Add AMP for TF Albert
Author: jplu
Pull request: yes

# What does this PR do?

This PR adds the following features to TF Albert:

- AMP compliancy
- Loss computation for TFAlbertForPreTraining
- Cleaning source code

Comments:
I can split this PR into two different ones, but the one on AMP will be very short (only two single line to update, see the review above). Are you agree with a that tiny PR? If it is still ok, I will split this one^^

It's ok, thanks for showing me the changes!

@patrickvonplaten feel free to merge if it looks ok for you!
Uploaded a new model but is not found on the hub.
Author: zolekode
Pull request: no
Labels: name: New model

# 🌟 New model addition
I recently added this model: https://huggingface.co/flexudy/t5-small-wav2vec2-grammar-fixer

However, I get this error whilst trying to download it.
```
Can't load tokenizer for 'flexudy/t5-small-wav2vec2-grammar-fixer'
```
How can I fix it please?


Comments:
Hi @zolekode ,

the folder structure is not quite correct:

https://huggingface.co/flexudy/t5-small-wav2vec2-grammar-fixer/tree/main

You just need to move everything from the `t5-small-wav2vec2-grammar-fixer` folder to the root folder. Then it should work :hugs: 

Ah I see. awesome. Thanks alot @stefan-it
fix run_seq2seq.py; porting trainer tests to it
Author: stas00
Pull request: yes

This PR:

- restores some of the essential dropped functionality from `finetune_trainer.py` - I'm almost sure this is far far from complete since so much was just dropped 
- ports wmt_en_ro test data to `jsonlines` - I move the tests dataset into the root of examples so that it can be accessed by a variety of sub-projects.
- ports DeepSpeed tests to use `run_seq2seq.py`
- ports the other trainer script to use `run_seq2seq.py`

@sgugger 

Comments:
OK, I decided to go ahead and port the other scripts instead of waiting for merging of the first set. Had to make some more fixes in the script while at it.
ONNX Export for Fine-Tuned DistilBertForTokenClassification
Author: biro-mark
Pull request: no

# 🚀 Feature request

I'd like to export a fine-tuned DistilBertForTokenClassification model to ONNX. Right now the conversion script convert_graph_to_onnx.py looks like it takes a string like "--model bert-base-cased" but I'd like to pass in the model object that I've fine tuned on my data set. 

## Motivation

I've trained a custom token classifier that I want to run on the ONNX runtime. 


Comments:
I figured out how to do it. Within python, you have to use `model.save_pretrained("path/to/output_dir")` where `model` is the fine-tuned `model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased', num_labels=...)`

Then inside an empty directory, run
`python -m transformers.convert_graph_to_onnx --model path/to/output_dir --framework pt --tokenizer distilbert-base-uncased out.onnx`
Add SageMakerTrainer for model paralellism
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a subclass of `Trainer` to use model parallelism in SageMaker. This new `Trainer` still retains all the functionality of the previous trainer (e.g. when not run in SageMaker it will work like the normal one) while automatically enabling model parallelism when an example script is launched via SageMaker with that option activated.

The easiest way to enable this in any example script is to replace the `Trainer` and `TrainingArguments` imports by:
```python
from transformers.sagemaker import SageMakerTrainingArguments as TrainingArguments, SageMakerTrainer as Trainer
```

Along the way, I had to refactor a few things in `Trainer` to make it easier to deal with stuff in the subclass (without having to rewrite the whole train method for instance), mainly the part that does the model wrapping. Also, there was a subtle bug coming from the fact SageMaker wrapper for the model for model parallelism changes the forward method of the model, so the `Trainer` will now store the arguments in the signature (in case that signature changes after wrapping the model).

Comments:
`super()` does not have `prepare_seq2seq_batch()`  in `transformers/models/rag/tokenization_rag.py` 
Author: moyapchen
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.12
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: nope

### Who can help

Models:
- rag: @patrickvonplaten, @lhoestq

## Information

Model I am using (Bert, XLNet ...): RAG

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run any of the scripts in the examples on https://huggingface.co/transformers/model_doc/rag.html#overview , ex. 
```
from transformers import RagTokenizer, RagRetriever, RagModel
import torch
tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
retriever = RagRetriever.from_pretrained("facebook/rag-token-base", index_name="exact", use_dummy_dataset=True)
# initialize with RagRetriever to do everything in one forward call
model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever)
input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="pt")
input_ids = input_dict["input_ids"]
outputs = model(input_ids=input_ids)
```

2. Get an error on https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/tokenization_rag.py#L77 about how `super()` does not have `prepare_seq2seq_batch()` 
   * Indeed, looking at the relevant file, RagTokenizer does not inherit from any other class. 

## Expected behavior

RAG works properly. 

Note that if I copy/paste the code in the file prior to https://github.com/huggingface/transformers/pull/9524 , it works fine. CC: @sgugger of that change. 


Comments:
Hi ! Thanks for reporting
#10167 should fix this issue

Convenient to see that the fix was already in the pipeline. Thanks!
How to fine-tune T5/Bart for other languages on summarization?
Author: HeroadZ
Pull request: no

Assume that I have a Japanese dataset for fine-tuning. How could I fine-tune it? 
I think that the original tokenizer like `BartTokenizer` or `T5Tokenizer` can't be used for Japanese, right?
So is it possible to use a Japanese tokenizer like `BertJapaneseTokenizer` to fine-tune a Bart model? Please give me some advice. Thank you very much.

Comments:
Hi! The `BertJapaneseTokenizer` you mention was created specifically for Japanese, so it should encode the Japanese language well.

You can find the list of models that have Japanese checkpoints [here](https://huggingface.co/models?filter=ja).

Tokenizers can be decoupled from their models, so you can indeed use a BERT tokenizer with a BART model; however, this requires the tokenizer and model to be trained together.

Thanks for the quick reply! 
I don't know the exact procedure to train tokenizer and model **together**. Could you explain it in detail?