noamaanMulla-03/transformers-issues huggingface dataset:
Multiple Mask support in Pipeline
Author: naveenjafer
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

The [fill mask](https://huggingface.co/bert-base-uncased?text=Paris+is+the+capital+of+%5BMASK%5D+%3F) feature as a part of the pipeline currently only supports a single mask for the inputs. It could be expanded to predict and return the results for multiple masks in the same sentence too.

## Motivation

There are use cases where one would ideally have more than just a single mask where they would need a prediction from the model. For example, smarter template filling in outputs returned to users etc. Could also be used in better study of the implicit knowledge that BERT models have accumulated during pre-training.

## Your contribution

I should be able to raise a PR for the same. The output JSON schema would have to be slightly modified, but I can go ahead and complete the same if there is no other obvious issue that slipped my mind as to why only a single [MASK] token needs to be supported.

Comments:
@LysandreJik 
The current implementation for a single mask returns the data as a list of 

```
{  
   "sequence" : "the final sequence with the mask added",  
   "score" :  "the softmax score",  
   "token" : "the token ID used in filling the MASK",  
   "token_str" : "the token string used in filling the MASK"  
}  
```

When returning the results for sentences with multiple masks, it is not possible to maintain the same return format of the JSON. I propose to have a different pipeline call for this 'fill-mask-multiple' or something along those lines.  The return format I have proceeded with is

```
{  
   "sequence" : "the final sequence with all the masks filled by the model,  
   "scores" :  ["the softmax score of mask 1",  "the softmax score of mask 2", ...]
   "tokens" : ["the token ID used in filling mask 1",  "the token ID used in filling mask 2", ...]
   "token_strs" : ["the token string used in filling mask 1",  "the token string used in filling mask 2", ...]
}  
```
Some minor changes will be made to the input param "targets" to support optional targets for each of the mask. 

If having 2 separate pipelines does not seem a great idea, we could just club them both right now into one single pipeline call irrespective of whether it is a single mask or multiple mask. The return json type would change, I am not sure about the impact/how feasible it would be to bring that across in minor version updates.  

Would really benefit from some expert advice since I am sort of new here.

PS: I have currently implemented the functionality for the pytorch framework, getting the same done in tf too.

This change seems okay to me. Since you have already some functionality for PyTorch, do you mind opening a PR (even a draft PR), so that we may play around with it and talk about the potential improvements? Thanks! Pinging @Narsil too
Direct way to apply different learning rate for different group of parameters in Trainer.
Author: liyucheng09
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

For now, if I want to specify learning rate to different parameter groups, I need to define an AdamW optimizer in my main function like the following:
```
optimizer = AdamW([{'params': model.classifier.parameters(), 'lr': 0.03 }], 
                 model.bert.parameters(), lr=5e-5
)
```
and new a lr_schedule like the following:
```
lr_scheduler = get_linear_schedule_with_warmup(
                self.optimizer, num_warmup_steps=self.args.warmup_steps, num_training_steps=num_training_steps
            )
```

I believe that adding the feature of specifying different learning rates to `Trainer` for networks is quite convenient for fine-tuning processing.

Like the following:

```
trainer = Trainer(
    ...
    grouped_parameters={
                    "params": ..,
                },
                {
                    "params": ..,
                },
            ]
```


## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I am not a professional Github user, but I think a can make a PR if necessary.


Comments:
What if instead you derive from `Trainer` and override `create_optimizer_and_scheduler()` and have that function set your different learning rates?
Store FLOS as floats to avoid overflow.
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in #10212, the `total_flos` stored as ints can result in overflowing errors: when in Python ints there is no risk, but when in distributed training, we use torch.int64 to gather all FLOS on all processes which can trigger that error.

Fixes #10212

Comments:
different behavior for get_input_embeddings()  between 4.2.x  and 4.3.x in Tensorflow
Author: AndyTheFactory
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.x vs 4.3.x
- Platform: Colab
- Python version: 3.6
- Tensorflow version (GPU?): 2.4.1

@jplu 

## Information

Model I am using (Bert, XLNet ...): Bert

## To reproduce

Steps to reproduce the behavior:

In version 4.3.x the following code 
```
model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
input = tokenizer.batch_encode_plus(['this is a test'])

embeddings = model.get_input_embeddings()
embeddings(input_ids=np.array(input['input_ids']), token_type_ids=np.array(input['token_type_ids']))

```

Throws an error:

> TypeError: call() got an unexpected keyword argument 'token_type_ids'

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

In version 4.2.x the code returned a tensor:

> <tf.Tensor: shape=(1, 6, 768), dtype=float32, [...]


Test also as [colab](https://colab.research.google.com/drive/1z5rboqdz8y8IM90FX8fDzixez9K_Efdy?usp=sharing) 




Comments:
Hello!

This is because in the 4.3 version the implementation of the embedding have changed and `get_input_embeddings()` returns only the word embeddings layer, hence only `input_ids` can be passed.

This was an unexpected behavior and will be fixed for the next release (the fix is already in master if you want). Sorry for the inconvenience.

thank you for the quick reply!

Ok, i was unaware it was already fixed in master

Thank you!
StopIteration Error when running beam search for squad 2.0
Author: neufang
Pull request: no

I'm using  `huggingface/transformers-pytorch-gpu:4.3.0` on Ubuntu DGX1 server with 8 V100 GPUs.
`NVIDIA-SMI 418.126.02   Driver Version: 418.126.02   CUDA Version: 10.1`

When running the step in `examples/question_answering/README.md` for beam search for squad 2.0

```
python run_qa_beam_search.py \
    --model_name_or_path xlnet-large-cased \
    --dataset_name squad_v2 \
    --do_train \
    --do_eval \
    --version_2_with_negative \
    --learning_rate 3e-5 \
    --num_train_epochs 4 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ./wwm_cased_finetuned_squad/ \
    --per_device_eval_batch_size=2  \
    --per_device_train_batch_size=2   \
    --save_steps 5000
```
Error 
```
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/xlnet/modeling_xlnet.py", line 1978, in forward
    start_logits = self.start_logits(hidden_states, p_mask=p_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 1241, in forward
    if next(self.parameters()).dtype == torch.float16:
StopIteration
```

Comments:
Indeed! I can reproduce, will fix.

Can you tell me if https://github.com/huggingface/transformers/pull/10215 fixes it? You can try by installing the following:

```
pip install git+https://github.com/huggingface/transformers@parameter-device-dtype
```

Thanks a lot for the quick fix. I'm running it right now. I will post whether the training is done in  ca. 7 hours.

> Can you tell me if #10215 fixes it? You can try by installing the following:
> 
> ```
> pip install git+https://github.com/huggingface/transformers@parameter-device-dtype
> ```

Hi, the issue is fixed. Thanks a lot.
No module named 'tasks'
Author: daniel-z-kaplan
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: 5.10.8-200.fc33.x86_64
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (GPU)
- Tensorflow version (GPU?): 2.4.1 (GPU)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

@sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): allenai/scibert_scivocab_uncased

The problem arises when using:
* [X ] the official example scripts: (give details below)

I'm using the old NER script, since the model I'm using doesn't support Fast Tokenizers. 
https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py
However, I get an error on trying to import "tasks.py". I did find a previous git issue for this, which recommended downloading said file.... but it no longer exists. 

The tasks I am working on is:
* [X ] my own task or dataset: (give details below)
I'm using the bc2gm-corpus dataset.

## To reproduce

Steps to reproduce the behaviour:

1. Download the script from the provided github link.
2. Run it with any real model name, a real directory for data (doesn't need to include data), and an output directory.
I used the following command: python3 run_ner.py --model_name_or_path allenai/scibert_scivocab_uncased --data_dir bc2gm-corpus/conll --output_dir ./output 

`  File "run_ner.py", line 323, in <module>
    main()
  File "run_ner.py", line 122, in main
    module = import_module("tasks")
  File "/home/dkaplan/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tasks' `
## Expected behavior

I expect the NER script to run, finetuning a model on the provided dataset. 


Comments:
I think you did not clone the repository properly or are not running the command from the folder `examples/legacy/token-classification`, since that folder does have a task.py file.

Ah, I was searching for "tasks.py" instead. Just user error, thanks for the fast reply.
T5 relative attention bias: Discrepancy to original implementation
Author: maurice-g
Pull request: no

### Who can help
@patrickvonplaten 

## Information

Model I am using: T5

In the huggingface TF T5 implementation, the relative attention bias only seems to be applied to the first layer of the stack. If I understand the original implementation correctly, though, it is applied to all layers there.

HF: https://github.com/huggingface/transformers/blob/master/src/transformers/models/t5/modeling_tf_t5.py#L570
Mesh: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/transformer_layers.py#L263

Comments:
The implementation of T5 is based on the original implementation which can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).

The implementation you are referring to is a general Transformer implementation from Tensorflow Mesh. It seems like this repo does not implement T5, but other models like the Funnel Transformer and the Evolved Transformer.



But the T5 repo actually calls the Tensorflow Mesh implementation internally. AFAIK there is no standalone t5 implementation (apart from the HF one).

Ok I see. In that case I'll leave it on to Patrick to help you. 

Hey @maurice-g, we compute it ones and then forward it to all the follow-up layers since the result will be the same for all layers. 2 month ago, I made sure that our implementation is exactly the same as the original implementation - see those tests: https://github.com/huggingface/transformers/blob/e94d63f6cbf5efe288e41d9840a96a5857090617/tests/test_modeling_t5.py#L734 and https://github.com/huggingface/transformers/blob/e94d63f6cbf5efe288e41d9840a96a5857090617/tests/test_modeling_tf_t5.py#L492 
So I'm confident that the model behaves as expected, but in case you have a reproducible code snippet showcasing differences between the 2 implementations, I'm more than happy to take a look :-)

You're right, missed that they were returned and forwarded to the other layers. Sorry for that.

No worries! Thanks for checking in-detail. I think it's always a very good practice to check things in-detail or more often than not you will find subtle bugs in Transformers that will help us improve the code :-)
Factor out methods
Author: LysandreJik
Pull request: yes

With PyTorch's DataParallel, it is not possible to simply iterate over parameters in order to find the `nn.Module`'s dtype or device.

Some efforts were made to catch the error (`StopIteration`) in most cases, but the some were forgotten. This PR factors the try/except in a method, which is applied everywhere instead.

Closes #10214

Comments:
@sgugger @patrickvonplaten verified this fixed the issue in #10214
Fix head masking for TFT5 models
Author: stancld
Pull request: yes

* This PR fixes head masking in TFT5 models (#9859)
* This PR further fixes the name of an error message variable from `__HEAD_MASK_WARNING_MSG` to `_HEAD_MASK_WARNING_MSG` as the former one was not working properly and raised error (double underscore made troubles)

<hr>

Fixes: #9859

Reviewers: @jplu 

Comments:
@stancld can you please rebase on master in order to solve the conflicts?

Thanks! The PR should be merged once @LysandreJik and @patrickvonplaten will have reviewed it.

Thanks for fixing it!
no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
Author: RayXu14
Pull request: no

what does this means? Whay these three kind no decay?

Comments:
Yes. We are reproducing the behavior of the original optimizer, see [here](https://github.com/google-research/bert/blob/master/optimization.py#L65).

thanks~

but why?

I have the same question, but did this prove to be better? Or is it just to speed up calculations?
Making TF BART-like models XLA and AMP compliant
Author: jplu
Pull request: yes

# What does this PR do?

This PR makes the TF BART-like models compliant with AMP and XLA. The main issue for XLA was all the asserts, XLA is not compliant with them (see the [TF doc](https://www.tensorflow.org/xla/known_issues)), so I had to disable them if the model is run with another mode than eager.

TF Marian and Pegasus have still their XLA test locked because they are not working for XLA_GPU. I need to investigate more in order to better understand why. My first guess is because of the `TFXSinusoidalPositionalEmbedding` class.

Comments:
I succeed to fix Marian and Pegasus, and my first guess was the good one. I basically reworked a bit how the embedding was created, and now it works in XLA_GPU. Of course, all the corresponding slow tests are passing, and the weights are properly loaded.
[trainer] refactor place_model_on_device logic, add deepspeed
Author: stas00
Pull request: yes

This PR:

* refactors 3 places of `place_model_on_device` logic - into one public attribute with the same name as the `TrainingArguments.place_model_on_device` attribute
* adds deepspeed to that logic (it was missing in 2 places)

@sgugger 

Comments:
Showing individual token and corresponding score during beam search
Author: monmanuela
Pull request: no

## Who can help
@patrickvonplaten

## Information

Hello,
 
I am using beam search with a pre-trained T5 model for summarization. I would like to visualize the beam search process by showing the tokens with the highest scores, and eventually the chosen beam like this diagram:
![image](https://user-images.githubusercontent.com/35655790/108174867-fc4eda00-713a-11eb-8565-dc3b0c87cf14.png)
(Taken from https://huggingface.co/blog/how-to-generate)
 
**I am unsure how I can show the tokens and their corresponding scores.**
 
I followed the discussion https://discuss.huggingface.co/t/announcement-generationoutputs-scores-attentions-and-hidden-states-now-available-as-outputs-to-generate/3094 and https://github.com/huggingface/transformers/pull/9150.
 
Following the docs, upon calling `generate`, I have set `return_dict_in_generate=True`, `output_scores=True`
 
```
generated_outputs = model_t5summary.generate(
  input_ids=input_ids.to(device),
  attention_mask=features['attention_mask'].to(device),
  max_length=input_ids.shape[-1] + 2,
  return_dict_in_generate=True,
  output_scores=True,
  output_hidden_states=True,
  output_attentions=True,
  no_repeat_ngram_size=2,
  early_stopping=True,
  num_return_sequences=3,
  num_beams=5,
)
```
 
Now I have an instance of `BeamSearchEncoderDecoderOutput`.
 
If I understand the docs (https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputs) correctly, `scores` will provide me with what I want but I am unsure on how to use the `scores`.

Any help/pointers from the community would be greatly appreciated, thank you 🙏 

Comments:
Hey @monmanuela,

Thanks for checking out the post! We try to keep the repository for github issues and kindly ask you to post these kinds of questions on the [forum](https://discuss.huggingface.co/). Feel free to tag me there (@patrickvonplaten) :-)

@patrickvonplaten thanks for your quick reply! Posted a topic on the forum, closing this issue.
Colab can't import trim_batch for T5, anything changed in transformers.tokenization_utils?
Author: yxu1168
Pull request: no
Labels: name: wontfix

from transformers.tokenization_utils import trim_batch

ImportError: cannot import name 'trim_batch'

Any solutions? Thanks a lot.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Had the same issue. Found a trim_batch implementation on the transformers repo, used it, didn't face any issues so far.

[Link to implementation](https://github.com/huggingface/transformers/blob/783d7d2629e97c5f0c5f9ef01b8c66410275c204/examples/research_projects/rag/utils_rag.py#L35)

Code for reference:

```python
def trim_batch(
    input_ids,
    pad_token_id,
    attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:
        return (input_ids[:, keep_column_mask], attention_mask[:, keep_column_mask])
```
microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract not available for tensorflow
Author: abhijithneilabraham
Pull request: no

@jplu
The above said model is available for pytorch but not for tensorflow. How to convert a pytorch checkpoint to tensorflow for this one? Is it possible for doing a contribution for the same?


Comments:
Hello!

You can load PyTorch weights into Tensorflow with `TFBertModel.from_pretrained("microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract", from_pt=True)`

That works, thanks!
Question about (no_decay = ['bias', 'LayerNorm.weight']) in BERT(Transformer-based) 
Author: fightnyy
Pull request: no

Hi, I have a Question about the BERT model code.
I saw "no_decay = ['bias', 'LayerNorm.weight']" in BERT code(especially, in Optimizer part). It seemed reasonable, however, did this prove to be better performance? Or is it just to speed up calculations?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Reduce the time spent for the TF slow tests
Author: jplu
Pull request: yes

# What does this PR do?

This PR reduces by half the time spent on running the all the tests (including the slow tests). Here are the time comparison (time recorded on my machine with the models already downloaded):

- albert: from 13mins to 6mins
- bart: from 19mins to 9mins
- bert: from 17mins to 9mins
- blenderbot_small: from 19mins to 9mins
- blenderbot: from 19mins to 9mins
- convbert: from 21mins to 11mins
- ctrl: from 10mins to 7mins
- distilbert: from 13mins to 7mins
- dpr: from 6mins to 3mins
- electra: from 15mins to 7mins
- flaubert: from 13mins to 7mins
- funnel: from 28mins to 13mins
- gpt2: from 8mins to 4mins
- led: from 44mins to 20mins
- longformer: from 1h30mins to 40mins
- lxmert: from 6mins to 3mins
- marian: from 19mins to 9mins
- mbart: from 19mins to 9mins
- mobilebert: from 33mins to 16mins
- mpnet: from 13mins to 7mins
- openai gpt: from 8mins to 4mins
- pegasus: from 19mins to 9mins
- roberta: from 10mins to 6mins
- t5: from 12mins to 7mins
- transfo_xl: from 8mins to 5mins
- xlm: from 13mins to 7mins
- xlnet: from 9mins to 5mins

Total: from 8h5mins to 4h8mins

The total time spent on running the entire tests has been reduced by half by merging three tests about the SavedModel into a single one.

Comments:
@sgugger Yes, this is exactly that. There was an important overlap across these three tests (all based on creating a saved model and two on testing the output), so merging them was IMO the best way to keep the coverage and reduce the time.

@patrickvonplaten feel free to merge if the PR looks ok for you!
[ci] scheduled job test
Author: stas00
Pull request: yes

please ignore

this time a branch on huggingface and not a fork

Comments:
well, the job never finished, something or something aborted the workflow - so the test wasn't complete.

The test was probably too long (>6 hours) and was stopped by CircleCI. This PR that was just merged should help in that regard: https://github.com/huggingface/transformers/pull/10152

That PR won't help, since in this test I removed both tf jobs - it was just one pt set of jobs per runner. Needed to do it since tf jobs were getting to run first.

But otherwise this is an awesome improvement for TF jobs!