noamaanMulla-03/transformers-issues huggingface dataset:
Speedup T5 Flax training by using Numpy instead of JAX for batch shuffling
Author: yhavinga
Pull request: yes

# What does this PR do?

The example Flax T5 MLM script was modified to not use JAX for batch shuffling.

With this change, training time per step decreased from 0.246 s to 0.185 s, a speedup of 1.3 (t5-base model trained on TPU v3-8 VM, batch size 64, Adafactor optimizer).

Shuffling on CPU also prevents using memory on the accelerator devices for the batch index array, which is significant for the large datasets that are typically used with pre-training. For instance, the batch index array size for a 30GB dataset with sequence length 512 is 500MB.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. @patrickvonplaten 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15963). All of your documentation changes will be reflected on that endpoint.

Think I'm in favor of this. However could we also set a numpy random seed in the script to make the runs reproducible? 

cc @patil-suraj - what do you think here? 

> However could we also set a numpy random seed in the script to make the runs reproducible?

The script calls trainer_utils.py's set_seed() which amongst others sets numpy's seed.
Do a pull in case docs were updated during build
Author: sgugger
Pull request: yes

# What does this PR do?

Now that the build of the documentation takes a long time, it can happen that the doc has changed when trying to update the doc-build repo in the case of a release (for instance there was a failure with the v4.17.0 release doc update since the master doc was updated during its build).
This PR stashes, pulls and pops the stash to avoid that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15922). All of your documentation changes will be reflected on that endpoint.
Fix TFEncDecModelTest - Pytorch device
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix pytorch device issue for `test_pt_tf_equivalence` in `TFEncoderDecoderMixin`.

@patrickvonplaten @sgugger 

## Remark:

There are other issues to fix for TFEncoderDecoder test pass:
  1. use small models for `test_real_model_save_load_from_pretrained`
  2. `test_bert2bert_summarization` test fails after #15562
 
Think these should be addressed in separate PRs (especially for 2.)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15979). All of your documentation changes will be reflected on that endpoint.
add special tokens does not work in GPT2Tokenizer
Author: ahxchzt
Pull request: no

## Environment info

- `transformers` version: 3.5.0
- Platform: Pytorch
- Python version: 3.7.5
- PyTorch version (GPU?): torch-1.6.0 cuda102

Models: GPT-2 @patrickvonplaten

- Tokenizers: @SaulLu


## Information

I want to add some special tokens to the GPT2 vocab but the function "add_ special_tokens" do not work.
My code just like this

`tokenizer=GPT2Tokenizer.from_pretrained(gpt2path)`

`kb=torch.load(kbpath)`

`tokenizer.add_special_tokens({'additional_special_tokens':list(kb)})`

`print(tokenizer.tokenize("i'd like to book a table at resto_bombay_expensive_british_4stars_2",add_special_tokens=True))`

`print("resto_bombay_expensive_british_4stars_2" in list(kb))`

The output of last print is 
**"True"**, 

so I think the special tokens in KB has been added into GPT2 vocab.
But the output of the tokenizer.tokenize(...) is 
**['i', "'d", 'Ġlike', 'Ġto', 'Ġbook', 'Ġa', 'Ġtable', 'Ġat', 'Ġrest', 'o', '_', 'bomb', 'ay', '_', 'expensive', '_', 'b', 'rit', 'ish', '_', '4', 'stars', '_', '2']**

The special tokens is splited. Why?

## Expected behavior

**['i', "'d", 'Ġlike', 'Ġto', 'Ġbook', 'Ġa', 'Ġtable', 'Ġat', 'Ġresto_bombay_expensive_british_4stars_2']**


Comments:
Hey @ahxchzt,

Thanks for your issue.
Could you please provide at code-snippet that one can copy-paste into a Python environment and execute directly. Your code snippet doesn't work because:

- 1) `GPT2Tokenizer` is not imported 
- 2) No one has access to `gpt2path` nor `kbpath` so we can't reproduce the error at all even though we know how to do 1)

Could you make sure the code snippet is reproducible? 

Thanks for your reply.@patrickvonplaten
I updated my transformers to 4.17.0 and the problem had been solved. I think the issue can be closed.
TF: Update multiple choice example
Author: gante
Pull request: yes

# What does this PR do?

Updates the TF example for multiple choice.

This example update is not as clean as the previous ones -- despite the deletion of a custom dataset function, it required the addition of a custom data collator ([just like in its PT counterpart](https://github.com/huggingface/transformers/blob/master/examples/pytorch/multiple-choice/run_swag.py#L150)).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15868). All of your documentation changes will be reflected on that endpoint.
[Env Command] Add hf hub to env version command
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

Adds the HF hub to the env command


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15981). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15981). All of your documentation changes will be reflected on that endpoint.

Doc dev test is green so good to merge no?
getting error while training bert language model. "ValueError: Expected input batch_size (8) to match target batch_size (1024)."
Author: bharathrajcl
Pull request: no
Labels: name: wontfix

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
from transformers import BertForSequenceClassification
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

%%time
from transformers import LineByLineTextDataset,TextDataset

paths = '/content/drive/My Drive/MyFile.txt'
dataset = TextDataset(
    tokenizer=tokenizer,
    file_path=paths,
    block_size=128,
)

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./EsperBERTo",
    overwrite_output_dir=True,
    num_train_epochs=1,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

%%time
trainer.train()

**'**''''''''''''''''''''''''''''''''''''''Getting error after executing trainer.train() ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''****

ValueError                                Traceback (most recent call last)
<ipython-input-12-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

10 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2214     if input.size(0) != target.size(0):
   2215         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2216                          .format(input.size(0), target.size(0)))
   2217     if dim == 2:
   2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

**ValueError: Expected input batch_size (8) to match target batch_size (1024).**

please help me to resolve this issue.

Comments:
the same probleme ??

Expected input batch_size (15) to match target batch_size (0).

Same problem

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same problem.

Same here! Anyone resolved it yet? @bharathrajcl @chaima-ai @gborodin @RufusGladiuz
is_index_masked and is_index_global_attn in Longformer
Author: lightxu
Pull request: no

I noticed that in [LongformerEncoder](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1255), is_index_masked and is_index_global_attn is set like this:

```
is_index_masked = attention_mask < 0
is_index_global_attn = attention_mask > 0
```

Can anyone help me understand what does these two variables do and why they are set in this way?

Note that in [LongformerModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1599), we have final attention_mask => 0 for no attention, 1 for local attention 2 for global attention.

Which means in LongformerEncoder, we will get is_index_masked as all False, and is_index_global_attn as all True except for no attention token. This is quite confusing to me.

It looks to me that the correct value should be like this, though I could be wrong:

```
is_index_masked = attention_mask == 0
is_index_global_attn = attention_mask == 2
```

Any help is appreciated!

Comments:
cc @patrickvonplaten 

Hey @lightxu,

What we usually do in `transformers` is to convert an attention_mask which looks like:

```python
attention_mask = [[1, 1, 1, 1], [1, 0, 0, 0]]
```

to something like:

```python
attention_mask = torch.broadcast_to([[0.0, 0.0, 0.0, 0.0], [0, -10000.0, -10000,0, -10000,0]], (1, 1, 4, 4))
```

since a `1` means that the vector at this time step of the `QK^T` matrix is **not** masked (+ 0.0) and `0` means that it is masked (+-10000.0). This happens for all models and is standard practice for transformer models. 

Now Longformer is special because it requires2 types of attention mechanism inside meaning that in addition to `0.0` and `-10000.0` we also add `10000.0` which is why we have the statement:

```python
is_index_masked = attention_mask < 0
is_index_global_attn = attention_mask > 0
```

Does that make sense?

Hi @patrickvonplaten 

This does make sense, since I also see corresponding comments in [LongformerSelfAttention](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L561). I am trying to understand where does this conversion happen, is it happenining here in [LongformerModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1699)?

For the context, I am trying to adapt distilbert model to use longformer self attention. The structure of [distilbert model](https://github.com/huggingface/transformers/blob/v4.3.0/src/transformers/models/distilbert/modeling_distilbert.py#L474) looks pretty simple, and I couldn't find anything that broadcast the attention_mask.

Found the conversion in [`get_extended_attention_mask`](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L311) method. So I think the logic in Longformer is correct, I will try to make similar changes for my long version of distilbert model. Thanks!
Backprop Test for Freeze FlaxWav2Vec2 Feature Encoder
Author: sanchit-gandhi
Pull request: yes

This PR correctly implements a back propagation test to verify the functionality of the `freeze_feature_encoder` argument added to the FlaxWav2Vec2 Model in #15873. It tests:

1. That the computed loss for the frozen feature encoder model and unfrozen model are **equal**.
2. That the gradients of the frozen feature encoder **differ** to those of the unfrozen feature encoder.
3. That the gradients of all other unfrozen layers remain **equal**.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15938). All of your documentation changes will be reflected on that endpoint.

If @patil-suraj is happy with this test I'll merge!
How to insert a hidden output from GPT2 model directly into a BERT layer?
Author: h56cho
Pull request: no

Hello,

I am trying to do the following:
1. Feed in the input_ids into the embedding layer of the pre-trained GPT2 model, and get the resulting embedding.
2. directly feed the embedding from 1. to each layer of the pre-trained GPT2, by using `GPT2_model.transformers.h`.
Store the resulting hidden output from each layer of the GPT-2 in a tensor named `layer_hidden_state`.
3. Directly input the `layer_hidden_state` into the 1st layer (which is on the top of the embedding layer) of the pre-trained BertModel, and let BertModel process the `layer_hidden_state` until it reaches the uppermost layer of BertModel.

My attempts for carrying out the above steps are shown below.
But I am getting an error when I try to do the step 3...how can I fix my error? the error is shown at the bottom of my code.

Thank you for the help,

```Python
    # turn on the evaluation mode
    # (to prevent the dropout for evaluation purpose).
    gpt2DoubleHeadsModel.eval()
    len_input_ids = len(input_ids)

    
    # get the hidden state vector from the embedding layer.
    # we will use this hidden state vector as an input to each layer.
    input_hidden_state = gpt2DoubleHeadsModel(input_ids=input_ids,
                                                         mc_token_ids = mc_token_ids,
                                                         token_type_ids = token_type_ids,
                                                         attention_mask = attention_mask)[3][0][:,:,:].detach()

    for j in range(num_layer_gpt2):
            
        # directly feed in the embedding hidden state vector into each layer of the GPT2DoubleHeadsModel,
        # and retrieve the resulting hidden state vector from each layer.
        layer_hidden_state = \
                    gpt2DoubleHeadsModel.transformer.h[j](input_hidden_state)[0][:,(len_input_ids-1),:]
            
        # store the hidden state vectors of the last token from each layer in `last_hidden_output_tensor`.
        last_hidden_output_tensor[:,j,:] = layer_hidden_state
        
    last_hidden_output_tensor = tuple(last_hidden_output_tensor)
        
    best_model_bert = BertModel.from_pretrained('bert-large-uncased', output_hidden_states=True)
    
    # an error is generated here the error is shown below:    
    for k in range(nlayer_bert):
        last_hidden_output_tensor = best_model_bert.encoder.layer[k]((last_hidden_output_tensor)[0])

"""
 error: 
  File "/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 239, in transpose_for_scores
    return x.permute(0, 2, 1, 3)

RuntimeError: number of dims don't match in permute
"""
```

Comments:
how do you fix it ?
Removed an outdated check about hdf5_version
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove a check `self.assertTrue(h5py.version.hdf5_version.startswith("1.10"))` which is outdated and makes the CI daily test fail.

@LysandreJik @sgugger 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16011). All of your documentation changes will be reflected on that endpoint.
Swag example: Update doc format
Author: gante
Pull request: yes

# What does this PR do?

Updates the doc format in the swag examples (requested [here](https://github.com/huggingface/transformers/pull/15868#discussion_r816869590))

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16014). All of your documentation changes will be reflected on that endpoint.
Parse in tensorflow strings as well as normal strings
Author: sachinruk
Pull request: no
Labels: name: wontfix

# 🚀 Feature request
So this is something I came across while trying to get a (fast) streaming dataset via the tensorflow data API for TPUs. This is a crosspost of [this SO question](https://stackoverflow.com/questions/61555097/mapping-text-data-through-huggingface-tokenizer/). 

I have my encode function that looks like this:
```python
from transformers import BertTokenizer, BertModel

MODEL = 'bert-base-multilingual-uncased'
tokenizer = BertTokenizer.from_pretrained(MODEL)

def encode(texts, tokenizer=tokenizer, maxlen=10):
#     import pdb; pdb.set_trace()
    inputs = tokenizer.encode_plus(
        texts,
        return_tensors='tf',
        return_attention_masks=True, 
        return_token_type_ids=True,
        pad_to_max_length=True,
        max_length=maxlen
    )
    
    return inputs['input_ids'], inputs["token_type_ids"], inputs["attention_mask"]
```
I want to get my data **encoded on the fly** by doing this:
```python
x_train = (tf.data.Dataset.from_tensor_slices(df_train.comment_text.astype(str).values)
           .map(encode))
```
However, this chucks the error:
```
ValueError: Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
```

Now from my understanding when I set a breakpoint inside `encode` it was because I was sending a non-numpy array. How do I get huggingface transformers to play nice with tensorflow strings as inputs?

If you need a dummy dataframe here it is:
```
df_train = pd.DataFrame({'comment_text': ['Today was a good day']*5})
```
## What I tried
So I tried to use `from_generator` so that I can parse in the strings to the `encode_plus` function. However, this does not work with TPUs.
```python
AUTO = tf.data.experimental.AUTOTUNE

def get_gen(df):
    def gen():
        for i in range(len(df)):
            yield encode(df.loc[i, 'comment_text']) , df.loc[i, 'toxic']
    return gen

shapes = ((tf.TensorShape([maxlen]), tf.TensorShape([maxlen]), tf.TensorShape([maxlen])), tf.TensorShape([]))

train_dataset = tf.data.Dataset.from_generator(
    get_gen(df_train),
    ((tf.int32, tf.int32, tf.int32), tf.int32),
    shapes
)
train_dataset = train_dataset.batch(BATCH_SIZE).prefetch(AUTO)
```

My final solution was to encode the all the strings using `batch_encode_plus` but to write those tensors out to disk. This currently takes ~1 hour, but seems like an inefficient way of prepping data for training.
## Version Info:
`transformers.__version__, tf.__version__` => `('2.7.0', '2.1.0')`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey, any updates about this issue? I'm facing the exact same problem!

@sachinruk @joaopedromattos any luck here in the end :)
Make `pos` optional in `PerceiverAudioPreprocessor` to avoid crashing `PerceiverModel` operation
Author: basilevh
Pull request: yes

Updates `PerceiverAudioPreprocessor` `forward()` implementation to match most other preprocessors / postprocessors.

Fixes #15971.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging this to solve your issue in the meantime.
Fix MaskFormer failing test on master
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This PR fixes a failing test on master, the fix was easy. In the test I was checking for `transformer_decoder_hidden_states` that only exists when `output_hidden_states == True`, what I needed to check was `transformer_decoder_last_hidden_state`. 

I have also made `output_hidden_states` depend on `config.use_auxiliary_loss` for the `MaskFormerModel` inside `MaskFormerForInstanceSegmentation` since we need all the `hidden_states` when computing the auxiliary loss

Comments:
_The documentation is not available anymore as the PR was closed or merged._