noamaanMulla-03/transformers-issues huggingface dataset:
'BertTokenizer' object has no attribute 'decode'
Author: merleyc
Pull request: no

## Environment info
- `transformers` version: 4.4.2 and 3.5.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core (centos 7)
- Python version: 3.7.7
- PyTorch version (GPU?): CPU torch=1.6.0
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
- tokenizers-0.10.2 
- pytorch-pretrained-bert=0.6.2 (installed by using 'pip install pytorch_pretrained_bert'
- Using jupyter notebook

### Who can help
- tokenizers: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): BertTokenizer

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
When I run the code below I am getting the following error:
```
AttributeError                            Traceback (most recent call last)
<timed exec> in <module>
<timed exec> in <listcomp>(.0)
AttributeError: 'BertTokenizer' object has no attribute 'decode'
```

```
from pytorch_pretrained_bert import BertTokenizer
import pandas as pd
from tqdm.notebook import tqdm
import transformers
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)
df = pd.read_parquet('file.parquet', engine="pyarrow")
df['coll'] = [ tokenizer.decode( [ int(n) for n in t.split('\t') ] ) for t in tqdm(df.text_tokens.values) ] 
```

## Expected behavior
Get the tokenized words.

Thank you!

Comments:
@merleyc Instead of using pytorch_pretrained_bert try using transformers:- `from transformers import BertTokenizer`.

Thanks a lot, @frankhart2018! It works.
[WIP] GPT Neo cleanup
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR refactors the `GPTNeoLocalSelfAttention` layer and adds more tests for it.

This PR
- adds the `AttentionMixin` class which contains the shared utilities for both global and local attention. The class is meant to be used as a mixin and makes it easy to test it.
- the `look_around` method is now replaced by the `AttentionMixin._look_back` method, which is now vectorized and can give up to 300x speed-up compared to old `look_around`
- The `GPTNeoLocalSelfAttention._create_attention_mask`  is now simplified and is also giving nice speed-up as it uses `_look_back`. I've added more detailed comments to explain the mask creation logic.
- I've added multiple shape checks in the  `AttentionMixin` to make it as robust as possible.

I didn't do a thorough benchmarking but I'm observing around ~3.9x speed-up when generating sequences of length 1024.

Verified that all slow-tets are passing.

Comments:
As explained in https://github.com/huggingface/transformers/issues/11076#issuecomment-814218202, the loss did decrease over time on this small sample so it looks like there are no regressions w.r.t training.

~Merge when ready @patil-suraj.~ :point_down: 

It seems I'm not passing the slow tests locally, `test_gpt_neo_sample` fails with:

```
AssertionError: 'Today is a nice day and a wonderful time to be in Rome, though the sun won’' != 'Today is a nice day and if you don’t get the memo here is what you can'
- Today is a nice day and a wonderful time to be in Rome, though the sun won’
+ Today is a nice day and if you don’t get the memo here is what you can
```

As seen with @patil-suraj, this is due to a wrongly initialized seed; and the other tests ensure that we have a correct attention mask and generation. Merging!
Auto feature extractor
Author: sgugger
Pull request: yes

# What does this PR do?

This PR does a few things that are all tangled together to bring a new `AutoFeatureExtractor` API.

First it removes the check at init for `Speech2TextFeatureExtractor` to only import this object when the `speech` dep is installed and add a dummy class if it's not, so it's easier to add speech objects in the future (similar to what we did for vision) cc @patil-suraj

Then it adds a new `AutoFeatureExtractor` that works like `AutoConfig`: it picks the right class by either looking through the config to check if a `feature_extractor_type` is present then falls back to pattern matching on the name of the checkpoint (so it will work with existing checkpoint if their names contains the model code).



Comments:
Really awesome! Thanks a lot Sylvain!
Using BERTModel for learning a siamese encoder
Author: wickedsprout
Pull request: no

Hi! This is more of a question than a bug report. Can I use BERTModel without any modifications to train a BERT-based siamese encoder?

(Not sure if this really is a BERT-specific question, but I will tag @LysandreJik just in case)

This is how my training step looks like:

```
optimizer.zero_grad()

outputs_a = model(input_ids_a, attention_mask=attention_mask_a)
outputs_b = model(input_ids_b, attention_mask=attention_mask_b)
a = torch.mean(outputs_a['last_hidden_state'], axis=1)
b = torch.mean(outputs_b['last_hidden_state'], axis=1)
cossim_normalized = (cossim(a, b) + 1) / 2

loss = bcentropy(cossim_normalized, labels)
loss.backward()
optimizer.step()
```

Should this work? Most other examples of siamese models in PyTorch simply modify the forward pass to include the second input, but I don't see why gradients shouldn't accumulate properly in my case.

Comments:
Hi,

Can you please ask this question on the [forum](https://discuss.huggingface.co/) rather than here? For example, [this comment](https://discuss.huggingface.co/t/what-are-some-recommended-pretrained-models-for-extracting-semantic-feature-on-single-sentence/2698/3?) might help you already.

The authors of HuggingFace like to keep Github issues for bugs/feature requests.

Thank you!


Thank you, closing this.
dead link fixed
Author: cronoik
Pull request: yes

# What does this PR do?

Fixes a dead link of the documentation.


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR:

@patrickvonplaten: @sgugger


Comments:
GPTNeo: importing model with padded vocab size should truncate wte
Author: leogao2
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: latest from `master`

### Who can help


@LysandreJik

## Information

Model I am using (Bert, XLNet ...): GPTNeo

Script: https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt_neo/convert_gpt_neo_mesh_tf_to_pytorch.py

Some GPTNeo models are trained with a vocab size greater than the actual used vocab size (i.e 50304 in config when the actual vocab size is 50257) where all tokens after the first i.e 50257 are unused. These models cannot currently be converted using the script because there is no way to cut the extra embeddings out of wte.

Comments:
Inconsistency in how Electra doing sentence level prediction
Author: richarddwang
Pull request: no
Labels: name: wontfix

In `ElectraForSequenceClassification`:
We have docstring says, `ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of
    the pooled output) e.g. for GLUE tasks.`
This is also what I observed in the official repository.
https://github.com/google-research/electra/blob/81f7e5fc98b0ad8bfd20b641aa8bc9e6ac00c8eb/finetune/classification/classification_tasks.py#L270
https://github.com/google-research/electra/blob/79111328070e491b287c307906701ebc61091eb2/model/modeling.py#L254
which is
```
nn.Sequential( nn.Dropout(config.hidden_dropout_prob),
                        nn.Linear(config.hidden_size, config.num_labels)
```

**But** the implementation of `ElectraClassificationHead` (used by `ElectraForSequenceClassification`) is
```
def forward(self, features, **kwargs):
        x = features[:, 0, :]  # take 
 token (equiv. to [CLS])
        x = self.dropout(x)
        x = self.dense(x)
        x = get_activation("gelu")(x)  # although BERT uses tanh here, it seems Electra authors used gelu here
        x = self.dropout(x)
        x = self.out_proj(x)
        return x
```

Is there something I overlooked in the official repository ? Hot to explain the inconsistency in doc and implementaion of `ElectraForSequenceClassification`.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Are there any updates on this issue?
XLNET tokenization changes after saving and loading
Author: Timoeller
Pull request: no

## Environment info
- Fresh Colab env with !pip install transformers[sentencepiece]
- `transformers` version: 4.5.0 
- Behavior also observed in previous Transformers versions >= 4.3.0

### Who can help
xlnet tokenization: @patrickvonplaten

## Information
Our CI reported different tokenization when saving and loading the xlnet tokenizer with added tokens:
```python
Before loading: ['▁Some', '▁Text', '▁with', 'neverseentokens', '▁and', '▁more', '<sep>', '<cls>']
After loading: ['▁Some', '▁Text', '▁with', '▁', 'neverseentokens', '▁and', '▁more', '<sep>', '<cls>']
```
## To reproduce

```python
from transformers import AutoTokenizer, AutoConfig

def test_tokenizers():
    basic_text = "Some Text with neverseentokens and more"

    tokenizer_orig = AutoTokenizer.from_pretrained("xlnet-base-cased")
    config_orig = AutoConfig.from_pretrained("xlnet-base-cased")
    tokenizer_orig.add_tokens(new_tokens=["neverseentokens"])
    tokenizer_orig.save_pretrained("testsave")
    config_orig.save_pretrained("testsave")
    tokens_before = tokenizer_orig.encode_plus(basic_text).encodings[0].tokens
    
    tokenizer_loaded = AutoTokenizer.from_pretrained("testsave")
    tokens_after = tokenizer_loaded.encode_plus(basic_text).encodings[0].tokens

    print(f"Before loading: {tokens_before}")
    print(f"After loading: {tokens_after}")
    assert tokens_before == tokens_after

if __name__ == "__main__":
    test_tokenizers()
```

## Expected behavior

Same tokenization before and after loading.


Comments:
Wow, that was quick, I just saw @stefan-it recent comment in our failing PR: https://github.com/deepset-ai/FARM/pull/727#issuecomment-814641885

So loading the XLNet Tokenizer in slow mode with
```python
tokenizer_orig = AutoTokenizer.from_pretrained("xlnet-base-cased", use_fast=True, from_slow=True)
```

resolved the issue.

Thanks for opening an issue @Timoeller. The `from_slow=True` is a bandaid over the underlying issue.

We're actively working on solving this issue, but it will take a bit of time as it's hard to do in a backwards compatible way.
Some styling of the training table in Notebooks
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes the custom styling of the progress bar, because the default one is actually prettier and it can cause some lag on some browsers (like Safari) to recompute the style at each update of the progress bar.

It also removes the timing metrics which do not make much sense form the table (there are still in the log history).

Comments:
Adds a note to resize the token embedding matrix when adding special …
Author: LysandreJik
Pull request: yes

…tokens

This was added to the `add_tokens` method, but was forgotten on the `add_special_tokens` method.

See the updated docs: https://191874-155220641-gh.circle-artifacts.com/0/docs/_build/html/internal/tokenization_utils.html?highlight=add_special_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_special_tokens

closes https://github.com/huggingface/transformers/issues/11102

Comments:
[BigBird] fix bigbird slow tests
Author: thevasudevgupta
Pull request: yes

# What does this PR do?

This PR will fix bigbird's slow tests as per our discussion @LysandreJik.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@LysandreJik, @sgugger 

Comments:
[versions] handle version requirement ranges
Author: stas00
Pull request: yes

As discussed in https://github.com/huggingface/transformers/issues/11107 our `require_version` wasn't handling ranges like `"tokenizers>=0.10.1,<0.11"`. This PR fixes it.

I don't know if it fixes the problem reported in https://github.com/huggingface/transformers/issues/11107 as I can't reproduce it.

One odd thing though in `numpy`:
```
$ python -c "import numpy; print(numpy.__version__)"
1.19.2
$ python -c "from importlib.metadata import version; print(version('numpy'))"
1.18.5
$ pip uninstall numpy
Found existing installation: numpy 1.19.2
Uninstalling numpy-1.19.2:
  Would remove:
    /mnt/nvme1/anaconda3/envs/main-38/bin/f2py
    /mnt/nvme1/anaconda3/envs/main-38/bin/f2py3
    /mnt/nvme1/anaconda3/envs/main-38/bin/f2py3.8
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy-1.19.2.dist-info/*
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy/*
  Would not remove (might be manually added):
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy/core/tests/test_issue14735.py
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy/distutils/compat.py
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy/random/_bit_generator.cpython-38-x86_64-linux-gnu.so
    /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/numpy/random/_bit_generator.pxd
Proceed (y/n)? y
  Successfully uninstalled numpy-1.19.2
$ python -c "from importlib.metadata import version; print(version('numpy'))"
1.18.5
$ python -c "import numpy; print(numpy.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'numpy' has no attribute '__version__'

$ pip install numpy -U
Requirement already satisfied: numpy in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (1.18.5)
Collecting numpy
  Downloading numpy-1.20.2-cp38-cp38-manylinux2010_x86_64.whl (15.4 MB)
     |████████████████████████████████| 15.4 MB 4.6 MB/s
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.18.5
    Uninstalling numpy-1.18.5:
      Successfully uninstalled numpy-1.18.5
$ python -c "import numpy; print(numpy.__version__)"
1.20.2
$  python -c "from importlib.metadata import version; print(version('numpy'))"
1.20.2
```

Perhaps it's just `numpy` and the extra files it leaves behind? I adjusted the test not to rely on `numpy.__version__` as it seems just invites failing tests. 

@sgugger, @LysandreJik 

Comments:
fix: The 'warn' method is deprecated
Author: stas00
Pull request: yes

```
examples/language-modeling/run_clm.py:340: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn(
```
This PR is fixing many dozes of those with:
```
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's|logger.warn\(|logger.warning(|g' {} \;
```

@LysandreJik 

Comments:
[WIP] Pad to multiple of 8 for run mlm example
Author: ak314
Pull request: yes

# What does this PR do?
This PR uses padding to a multiple of 8 in the run_mlm.py language modeling example, when fp16 is used. Since the DataCollatorForLanguageModeling did not initially accept the pad_to_multiple_of option, that functionality was added.
Fixes #10627


## Before submitting
- [x] Did you write any new necessary tests?


## Who can review?
@sgugger 

Comments:
@sgugger I fixed an edge case in the `_collate_batch` method and added tests for that, but if it's ok I'd like to close this PR and open a fresh one.

Closing in favor of #11128
Fix and refactor check_repo
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes issues that a user may have with `make quality`/`make fixup` when all backends are not installed: in that case `requires_backends` is imported in the main init on top of the dummy objects and the script complains it's not documented.

The PR also refactors the white-list for the model that are not in an Auto-class, which was mostly containing Encoder and Decoder pieces of seq2seq models.

Comments:
Typo fix of the name of BertLMHeadModel in BERT doc
Author: forest1988
Pull request: yes

# What does this PR do?

Typo fix in BERT doc.
I was confused that I couldn't find the implementation and discussion log of `BertModelLMHeadModel,` and found that `BertLMHeadModel` is the correct name. 
It was titled `BertModelLMHeadModel` in the BERT doc, and it seems `BertLMHeadModel` is the intended name.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @sgugger



Comments:
[run_clm] clarify why we get the tokenizer warning on long input
Author: stas00
Pull request: yes

Solving https://github.com/huggingface/transformers/issues/11108 this PR adds a clarification of why the warning is printed by the tokenizer when `run_clm.py` sends a huge input to tokenize against a short `block_size`. It's not great, but at least now the user will know that the warning is not warranted to be a warning in this particular situation.

> [WARNING|tokenization_utils_base.py:3138] 2021-04-06 21:29:29,790 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1462828 > 1024). Running this sequence through the model will result in indexing errors

So after this PR the we end up an extra warning:

```
[WARNING|tokenization_utils_base.py:3143] 2021-04-07 21:09:22,144 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1462828 > 1024). Running this sequence through the model will result in indexing errors
[WARNING|run_clm.py:326] 2021-04-07 21:13:14,300 >> ^^^^^^^^^^^^^^^^ Please ignore the warning above - it's just a long input
```

The correct solution would be to redesign the API to notify the tokenizer that in some cases the input doesn't have to be less than `block_size`.

Fixes: https://github.com/huggingface/transformers/issues/11108

@sgugger 

Comments: