noamaanMulla-03/transformers-issues huggingface dataset:
wav2vec 2.0 loaded from a fairseq checkpoint isn't strictly equivalent
Author: TParcollet
Pull request: no

## Environment info
- `transformers` version: 4.9.2
- Platform: Linux-4.18.0-147.51.2.el8_1.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1 
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

## Who can help
@patrickvonplaten @anton-l 

## Information

This is related to wav2vec 2.0 loaded with Wav2Vec2Model

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. train a base model with fairseq (official base config) and convert the checkpoint with the official HF script.
2. Load the converted model with Wav2Vec2Model
3. Load the fairseq checkpoint with fairseq.checkpoint_utils.load_model_ensemble_and_task
4. Print a sum of all parameters and the output of identical input wav file.

## Expected behavior

In practice, we should see two identical sum as the models should be exactly identical. This is not the case in my test. Then, if you pass the same wav file in the two models (HF and fairseq), you'll get different outputs as well. 

What I did to compute the sum:
![image](https://user-images.githubusercontent.com/11910731/134816102-8bd233af-0666-42e3-aae8-d71f2a3572b8.png)

The output:
![image](https://user-images.githubusercontent.com/11910731/134816107-291c9a9d-2d21-4f53-9215-af217a5ea0bb.png)

As you can see, we have the same number of parameters, but not the same values as they differ slightly. 

Then, I let both models train on the exact same downstream task (everything runs within SpeechBrain). The downstream setups are **identical**. At the end, the fairseq models as a WER 2% lower while we should see the exact same results.


Comments:
Hey @TParcollet,

Thanks for the issue. Could you try to make a fully reproducible code snippet? I image that one wouldn't have to train a whole new model, but could also just use one of the already pretrained wav2vec2 models from fairseq to see this difference no?

Also it would be great if you could replace the screenshots by code snippets which would make it easier to reproduce the error for us. Thanks!

I'll do this as soon as I have 5 minutes of spare time :-P 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Alphafold 2.0
Author: elk-cloner
Pull request: no
Labels: name: New model

# 🌟 New model addition

It would be so amazing to have Alphafold model in huggingface. 😍
I don't know if there is any plan to add these kind of models to huggingface repo.

<!-- Important information -->
## Model description


## Open source status

* [x] the model implementation is available: ([github](https://github.com/deepmind/alphafold))
* [] the model weights are available: ([github](https://github.com/deepmind/alphafold))
* [x] who are the authors: (@deepmind)

Comments:
cc @Rocketknight1 :)

cc @Narsil too, this is something we've been looking at internally!

Any updates? 

We roughly looked into it, but alphafold requires quite a bit of data to work with (other known proteins configurations), and it seems just transposing the model would be not super relevant on its own.

Feel free to take a look on your own, I think it would be a great addition if we could make it work, but it seemed the overall `alphafold` thing would be quite the undertaking and require a lot of other dependencies, which is why we haven't started a proper integration.
Document or add more detail on `DPRPretrainedModel`
Author: xhluca
Pull request: no

What exactly is [`DPRPretrainedModel`](https://huggingface.co/transformers/model_doc/dpr.html#dprpretrainedmodel)? There's no documentation about it, and it's only subclassed by a undocumented model when I look at the source code. DPR is also a 3-model system, so it's confusing which one this is supposed to correspond to, and how the end user can use it.

It would be nice to have more documentation or description on that.

Comments:
`DPRPretrainedModel` shouldn't actually be included in the documentation, as `BertPretrainedModel` for example is also not documented.

Each model in the Transformers library defines a `<model_name>PretrainedModel`, which is just an abstract class defining the weights initialization of all models (base model + head models) and a simple interface for downloading and loading pretrained models.

Feel free to open a PR to remove it from the docs.
How to use LayoutLM Tensorflow version on Google Colab?
Author: alucor-it
Pull request: no

Hi,
Could you please help me implement LayoutLM's tensorflow version in Google Colab?
Previously I have implemented using the following link: https://github.com/microsoft/unilm.git

I would like to understand how to implement the Tensorflow version.
https://huggingface.co/atahmasb/tf-layoutlm-base-uncased

Thank you.

Comments:
You can check out the documentation in order to use the model: https://huggingface.co/transformers/model_doc/layoutlm.html#tflayoutlmmodel

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[performance doc] add tables with DDP/DP/no-DP benchmarks
Author: stas00
Pull request: no
Labels: name: WIP

Users ask of DDP vs. DP vs. single GPU vs w/ w/o Nvlink and this summary is insufficient 
https://huggingface.co/transformers/performance.html#nvlink
luckily we have great numbers posted - so should link to those from the guide:

1. table 1 where DDP wins by miles over any other setup: https://github.com/huggingface/transformers/issues/9371#issuecomment-768656711
2. table 2 where DDP is terrible https://github.com/huggingface/transformers/issues/9371#issuecomment-771302472



Comments:
Question: dropping transformers layers
Author: JulesBelveze
Pull request: no

Hi everyone,

I came across this interesting paper: _[On the Effect of Dropping Layers of Pre-trained Transformer Models](https://arxiv.org/pdf/2004.03844.pdf)_ and I am actually trying to implement it. However, I am wondering what would be a good practice to perform _layer dropping_.

Would it be safer to just mask the unwanted layers (some sort of pruning)? Or maybe copy the wanted layers into a new model? I've also seen that you guys have a `nn_pruning` library involving some sort of `SparseTrainers`, would that be the way to go?
Would love to hear your suggestions on this one 😃 

Thanks for your help and thanks a lot for the amazing repo!
Cheers,
Jules

Comments:
Hey @JulesBelveze, we generally try to keep the github issues for bugs/feature requests, and foster discussions on the [forum](https://discuss.huggingface.co) instead.

For your question regarding layer dropping, we actually have a few architectures that support it! You can search for the term `layerdrop`: https://github.com/huggingface/transformers/search?q=layerdrop

Examples of those are the BART, Marian, mBART, Pegasus family of models.

Thanks!

Hey @LysandreJik, really sorry for asking the question here.
Thanks for your answer, that's exactly what I was looking for!

Cheers
[Tests] Update audio classification tests to support torch 1.10
Author: anton-l
Pull request: yes

# What does this PR do?

The recent changes to `torch.nn.GroupNorm()` algorithm made it work slightly differently for inputs with small standard deviations. This is most noticeable (`~1e-3`) for uniform inputs like `np.ones()` in `test_small_model_pt()` and less noticeable (`<1e-5`) for other audio classification tests. 
The ASR models are not affected, since they don't use mean-pooling over outputs and thus don't amplify the difference as much.

PyTorch response: https://github.com/pytorch/pytorch/issues/67907


Comments:
Shapes mismatch triggered at modeling_flax_utils
Author: javismiles
Pull request: no

Good day,
while using the MiniDalle repo at:
https://github.com/borisdayma/dalle-mini/issues/99

we are suddenly getting this error which was not happening before:
"Trying to load the pretrained weight for ('decoder', 'mid', 'attn_1', 'norm', 'bias') failed: checkpoint has shape (1, 1, 1, 512) which is incompatible with the model shape (512,). Using `ignore_mismatched_sizes=True` if you really want to load this checkpoint inside this model."

This is being triggered here:
https://huggingface.co/transformers/_modules/transformers/modeling_flax_utils.html

in this area:
`# Mistmatched keys contains tuples key/shape1/shape2 of weights in the checkpoint that have a shape not
        # matching the weights in the model.
        mismatched_keys = []
        for key in state.keys():
            if key in random_state and state[key].shape != random_state[key].shape:
                if ignore_mismatched_sizes:
                    mismatched_keys.append((key, state[key].shape, random_state[key].shape))
                    state[key] = random_state[key]
                else:
                    raise ValueError(
                        f"Trying to load the pretrained weight for {key} failed: checkpoint has shape "
                        f"{state[key].shape} which is incompatible with the model shape {random_state[key].shape}. "
                        "Using `ignore_mismatched_sizes=True` if you really want to load this checkpoint inside this "
                        "model."
                    )`


There is a way to avoid halting the execution by going into the code and adding "ignore_mismatched_sizes=True" in the call. However, this does not fix the problem. If we do that, the execution continues but the results obtained by the minidalle model are wrong all washed out and with the wrong colors and contrast (which was not happening some days ago, so something has changed that is producing this problem).
So this seems to be a bug coming from this file. Any tips are super welcome, thank you :)


Comments:
cc @patil-suraj 

It seems to work with `flax==0.3.5`.
My guess is that weights are now being squeezed. Maybe need to reupload a new checkpoint?
Actually here a shape of (512,) seems to make more sense than (1,1,1,512)

Do we know which commit in Flax is responsible for this bug?

> It seems to work with `flax==0.3.5`. My guess is that weights are now being squeezed. Maybe need to reupload a new checkpoint? Actually here a shape of (512,) seems to make more sense than (1,1,1,512)

Could you please elaborate ?
where shall i add  flax==0.3.5 ?

Thanks!

Left a comment here, https://github.com/borisdayma/dalle-mini/issues/99#issuecomment-963103973
Closing this issue, since it's not related to `transformers`.
[TFWav2Vec2Model] Fix input shapes in TFWav2Vec2WeightNormConv1D
Author: anton-l
Pull request: yes

# What does this PR do?
**Context**
After updating to TF 2.7 the tests started failing with `ValueError('One of the dimensions in the output is <= 0...')` which is caused by the recently added check to `keras.layers.Conv.build()` https://github.com/keras-team/keras/blob/v2.7.0/keras/layers/convolutional.py#L197

This PR fixes the input shapes passed during `TFWav2Vec2WeightNormConv1D.build()` to account for padding.



Comments:
Fixing tests on master.
Author: Narsil
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik
@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
question answering pipeline throws error when handle_impossible_answer=True
Author: jsalbr
Pull request: no

## Environment info
- `transformers` version: 4.12.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

Pipelines: @Narsil 

## Information

The problem arises when using:
my own modified scripts: (give details below)  
with any model

## To reproduce

See an example here https://github.com/jsalbr/PythonIntro/blob/master/Transformers_Issue.ipynb
or directly in colab: https://colab.research.google.com/github/jsalbr/PythonIntro/blob/master/Transformers_Issue.ipynb

or briefly here
```python
from transformers import pipeline

model_name = "deepset/minilm-uncased-squad2"
qa = pipeline("question-answering", model=model_name,
              tokenizer=model_name, device=0)

question = "How is the weather?"
context = """Washington is the capital of the United States."""
# throws exception in transformers 4.12.3
answer = qa(question, context, handle_impossible_answer=True)
answer
```

Error message:
```
/usr/local/lib/python3.7/dist-packages/transformers/pipelines/question_answering.py in postprocess(self, model_outputs, top_k, handle_impossible_answer, max_answer_len)
    407 
    408             if handle_impossible_answer:
--> 409                 min_null_score = min(min_null_score, (start_[0] * end_[0]).item())
    410 
    411             # Mask CLS

ValueError: can only convert an array of size 1 to a Python scalar
```

## Expected behavior

No exception ;-)

I patched it this way, and it seems to work (though probably not correct):

```patch
409c409
<                 min_null_score = min(min_null_score, (start_[0] * end_[0]).item())
---
>                 min_null_score = min(min_null_score, (np.max(start_[0] * end_[0])).item())
```


Comments:
This has been (hopefully) fixed in master.

Can you confirm: https://github.com/huggingface/transformers/pull/14279 ?

Yes, no error anymore.
About qa example
Author: wutong4012
Pull request: no

https://github.com/huggingface/transformers/blob/ce01122a3bb690a944054bef216f87b90c2c9e63/examples/pytorch/question-answering/run_qa.py#L400

According to my understanding, although this is the last 1 that reaches the sequence_id, it should correspond to the [SEP] mark in the real text? So I think it should be reduced by one more here?

Comments:
cc @sgugger 

The [SEP] token has a sequence_id of `None`, not `1`, so it's not going to be taken into account.
RuntimeError: Could not infer dtype of numpy.int64 on Squad T5
Author: helloworld123-lab
Pull request: no

Hello,

I try to run the code for T5 on Squad dataset in [https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb](url)
I install the required libraries as:
```
!pip install transformers==2.9.1
!pip install -U nlp
!pip install sentencepiece
```
I fixed the xla error as in [https://stackoverflow.com/questions/67257008/oserror-libmkl-intel-lp64-so-1-cannot-open-shared-object-file-no-such-file-or](url)


However, when the training starts, it gives the following error:
Exception in thread Thread-17:
```
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/parallel_loader.py", line 139, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/nlp/arrow_dataset.py", line 719, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/nlp/arrow_dataset.py", line 707, in _getitem
    format_kwargs=format_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/nlp/arrow_dataset.py", line 619, in _convert_outputs
    v = map_nested(command, v, **map_nested_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/nlp/utils/py_utils.py", line 191, in map_nested
    return function(data_struct)
RuntimeError: Could not infer dtype of numpy.int64
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I think this issue is still occuring.

Hi @wanglec , that's an old notebook and has not been updated since, so I don't recommend it anymore.
There's a new example in transformers for fine-tuning T5 for qa, [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering#fine-tuning-t5-on-squad20). It also uses `Trainer`, so supports training on TPUs. [Here's](https://github.com/huggingface/transformers/tree/master/examples/pytorch#running-on-tpus) a short guide about how to run these scripts on tpu
Could not load from pretrain even with the same code
Author: unbreading
Pull request: no

environment: pytorch_transformers 1.1.0

I copy the code of `RobertaModel` from `pytorch_transformers.modeling_roberta` to a local directory with minimal requirements. 

```
from pytorch_transformers.modeling_bert import BertModel
from pytorch_transformers.modeling_roberta import RobertaConfig, ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP, RobertaEmbeddings

class RobertaModel(BertModel):
    config_class = RobertaConfig
    pretrained_model_archive_map = ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
    base_model_prefix = "roberta"

    def __init__(self, config):
        super(RobertaModel, self).__init__(config)

        self.embeddings = RobertaEmbeddings(config)
        self.init_weights()

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, position_ids=None, head_mask=None):
        if input_ids[:, 0].sum().item() != 0:
            print("A sequence with no special tokens has been passed to the RoBERTa model. "
                           "This model requires special tokens in order to work. "
                           "Please specify add_special_tokens=True in your encoding.")
        return super(RobertaModel, self).forward(input_ids, token_type_ids, attention_mask, position_ids, head_mask)
```


And I failed to load from pretrain in my main script:

```
from MyRoberta.RobertaTest import RobertaModel
bert_model = RobertaModel.from_pretrained(args.roberta_model)
```

exception:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/bs/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 474, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File ".../MyRoberta/RobertaTest.py", line 51, in __init__
    self.init_weights()
TypeError: init_weights() missing 1 required positional argument: 'module'
```

What's wrong? It should load successfully.

The reason I made the copy is that, I want to add a new embedding into roberta model, so I begin with running the original roberta code in my directory.

Thanks a lot.

Comments:
Problem solved using transformers instead of pytorch_transformers.
But I'm still curious about the reason behind such problem.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.