noamaanMulla-03/transformers-issues huggingface dataset:
[Wav2Vec2, Hubert] Fix ctc loss test
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR fixes the `ctc_loss` test. The problem is that PyTorch ctc_loss implementation https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html seems to use too many approximation to ensure deterministic behavior for a test. So in the PR I make the test much less aggressive by simply checking that the loss can be computed.

We've trained 250 models during the Wav2Vec2 sprint using this function so I think we can be quite certain that it works as expected. 

Not an ideal solution, but I don't really see how to make this "sum == mean" test really work since it's so flaky

Comments:
Fix bug in Masked Language Modeling example scripts (#11840))
Author: bzantium
Pull request: yes

# What does this PR do?
when `data_args.line_by_line == False`, the script firstly converts given examples into input_ids, token_type_ids, attention_mask and special_tokens_mask including cls_token, sep_token. Then it concatenates all tokenized outputs and generate chunks of max_seq_length. However, it will generate unintended training examples such as [871, 512, 2492, 1111, 947, 533] not [2 (cls_token), 512, 2492, 1111, 947, 3 (sep_token)]. I fix this problem.

Fixes #11840 
@sgugger, @patil-suraj

Comments:
The `line_by_line=False` argument is not supposed to be used for BERT-like pretraining objectives, it it there to do GPT-like pretraining. Maybe it does not make sense to have it in `run_mlm` at all.
In any case this fix will not necessarily work for all models supported by the script, as the special tokens may be slightly different than what is hard-coded.

#10737 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Is 10% in annotation different from 0.5 in code？
Author: aixuedegege
Pull request: no

https://github.com/huggingface/transformers/blob/0cbddfb190ab9b05b6575fbf818aae17bad4d24a/src/transformers/data/data_collator.py#L387

```python
        # 10% of the time, we replace masked input tokens with random word
        indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
        random_words = torch.randint(len(self.tokenizer), labels.shape, dtype=torch.long)
        inputs[indices_random] = random_words[indices_random]
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add TPU README
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds intructions on how to access a TPU. 

@patil-suraj - can you check whether it works as expected? Can you follow the steps of the PR? I've sent you two mails to @huggingface.co

@avital - could you also review? :-)



Comments:
LGTM!

One thing -- In the email you sent to others, or in this README, or both we should say that "please only SSH into machines where you're collaborating on a project. If there are any concerns about access, we may look at the audit logs to see who accessed which TPUs" or something like that.
Import check_inits handling of duplicate definitions.
Author: Iwontbecreative
Pull request: yes

# Improves the handling of duplicate definitions for `utils/check_inits.py`

Currently if there are duplicate definitions of import structure or type hints, the model may or not may fail depending on if the import_structure and TYPE_HINTS are duplicated in the same way. If it fails, it will show the wrong error message.

This ensures this test always fails and displays the appropriate error message.

This is particularly useful if things go wrong while merging master into a model PR after a long time.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you write any new necessary tests? -> N/A.


## Who can review?
 @sgugger since he first wrote the tool

Comments:
Validation split added: custom data files @sgugger, @patil-suraj
Author: Souvic
Pull request: yes



# What does this PR do?
Validation split added in case of no validation file and loading custom data for TensorFlow examples run_mlm.py file
Fixes # (issue)

Issue #12406 fixed.
Docs on language modeling TensorFlow updated.




Comments:
@sgugger 
@patil-suraj 


@sgugger
@patil-suraj
Made the necessary changes.. please see and comment.

Hopefully done now!

All good now, thanks!
[roberta] lm_head.decoder save/load needs fixing
Author: stas00
Pull request: no

```
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("distilroberta-base")

print('decoder.weight' in dict(model.lm_head.named_parameters()).keys())
print('dense.weight' in dict(model.lm_head.named_parameters()).keys())

print('lm_head.decoder.weight' in dict(model.named_parameters()).keys())
print('lm_head.dense.weight' in dict(model.named_parameters()).keys())
```

gives:
```
True
True
False
True
```

So if we query `lm_head` we can see `lm_head.decoder.weight`, however it's not visible to the whole model via `parameters()` (named or not).

The problem comes from `tie_weights`:
```
       output_embeddings = self.get_output_embeddings()
        if output_embeddings is not None and self.config.tie_word_embeddings:
            self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
```

which essentially does:
```
lm_head.decoder.weight = embeddings.word_embeddings.weight
```

which disconnects it from the module list.

This is inconsistent behavior.

But the main issue is that `lm_head.decoder.weight` is saved in the `save_pretrained` and then is expected to be there on `torch.load` but since it's tied frameworks like deepspeed won't save it.

So if `config.tie_word_embeddings` is `True`, it shouldn't save that key and not expect to load it.

Please correct me if I have missed something obvious.

@LysandreJik, @sgugger 


Comments:
Indeed, the key should be added to the keys to ignore on save, and expected missing keys.

Thanks for your investigation!

What if someone sets `config.tie_word_embeddings=False`. Should the save/expected keys be dynamically adjusted in `__init__`? 

or do it the other way around - have `lm_head.decoder.weight` on the list and instead remove it in `__init__` if  `config.tie_word_embeddings=False` as the former is the most likely behavior.

Yes, I think the second proposition makes a lot of sense! Great catch.

We'll need to check, but I wouldn't be surprised if other models have their lm head on the list - and nothing set in the `__init__` to prevent the save/expected keys from interacting with that layer. If so, it's not a high priority issue as no one has brought it up - but it is an issue nonetheless that potentially affects several models.
gpt2 causal mask to skip learning context input? (beginner question)
Author: gyin94
Pull request: no

Suppose we use gpt2 to train a text generation with context or translation with input sequence. 

[google translate<sep> google Übersetzer]. 

Can I ask how we can change the casual mask to let the training only from the token generation after the context input (\<en-de>\)? Like,
[google translate\<en-de\>] to [google]
[google translate\<en-de\> google] to [Übersetzer]

instead of 
[google] to [google translate]

I saw we currently added the casual mask here. 
```
causal_mask = self.bias[:, :, key_length - query_length : key_length, :key_length].bool()
```
https://github.com/huggingface/transformers/blob/b655f16d4eb405d286821bdb68b46b8a989f9c04/src/transformers/models/gpt2/modeling_gpt2.py#L186

Note using the default DataCollatorForLanguageModeling can work for the above task. I am thinking how we can let the training focus on text generation after the context input. 

Can we use the special mask tokens to achieve it? I am a bit doubtful on it. Since it is used for mlm from the code
https://github.com/huggingface/transformers/blob/b655f16d4eb405d286821bdb68b46b8a989f9c04/src/transformers/data/data_collator.py#L350
```
 # If special token mask has been preprocessed, pop it from the dict.
        special_tokens_mask = batch.pop("special_tokens_mask", None)
        if self.mlm:
            batch["input_ids"], batch["labels"] = self.mask_tokens(
                batch["input_ids"], special_tokens_mask=special_tokens_mask
            )
        else:
            labels = batch["input_ids"].clone()
            if self.tokenizer.pad_token_id is not None:
                labels[labels == self.tokenizer.pad_token_id] = -100
            batch["labels"] = labels
        return batch
```

or should we mask all context token input (before \<en-de\>)as -100 in labels so that they would be ignored in the below loss function?
```
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            # Flatten the tokens
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
```

Thanks a lot. cc @patrickvonplaten 

Comments:
MLM training fails with no validation file
Author: Souvic
Pull request: no

## Environment info

- `transformers` version: 4.9.0.dev0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes



## Information

Model I am using (Bert, XLNet ...): distilbert-base-cased

The problem arises when using: the official example scripts: (give details below)

The tasks I am working on is: MLM finetuning

## To reproduce

Steps to reproduce the behavior:

1. Just run the tensorflow  examples
2. python3 ./transformers/examples/tensorflow/language-modeling/run_mlm.py\
 --model_name_or_path distilbert-base-cased \
 --output_dir ./g \
 --train_file "customdata.txt" \
3. The model fails with error message that no validation file is there.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
it should use the validation split percentage parameter to divide the training set into training and eval samples.

Comments:
Log:
Grouping texts in chunks of 512: 100% 87/87 [00:26<00:00,  3.25ba/s]
Traceback (most recent call last):
  File "./transformers/examples/tensorflow/language-modeling/run_mlm.py", line 604, in <module>
    main()
  File "./transformers/examples/tensorflow/language-modeling/run_mlm.py", line 493, in main
    eval_dataset = tokenized_datasets["validation"]
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 37, in __getitem__
    return super().__getitem__(k)
KeyError: 'validation'

Fixed.
Fix TAPAS test uncovered by #12446
Author: LysandreJik
Pull request: yes

#12446 cleaned up a typo in a test name that uncovered a failure in the TAPAS test.

The failure in the TAPAS test is the following: `test_save_load_fast_init_from_base`. It adds a mock initialization. However, TAPAS has `nn.Parameter` attributes which are not impacted by that initialization, and end up having random values when loaded from the base model.

Updating this configuration parameter ensures that these values get initialized to 0.


Comments:
Thanks for solving this!
OSError: Unable to load weights from pytorch checkpoint file. 
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

When I am trying to load the `Roberta-large` pre-trained model, I get the following error:

```python
model_RobertaForMultipleChoice = RobertaForMultipleChoice.from_pretrained('roberta-large', output_hidden_states = True)

OUT:
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 
```

How can I solve this issue? Thank you,

Comments:
Hi! Do you mind pasting your environment information here so that we may take a look?

Try to delete cache directory files.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey, I trained my model on GPT2-small but I am not able to load it! It gives off the following error:

    Unable to load weights from pytorch checkpoint file for '{pretrained_model_name_or_path}' OSError: Unable to load weights 
    from pytorch checkpoint file for '/mounted/models/train-on-test1/' at '/mounted/models/train-on-test1/pytorch_model.bin' If you 
    tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

@leeivan @LysandreJik 

Please open a new issue with your specific problem, alongside all the information related to your environment as asked in the template. Thank you.

Same as <https://github.com/huggingface/transformers/issues/6620>, i guess it is because the checkpoint file is not a true checkpoint file: such as a git lfs file.

#6159, #6970,  #6620 are all same issue. 
In my case I cloned the checkpoint file using git lfs and issue was resolved. Earlier I had used pointer to avoid git lfs however it gave this error. 
For some changing torch, transformer, tokenizer versions helped. 
Also you can go through the docs and check some from_pretrained parameters like force_download, from_tf and try.

For futuer visitors:  [check this](https://discuss.pytorch.org/t/getting-an-error-unpicklingerror-invalid-load-key-v-in-pytorch-model-deploying-in-streamlit/107768/4)
Wrong BlenderbotConfig description (max_position_embeddings)
Author: shinishiho
Pull request: no

Hi there, the documentation page for BlenderbotConfiguration has a wrong parameter description
[https://huggingface.co/transformers/model_doc/blenderbot.html](url)

max_position_embeddings (int, optional, defaults to 1024) – The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).

But it's actually 128, as shown in the source code:
[https://huggingface.co/transformers/_modules/transformers/models/blenderbot/configuration_blenderbot.html#BlenderbotConfig](url)

def __init__( self, vocab_size=8008, max_position_embeddings=128, encoder_layers=2, encoder_ffn_dim=10240, ...

And by the way, do anyone know how to increase the maximum sequence length of this model? If I modify the config.max_position_embeddings, it will result in an error: (BlenderbotForConditionalGeneration)

`size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([128, 2560]) from checkpoint, the shape in current model is torch.Size([1024, 2560]).`
`size mismatch for model.decoder.embed_positions.weight: copying a param with shape torch.Size([128, 2560]) from checkpoint, the shape in current model is torch.Size([1024, 2560]).`

With the length of 128 tokens, it will "forget" the conversation's topic quite fast, since the input has to be trimmed.

Thanks in advance.

@patrickvonplaten @patil-suraj


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ONNX model conversion
Author: fdlci
Pull request: no

Hi,

I have been comparing inference speeds between pytorch models and their ONNX versions. To convert a model from pytorch to ONNX I have used the code your provided in convert_graph_to_onnx.py.

I have built my onnx model as follows as I am applying it to QA: python transformers/src/transformers/convert_graph_to_onnx.py --framework pt --model Camembert-base-ccnet-fquad11 --quantize cam_onnx/camembert-base.onnx --pipeline 'question-answering'

This code outputs 3 models, camembert-base.onnx, camembert-base-optimized.onnx, camembert-base-optimized-quantize.onnx.

I run inference with the three models and I was expecting the quantize version to be much faster than the camembert-base.onnx, but it was the complete opposite. I don't understand why quantization doesn't increase the speedup in this case?

Thank you for your answer!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add guide on how to build demos for the Flax sprint
Author: osanseviero
Pull request: yes

Add a guide on how to build a demo for the Flax sprint. This guide will be also updated early next week, but I would like to submit this version on Friday.

Note: this will be extended more once the evaluation is a bit more defined and with the launch of S.

Comments:
Some TODO ideas (for a later iteration)
* Add small guide on Streamlit
* Add small guide on Gradio
* Add small guide on how to use S (or a video)
* Explain why the importance of a demo

I'll merge this today. I'll do a follow up PR on Monday extending the demo section.
Get core dump after import FlaxRobertaModel
Author: cahya-wirawan
Pull request: no

## Environment info
I tried to run "transformers-cli env" but I get also this error message:
2021-07-03 05:37:33.144628: F external/org_tensorflow/tensorflow/core/tpu/tpu_executor_init_fns.inc:110] TpuTransferManager_ReadDynamicShapes not available in this library.
Aborted (core dumped)

- `transformers` version: 4.9.0.dev0
- Platform: TPU vm from the Jax/Flax event
- Python version: 3.8
- PyTorch version (GPU?): not installed or pytorch 1.8.1, the same issue 
- Tensorflow version (GPU?): not installed or tensorflow 2.4.1, the same issue 
- Using GPU in script?:
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten 

## Information

I followed the instruction https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#how-to-install-relevant-libraries in a TPU vm for the JAX/Flax event. But after the installation of transformers and the datasets, I get following error message when I import FlaxRobertaModel:
```
$ python
Python 3.8.10 (default, Jun  4 2021, 15:09:15) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import FlaxRobertaModel
2021-07-03 05:55:17.729321: F external/org_tensorflow/tensorflow/core/tpu/tpu_executor_init_fns.inc:110] TpuTransferManager_ReadDynamicShapes not available in this library.
Aborted (core dumped)

```
or run the transformers-cli:
```
$ transformers-cli env
WARNING:tensorflow:From /home/cahya/transformers/src/transformers/commands/env.py:50: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2021-07-03 05:56:23.495538: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-03 05:56:23.555453: F external/org_tensorflow/tensorflow/core/tpu/tpu_executor_init_fns.inc:110] TpuTransferManager_ReadDynamicShapes not available in this library.
Aborted (core dumped)

```
The problem arises when using:
* [ x] the official example scripts: (see above)

The tasks I am working on is:
* [ ] JAX/Flax project
* [ ] 

## To reproduce
see above


## Expected behavior
There should be no core dump

Comments:
It seems that the jax version 0.2.16 installed automatically during transformers installation make this issue. I can fix it after reinstalling the jax using following command:
`pip install "jax[tpu]>=0.2.16" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html`
Find the requested files in the cached path without the internet
Author: Bachstelze
Pull request: no

# 🚀 Feature request
The pipeline needs an internet connection to find the cached path. A request usually consumes time. 

## Motivation
Could we search only localy for a better performance?

## Your contribution
 My test code:
```
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-de-en")
model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-de-en")

de_en_translator = pipeline("translation_de_to_en", model=model, tokenizer=tokenizer)
translation = de_en_translator("Ein kleiner Test.")
```

Comments:
You can save your model and tokenizer to a directory using `save_pretrained` and load them from there! You only need to specify the directory path to the model/tokenizer arguments you pass to the `pipeline` method.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.