noamaanMulla-03/transformers-issues huggingface dataset:
Update return introduction of `forward` method
Author: kouyk
Pull request: yes

# What does this PR do?
Make it clear that the `forward` method now returns a dict instead of tuple.

PR #8530 switch the default value of `return_dict` in configurations to `True`. This caused some older code that relied on `return_dict` being set to `False` to break since assigned a dictionary to tuples can have unexpected outcomes such as receiving strings instead.

The phrasing of the introduction under the return section of the `forward` method currently state that a dictionary will be returned only if `return_dict=True` is passed or that `config.return_dict` is set to `True`. This is no longer valid ever since the default configuration changed, thus it will be beneficial for the readers to update this portion to indicate that those values need to be `False` for a tuple to be returned.

This will likely save readers some time when adapting old code :)

Comments:
Requesting review from @sgugger

Looks like you just need to run `make style` on your branch and we should be good to merge!

I have fixed the styling issues!

Thanks!
Flax text-classification multi-optimizer incorrect
Author: n2cholas
Pull request: no

The below code is incorrect:

https://github.com/huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/examples/flax/text-classification/run_flax_glue.py#L173-L188

- The `dict` from `traverse_util.flatten_dict` has keys which are tuples of strings, not one long string with the path separated by periods.
- `optax.masked` applies the transformation wherever the mask is True, so the masks are flipped.
- Flax's LayerNorm calls the scale parameter `scale` not `weight`

I believe following code would be correct:

```py
# We use Optax's "masking" functionality to create a multi-optimizer, one
# with weight decay and the other without. Each sub-optimizer will be applied
# wherever the mask is True
decay_path = lambda p: p[-1] != "bias" and p[-2:] != ("LayerNorm", "scale")  # noqa: E731

tx = optax.chain(
    optax.masked(adamw(0.0), mask=traverse(lambda path, _: not decay_path(path))),
    optax.masked(adamw(weight_decay), mask=traverse(lambda path, _: decay_path(path))),
)
```

But, for the networks in that example, the parameters that shouldn't be decayed all have 1 dimension (biases and layernorm scales). L173-L188 can therefore be simplified to:

```py
# We use Optax's "masking" functionality to create a multi-optimizer, one
# with weight decay and the other without. Each sub-optimizer will be applied
# wherever the mask is True. The bias parameters and LayerNorm scale
# parameters should not be decayed, and these are the only parameters
# with 1 dimension.
tx = optax.chain(
    optax.masked(adamw(0.0), mask=partial(jax.tree_map, lambda p: p.ndim == 1)),
    optax.masked(adamw(weight_decay), mask=partial(jax.tree_map, lambda p: p.ndim != 1)),
)
```

Though the latter solution is simpler, perhaps the first one is better since it illustrates a more general way for users to construct a multi-optimizer. I'd be happy to open a PR with either.

cc: @marcvanzee @patrickvonplaten 

Comments:
Actually `optax.adamw` can take a `mask` directly to mask weight decay (since that's pretty common):

```py
tx = optax.adamw(learning_rate=learning_rate_fn, b1=0.9, b2=0.999, eps=1e-6, weight_decay=weight_decay, 
                 mask=partial(jax.tree_map, lambda p: p.ndim != 1))
```

Hey @n2cholas,

Thanks for diving into this! 

I think a nice solution would be:

```python
tx = optax.adamw(learning_rate=learning_rate_fn, b1=0.9, b2=0.999, eps=1e-6, weight_decay=weight_decay, 
                 mask=partial(jax.tree_map, lambda p: p[-1] != "bias" and p[-2:] != ("LayerNorm", "scale")))
```

This way, it's consistent, but we also show the user clearly that `bias` and `LayerNorm` are not included in the weight decay. It would be great if you could open a PR for this :-) I can re-run the experiments afterward


Great, will do!
Fix weight decay masking in `run_flax_glue.py`
Author: n2cholas
Pull request: yes
Labels: name: Flax

# What does this PR do?

Fixes #11936

In addition to the changes discussed in the issue, I combined `traverse` and `decay_path` into one function `decay_mask_fn` to simplify the implementation. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten 

Comments:
That's great - thanks a lot for the fix @n2cholas :-) I'll re-run the FlaxGlue suit today with your fix to check if results improve 

Thanks a lot for the fix @n2cholas - I reran the eval train+eval script & updated the results accordingly
convert gpt2 from tensorflow to pytorch
Author: 7AM7
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers`  = 4.5.1
- PyTorch version (GPU?) = = 1.8.1+cu101

command :
!python3 /content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py \
--gpt2_checkpoint_path=/content/drive/MyDrive/tensorflowCheckpoints/model.ckpt-50000 \
--pytorch_dump_folder_path=/content/drive/MyDrive/convertpyorch/torch_model-500gpt2.bin \
--gpt2_config_file=/content/drive/MyDrive/tensorflowCheckpoints/config2.json 

Error :
Traceback (most recent call last):
  File "/content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 68, in <module>
    convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path, args.gpt2_config_file, args.pytorch_dump_folder_path)
  File "/content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 39, in convert_gpt2_checkpoint_to_pytorch
    load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 109, in load_tf_weights_in_gpt2
    pointer = getattr(pointer, scope_names[0])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 948, in __getattr__
    type(self).__name__, name))
AttributeError: 'GPT2Model' object has no attribute '_step'



Comments:
Hi @7AM7 

I think this is because there is `_step` in TF checkpoint, which should be ignored when loading the weights.

for this you should write your own conversion script. You could take and modify this function
https://github.com/huggingface/transformers/blob/30f065890e77f2917895b175b9a1df503b89e202/src/transformers/models/gpt2/modeling_gpt2.py#L68

adding some check like this here would solve this
```python
for name, shape in init_vars:
        if "_step" not in name name:
```


i modified this function def load_tf_weights_in_gpt2 and ignored "_step" Like if name!="_step" or  if "_step" not in name And still same error



in that case, you could check what extra variables are there in the `names` and then remove those from `names` and `arrays`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
tensorflow version is not able to pick the trained model from local directory in an air gapped system
Author: hiteshkum123
Pull request: no

hi ,
i have trained  the TFBertForSequenceClassification model and i have to deploy the trained model in an air gapped server.

code -
from transformers import BertTokenizer, TFBertForSequenceClassification
from transformers import InputExample, InputFeatures

model1=TFBertForSequenceClassification.from_pretrained(local_path)
tokenizer1=BertTokenizer.from_pretrained(loacal_path)


 ImportError: cannot import name 'TFBertForSequenceClassification' from 'transformers' (unknown location)


Same code works if i am using the PyTorch version( BertForSequenceClassification)

Comments:
Did you install TensorFlow in your environment? You might need a more recent TensorFlow version if so.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Simple questions about EncoderDecoderModel
Author: dobby-seo
Pull request: no

First, thank you for great works!

1. Does that tie function work for sharing pretrained weight of encoder's embedding with weight of decoder's embedding?https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L185

If i want to use different tokenizer between encoder and decoder inputs, does tie function ignore sharing embedding such as strict option?

Comments:
Hi @qute012 

The `tie_weights` method ties all the weights of the encoder and decoder including the embedding. For this to work, the encoder and decoder need to be the same model (same class) i.e either BERT2BERT or ROBERTA2ROBERTA2 and with same size.

> If i want to use different tokenizer between encoder and decoder inputs, does tie function ignore sharing embedding such as strict option?

no, it does not ignore sharing embedding in that case because as I wrote above it expects both encoder and decoder to be the same model so implicitly assumes that the tokenizer will also be the same.

But if that's what you want to do you could manually untie the embeddings or just re-initialize both of them so they won't be shared/tied.

Thanks to reply @patil-suraj 

For example, is it right that encoder's embedding weights can be adjusted by decoder's input?

Then if i want to untie, should i remove or comment out below code manually?
https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L183

do you have any plan to add parameter that choosing tie function in EncoderDecoderModel class? I know bert2bert is better performance than random decoder's embedding weights,  but it requires to extending for experiment newly when each encoder and decoder use different vocabulary. If you okay, i will PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Question] Implementing character based tokenizer
Author: ethen8181
Pull request: no

Hi team, what's the recommended approach for implementing a character based tokenizer? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Transformers Pegasus - how do I fine tune another language?
Author: seregadgl20-oss
Pull request: no

How do I fine tune another language? Who will tell you?

Comments:
Hi @seregadgl20-oss 

It would be nice if you use the forum (https://discuss.huggingface.co/) to ask such general questions. Issues are for bugs and feature requests. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
some issue in loading local txt file as Dataset for run_mlm.py
Author: alighofrani95
Pull request: no

![image](https://user-images.githubusercontent.com/14968123/115773877-18cef300-a3c6-11eb-8e58-a9cbfd1001ec.png)

first of all, I tried to load 3 .txt files as a dataset (sure that the directory and permission is OK.), I face with the below error.

> FileNotFoundError: [Errno 2] No such file or directory: 'c'

by removing one of the training .txt files It's fixed and although if I put all file as training it's ok
![image](https://user-images.githubusercontent.com/14968123/115774207-867b1f00-a3c6-11eb-953b-905cfb112d25.png)
![image](https://user-images.githubusercontent.com/14968123/115774264-9b57b280-a3c6-11eb-9f36-7b109f0e5a31.png)


after this, my question is how could I use this defined Dataset for run_mlm.py for from scratch pretraining.
by using  --train_file path_to_train_file just can use one .txt , .csv or, .json file. I tried to set my defined Dataset as --dataset_name but the below issue occurs.


> Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 336, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/dataset/dataset.py

> During handling of the above exception, another exception occurred:

> Traceback (most recent call last):
  File "run_mlm.py", line 486, in <module>
    main()
  File "run_mlm.py", line 242, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 719, in load_dataset
    use_auth_token=use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 347, in prepare_module
    combined_path, github_file_path
FileNotFoundError: Couldn't find file locally at dataset/dataset.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.6.0/datasets/dataset/dataset.py.
The file is also not present on the master branch on github.


Comments:
_I tried to load 3 .txt files as a dataset_ - bad idea, try to merge them in one, then let's see what you'll get (which error message or result).

Try this one:
```
from pathlib import Path
paths = [str(x) for x in Path(".").glob("**/*.txt")]
```

P.S. Next time just copy-paste code, don't screen it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add clipping to relative positional embedding
Author: hadaev8
Pull request: yes

# What does this PR do?

This PR adds clipping to distance embedding as described in the paper [Improve Transformer Models with Better Relative Position Embeddings](https://arxiv.org/abs/2009.13658)
Without this addition, the model with relative positional embedding will return an error if the input length above max_position_embeddings param.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
@patrickvonplaten @LysandreJik @julien-c @zhiheng-huang

Hey @hadaev8,

Thanks for your PR. It would be awesome if you could provide a code snippet to this PR that shows a case where the current implementation of BERT's relative positional embeddings would fail. Hope it's fine to tag the original author here: @zhiheng-huang for some discussion. 

@patrickvonplaten 
Colab notebook
https://colab.research.google.com/drive/1bAwwNMbh27JW0H6uWG9rZgyjz_b0eB1M?usp=sharing


@patrickvonplaten 
Any update?

Hey @hadaev8,

I'm having a hard time deciding here without the opinion of the official author @zhiheng-huang . @zhiheng-huang it would be great if you could leave a comment here :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@patrickvonplaten 
Well, in current state model with rel pos just would not work as expected eg process longer input.

@hadaev8, 

Alright, let's try to fix it together! It would be awesome if you could post a reproducible code snippet here so that we can see when an error arises

@patrickvonplaten 
Here it doesnt work
https://colab.research.google.com/drive/1bAwwNMbh27JW0H6uWG9rZgyjz_b0eB1M?usp=sharing
Here it work
https://colab.research.google.com/drive/1OIHjR2kVndDzwro7n_SgkpJG4YwjgGQY?usp=sharing

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Why DataCollatorForLanguageModeling do not make attention_mask feature?
Author: makailove123
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
As we know, according to padding on batch of input_ids, we should mask the padding ids as invalidate to avoid attention on them. But I found that collate function of DataCollatorForLanguageModeling didn't implement attention_mask feature.
Is this expected? Or do you think attention_mask is not necessary?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@makailove123 Have you figured out why?
Add GPT Neo models to Write With Transformer
Author: zxv
Pull request: no
Labels: name: Feature request

# 🚀 Feature request
Would it be possible to get the newly-added GPT Neo models usable on Write With Transformer?

## Motivation
It would be helpful to use the new models in the Write With Transformer app since it supports newlines.

CC @LysandreJik 

Comments:
Very much needed!
Train GPT2 with Trainer & TrainingArguments using/specifying attention_mask
Author: alexol91
Pull request: no

Hi, I'm using Trainer & TrainingArguments to train GPT2 Model, but it seems that this does not work well.

My datasets have the ids of the tokens of my corpus and the mask of each text, to indicate where to apply the attention:
```
Dataset({
     features: ['attention_mask', 'input_ids', 'labels'],
     num_rows: 2012860
 }))
```
I am doing the training with Trainer & TrainingArguments, passing my model and my previous dataset as follows. But nowhere do I specify anything about the attention_mask:
```
training_args = TrainingArguments(
    output_dir=path_save_checkpoints,
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size = 4, 
    gradient_accumulation_steps = 4, 
    logging_steps = 5_000, save_steps=5_000,
    fp16=True,
    deepspeed="ds_config.json",
    remove_unused_columns = True,   
    debug = True
    )


trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train() 
```

How should I tell the Trainer to use this feature (attention_mask)?
If you take a look at the file /transformers/trainer.py there is no reference to "attention" or "mask".

Thanks in advance!

Comments:
Hi there,

If your dataset or collator returns the `attention_mask`, then you don't need to pass it separately. With `Trainer`, all the input that you want to pass to model's `forward` should be returned by the dataset/collator and it will be passed to `model.forward` by `Trainer`.

Thank you very much for your quick reply @patil-suraj 

Is it possible that the problem is that more than 50% of the input is padding? Could this be too much? Do you think that training more would solve it? It currently returns a minimal loss. Having such a low loss is what made me think that I was not considering the `attention_mask` and that is why the loss was low (obviously it is easy to predict 600 padding tokens xD)

What I am doing now is changing the size of the input, instead of 1024 (default value) I am testing with 400 (size of the longest text in my dataset).

Best regards!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.