noamaanMulla-03/transformers-issues huggingface dataset:
Fix a bug in run_glue.py (#9812)
Author: forest1988
Pull request: yes

# What does this PR do?

It seems the `if` statement for `label_to_id` seems to be strange.
There should be `not` before `is_regression`.

Fixes #9812

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

Comments:
Thanks for the fix!

Thank you for your quick response!
padding='max_length' allowing more than max length
Author: SimplyLucKey
Pull request: no

I compared my tokenized data with `pad_to_max_length=True` and `padding='max_length'`. My `max_length` was set to 160. However I noticed there were a couple of data that were tokenized more than 160 when I used `padding='max_length'` versus `pad_to_max_length=True`

Comments:
Could you post a short code snippet so we can reproduce ?

> Could you post a short code snippet so we can reproduce ?

This is my code.

```
class GPReviewDataset(Dataset):
    def __init__(self, reviews, targets, tokenizer, max_len):
        self.reviews = reviews
        self.targets = targets
        self.tokenizer = tokenizer
        self.max_len = max_len
        
    def __len__(self):
        return len(self.reviews)
    
    def __getitem__(self, item):
        review = str(self.reviews[item])
        target = self.targets[item]
        
        encoding = self.tokenizer.encode_plus(text=review, max_length=self.max_len,
                                         add_special_tokens=True, padding='max_length', 
                                         return_attention_mask=True, 
                                         return_token_type_ids=False, return_tensors='pt')
        
        return {'review': review,
                'input_ids': encoding['input_ids'].flatten(), 
                'attention_mask': encoding['attention_mask'].flatten(),
                'targets': torch.tensor(target, dtype=torch.long)}


error_list = []

for i in range(len(free_df)):
    if len(GPReviewDataset(free_df['content'].to_numpy(), free_df['score'].to_numpy(), tokenizer, 160).__getitem__(i)['attention_mask']) != 160:
        error_list.append((i, len(GPReviewDataset(free_df['content'].to_numpy(), free_df['score'].to_numpy(), tokenizer, 160).__getitem__(i)['input_ids'])))
        
error_list
```

and the results
```
[(95, 184),
 (948, 218),
 (1025, 162),
 (3679, 204),
 (3680, 164),
 (4150, 220),
 (6139, 185),
 (7139, 165),
 (7201, 166),
 (7237, 256),
 (7381, 181),
 (7599, 254),
 (7600, 204),
 (7679, 170),
 (8111, 202),
 (8378, 193),
 (8773, 583),
 (9041, 583),
 (9321, 161),
 (10466, 279)]
```

You should set your truncation parameter as well, otherwise it won't truncate the texts that are too long. See the docs about the tokenizer methods  [here (look for `truncation`)](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=truncation#transformers.tokenization_utils_base.PreTrainedTokenizerBase.__call__)

> You should set your truncation parameter as well, otherwise it won't truncate the texts that are too long. See the docs about the tokenizer methods [here (look for `truncation`)](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=truncation#transformers.tokenization_utils_base.PreTrainedTokenizerBase.__call__)

Thank you! That worked.
Can I use a smaller base model than allenai/led-base-16384 for LED?
Author: mmoya01
Pull request: no

Hello, I'm trying to fine tune my own Longformer Encoder Decoder following this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=jpUr9QeebZ-n). However, I was wondering if there was a way to consider a base model like 

`allenai/longformer-base-4096`

instead of 

`led-base-16384`?

When I try doing
```python
led = AutoModelForSeq2SeqLM.from_pretrained(
    "allenai/longformer-base-4096",
    config="roberta-base",
    gradient_checkpointing=True,
    use_cache=False,
)
```
but that gives me
```
   led = AutoModelForSeq2SeqLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/modeling_auto.py", line 1221, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.longformer.configuration_longformer.LongformerConfig'> for this kind of AutoModel: AutoModelForSeq2SeqLM.
Model type should be one of LEDConfig, BlenderbotSmallConfig, MT5Config, T5Config, PegasusConfig, MarianConfig, MBartConfig, BlenderbotConfig, BartConfig, FSMTConfig, EncoderDecoderConfig, XLMProphetNetConfig, ProphetNetConfig.
```

my `encoder_max_length` is only 2048 since I'm not planning on feeding transcripts of 8k word to summarize but rather transcripts of 2k words. So using the smaller base model would work great. I'm basically trying to replicate the fine tuning for this
https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16
^but I would want to use that model more as a checkpoint to fine tune further on domain specific data. Hence why I'm trying to create:
1.) a model fine tuned on cnn dailymail data(basically replicate the [longformer2roberta](https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16) but a version that we can fine tune further)
2.) using part 1 as a checkpoint, fine tune that on domain specific transcript+summary data

@patrickvonplaten or others in the community, I'd greatly appreciate any advice on this

Comments:
I'd suggest to still use `allenai/led-base-16384` and just pad the input to a maximum length of only `2048`. Also you could think about reducing `config.attention_window` of the model's config to something like 512 or 256 to make your model more efficient for `2048` input

E.g. in this notebook: https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing, I just pad the max length of every input to `8192` and use `allenai/led-base-16384` which works very well!

Sounds great, I'll look into setting`led.config.attention_window=512` instead of 1024 and the `max_input_length=2048` for the encoder. Thank you for your feedback!
Remove a TF usage warning and rework the documentation
Author: jplu
Pull request: yes

# What does this PR do?

Recently we moved the warning for boolean argument misused in TF models from `warnings.warning(...)` to `tf.Print` to avoid the overhelm of messages in the output. Nevertheless, the usage of `tf.Print` prevent all our TF models to be compiled and executed with XLA and quantized with TFLite because the `Print` operator is not supported in XLA and TFLite.

As a solution for both issues (logging overhelm + XLA compilation/execution) I propose to simply remove the logs and state the use case directly inside the documentation for all the TF models.


Comments:
Not in favor of this to be honest...Warnings / Print statements are super important IMO. It's weird that `tf.Print` is not supported in XLA -> how are logs / warnings / print statements then produced in XLA?

XLA/Graph/TFlite execution are not made to have things to be printed, this includes some of the assert usage as well.

@patrickvonplaten  TF XLA has few other known issues https://www.tensorflow.org/xla/known_issues

> @patrickvonplaten TF XLA has few other known issues https://www.tensorflow.org/xla/known_issues

Thanks for sharing! I'm still very surprised by your message:

> Nevertheless, the usage of tf.Print prevent all our TF models to be compiled and executed with XLA and quantized with TFLite

To me, this means that every TF Repo that wants to be executable with XLA has no `tf.Print(...)` ops. That's a pretty hard requirement no? 

> To me, this means that every TF Repo that wants to be executable with XLA has no tf.Print(...) ops. That's a pretty hard requirement no?

I agree, it is, but I see very rarely `tf.print(..)` to be used. As far as I know I never seen it implemented in official TF models during runtime (you can easily check with a quick search on https://github.com/tensorflow/models), usually it is used when evaluating a model, which is a use case that is not XLA related.

Can we use
```
import tensorflow as tf
tf_logger = tf.get_logger()
tf_logger.warn(xxx)
```
intead?

@sgugger yes, using the internal TF logger should work as expected, but I'm not sure if it will bring any conflict with the actual transformers logger in terms of configuration.

Do you want me to use it instead, and we will see later if there is indeed any conflict?

Yeah I think it would be best to have a warning the user can't silence with our centralized logging that none.

Ok done! I restored the warning but with the internal TF logger.
I am trying to Fine tune on BartForConditionalGeneration but I end up getting all <pad_tokens>. Can you please help resolve it? 
Author: Sai-Ashish
Pull request: no


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Mixed Precision support and avoid ‘分咯啊太’
Author: xuxingya
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
[trainer] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters
Author: stas00
Pull request: no

When running DDP:
```
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 \
run_clm.py --model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
```
I get:

> [W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())


but it's not possible to turn it off from the trainer. i.e. it's hardwired.

@sgugger

Comments:
Edit: I am not sure why this param is always set to `True` except for gradient checkpointing. We can certainly make a training argument control its value to avoid hard-coding it. At least to experiment and benchmark whether it's best at True/False.
Clean TF Bert
Author: jplu
Pull request: yes

# What does this PR do?

This PR aims to clean the code base of BERT and the other models that depends of it because of the `#Copied from...`. It also clean the template accordingly to the same changes applied in BERT.

The other models will receive the same type of cleaning, but each model will have its own PR.

Comments:
I fully rework the keywords addition part to keep only those that seemed the most meaningful.
Fix auto-resume training from checkpoint
Author: jncasey
Pull request: yes

# What does this PR do?

This fixes a few minor issues with training auto-resume, as discussed [here](https://github.com/huggingface/transformers/pull/9776#issuecomment-767841895)

1. `checkpoints = [path for path in content if _re_checkpoint.search(path) is not None and os.path.isdir(path)]` was returning empty. I changed `os.path.isdir(path)` to `os.path.isdir(os.path.join(folder, path))` and now it returns a list of the checkpoint folders as expected.
2. Similarly, the `get_last_checkpoint` function was returning the basename of the checkpoint folder, not the full path, which seems to be expected based on the updates to the example scripts. I changed the last line of the function to `return os.path.join(folder, max(checkpoints, key=lambda x: int(_re_checkpoint.search(x).groups()[0])))`
3. After I made those update, it was resuming from the oldest checkpoint, not the newest. I noticed the checkpoint regex was only capturing the final digit in the directory name. I changed it to `_re_checkpoint = re.compile(r"^" + PREFIX_CHECKPOINT_DIR + r"\-(\d+)$")` with the `+` inside the capture group, and now `get_last_checkpoint` is giving me the newest checkpoint as expected.


## Who can review?

- trainer: @sgugger

Comments:
Oh you need to run `make style` on your branch for the styling test to pass. Let me know if you run into any issue doing that!

Sorry! I'm pretty rusty on the software dev stuff - college was 16 years ago. I think I've fixed it now.
When resuming training from checkpoint, Trainer loads model
Author: sgugger
Pull request: yes

# What does this PR do?

Trainer was not reloading model when resuming training from a checkpoint, which was confusing for users (see #9099) and also was preventing the recent auto-reload from checkpoint to fully work.

This isn't a breaking change (if users were passing a model with the checkpoint already loaded, it is just loaded twice).

Comments:
If I may add, under `def train():`, I think the initialisation of `self._globalstep_last_logged = 0` should be `self._globalstep_last_logged=self.state.global_step`, to ensure that the first logging of the loss is correct when you later divide by `self.state.global_step-self._globalstep_last_logged`?
Allow --arg Value for booleans in HfArgumentParser
Author: sgugger
Pull request: yes

# What does this PR do?

The primary reason I dived into this PR is because when launching a training with sagemaker, bool arguments need to be passed along with something (e.g. we can't do `--do_train`, we have to do `--do_train True` because the arguments are passed as a dict). This is refused by the current argparser.

This PR changes a little bit the way `HfArgumentParser` handles bool fields in dataclasses. Up until now:
- a bool arg `foo` with `True` as a default gives a flag `--no_foo` that stores `False` in foo
- a bool arg `bar` with no default value or `False` as a default value gives a flag `bar` that stores `True` in bar
- an optional bool arg `opt` gives the same as a bool if its default is True or False. If the default is None, it gives a flag `--opt` that requires an argument that accepts anything and stores the value as a string (which is obviously a bug)

After this PR, the following happens:
- a bool arg `foo` with `True` as a default gives a flag `--no_foo` that stores `False` in foo, it also gives a flag `--foo` that can be used as is (will store `True` in foo), or by using any truthy/falsy value (`--foo yes`, `--foo True`, `--foo no`...) that will store the result as a proper bool.
- a bool arg  `bar` with no default value or `False` gives a flag `bar` that can be used as is (will store `True` in bar), or by using any truthy/falsy value (`--bar yes`, `--bar True`, `--bar no`...) that will store the result as a proper bool.
- an optional bool arg `opt` gives the same as a bool if its default is True or False. If the default is None, it gives a flag `--opt` that requires an argument that accepts a truthy value and stores the value as a proper bool.

In all cases above, when a truthy value is expected but something else is passed (that is not `true`, `false`, `yes`, `no`, `1`, `0`, `t`, `f`, `y`, `n`), an error is raised.

So no breaking changes at all and all bool values can be used with an argument so that sagemaker is happy. Tests are updated and improved to check the behaviors summarized above are all correct.

Comments:
Load pretrained roberta model from fairseq?
Author: frankxu2004
Pull request: no

# ❓ Questions & Help
Would it be possible to load pretrained model of roberta-base that is trained by myself using fairseq, following the instructions in https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md?

Since the current transformers library is not suitable for pretraining from scratch, I think it would be nice to be able to load pre-trained model trained on fairseq. I think it might be possible but I am not sure how the current transformers' roberta pretrained model is translated/loaded?

thanks!

Comments:
Try https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py

You might have to make some modifications, I have never tried this. Good luck!


huggingface/transformers can be used to train from scratch. See [how to train a LM from scratch](https://huggingface.co/blog/how-to-train).

Feel free to open another issue if more specific 

related official blog: "Porting fairseq wmt19 translation system to transformers" by @stas00 
https://huggingface.co/blog/porting-fsmt

Might be able to convert fairseq language models following similar steps.

> Try https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
> 
> You might have to make some modifications, I have never tried this. Good luck!

Hey :) I'm interested in this but the link doesn't seem to work.

when pasting links to a repo one needs to hit `y` to get a fixed link to the current revision which would never break (as long as the repo is in place), here you go:

https://github.com/huggingface/transformers/blob/7c6d63298f27b4a386f4603262a4603a2a6bf057/src/transformers/models/roberta/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
Partial local tokenizer load
Author: LysandreJik
Pull request: yes

This PR aims to allow partial loading of a cached tokenizer.

Fixes #9147 which explains the issue in a lot of detail.

Currently, if we download a tokenizer from the hub using the `from_pretrained` method:

```py
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("google/bert_uncased_L-2_H-128_A-2", local_files_only=True)
```

It caches the files to be reused later. Reloading the tokenizer while specifying `local_files_only=True`

```py
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("google/bert_uncased_L-2_H-128_A-2", local_files_only=True)
```

results in a failure as it tries to fetch all of the tokenizer files, even those that are necessary. It currently fails with a hard error.

This PR changes that error to an info log, and prints a single log containing all the files that were not loaded. I put it as an `info` and not as a `warning` or an `error`, because the situation where this is an actual issue is imo very rare; it is a real issue only when the initial `from_pretrained` managed to obtain only some of the necessary files, i.e., when the download was interrupted.

Running the last snippet results in the following warning:

```
Can't load following files from cache: ['added_tokens_file', 'special_tokens_map_file', 'tokenizer_config_file', 'tokenizer_file'] and cannot check if these files are necessary for the tokenizer to operate.
```

Comments:
Add tpu_zone and gcp_project in training_args_tf.py
Author: kiyoungkim1
Pull request: yes

# What does this PR do?
Add ```tpu_zone``` and ```gcp_project``` in ```training_args_tf.py```.
For using TPUs created in a zone different from the VM zone, ```tpu_zone``` must be allocated.

See official Bert repo,
https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/run_pretraining.py#L426

- trainer: @sgugger 

Comments:
@sgugger 
I got the error message with ```make style```.

```
kiyoung@medical-ubuntu:~/transformers$ make style
running deps_table_update
updating src/transformers/dependency_versions_table.py
black examples tests src utils
make: black: Command not found
Makefile:42: recipe for target 'style' failed
make: *** [style] Error 127
```

You need to follow the steps of the [contributing guide](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) do be able to make PRs. In particular you didn't follow the installation part by running `pip install -e ".[dev]"` since you don't have `black` installed.

@sgugger 
Thanks, I did it.

Thanks for fixing! Now the problem comes from a new release of jax, which has been fixed in master so this is safe to merge.

This PR introduced a `datasets` submodule. I'm removing it in #9868.