noamaanMulla-03/transformers-issues huggingface dataset:
Improve LayoutLM
Author: NielsRogge
Pull request: yes

# What does this PR do?

- [x] Improve documentation of `LayoutLM`, explaining how people can normalize bounding boxes before passing them to the model, add links to the various datasets on which the model achieves state-of-the-art results, add code examples in the documentation for the various models
- [x] Add notebook to the list of community notebooks showcasing how to fine-tune `LayoutLMForTokenClassification` on the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (on which the model achieves SOTA results)
- [x] Add integration tests, which confirm that the model outputs the same tensors as the original implementation on the same input data
- [x] Add `LayoutLMForSequenceClassification`, which makes it possible to fine-tune LayoutLM for document image classification tasks (such as the [RVL-CLIP dataset](https://www.cs.cmu.edu/~aharley/rvl-cdip/)), extra tests included.

Fixes the following issues:
- #9228 
- #9097 
- #8866 
- #8524

## Who can review?

@LysandreJik, @patrickvonplaten, @sgugger
 


Comments:
Thanks for the reviews, I've addressed all comments. There are 2 things remaining:

- in the code examples, I use both `tokenize()` and `convert_tokens_to_ids` as the bounding boxes (which are at word-level) need to be converted to token-level. Is there a better solution?
```
words = ["Hello", "world"]
normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782]
tokens = []
token_boxes = []
for word, box in zip(words, normalized_word_boxes):
    word_tokens = tokenizer.tokenize(word)
    tokens.extend(word_tokens)
    token_boxes.extend([box] * len(word_tokens))
```
- according to @sgugger the input data on which the integration tests are ran are maybe too long, and black formatting causes them to be flattened vertically. Could you maybe fix this @LysandreJik? 

I pushed the reformat you asked for @NielsRogge, make sure to pull before doing any more changes!

Ok thank you, so the only thing remaining is make the code examples more efficient? Is there a way to make the code block (see comment above) better?
Generating sequence from two input sequences
Author: MiriamFarber
Pull request: no

The code here

https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_trainer.py

enables training `distilbart-xsum-12-6` model that gets an input sequence and outputs an input sequence. Is there a simple way to adapt the code to enable it to get two input sequences (e.g sentence and context sentence) and output a sequence? 

It seems to me that I need to use something like that: https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences, but wasn't sure how to combine it with the bart code.



Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead? You'll get more answers there.

Thanks!
BertForTokenClassificiation save
Author: Foysal87
Pull request: no

How can I save the BertForTokenClassificiation model?


Comments:
You can save a model using the `.save_pretrained()` method. So given that your model is called `model`, you can save it as follows:

`model.save_pretrained(path_to_directory) `

Thank you for your reply.  I already tried it. But got an error like this.

```
'BertForTokenClassification' object has no attribute 'save_pretrained'
```


Can you share some more code about how you created the model?

```
from pytorch_pretrained_bert import BertForTokenClassification
model = BertForTokenClassification.from_pretrained("bert-base-cased", num_labels=len(tag2idx))
```

after that fine-tuning it. Then want to save this model.

ok, I got it.. I can save it by torch.. thank you
Fix barthez tokenizer
Author: LysandreJik
Pull request: yes

The barthez tokenizer should be put in the "no config tokenizer", as two tokenizers with the same configs can't be put together.

Running the [following code](https://github.com/huggingface/transformers/issues/9422#issuecomment-759327863) works now:

```py
from transformers import AutoTokenizer

barthez_tokenizer = AutoTokenizer.from_pretrained("moussaKam/barthez")
```

Comments:
SMITH Google
Author: miketrimmel
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Google´s SMITH Algorithm


## https://github.com/google-research/google-research/tree/master/smith

* [x] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
This is a duplicate. See #9526 

Oh ok, thanks.
Dynamic padding + truncation in classification script
Author: pashok3d
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

@VictorSanh
@sgugger 

It seems that it is not possible to use dynamic padding along with truncation in classification script, because when `tokenizer` gets `max_length = None` it just skips truncation. 
https://github.com/huggingface/transformers/blob/063d8d27f4e1d089dc76f22e378b86b219167e3b/examples/text-classification/run_glue.py#L290

On the other hand, in language modeling script it works.
https://github.com/huggingface/transformers/blob/063d8d27f4e1d089dc76f22e378b86b219167e3b/examples/language-modeling/run_mlm.py#L311


Comments:
Yes, the `max_length` should be passed the same way. Would like to open a PR to fix `run_glue.py`?

Okay, I will do that.
[examples/text-classification] `do_predict` for the test set of local datasets
Author: forest1988
Pull request: no

# 🚀 Feature request

It seems that `run_glue.py` has the train set and the validation set management for local CSV/JSON files, but it doesn't have args for managing the test set of the local datasets.

https://github.com/huggingface/transformers/blob/7a9f1b5c99e9a5d1772649d029acdf5160419239/examples/text-classification/run_glue.py#L90-L95

I think the script is intended to be used not only for the train/validation but the test, as `glue` tasks test sets are downloaded as shown in https://huggingface.co/docs/datasets/loading_datasets.html#selecting-a-configuration.
It has the `--do_predict` option for the test sets.

If there is no particular reason for not having the ability to read the test set in the local dataset, would it be ok for me to add the feature?
Or is there some intention behind this implementation?

## Motivation

I'd like to train, validate, and test my own local dataset.

## Your contribution

I think some modifications like the below may help to add the feature.

``` python
    test_file: Optional[str] = field(
        default=None, metadata={"help": "A csv or a json file containing the test data."}
    )
```

``` python
        datasets = load_dataset(
            "csv", data_files={"train": data_args.train_file, "validation": data_args.validation_file, "test": data_args.test_file}
        )
```

``` python
    # if data_args.task_name is not None:
    #     test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
    test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
```

Thank you in advance.


Comments:
As long as it's kept simple (we want short and focused example scripts, so that they are easy to understand and tweak), I don't mind adding this feature. Feel free to open a PR with your suggestions!

Thanks, I'll try a modification as simple as possible, and if it can fix this issue without making the example difficult to understand, I’ll open a PR!
Fix classification script: enable dynamic padding with truncation
Author: pashok3d
Pull request: yes

# What does this PR do?

To fix the issue (below) in the run_glue.pl script, tokenizer's `max_length` value is now assigned directly with `max_seq_length` argument. Now it is possible to truncate the sequence and use dynamic padding. By default `max_length` is 128, which means truncation to 128 tokens. To disable truncation, set `max_length = None`.

Fixes #9551 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/9551#issue-784679542
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@sgugger 

Comments:
Fix data parallelism in Trainer
Author: sgugger
Pull request: yes

# What does this PR do?

A bug in data parallelism was introduced in #9451 (mostly because of some weird behavior of dataclasses in python) and data was... well not parallelized anymore (more like the batch size ended up divided by the number of GPUs).

This PR fixes that and to make sure it didn't break the behavior introduced in #9451 for model parallelism, adds a multiGPU test (passing locally) to ensure data is not parallelized when the model is parallel.


Comments:
Update run_glue for do_predict with local test data (#9442)
Author: forest1988
Pull request: yes

# What does this PR do?

Currently, run_glue.py cannot use the test set (`do_predict`) unless we give it a GLUE task name.
This PR will allow us to use the local test dataset.

As commented in #9442, I tried to achieve the functionality with only simple changes.

- It still works with only the local train and valid files (in other words, this PR does not break the current operation.).
- If we add `--do_predict` with out adding specific params, we will get an error statement saying that we need either the GLUE task name or the path of the local test file.

Fixes #9442 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger

Thank you for your kind comments on the issue. 
I have tried to keep it simple and hope there is no problem as an example script.

Comments:
Error messages of the CircleCI are:

```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_pipelines_conversational.py::SimpleConversationPipelineTests::test_history_cache
FAILED tests/test_pipelines_conversational.py::SimpleConversationPipelineTests::test_integration_torch_conversation
==== 2 failed, 4207 passed, 1744 skipped, 734 warnings in 190.84s (0:03:10) ====
```

```
FAILED tests/test_pipelines_conversational.py::SimpleConversationPipelineTests::test_history_cache
==== 1 failed, 4178 passed, 1774 skipped, 735 warnings in 260.31s (0:04:20) ====
```

I'm sorry but I'd like to ask you if `run_glue.py` is related to the conversation pipeline.


@sgugger @LysandreJik 
Thank you for reviewing and merging!
Fix Trainer with a parallel model
Author: sgugger
Pull request: yes

# What does this PR do?

The test introduced in #9566 wasn't actually working as the default batch size is 8, not 16...
So the problem was still there, the reason because `_setup_devices` in `TrainingArguments` is a `cached_property`, so its result is computed once and for all at init. Had to change the behavior slightly, but it should be okay since it's a private method.

Fixes #9577 (model is getting wrapped into DataParallel because the value of `self.args.n_gpu` is not updated.

Comments:
Entity level F-1 scores in run_ner.py
Author: pranav-s
Pull request: no

# 🚀 Feature request

The run_ner.py script in examples/token_classification reports evaluation metrics in terms of token F-1 scores (from what I can tell by examining the code). This issue is to request entity level F-1 scores as an evaluation metric. Token level scores will evaluate the F-1 score over an input such as ["I", "work", "for", "ABC", "com", "##pany", "in", "New", "York", "City"] with the label for each token considered separately. In this example, if the ground truth labels are ["O", "O", "O", "ORG", "O", "O", "O", "GPE", "GPE", "GPE"] the label for "New" , "York" and "City" from the predictions would be compared against ground truth separately.  Entity level scores would mark a true positive if for instance all 3 tokens in the span "New York City" are labelled correctly.

## Motivation

Token level F-1 scores can be a more lenient metric compared to entity level scores since labels on sub-words/entities are considered separately.

## Contribution

I can also help implement this but I am not certain how I should get started with this. Any help is appreciated.

Comments:
Oh this makes me realize this script hasn't been switched to use the datasets metrics for seqeval. This would solve the issue as it computes all scores. Will do that tomorrow.

Thank you @sgugger
Custom mask when performing forward pass
Author: hoangphuc1998
Pull request: no

Suppose I have a sequence that consists of 2 sentences separated by \<\/SEP\> tokens like A \<\/SEP\> B. When performing forward pass with RoBERTa model, I want tokens in sentence A only attend to tokens in sentence A and vice versa for sentence B. The mask will be look like this:
![image](https://user-images.githubusercontent.com/24960238/104556288-e3ea2c00-5671-11eb-8dcd-1735761722b4.png)
In summary, is there any way to explicitly pass a custom attention mask to the model? 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Make logs TF compliant
Author: jplu
Pull request: yes

# What does this PR do?

Currently when TensorFlow model is run in graph mode, the logs are displayed as many time the method is called even if the confition is not respected. This is because in graph mode the logs are not compiled in the graph and then are displayed all the time. To fix this, we use now `tf.print` that compiles the message inside the graph and will be displayed only when the conditions are respected.

## Fixes issue
#9285 

Comments:
Okey for me
How to use different learning rates in the classifier example.
Author: svishnu88
Pull request: no
Labels: name: wontfix

HI,

I am trying to use different learning rates for the bert and classifier. I am assuming that I can just say model.parameters and classifier.parameters like below.

`optimizer_grouped_parameters = [
    {'params': model.bert.parameters(), 'lr': 0.001},
    {'params': model.classifier.parameters(), 'lr': 0.01}
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
    ]`

Can someone confirm if its correct way of using it, and also why no weight decay is specified for particular layers.

Thanks,
Vishnu

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Did anyone conduct different learning rate in different layers when fine-tuning BERT?

Thanks. 

[This paper](https://arxiv.org/abs/1905.05583) suggests a "layer-wise decreasing learning rate" improves BERT for text classification tasks, but I can't see any option for it in the [AdamW](https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adamw-pytorch) optimizer. If it is helpful for text classification, it would be great to see it implemented.

Did anyone conduct different learning rate in different layers when fine-tuning BERT?

Thanks

You can do this in PyTorch (not sure about tf) and my understanding from this thread is that `huggingface`'s `AdamW` is now equivalent to PT's `AdamW` (see https://github.com/huggingface/transformers/issues/3407) so it should be equivalent - it would be great to get confirmation of this from someone more familiar with the huggingface codebase.  

See here for PT multiple rates: https://discuss.pytorch.org/t/how-to-set-a-different-learning-rate-for-a-single-layer-in-a-network/48552/4



As an update to the above - it actually _is_ possible to use the `huggingface` `AdamW` directly with different learning rates. 

Say you wanted to train your new parameters at x10 the learning rate of the pre-trained bert-variant parameters (in this case held as `model.bert`) you would do:
```python
from transformers import AdamW
# define model etc.
...

pretrained = model.bert.parameters()
# Get names of pretrained parameters (including `bert.` prefix)
pretrained_names = [f'bert.{k}' for (k, v) in model.bert.named_parameters()]

new_params= [v for k, v in model.named_parameters() if k not in pretrained_names]

optimizer = AdamW(
    [{'params': pretrained}, {'params': new_params, 'lr': learning_rate * 10}],
    lr=learning_rate,
)
```
disable message "Some layers from the model checkpoint at bert-base-cased were not used when initializing"
Author: tide90
Pull request: no

I wonder how can I disable this message? v3.4

```
Some layers from the model checkpoint at bert-base-cased were not used when initializing TFBertForSequenceClassification: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['dropout_113', 'classifier']
```
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Comments:
You can change the logging level:

```
from transformers import logging as hf_logging

hf_logging.set_verbosity_error()
```
disable message "Some layers from the model checkpoint ..."
Author: tide90
Pull request: no

I wonder how can I disable this message? v3.4

```
Some layers from the model checkpoint at bert-base-cased were not used when initializing TFBertForSequenceClassification: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['dropout_113', 'classifier']
```
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Comments:
You can change the logging level:

```py
from transformers import logging as hf_logging

hf_logging.set_verbosity_error()
```
BatchEncoding.to with device with tests
Author: LysandreJik
Pull request: yes

Closes https://github.com/huggingface/transformers/issues/9580

The `torch` module isn't imported directly in the `tokenization_utils.py` file. In a similar fashion to the tensor checks, this PR adds a device check to identify if a variable is a torch device.

Adds a test that fails previous to this PR.

Comments:
Upstream (and rename) sortish sampler
Author: sgugger
Pull request: yes

# What does this PR do?

This PR moves the logic of the "sortish sampler" from examples/seq2seq utils to `trainer_pt_utils` to make this behavior available for all types of training in the main `Trainer`. It also fixes a bug in the previous implementation of the distributed sortish sampler that did not synchronize the random generator used for the shuffling (thus the data returned on the two processes joined together was not a permutation of the whole dataset).

The sortish sampler logic is to group items of the training datasets that have similar lengths together to minimize padding while retaining a bit of randomness. It does some sorting for this, but that's not the main feature, and it's unclear for anyone reading it what it might do, so the argument name was badly chose in my opinion. I chose to name it `group_by_length` when introducing it in `TrainingArguments` (while keeping the old `sortish_sampler` argument in `Seq2SeqTrainingArguments` for backward compatibility, for now).

The actual samplers are given by the two introduced classes `LengthGroupedSampler` and `DistributedLengthGroupedSampler`. They are both tested in the test, and in particular, the distributed one has a test it is using the same random generator for the bit of randomness.

Renaming the old `sortish_sampler` arg is just done in the tests of seq2seq examples for now, it will be done more generally when the seq2seq finetuning script is rewritten to use `datasets`.

Comments:
saving the model during run_mlm.py
Author: lkcao
Pull request: no

Hi friends-
I am trying to train a Roberta on a large corpus with a server with time limitation.
Is there any way to save the model like every 3000 steps to keep record of the training, and resume it later?
Really need it with the project…Thanks for helping.


Comments:
Please avoid spamming the repository with multiple duplicate issues.
Also, those questions should go in the [forums](https://discuss.huggingface.co/), the issues are kept for bugs and feature requests only.
saving the model during run_mlm
Author: lkcao
Pull request: no

Hi friends-
I am trying to train a Roberta on a large corpus with a server with time limitation.
Is there any way to save the model like every 3000 steps to keep record of the training, and resume it later?
Really need it with the project…Thanks for helping.

Comments:
Please avoid spamming the repository with multiple duplicate issues.
Also, those questions should go in the [forums](https://discuss.huggingface.co/), the issues are kept for bugs and feature requests only.

sorry...I created two streams by mistake.