noamaanMulla-03/transformers-issues huggingface dataset:
Extend Transformers Trainer Class to Enable PyTorch SGD/Adagrad Optimizers for Training
Author: jianan-gu
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR intends to extend Transformers Trainer Class with two PyTorch common optimizer algorithms, [SGD](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html?highlight=sgd#torch.optim.SGD) and [Adagrad ](https://pytorch.org/docs/stable/generated/torch.optim.Adagrad.html),  for model training.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update CLIPFeatureExtractor to convert PIL image to RGB
Author: hengkuanwee
Pull request: yes

Currently PIL images with RGBA format throws an error when being processed by CLIPFeatureExtractor.

CLIPFeatureExtractor.normalize() throws the following error:
.../sentence-transformer/lib/python3.9/site-packages/transformers/image_utils.py", line 185, in normalize return (image - mean) / std ValueError: operands could not be broadcast together with shapes (4,224,224) (3,)

The original [clip model preprocesses PIL Images](https://github.com/openai/CLIP/blob/main/clip/clip.py#L74) by converting all PIL images into RGB format.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

hi @patil-suraj, re-created the PR as discussed 😄
How loss is calculated in MLM transformers training and the calculation/Intuition behind it?
Author: pratikchhapolika
Pull request: no

I am training a MLM model using `Roberta-XLM large` model.

Here is the standard code.


```
tokenizer = tr.XLMRobertaTokenizer.from_pretrained("xlm-roberta-large",local_files_only=True)
model = tr.XLMRobertaForMaskedLM.from_pretrained("xlm-roberta-large", return_dict=True,local_files_only=True)

df=pd.read_csv("training_data_multilingual.csv") 
train_df=df.message_text.tolist()
train_df=list(set(train_df))
train_df = [x for x in train_df if str(x) != 'nan']

train_encodings = tokenizer(train_df, truncation=True, padding=True, max_length=512, return_tensors="pt")

class SEDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings
        
    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        return item

    def __len__(self):
        return len(self.encodings["attention_mask"])

train_data = SEDataset(train_encodings)

# print("train data created")

training_args = tr.TrainingArguments(

     output_dir='results_mlm_vocab_exp'
    ,logging_dir='logs_mlm_vocab_exp'        # directory for storing logs
    ,save_strategy="epoch"
    ,learning_rate=2e-5
    ,logging_steps=6000
    ,overwrite_output_dir=True
    ,num_train_epochs=10
    ,per_device_train_batch_size=2
    ,prediction_loss_only=True
    ,gradient_accumulation_steps=4
    ,bf16=True #Ampere GPU
    ,optim="adamw_hf"
)


trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data
)

trainer.train()
```

I have few question related to this:

* How loss is calculated in MLM training? I see during training these logs are printed `{'loss': 1.6117, 'learning_rate': 1.751861042183623e-05, 'epoch': 2.48}`.  I guess it's training loss? If so how its calculated?
* How to pass validation data inside `TrainingArguments` ?  Is it same as training data?
* Is it logical to get precision, recall, F1 score for training and validation data for MLM training? If so then how to achieve it using `Trainer`?

Any reading links would also be appreciated.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update TAPAS tokenization's add_numeric_table_values
Author: xhluca
Pull request: yes

Using `table.iloc[row_index, col_index]` will use the index label, whereas `table.loc[row_index, col_index]` will use the location. So if the original dataframe had an index, using `iloc` would try to access the incorrect index and fall out of range.

Comments:
@NielsRogge could you take a look at this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik is that something useful to merge? It solves the following issues https://github.com/huggingface/transformers/issues/14544 and https://github.com/huggingface/transformers/issues/10265

They are marked as closed since they were auto-closed by the bot, but the issues still persist. I'm also not sure why the tests fail, but i can look into it if someone is willing to review & merge this PR (if no one is available for that then i can just close it).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @xhlulu,

sorry for the very late reply. Could you rebase your PR with master, such that I can take a look to merge it?

Thanks,

Niels

_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm not sure if iloc even works here and not familiar with the tests. Will close this as this isn't a priority and there's an easy workaround.
shift_tokens_right function missing for mt5 models
Author: bduclaux
Pull request: no

## Environment info
- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.11.0-1028-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.11.0a0+git06838ce (False)
- Tensorflow version (GPU?): 2.9.0-dev20220221 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.1
- JaxLib version: 0.3.0

### Who can help
@patil-suraj 

## Information

Hello

I am trying to finetune with Flax on TPU a mt5-small model on a summarization task, using the examples/flax/summarization/run_summarization_flax.py script.
When I run the script, I get an error about the fact that shift_tokens_right is not defined for mt5 models:

``  
File "/home/prod/transformers/examples/flax/summarization/run_summarization_flax.py", line 521, in main
    # shift_tokens_right_fn = getattr(model_module, "shift_tokens_right")
AttributeError: module 'transformers.models.mt5.modeling_flax_mt5' has no attribute 'shift_tokens_right'
``

Moreover, the current flax summarization script has a typo line 516 :

See https://github.com/huggingface/transformers/blob/master/examples/flax/summarization/run_summarization_flax.py#L516 :
``
 model_module = __import__(model.__module__, fromlist=["shift_tokens_tight"]) 
``

(tight should be right !)

I have been able to fix the issue by copying the function shift_tokens_right defined in `src/transformers/models/t5/modeling_flax_mt5.py` into the file `src/transformers/models/mt5/modeling_flax_mt5.py` .
Now the Flax summarization script works fine. 

Hope you can fix the mt5 code accordingly !


Comments:
Good catch, thanks for opening the issue! 
Would you be interested to open a PR to add this function in flax mT5?

Hey Suraj,

Sure ! Will do it tomorrow and keep you posted. Thanks !

Also noticed that `adafactor` parameter is not used to instantiate the optimizer in the `run_summarization_flax.py` script.
Will add it in my PR, to have support for both adamW and adafactor.


> Also noticed that `adafactor` parameter is not used to instantiate the optimizer in the `run_summarization_flax.py` script. Will add it in my PR, to have support for both adamW and adafactor.

Ahh, Good catch! It would be better two open a new PR for this since these are two different changes.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Bump - will do PR soon

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Will make a PR for this shortly :)
[feat] Add FLAVA model
Author: apsdehal
Pull request: yes

This PR aims to add [FLAVA](ihttps://arxiv.org/abs/2112.04482) model to the transformers repo.

Following checklist delineates the list of things to be done for this PR
to be complete:

- [x] Flava init
- [x] Flava base models
- [x] Flava layers
- [x] Flava Configs
- [x] Flava encoders
- [x] Flava pretraining models
- [x] Flava codebook
- [x] Flava feature extractors
- [ ] Flava classification/retrieval models (in progress)
- [x] Documentation updates 
- [x] Encoding utilities
- [x] Imports updates
- [x] Argstring updates
- [x] Flava pretrained checkpoints
- [x] Flava unit tests
- [x] Flava integration tests
- [x] Sanity check
- [x] Lint
- [x] Flava Processors




Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Thanks for your review on the PR. About lowercasing things, I followed what I saw in the repo as not all models were lower case. An example of this would be CLIP where classes are named as `CLIPConfig`, `CLIPTextEncoder` and so on. Would it be possible to keep `FLAVA` as uppercase?

`CLIP` was a mistake IMO, and it makes it harder on users to guess the right names of the models if we have different casings all the time. It's not because we make a mistake once we should repeat it again ang again just because there is a precedent ;-)

Hi! I was wondering if there are any updates on this and if this will be merged soon?

@kshitijkg This is almost ready and should land in the core soon.

@sgugger @patrickvonplaten This is ready for another round of review. I have resolved all of your comments except a few where I have asked clarifications. Hoping to land it soon.
Remove duplicated os.path.join in Trainer._load_rng_state
Author: shijie-wu
Pull request: yes

# What does this PR do?

Remove duplicated os.path.join in `Trainer._load_rng_state`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._
No separation between Torch and TF examples in create_a_model.md
Author: ignacioct
Pull request: no

Hi! 

While writing the Spanish translation of the guide to create a custom mode (#15947), I've come across something that I think it is not intentional. In the section Model, we have this first paragraph explaining how to start initializing and customizing a - custom - model. 

![image](https://user-images.githubusercontent.com/56955040/166233739-9f83de75-0dfc-46c8-b0be-e169d6f7ef3e.png)

And then, immediately after that, the texts repeats itself, but this time using TF models.

![image](https://user-images.githubusercontent.com/56955040/166234019-6887b5b4-6ee0-4a74-a663-d8ba0550e2a9.png)
 
I think there is a missing subsection title separating both ways of doing the procedure, or the information is redundant and one of then should be removed. One way or the other, is strange to be reading this from top to bottom and see some text repeating itself without any clue. 

Comments:
Hi, thanks for your help in translating the docs! :)

From my end on the `main` version of the docs, there are two separate blocks for PyTorch and TensorFlow content:

![Screen Shot 2022-05-03 at 10 30 01 AM](https://user-images.githubusercontent.com/59462357/166507509-7aabe3d4-ef88-4657-a57e-44ec4d841518.png)

Oh, I was directly looking using https://github.com/huggingface/transformers/blob/main/docs/source/en/create_a_model.mdx, and there is no separation there I think (that should also be the main version). If that is handled in the docs, I can just go on and replace the text in the same fashion. 

Thank you!
Added es version of language_modeling.mdx doc
Author: jQuinRivero
Pull request: yes

# What does this PR do?

Fixes(#15947)

Added spanish version of language_modeling.mdx documentation file. 


### Before submitting

- [X ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



@sgugger 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@omarespejel Could you confirm this is good to merge?
Forward outputs on multiple sequences is wrong
Author: rafikg
Pull request: no
Labels: name: bug

### System Info

```shell
latest version of transformers
pytorch
python 3.10
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
model.resize_token_embeddings(len(tokenizer))
model.to("cuda")
model.eval()



# sequences
seq1 = "summarize: Calling the model (which means the forward method) uses the labels for teacher forcing. This means inputs to the decoder are the labels shifted by one"
output1 = "calling the model uses the labels for teacher forcing. inputs to the decoder"

seq2 = "summarize: When you call the generate method, the model is used in the autoregressive fashion"
output2 = "the model is used in the auto-aggressive fashion."

seq3 = "summarize: However, selecting the token is a hard decision, and the gradient cannot be propagated through this decision"
output3 = "the token is a hard decision, and the gradient cannot be propagated through this decision"

input_sequences = [seq1, seq2, seq3]
output_seq = [output1, output2, output3]

# encoding input and attention mask
encoding = tokenizer(
    input_sequences,
    padding="longest",
    max_length=128,
    truncation=True,
    return_tensors="pt",
)

input_ids, attention_mask = encoding.input_ids.to("cuda"), encoding.attention_mask.to("cuda")

# labels
target_encoding = tokenizer(
    output_seq, padding="longest", max_length=128, truncation=True
)
labels = target_encoding.input_ids
labels = torch.tensor(labels).to("cuda")
labels[labels == tokenizer.pad_token_id] = -100

# Call the models
logits = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).logits

# Apply softamx() and batch_decode()

X = logits
X = F.softmax(X, dim=-1)
ids = X.argmax(dim=-1)
y = tokenizer.batch_decode(sequences=ids, skip_special_tokens=True)

# results: batch_size=3

['call the model uses the labels for teacher forcing  inputs to the decoder are',
 'the model is used in the auto-aggressive fashion  the the the',
 'the token is a hard decision, and the gradient cannot be propagated through this decision ']

# results: batch_size =1 i.e. consider 1 seq each time

['call the model uses the labels for teacher forcing  inputs to the decoder are']

['the model is used in the auto-aggressive fashion ']

['the token is a hard decision, and the gradient cannot be propagated through this decision ']
```

### Expected behavior

```shell
running model on a batch should give the same result as running on a single sequence
```


Comments:
How to change the Text embedder(Layoutlmv2Tokenizer) in huggingface LayoutLMv2 model?
Author: yellowjs0304
Pull request: no

Hi, I’m a beginner in Transformers huggingface.
I wonder How to change the text embedding models in LayoutLMv2(original to KoBERT)
I used those model by huggingface, so I used each of processor for LayoutLMv2, LayoutXLM.

I think i need to change the text tokenizer for data loading, and change the text encoding weights (in Original LayoutLMv2 model) as KoBERT’s like below codes.

```
kobert_name = "monologg/kobert"
bert_model = BertModel.from_pretrained(kobert_name)
kobert_tokenizer = KoBertTokenizer.from_pretrained(kobert_name)

feature_extractor = LayoutLMv2FeatureExtractor(apply_ocr=False)
processor = LayoutLMv2Processor(feature_extractor, kobert_tokenizer) => Returned below error

# ValueError: Received a KoBertTokenizer for argument tokenizer, but a ('LayoutLMv2Tokenizer', 'LayoutLMv2TokenizerFast') was expected.

model = LayoutLMv2ForTokenClassification.from_pretrained("microsoft/layoutxlm-base", num_labels=len(labels))
# Need to exchange the layoutlmv2.embeddings. as kobert parameters(weights)
```
But I got error in processor defined part… I think the original LayoutLMv2Processor only define the original.
I’m used this [code](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv2/CORD)
What point should i modify for changing text embedding?
Please share any tips for beginner. 😂

Comments:
Hi,

It's not possible to plug in a custom tokenizer into `LayoutLMv2Processor`, as `LayoutLMv2Tokenizer` was designed specifically to handle bounding boxes.

If you plan to use a custom tokenizer, then it means that you need to handle addition of special tokens, padding/truncation yourself, which is also what the LayoutLMv2 authors did, as seen [here](https://github.com/microsoft/unilm/blob/ca82fd4e5c7b0e4594eee33f8dd08388183787cd/layoutlmft/examples/run_funsd.py#L165) when tokenizing the words, then defining a custom data collator (to batch examples together) [here](https://github.com/microsoft/unilm/blob/master/layoutlmft/layoutlmft/data/data_collator.py). 

Thank you for kindly reply. It's really helpful.
Fix minor style error in Spanish docs
Author: osanseviero
Pull request: yes

# What does this PR do?

This PR is a minor fix of style since CircleCI is red due to style. FYI I did this in clean environment

```
pip install hf-doc-builder -U
pip install -e ".[dev]"
make style
```

Part of #15947

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sgugger 

Thank you very much @osanseviero! I was missing this part: `pip install -e ".[dev]"`. Was really frustrating but a lesson learned. 🚀