noamaanMulla-03/transformers-issues huggingface dataset:
XLM-ROBERTA
Author: nooralahzadeh
Pull request: no
Labels: name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):  XLM-ROBERTA

In order to fine-tune XLM-ROBERTA from pre_trained file, since the config file does not have the model_type field, all the required files should be in a folder which contains "xlm-robert" in its name! otherwise the "XLM" config will be assigned !



Comments:
That's a good point @nooralahzadeh! We should probably add the `model_type` field to the XLM-ROBERTA config. 

@julien-c - If it's fine for you I can add the `model_type` `xlm-roberta` to all `xlm-roberta` configs of the following models: 
https://huggingface.co/models?search=xlm-rob 

As @nooralahzadeh pointed out, the fine-tuned XLM-ROBERTA models would otherwise default to the XLM model (because of its name and the implemented Fallback pattern in https://github.com/huggingface/transformers/blob/f6a23d19116a62bd3c662d0aa381130b49abcff7/src/transformers/configuration_auto.py#L190) 

Should be easy with a S3 AWS script. 

Hmm, there is a `xlm-roberta` before `xlm` and `roberta` in [CONFIG_MAPPING](https://github.com/huggingface/transformers/blob/f6a23d19116a62bd3c662d0aa381130b49abcff7/src/transformers/configuration_auto.py#L65) so it should be correctly picked up, no?

True, @nooralahzadeh I guess in order to avoid falling to 'xlm' you have to name your files/folders 'xlm-roberta-something' (so you need to include the full xlm-roberta name) or you can just manually change the config of your xlm-roberta and add the `model_type`

Thanks, this is what I said in the first comment: need to have "xlm-roberta" in its name!

Hello thanks,   this command not work! can you help me? please
python  run_squad.py \
--model_type xlm-roberta



Hey @Forutanrad,

could you please open a new issue for this?

Hello, yes thanks.

On Thu, 3 Feb 2022, 13:28 Patrick von Platen, ***@***.***>
wrote:

> Hey @Forutanrad <https://github.com/Forutanrad>,
>
> could you please open a new issue for this?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3498#issuecomment-1028806963>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ASUDOKXALOAGOJF5NSHE4MDUZJGSRANCNFSM4LVTH64A>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Add general vision docstrings
Author: NielsRogge
Pull request: yes

# What does this PR do?

This is an attempt to add general docstrings for vision models, specifically for the base model and the image classification ones.

To do:

- [x] ViT
- [x] DeiT
- [x] BEiT
- [x] SegFormer
- [x] Swin Transformer

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Preprocess/transform logits before caching them for computing metrics.
Author: davidleonfdez
Pull request: no

# 🚀 Feature request

I think it'd be nice to have a simple way to preprocess the logits before caching them for computing metrics.

## Motivation

When the `Trainer` `compute_metrics` are set, during evaluation the logits are accumulated (some in GPU memory, for `args.eval_accumulation_steps` steps; all in RAM). For some models, it will almost certainly lead to out of memory problems.

For instance, for a language model, this means storing in RAM a tensor of size [eval ds size, sequence length, vocab size].

In many cases, what is needed to compute metrics is just some reduction of the logits. For example: `logits.argmax(dim=-1)`.

I know I can subclass `Trainer` for this and redefine `evaluation_loop`, just wanted to know if you'd consider a more generic solution that prevents everyone that needs the feature from duplicating the rest of the code of `evaluation_loop`. I've seen more people running into the same issue. For instance:

https://github.com/huggingface/transformers/issues/8476
https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941
https://discuss.huggingface.co/t/cuda-out-of-memory-during-evaluation-but-training-is-fine/1783/4

## Your contribution

I was thinking about something like adding a `preprocess_logits_for_metrics` parameter to `TrainingArguments` of type Callable

If you don't set the parameter, the default is None and everything would work as always. If you set it, the logits are passed to `args.preprocess_logits_for_metrics` and its output is what's cached.

The main modification would be this in `Trainer.evaluation_loop`:

```
# Update containers on host
...
 if logits is not None:
    logits = self._pad_across_processes(logits)
    logits = self._nested_gather(logits)
    if self.args.preprocess_logits_for_metrics is not None:
        logits = self.args.preprocess_logits_for_metrics(logits)
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
```

Do you think it's worth it? If you do, I can submit a PR.

I tag @sgugger because I think he's worked quite a lot with the training loop, but I'm open to receive feedback from anyone.



Comments:
I think it would be a valuable addition, as you describe the problematic situation very well, when someone wants to compute perplexity with a language model having a very large vocab size, for instance.

The `TrainingArguments` can't have a new argument of type callable, but I think we could have a new argument in the init `preprocess_logits_for_metrics`.

I'm happy to review a PR for this, and if you could show inside how to use it in the examples `run_clm` or `run_mlm` to get the perplexity at each evaluation without getting OOM, that would be a very compelling argument for this new API!

cc @LysandreJik for info.
when I use "convert_pytorch_checkpoint_to_tf", I meet some problems
Author: sunM123
Pull request: no

I found that these args are not really useful, actually I can not run the code when use the arg—— "model_name" 
the origin code is like this, which I find it not work:
`    model = BertModel.from_pretrained(
        pretrained_model_name_or_path=args.model_name,
        state_dict=torch.load(args.pytorch_model_path),
        cache_dir=args.cache_dir,
    )
`
when I change the code as the follow , I can run it successfully
`    model = BertModel.from_pretrained(
        pretrained_model_name_or_path=args.cache_dir
    )
`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
A warning is raised when using DistributedDataParallel of PyTorch
Author: ForawardStar
Pull request: no

 When I train huggingface-transformers model in multi-cards and multi-machines using DistributedDataParallel (DDP) of PyTorch, a warning is always output at each epoch:
```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible

- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
```
Does this warning degrade the performace of parallel training? How to fix it? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to load bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6 model in offline mode in jupyter nb
Author: vs9205
Pull request: no

I have downloaded "bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6" model locally but while trying to use it in jupyter notebook getting error:

```
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models//bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6
```
How to fix this?
code I am using to use the model is as below:

```
import torch
import math
from transformers import BertModel, BertConfig, BertPreTrainedModel, BertTokenizer

class BertWSD(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.bert = BertModel(config)
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)

        self.ranking_linear = torch.nn.Linear(config.hidden_size, 1)

        self.init_weights()

    

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model_dir = "/bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6"

model = BertWSD.from_pretrained(model_dir)
tokenizer = BertTokenizer.from_pretrained(model_dir)
# add new special token
if '[TGT]' not in tokenizer.additional_special_tokens:
    tokenizer.add_special_tokens({'additional_special_tokens': ['[TGT]']})
    assert '[TGT]' in tokenizer.additional_special_tokens
    model.resize_token_embeddings(len(tokenizer))
    
model.to(DEVICE)
model.eval()
```

Comments:
Fix TFRemBertEncoder all_hidden_states
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `TFRemBertEncoder`  adds (initial, after a projection) `hidden_states` twice to `all_hidden_states`.
This PR fixes it.

In the test, it comments `# RemBERT also returns the upprojected word embeddings as an hidden layers`, but the code actually added the projected embedding twice. Furthermore, PyTorch's RemBERT only added the projected one.

@Rocketknight1 @gante 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Applied suggestion

https://github.com/huggingface/transformers/pull/15510#discussion_r799506824
Standardize semantic segmentation models outputs
Author: sgugger
Pull request: yes

# What does this PR do?

This PR standardizes the model outputs for semantic segmentation models and creates an `AutoModelForSemanticSegmentation` class. As discussed internally, models for semantic segmentation should return `logits` of shape `batch_size, num_labels, height, width`, one logit per pixel.

**Breaking change:** The `BeitForSemanticSegmentation` and `SegformerForSemanticSegmentation` models have logits with the same height and width as the input after this PR (instead of height/4 and width/4).

To maintain some level of backward compatibility, the `SemanticSegmentationModelOutput` has a field `legacy_logits` that users can pick to get the old logits value.

Another possible road to be less breaking is to create new classes `BeitForPixelClassification` and `SegformerForPixelClassification` while deprecating the current ones, then name "instance segmentation" "pixel classification" everywhere. It has the benefit of being more understandable to the beginner and look like our "ForTokenClassification" classes but it might surprise an expert more used to the "semantic segmentation" name.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
How can I see the masked words during pre-learning by MLM?
Author: tossyi
Pull request: no

I would like to know what words are masked during pre-learning by masked language modeling.
How can I see the masked words during pre-learning?

For example, Below is sample code.

```

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path=corpus,
    block_size=max_length,
)

data_collator = DataCollatorForLanguageModeling(
     tokenizer=tokenizer, 
     mlm=True,
     mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir=outputdir,
    overwrite_output_dir=False,
    num_train_epochs=epochs,
    per_device_train_batch_size=batch_size,
    save_steps=2000,
    save_total_limit=2,
    prediction_loss_only=True,
    logging_steps=2000,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset
)

trainer.train()
```

Thank you.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

Thank you for checking an issue! I'll ask my question in the huggingface forum.
[deepspeed docs] DeepSpeed ZeRO Inference
Author: stas00
Pull request: yes

A good demo example has been worked out for using DeepSpeed ZeRO Inference w/o using HF Trainer [here](https://github.com/huggingface/transformers/issues/15399#issuecomment-1025240005), so let's add it to the doc as we didn't have any.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
TrainingArguments --learning_rate should not be used to set both "lr" and "warmup_max_lr" in DeepSpeed
Author: dunalduck0
Pull request: no
Labels: name: DeepSpeed

According to this document: https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/deepspeed#shared-configuration, 
--learning_rate is used to set two parameters in DeepSpeed configuration: "lr" and "warmup_max_lr". This causes learning rate never changes, because the initial value is already the maximal value. 

Here is my DeepSpeed configuration:

```
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto"
    }
  },

  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto" ,
      "warmup_num_steps": "auto"
    }
  },

```

When running with this DeepSpeed configuration, I saw both "lr" and "warmup_max_lr" are set to 5e-5:

![image](https://user-images.githubusercontent.com/51274745/152467841-4c06b0bd-c882-4989-8370-60d09d3f70e3.png)

Unfortunately, the learning rate stayed at 5e-5 all the way because "lr" is bounded by "warmup_max_lr". When not using DeepSpeed, HF is in charge of learning rate, and I see learning rate changes.

I think --learning_rate should be used to set "warmup_min_lr", not the max.

Comments:
@stas00 

Let's look at the WarmupLR docs: https://deepspeed.readthedocs.io/en/latest/schedulers.html#warmuplr

> Increase the learning rate of each parameter group from min lr to max lr over warmup_num_steps steps, **and then fix at max lr**.

So max lr is the same as `args.learning_rate` in this context. And you're suggesting that it's not.

Perhaps you shouldn't use the `WarmupLR` scheduler if you don't want its behavior? I'd say perhaps you want this one instead?
https://deepspeed.readthedocs.io/en/latest/schedulers.html#warmupdecaylr

Please help me understand where you think it's not doing the right thing.



Ah, sorry. You're right. I misread the document. I thought it's saying learning rate increases from "lr" (not "min lr") to max lr. I also noticed that my warmup_step is 0. That may be the real cause why my rate is not changing at all.

This should probably go to another thread. But have you ever seen learning curve as odd as this one? The X is epoch and the Y is loss. At the 1st step of each epoch, training loss drops significantly while validation loss jumps. As early as 2nd epoch, the validation loss is overfitted. I've been investigating whole day today, but got no clue (and tried to blame LR scheduler :( ). This is from GPT-J model. If I switch the model to GPT-neo-1.3B, I have a perfectly fine learning curve and the validation loss doesn't overfit so quickly.

![image](https://user-images.githubusercontent.com/51274745/152485064-32c0ca45-c7d2-40e6-8b76-710d499f95b8.png)




Yes, please definitely open a new issue as it's not related to Deepspeed anymore and surely others might be able to help as I have no experience finetuning GPT-J.

The only thing I can suggest is to add more data if you can so that you don't need to repeat it more than once.

I'm closing this Issue otherwise.
Enable automatic creation of decoder_input_ids given labels in TF
Author: Carlosbogo
Pull request: yes

Added a way to automatically create the `decoder_input_ids` from the `labels` provided by the user the `TFEncoderDecoderModel`



Comments:
Hey @Carlosbogo, thanks for your contribution! I've pinged @patrickvonplaten to review your PR, but please note that Patrick if off until next Monday. Thanks for your patience!

Thanks for the PR @Carlosbogo! 

I think we should also move the loss computation into the `TFEncoderDecoderModel` class similar to how @NielsRogge has done it in this PR: https://github.com/huggingface/transformers/pull/14139 . Right now the loss is computed in the decoder. However we should compute the loss directly in the `TFEncoderDecoderModel` class.

Hi @Carlosbogo, thanks for your contribution! Let me know if you are able to take into account the comment above.

Thanks!

Yeah,  I think I can work on that.
Should I create a new PR for it when I finish?

No it can be included in this PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @Carlosbogo, 

Hope you had a good start to 2022! Let me know if I can help with anything to get this PR merged :-) 

I'm really sorry for the delay: I had my university exams and couldn't find time to work on it. I'll try to do it as soon as possible.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[WIP] Language modeling example for TF Trainer
Author: ydshieh
Pull request: yes
Labels: name: wontfix

To support language modeling for TF Trainer.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=h1) Report
> Merging [#6955](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c5d43a872f0e85ce069e921c5bda02374e5b9cbf?el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6955/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6955      +/-   ##
==========================================
- Coverage   80.02%   80.00%   -0.02%     
==========================================
  Files         161      161              
  Lines       30120    30120              
==========================================
- Hits        24104    24098       -6     
- Misses       6016     6022       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6955/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6955/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6955/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `92.30% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=footer). Last update [c5d43a8...8e24159](https://codecov.io/gh/huggingface/transformers/pull/6955?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to fine-tune CTRL?
Author: zhongpeixiang
Pull request: no
Labels: name: wontfix

How to fine-tune CTRL on a custom dataset with custom control codes using the transformers package?

I'm aware of the [guide](https://github.com/salesforce/ctrl/tree/master/training_utils) for tensorflow users. However, as a PyTorch user, the guide is not friendly to me. 

I'm also aware of the language modelling fine-tuning script [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py). However, it does not support CTRL right now.

Thanks,
Peixiang

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@zhongpeixiang do you have any info about finetuning ctrl?

@saippuakauppias No, in the end I chose the original CTRL repo from Salesforce to finetune.

Hey @zhongpeixiang, could you share some more information on how you fine-tuned the CTRL? I am also struggling to fine tune it using transformers. 

@ludoro I followed this repo to fine-tune the CTRL: https://github.com/salesforce/ctrl

also struggling through the fine-tuning of CTRL, if someone can show a notebook or just the code to do that, it will help a lot!

> also struggling through the fine-tuning of CTRL, if someone can show a notebook or just the code to do that, it will help a lot!

https://github.com/salesforce/ctrl/tree/master/training_utils