noamaanMulla-03/transformers-issues huggingface dataset:
Update model share tutorial
Author: stevhliu
Pull request: yes
Labels: name: Documentation

First draft of the updated model sharing tutorial. Main updates include:

- Suggest converting a model to other frameworks before uploading it to the Hub.
- Advocate for `push_to_hub` as best practice for uploading files to the Hub. Most notably, the `transformers-cli` method has been removed. Would love your thoughts on whether we should include using the web interface to share models so this step is more accessible a wider background of users!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

yes i would advocate to include using the web interface to share models
Fix missing eps arg for LayerNorm in ElectraGeneratorPredictions
Author: ydshieh
Pull request: yes

# What does this PR do?

`ElectraGeneratorPredictions`  doesn't specify `eps`

https://github.com/huggingface/transformers/blob/0501beb84601651bf6a44c5a16ab0e5b98948f78/src/transformers/models/electra/modeling_electra.py#L650

but`TFElectraGeneratorPredictions` does
https://github.com/huggingface/transformers/blob/0501beb84601651bf6a44c5a16ab0e5b98948f78/src/transformers/models/electra/modeling_tf_electra.py#L572

This causes the differences of logits/loss in PT/TF `ElectraForMaskedLM` as high as `1e-3`.
With this PR, the difference is in the range `5e-7 ~ 2e-6`.

This PR also makes the new added test introduced #15256 pass without error.

----
It's unclear to me why PyTorch `ElectraGeneratorPredictions` doesn't specify `eps` though. I assume it is a mistake.
I saw many other PyTorch models use this arg for `nn.LayerNorm`

## Question:
Should I add some deprecation warning in this case?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for checking all these issues @ydshieh 💯 

I've seen a few examples where the TF LayerNorm eps is specified and the PT eps is not, merely because PT's default value is assumed (e.g. in BART: [PT example](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L280); [TF example](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_tf_bart.py#L288)).

But if it is to be loaded from a config file and produces different results, then it seems like a different story -- i.e. this PR makes total sense. @LysandreJik can you confirm (or point to someone who can confirm)? :)
AutoTokenizer does not support XGLMTokenizer
Author: mrseeker
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  4.17.0.dev (f380bf2b612e6030ef8bc8904b287d274f035e29)
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): 1.9.0
- Tensorflow version (GPU?): n/a
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

### Who can help
@patil-suraj 

## Information

Model I am using (Bert, XLNet ...): XGLM

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Just execute this piece of code:
```
from transformers import AutoTokenizer
tokenizer=AutoTokenizer.from_pretrained(facebook/xglm-1.7B")
```

## Expected behavior

tokenizer to be of class "XGLMTokenizer"

Comments:
Thank you for reporting this issue! The fix is here #15406
Running MLM pretraining with not "line_by_line" big dataset
Author: lucadinidue
Pull request: no

# 🚀 Feature request

BERT pretraining with not line_by_line big dataset.

## Motivation
I'm trying to pretrain a BERT model with the classic masking task (without NSP), using examples/pytorch/language_modeling/run_mlm.py script. As with the original BERT, I would not like to construct each example of the batch using only one sentence, but would like to take a portion of text as long as max_sequence_lenght. This feature is provided if you set the "line_by_line" parameter to false with the "group_texts" function.
The problem is that I have a big dataset and I have to use "on-the-fly" tokenization with set_transform (as suggested in #10204). Given that, I can't map my dataset with group_texts anymore.

Is it possible to make it possible?

@sgugger 


Comments:
Hi there!

As mentioned in the README, examples are just examples. We can't show every single usecase everyone will want, so it's up to you to adapt them to your needs :-). If you need help, you can reach out on the [forums](https://discuss.huggingface.co/) or our Discord.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
dataset of Helsinki-NLP/opus-mt-en-zh
Author: QzzIsCoding
Pull request: no

Hi, thanks for your model. I have two questions of the train datasets of opus-mt-en-zh.

https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/data/README-v2021-08-07.md
English - Chinese eng-zho 10390 | 43075 | 129323178
Middle English (1100-1500) - Chinese enm-zho

In this website, there are two datasets from en to zh. Which is the dataset of opus-mt-en-zh?
When fine-tuning the model, does it need to add ">>cmn_Hans<< " before train_src?

Comments:
The README here mentions `eng-zho`, if I'm not mistaken: https://huggingface.co/Helsinki-NLP/opus-mt-en-zh

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
DebertaV2 For run_qa.py
Author: yinzhangyue
Pull request: no

## Environment info
- `transformers` version: 4.15.0
- Platform: Linux-4.4.0-142-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.3.5 (cpu)
- Jax version: 0.2.22
- JaxLib version: 0.1.72


## Models:

- [microsoft/deberta-v2-xlarge](https://huggingface.co/microsoft/deberta-v2-xlarge)


## Error:
```
Traceback (most recent call last):
  File "run_qa.py", line 642, in <module>
    main()
  File "run_qa.py", line 308, in main
    "This example script only works for models that have a fast tokenizer. Checkout the big table of models "
ValueError: This example script only works for models that have a fast tokenizer. Checkout the big table of models at https://huggingface.co/transformers/index.html#supported-frameworks to find the model types that meet this requirement
```

## Script
```
CUDA_VISIBLE_DEVICES=0 python run_qa.py \
      --model_name_or_path microsoft/deberta-v2-xlarge \
      --dataset_name squad \
      --do_train \
      --do_eval \
      --per_device_train_batch_size 12 \
      --learning_rate 3e-5 \
      --num_train_epochs 2 \
      --max_seq_length 384 \
      --doc_stride 128 \
      --output_dir /tmp/debug_squad/
```
[last run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py)

## My Problem
DebertaV2 has significant capabilities in NLI task. I want to use DebertaV2 for QA task. However, DebertaV2 don't have DebertaV2TokenizerFast. How can I use DevertaV2 for QA task? Is it possible to get offset_mapping without TokenizerFast? Thank you very much!


Comments:
DebertaV2TokenizerFast is close to being merged. If you look at the code in the PR, you can use it now. https://github.com/huggingface/transformers/pull/14928

OK! thank you~
Fix additional DataTrainingArguments documentation
Author: FremyCompany
Pull request: yes

(This is an editorial change only)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Defaults models for different pipelines
Author: p-christ
Pull request: no

Hey,

Where can I find the default models that are used for the different pipelines?

Comments:
Defaults are at https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L1459 . There is often a different default for tf and pt (pytorch).

Closing this as i think the question is resolved (also it's probably a better match for Stack Overflow)

What is the default model for the 'fill-mask' pipeline? I'm not able to tell from the previous answer in this thread. 

Any assistance much appreciated.

The defaults are defined [here](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L2987-L3087). The `fill-mask` pipeline uses the `distilroberta-base` checkpoint.

The above answers are out of date.
The defaults are now defined in [pipelines/__init__.py](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/__init__.py) (in the values of the SUPPORTED_TASKS dictionary).
adds decoding scores to Wav2Vec2DecoderWithLMOutput
Author: arampacha
Pull request: yes

# What does this PR do?


- Adds `logit_score` and `lm_score` produced by `BeamSearchDecoderCTC` to `Wav2Vec2DecoderWithLMOutput`. Useful for results analysis and pseudo-labeling for self-training.
- Removes padding from batched logits (as returned by `Trainer.predict`) at `batch_decode`. This prevents producing nonsensical tokens at the end of decoded texts.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
The PR is related to Robust ASR week.
@patrickvonplaten @anton-l 
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm fine with adding this actually. Is there a reason why you closed the PR? :-)

I see a new one was opened: https://github.com/huggingface/transformers/pull/15413/files . never mind
[Robust Speech Challenge] Add missing LR parameter
Author: jonatasgrosman
Pull request: yes

Some participants of the Robust Speech event complain about learning rate issues when training the model with the run_speech_recognition_ctc_bnb.py script. This problem is due to a missing LR parameter on the 8-bit optimizer initialization. This PR fixes this.

@patrickvonplaten

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Why I met Type 'seq(tensor(int64))' of operator (MemcpyFromHost) is invalid when using onnxruntime.InferenceSession() in GPU, and How to resolve it? On emergency hold，thanks!
Author: zzj-otw
Pull request: no

## Environment info
when I run exported onnx model of transformers (BARTBeamSearchGenerator model example) on GPU. I met this. Who knows how can resolve it?
<img width="1392" alt="截屏2021-12-24 下午3 32 55" src="https://user-images.githubusercontent.com/13781668/147331435-f518d5c7-b6fd-4550-9de9-940a9ec87003.png">

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):1.10.1
- Tensorflow version (GPU?):
- Using GPU in script?:YES
- Onnxruntime version: 1.8.0
- onnx version: 1.10.0






Comments:
Hi @yuanhuachao thanks for raising this issue! Can you please provide the exact command you used to export the model with? You might also be interested in a related issue #14882 about running this example script on a GPU.

Hi @lewtun ,i run the run_onnx_exporter.py with command 'python run_onnx_exporter.py --model_name_or_path facebook/bart-base  --device=cuda', in the directory of 'transformers/examples/onnx/pytorch/summarization'. And change the way to create onnxruntime.InferenceSession with CUDAExecutionProvider,( 'ort_sess = onnxruntime.InferenceSession(new_onnx_file_path, providers=['CUDAExecutionProvider']))'  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ConnectionError
Author: cjj-sunshine
Pull request: no

I encounterd the issue:ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.16.1/datasets/common_voice/common_voice.py when run run_speech_recognition_ctc.py. How can I solve this problem?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to generate output using custom embeddings?
Author: AzusaTsang
Pull request: no

My code is kinda like
```python
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
token_embeds, pos_embeds = custom_embeds()
output = model(inputs_embeds=token_embeds + pos_embeds, decoder_input_ids=torch.tensor([[tokenizer.bos_token_id]]))
```
How do I generate text output from this output? The `model.generate()` function requires `token_ids` so I can't use it.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Adding tokens to pretrained model "Helsinki-NLP/opus-tatoeba-en-ja" using tokens from vietnamese not working
Author: mhmd-mst
Pull request: no

Hello,
I am working on a code of the paper regarding multilingual model with multistage finetuning, I am using trainer API of huggingface for the finetuning of pretrained model from english to japanese with dataset containing vietnamese sentences, but before I want to modify the pretrained tokenizer adding tokens from other pretrained model tokenizer that recognize vietnamese 

model_checkpoint = "Helsinki-NLP/opus-tatoeba-en-ja"
model_mbart = "facebook/mbart-large-50-one-to-many-mmt"

mbart_tokenizer=MBart50Tokenizer.from_pretrained(model_mbart)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

tokenizer.add_tokens([vocab for vocab in mbart_tokenizer.get_vocab().keys())
model.resize_token_embeddings(len(tokenizer)) 

Before modifying the tokenizer, when tokenizing a vietnamese sentence i got 72 then i try to tokenize a vietnamese sentence with the modified tokenizer i got 66 tokens whereas using mbart50 i get 24 .
Can you please tell me what I am doing wrong??

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
GPT-J: Implement Memory Efficient Attention
Author: xloem
Pull request: no

Just a note that it would be great to add memory efficient attention ( https://github.com/AminRezaei0x443/memory-efficient-attention ) for larger models.  It might be nice to add an option to reuse the memory of input hidden states, too, if the user doesn't need them.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
XGLM Bug: break: module 'torch.utils' has no attribute 'checkpoint'
Author: mrseeker
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0.dev0
- Platform: Linux-4.9.253-tegra-aarch64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patil-suraj 

## Information

Model I am using (Bert, XLNet ...): XGLMForCausalLM

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I am trying to start training on a dataset using a converted model. 
2. As soon as I start training using the script `run_clm.py` (found in the examples folder) I get the following error back:
```
break: module 'torch.utils' has no attribute 'checkpoint'
```
Cause might be related to this: https://discuss.pytorch.org/t/attributeerror-module-torch-utils-has-no-attribute-checkpoint/101543/6

When adding the following line in the modeling_xglm.py, the error does not appear and training starts:

```
from torch.utils.checkpoint import checkpoint
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Training would start without any issues.

Comments:
Hi,
Could you please share a code snippet or colab so I could reproduce this issue? Thanks 

I can't share my dataset, but I used this one as the basis for the trainer:

https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py

EDIT: Just train any XGLM-compatible model with a default dataset (wikitext for example)

Thank you for reporting this. Fixed in #15427
[Hotfix] Fix Swin model outputs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR:

- fixes some critical outputs of `SwinModel`, to be exactly as `ViTModel`, `BeitModel`, `DeitModel`. More specifically, the `last_hidden_state` previously returned a pooled output, which shouldn't be the case. Instead, the output should be a tensor of shape (batch_size, seq_len, hidden_size), and the pooler output a tensor of shape (batch_size, hidden_size).
- adds `problem_type` support to `SwinForImageClassification`
- fixes the integration test

To do:

- [x] more of a nit, but the pooler in Swin is called `self.pool` rather than `self.pooler` (as in models like BERT, ViT). We could rename it to `self.pooler` for consistency.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
add t5 ner finetuning
Author: ToluClassics
Pull request: yes

# What does this PR do?

This PR adds a Colab notebook for fine-tuning Text-2-Text T5 for Named Entity Recognition

@patil-suraj @patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patil-suraj , thanks

I have reverted the commit
[deepspeed doc] fix import, extra notes
Author: stas00
Pull request: yes

As flagged by https://github.com/huggingface/transformers/issues/15399 the doc had a wrong import instruction so this PR is fixing it. Additionally adding some extra explanatory notes about non-HF Trainer DeepSpeed use.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix spurious warning in TF TokenClassification models
Author: Rocketknight1
Pull request: yes

This small fix switches `warnings.warn` for `tf.print` in the loss function for TF `TokenClassification` models. When TF compiles a model which has a conditional that depends on values inside a Tensor, it traces both branches. The result of this is that when one of the branches contains a `warnings.warn` statement, that warning is fired during tracing, before the model has seen any data.

Using `tf.print` instead of `warnings.warn` has a few downsides - in particular, it will possibly fire more than once. Still, this is better than firing the warning every time the model is used even if the data is correct, which is what happens now.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think if we use `FutureWarning` they are only printed once, no? Could you check in this instance @Rocketknight1 ?


@sgugger Either `FutureWarning` or `warnings.warn` are correctly only printed once - the problem is that they are **always** printed once, even if there are no -1s in the labels! This is because TF compiles the branch of the graph where the labels tensor contains `-1` and fires the warning during tracing. Only a TF graph op (like `tf.print()`) is correctly fired if and only if the tensor contains `-1`.

Ah thanks for explaining! I had completely misunderstood :-)

Don't worry, even the TF engineers don't understand TF