noamaanMulla-03/transformers-issues huggingface dataset:
Does the word embedding matrix of GPT2 load from the checkpoint,  during the fine-tuning?
Author: Choitsugun
Pull request: no

Does the word embedding matrix of GPT2 load from the checkpoint,  during the fine-tuning?
（transformer.wte.weight⇒shared token embedding layer Shared weights logic）
  
I tried to do the following:
  
  model = TFGPT2LMHeadModel.from_pretrained("gpt2-medium"）
  model.get_output_embeddings().weight
  
But the following error was occurred, probably because the parameters of wte.weight were not initialized.

  tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) Failed precondition: Error while reading resource variable tfgp_t2lm_head_model/transformer/wte/weight from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/tfgp_t2lm_head_model/transformer/wte/weight)
	 [[node tfgp_t2lm_head_model/transformer/wte/weight/Read/ReadVariableOp (defined at home/nak/cho/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
	 [[tfgp_t2lm_head_model/transformer/wte/weight/Read/ReadVariableOp/_1]]
  (1) Failed precondition: Error while reading resource variable tfgp_t2lm_head_model/transformer/wte/weight from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/tfgp_t2lm_head_model/transformer/wte/weight)
	 [[node tfgp_t2lm_head_model/transformer/wte/weight/Read/ReadVariableOp (defined at home/nak/cho/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]

I am very confused. Shouldn't the parameters of the word embedding table be restored directly from the checkpoint during fine-tuning? Why is it not initialized? 
 
Moreover, when adding special tokens through the following operations, the number of tokens in the dictionary changes, which means the dimension of the word embedding table is changed. Does this mean that the wte.weight needs to be reinitialized? Will the fine-tuning effect be good in this case? 

 num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
 model.resize_token_embeddings(len(tokenizer))
 
Many thanks!
 

Comments:
I was mistaken. Sorry.
Shift operation in loss computation for seq2seq model
Author: ZHANG-GuiGui
Pull request: no

Hello, I'm using the Seq2seqTrainer provided in Legacy here https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/seq2seq_trainer.py#L162 
I got a question when I read the `_compute_loss` function as it provides three different ways to calculate the loss.

In general, for text generation task, the labels and the logits need to be shifted before calculating loss for the purpose of predicting the next token.

**I wonder whether the methods in line 163 and in line 171 are lack of the shifting operation.** As the Seq2seqTrainer split the `labels` from the `inputs`, the model with `labels=None` will not compute the loss and will not shift the logits and labels in the `forward` function.
```
logits = model(**inputs, use_cache=False)[0] # no shift in the forward function
loss = self.loss_fn(logits.view(-1, logits.shape[-1]), labels.view(-1)) # no shift in the loss computation
```
While the method in line 168 will do the shift as the `labels` is provided to the `forward` function.

Do I understand right? Hopeful to get your advice.

- Text generation: @patrickvonplaten
- Trainer: @sgugger


Comments:
Please note that this code is no longer maintained, so questions about it should be asked on the [forums](https://discuss.huggingface.co/) since we keep the issues for bugs and feature requests only.

As for your question, I'm not sure why you want to shift labels in a sequence-to-sequence problem. This Trainer is used for problems like summarization and translation, where there is no label shifting. If you want to apply it to a causal LM problem, you will need to shift your labels before sending them to the model.

Thanks for your fast reply. 
Sorry for asking my question in the wrong place. I've opened a topic on the [forums](https://discuss.huggingface.co/t/seq2seq-loss-computation-in-trainer/10988).
As for your reply, I don't understand why there is no need for shifting in seq2seq problem. The decoder inputs and the labels are identicals. I think that the labels need to be shifted by one position, because we need to predict the next token referring to the decoder inputs.
Perhaps we could further discuss this question on the [forums](https://discuss.huggingface.co/t/seq2seq-loss-computation-in-trainer/10988)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
gramformer installation error.
Author: viru06
Pull request: no

I am using VS Code for **gramformer** and Python 3.7 & Pip version **pip==20.1.1**   and used the below link.

**pip install git+https://github.com/PrithivirajDamodaran/Gramformer.git**
I am getting this error, **How to resolve it**
ERROR: lm-scorer 0.4.2 has requirement transformers<3.0.0,>=2.9.0, but you'll have transformers 4.11.3 which is incompatible.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
hf wav2vec2 to fairseq
Author: natspan
Pull request: no

Hello, I was wondering if it's possible to convert the pretrained or finetuned hf wav2vec2 models to the fairseq pytorch format?


Comments:
Good question! We usually only do it the other way around :D 
I think the best way to start would be to try to invert the conversion script here: https://github.com/huggingface/transformers/blob/master/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
"Resource exhausted" when loading Flax GPT-Neo 2.7B
Author: rolandgvc
Pull request: no
Labels: name: WIP

## Environment info

- `transformers` version: 4.10.0.dev0
- Platform: Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.4 (cpu)
- Jax version: 0.2.19
- JaxLib version: 0.1.70
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@patrickvonplaten @patil-suraj @LysandreJik

## Information

I am not able to load the Flax GPT-Neo 2.7B model in my TPU VM v3-8 instance. 
```python
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B", pad_token="
", padding_side="left")
model = FlaxAutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B", pad_token_id=tokenizer.eos_token_id)
```
The model will download but will fail to load with
```
RuntimeError: Resource exhausted: Failed to allocate request for 100.00MiB (104857600B) on device ordinal 0
```
However, the pytorch version will load and run just fine.


Comments:
Hi, any updates on this?

Thanks for reporting this, I'm looking into it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Unsale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Running into this same issue when trying to load t5-3b as a Flax model from the Pytorch version on TPU3.8

Working on a feature that should fix this issue. This is probably because the model is initialized randomly and the weights are on the device, and then the pre-trained weights are also loaded directly on the device. So working on a feature that allows initializing the model only abstractly to consume less memory. Should be available in a couple of weeks :) 

Any update on this @patil-suraj 
We are experiencing this when trying to load RoBERTa using Flax with TPU v3-8
Fix a Bug, modeling_bert.py, erroneosly switched BCE and CrossEntropy
Author: xmvlad
Pull request: yes

_In BertForSequenceClassification when detecting what type of loss to use, based on self.config.problem_type  BCEWithLogitsLoss and CrossEntropyLoss erroneosly switched. It works correctly in case when self.config.problem_type not defined, because it switched in detection and application code, but if problem_type  correctly defined by model it fail(or if "single_label_classification" selected CrossEntropy works as replacement for BCE obviously without errors). I found it when trying to fine-tune "unitary/toxic-bert" model._

Update:
Sorry i'm incorectly read mulit-label vs multi-class problem. i'm just use this model "unitary/toxic-bert" with removed head to fine-tune multi-class and get errors with multi-label defined loss. :(
## 
@LysandreJik

Comments:
@LysandreJik Can you please review this PR. And help how to fix torch tests and check_code_quality?
LayoutLMv2FeatureExtractor now supports non-English languages when applying Tesseract OCR.
Author: Xargonus
Pull request: yes

# What does this PR do?

This PR adds an additional `ocr_lang` argument to the \_\_init\_\_ method of LayoutLMv2FeatureExtractor  which specifies which Teserract model to use when applying Tesseract OCR.

Fixes #14511 

@NielsRogge

Comments:
Can you verify the slow tests (which are not run by the CI) are passing as well?

i.e. `RUN_SLOW=yes pytest tests/test_feature_extraction_layoutlmv2.py` and `RUN_SLOW=yes pytest tests/test_feature_processor_layoutlmv2.py `

I assume you meant `RUN_SLOW=yes pytest tests/test_feature_extraction_layoutlmv2.py` and `RUN_SLOW=yes pytest tests/test_processor_layoutlmv2.py`?

I am developing on Windows, therefore my options when installing tesseract are limited to available installer versions. After moving from v5.0.0 to v4.1.0, which is the closest to v4.1.1, the version used to get bboxes in the `test_feature_extraction_layoutlmv2.py`, all of these tests ran successfully (There are no slow ones here).

As for `test_processor_layoutlmv2.py`, `test_processor_case1` failed for me, even when staying in the master branch. I was unable to reproduce the environment used to develop these tests on my computer. The reason is probably the different Tesseract version or model. The error comes from this line: https://github.com/huggingface/transformers/blob/3772af49ceba348f2c9c5bbbb7f7c12e35d2a6eb/tests/test_processor_layoutlmv2.py#L210
LayoutXLMProcessor applies the english Tesseract model
Author: Xargonus
Pull request: no

# 🚀 Feature request
LayoutXLMProcessor.\_\_call\_\_ should support a language argument for Tesseract OCR


## Motivation

LayoutXLM is a multilingual version of the successful LayoutLMv2 model. The main reason to use it over LayoutLMV2 is to handle different languages, yet the current API does not allow specifying the language to be used in apply_tesseract.

## Your contribution

I could submit a PR but I am not that familiar with the Transformers library to suggest the best place to add the lang argument.


Comments:
Hi,

Great point, thanks for the feature request.

It would be rather straightforward: one should just add a lang parameter to [this line](https://github.com/huggingface/transformers/blob/956a483173e77ebf655ca9636a5f7b6ef010b307/src/transformers/models/layoutlmv2/feature_extraction_layoutlmv2.py#L54).

Can you open a PR for this?
Doc new front github actions
Author: LysandreJik
Pull request: yes

New doc builder.

Before merging, will only need to change the branch name to which the script pushes.

Comments:
Doesn't have to be part of this PR. making a note here that: 
needs to be updated with each new build/release
https://github.com/huggingface/transformers/blob/765f6a4f1728050e487ba93d51512672ef3d6c4d/docs/source/versions.yml#L1-L5 just as this list is updated currently 
 https://github.com/huggingface/transformers/blob/c6c075544d95940b086c2aaae46e114a0e3b9ab2/.circleci/deploy.sh#L81-L84
Add a new gradient regularization feature
Author: RunxinXu
Pull request: no

# 🚀 Feature request

When we fine-tune a large-scale pre-trained model in the downstream task (especially with low-resource data), it would be helpful if we randomly discard some gradients, as introduced in this [paper](https://aclanthology.org/2021.emnlp-main.749.pdf).
Therefore, I wonder if I could add this simple feature to transformers?

## Motivation

Add a simple regularization feature to address the mismatch between large models and small training data.

## Your contribution

I implement the feature by adding the following codes in the `step()` function in the optimizer.

```
grad_mask = Bernoulli(grad.new_full(size=grad.size(), fill_value=self.reserve_p))  
grad *= grad_mask.sample() / self.reserve_p
```


Comments:
Maybe of interest to @sgugger or @Rocketknight1 :)

It seems like an interesting paper, but there are **a lot** of papers with new training recipes like this, and I don't think we can implement them all until they become very famous and widely-used. Also, this particular method should be relatively easy for users to implement in their custom training loops or TF `train_step` code, so I think we probably would not be able to accept this PR right now (although we definitely appreciate the enthusiasm!). @sgugger do you agree?

 I agree this is out of scope of the `Trainer` for now, unless it becomes a widely used technique. In the meantime, using our [`Accelerate`](https://github.com/huggingface/accelerate) library lets the user write their custom training loop where they can implement this feature easily :-)

Thanks so much :)
[BUG] Tokenizer offset
Author: SmileTM
Pull request: no

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
text = "k , u , o , 105 , 957 , 601 . "
out = tokenizer.encode_plus(text=text,
                            truncation=True,
                            max_length=128,
                            return_tensors='tf',
                            return_token_type_ids=True,
                            return_attention_mask=True,
                            return_offsets_mapping=True,
                            return_length=True)

print(tokenizer.tokenize(text))
text_offsets = out['offset_mapping'][0].numpy().tolist()
for i in text_offsets:
    s,e = i
    print(f"{(s,e)}:{text[s:e]}")
```

```
['▁k', '▁', ',', '▁u', '▁', ',', '▁o', '▁', ',', '▁105', '▁', ',', '▁9', '57', '▁', ',', '▁', '601', '▁', '.']
(0, 0):
(0, 1):k
(2, 3):,
(2, 3):,
(4, 5):u
(6, 7):,
(6, 7):,
(8, 9):o
(10, 11):,
(10, 11):,
(12, 15):105
(16, 17):,
(16, 17):,
(18, 19):9
(19, 21):57
(22, 23):,
(22, 23):,
(24, 25):6
(24, 27):601
(28, 29):.
(28, 29):.
(0, 0):
```
the offset  can't  return the Original text
1.the offset of '▁' will be  filled  by next token .   ????
2.  (24, 25):6   ????

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add a parameter "device " in Tokenizer.__call__()
Author: zchuz
Pull request: no

In method Tokenizer.__call__() Add a parameter "device" to get batchencoding whose tensors are on the specified device

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Target customized languages using multi-lingual sentence transformer models like "stsb-xlm-r-multilingual"
Author: Tortoise17
Pull request: no

I have seen these models with multi-lingual ability and they are very nice. 
But, I would like to ask that is there any possibility to have the specific target language / languages from the multi-language data search?
Like source language >> input, target language << output?
Main motivation is that, I have data in several languages. But I want only specific languages to be searched and get the results from that. 

Comments:
VisionTextDualEncoder
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds `VisionTextDualEncoder` model in PyTorch and Flax to be able to load any pre-trained vision (`ViT`, `DeiT`, `BeiT`, CLIP's vision model) and text (`BERT`, `ROBERTA`) model in the library for vision-text tasks like CLIP.

This model pairs a vision and text encoder and adds projection layers to project the embeddings to another embeddings space with similar dimensions. which can then be used to align the two modalities.

The API to load the config and model is similar to the API of `EncoderDecoder` and `VisionEncoderDecoder` models.

- load `vit-bert` model from config

```python3
config_vision = ViTConfig()
config_text = BertConfig()

config = VisionTextDualEncoderConfig.from_vision_text_configs(config_vision, config_text, projection_dim=512)
# Initializing a BERT and ViT model
 model = VisionTextDualEncoderModel(config=config)
```

- load using pre-trained vision and text model

```python3
model = VisionTextDualEncoderModel.from_vision_text_pretrained(
    "google/vit-base-patch16-224", "bert-base-uncased"
)
```

Since this is a multi-modal model, this PR also adds a generic `VisionTextDualEncoderProcessor`, which wraps any feature extractor and tokenizer


```python3
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
processor = VisionTextDualEncoderProcessor(feature_extractor, tokenizer)
```

Comments:
Let's merge this asap - it's been on the ToDo-List a bit too long now ;-)
Using `bf16` instead of `fp16`
Author: JamesDeAntonis
Pull request: no
Labels: name: WIP

# 🚀 Feature request

As seen in [this pr](https://github.com/huggingface/transformers/pull/10956), there is demand for `bf16` compatibility in training of transformers models.  The pytorch folks just [added this feature](https://github.com/pytorch/pytorch/pull/61002) to their master branch, so we are now able to work on adding it to this repo.

## Motivation

Related to [this issue](https://github.com/huggingface/transformers/pull/10956) and [this pytorch pr](https://github.com/pytorch/pytorch/pull/61002).  This feature would allow for proper half-precision training of google-trained models, for example any `T5` model.

## Your contribution

I am currently working on a PR for this [here](https://github.com/JamesDeAntonis/transformers/tree/bf16), and would gladly field any suggestions and contributions.

@stas00 

Comments:
Very much looking forward to enable bf16 in PyTorch :-) Think we should probably wait though until the next PyTorch release is out. But it would be a good idea to have it supported as soon as the release is out then. cc @LysandreJik @sgugger 

Indeed. I will install pt-nightly and work on this.

Thank you for staying on top of the pytorch development, @JamesDeAntonis 

@stas00 I'm going to open a pr shortly for my [branch](https://github.com/JamesDeAntonis/transformers/tree/bf16).  Feel free to check that out for starter
NotebookProgressBar can't display in Pycharm jupyter
Author: li-xuanhong
Pull request: no

this is full code
```
from transformers.utils.notebook import NotebookProgressBar
import time
pbar = NotebookProgressBar(100)
for val in range(100):
    pbar.update(val)
    time.sleep(0.07)
pbar.update(100)
```

when I use Pycharm jupyter, it display like this and not move
![QQ截图20210929100353](https://user-images.githubusercontent.com/44440077/135194799-b491a14a-e1b8-4caf-8b0b-90e00d6f74e3.png)

but when I use vscode, it run correctly
![QQ截图20210929100651](https://user-images.githubusercontent.com/44440077/135194869-1ae8ba61-e2f1-4a2c-94a3-56d1498e1811.png)

Can you give me some advise? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry I missed this! cc @sgugger if he has any idea of what might be going on

Not sure what the problem is with PyCharm notebooks. I haven't tried them before, so you should stick with the normal progress bars there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
LayoutLMv2 functionality.
Author: nataliebarcickikas
Pull request: yes

Support for new LayoutLMv2 as discussed in this issue: https://github.com/huggingface/transformers/issues/14160.

Namely, the addition of `LayoutLMv2ForMaskedLM`, which adds a language modeling head on top of the base model to replicate the first pre-training objective in the original paper. 

Comments:
Hi,

This is actually not the implementation, I see you just added text to a README file. If you want to add `LayoutLMv2ForMaskedLM`, you should implement it in `modeling_layoutlmv2.py` which can be found [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/layoutlmv2/modeling_layoutlmv2.py).