noamaanMulla-03/transformers-issues huggingface dataset:
fix layoutlmv2 doc page
Author: garyhlai
Pull request: yes

# What does this PR do?

Quick follow up PR to #17168 to address @NielsRogge 's comments

Clarify that the `torchvision` and `tesseract` packages are optional dependencies for LayoutLMv2.

## Who can review?
@sgugger @NielsRogge 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17409). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[bloom] fix alibi device placement
Author: stas00
Pull request: yes

This PR fixes alibi device placement - currently it's the default device which breaks things at times - it has to be set explicitly to the correct device.

The problem emerged when trying to get DeepSpeed-Inference working.

Kudos to @RezaYazdaniAminabadi for discovering the problem.

cc: @younesbelkada 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot for the fix !

btw, we also discussed to change the alibi creation logic.

it should create it once in init with a largish length (say 1k) and not change it again unless the input is longer.  

then the device and dtype will be automatically handled correctly.

this is the logic that all `transformers` positional embeddings use.

Yes agreed !
We are already addressing these issues in this PR together with refactoring the whole attention block which looks too complicated : https://github.com/huggingface/transformers/pull/17866 
Here alibi (including the shifting) and the attention mask is created only once
Add Visual Question Answering (VQA) pipeline
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

### Feature request

We currently have [ViLT](https://huggingface.co/docs/transformers/model_doc/vilt) in the library, which, among other tasks, is capable of performing visual question answering (VQA).

It would be great to have a pipeline for this task, with the following API:

```
from transformers import pipeline

pipe = pipeline("vqa")
pipe("cats.png", "how many cats are there?")
```

This pipeline could default to the https://huggingface.co/dandelin/vilt-b32-finetuned-vqa checkpoint. Also check out the [Space](https://huggingface.co/spaces/nielsr/vilt-vqa) that showcases the model.

This can be implemented similar to [other pipelines](https://github.com/huggingface/transformers/tree/main/src/transformers/pipelines). For an example PR that added a pipeline, see #11598.

### Motivation

A pipeline is required in order to have inference widgets + a task defined at hf.co/tasks.

Moreover, it would be great to do VQA in two lines of code.

### Your contribution

I can definitely assist in this, together with @Narsil, who's the pipeline expert. 

Comments:
Tagging @mishig25 for the widget 

Also LXMERT should handle this task, but likely has a very different API.

This sounds amazing. Happy to contribute in anyway I can

I'd love to pick this up!

Hey @sijunhe, I'm just starting out in open-source, but I'd like to help out however I can! 

@sabarish-srinivasan appreciate the help but I saw this a little late and I am almost done with the PR.


@sijunhe No problem, thanks for letting me know! 

@LysandreJik I looked at both ViLT and LXMERT and I don't think it's possible to combine these two into a single pipeline for the following reasons:

1. ViLT formats VQA as a classification task and LXMERT formats VQA as a squad-like QA task. It'd be hard to write a common post-processing
2. ViLT is self-contained within transformers but LXMERT expects some faster-RCNN model to generate the visual features that goes into the model.

Yes, don't think we should support LXMERT for the pipeline, since it isn't entirely included in the Transformers library.

Sounds good, let's go with ViLT then!

Now that #17286 is merged, this issue should be closed now?

Yes :) Thank you for your contribution @sijunhe!
train_new_from_iterator missing from GPT2Tokenizer
Author: ncoop57
Pull request: no

## Environment info

- `transformers` version: 4.15.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help
Library:

- Tokenizers: @LysandreJik

Examples:

- research_projects/codeparrot: @lvwerra 

## Information

Model I am using (Bert, XLNet ...): GPT2Tokenizer

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Was using the official research project script for codeparrto to train a new BPE tokenizer with the GPT2Tokenizer as the base: https://github.com/huggingface/transformers/blob/master/examples/research_projects/codeparrot/scripts/bpe_training.py

## To reproduce

Steps to reproduce the behavior: run the following colab: https://colab.research.google.com/drive/1G34QEjP9oNfcP_w_JO6DEnJuUTOsZe_D?usp=sharing

## Expected behavior

To be able to train a new GPT2Tokenizer from an iterator.

Comments:
Just found if I use `AutoTokenizer` rather than the `GPT2Tokenizer` class it works. Is this expected? If so the codeparrot example will need to change.

The `train_new_from_iterator` works with fast tokenizers only. `GPT2Tokenizer` is a slow tokenizer, with `GPT2TokenizerFast` being its fast counterpart.

Thanks @ncoop57 for flagging and @LysandreJik  for pointing out the issue. That is a bug I introduced when switching from `AutoTokenizer` to `GPT2Tokenizer` - I'll fix this shortly.

Hi @lvwerra, I hate to say that it's still giving issues. I am on kaggle (ubuntu) with transformers 4.5.1 and tokenizers 0.10.3 and so even with fast tokenizers, it still says `has no attribute train_new_from_iterator...`

Hi @maulberto3, could you share a minimal reproducible example? Does it work with the latest transformers version?

**Update**: if I read the commit history correctly that feature was added in `v4.9.0` so you need to update the transformers library.

Hi @lvwerra You are right, I fixed the version and it works. Thanks.

What are the `parameters` inside `train_new_from_iterator`? Can I pass `frequency` inside it?
Also which `tokenizer` does  it trains? Is it `BPE `tokenizer?

Hi @pratikchhapolika, such questions are best asked in the forum [discuss.huggingface.co](discuss.huggingface.co) or you can have a look at the documentation [here](https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast.train_new_from_iterator).
Untranslation of some words from an external dictionary
Author: Dmitry-Sn
Pull request: no

I use some pre-trained translator models from your library (for example, Helsinki-NLP). During the translation process, I would like to leave some words untranslatable (for example, acronyms, toponyms or names) due to the presence of errors in their translation.  I tried adding extra tokens and replacing these words with conditional tokens (<extra_id_0>), but this translation requires raising the num_beams parameter, which significantly slows down the translation. Unfortunately, I can't find any additional mechanisms to perform this task. Is it provided or creating it is a custom task?

Comments:
Hey @Dmitry-Sn,

It would be great if you could ask these kind of questions on the forum: https://discuss.huggingface.co/ . We try to keep github for issues and less for user-specific use cases. Thanks!

> 

Hi,  have you found some way to solve this problem? I meet the same situation, some proper nouns can' t translated properly, and I want to keep it as its native format.

Hi @vpegasus! There were not great ideas, generally.
As far as I remember, I decided to try to finetune the model for a special token. In my task, it was simple - since I wanted to keep geographical names in the original language, I replaced only them in the training data with a special token (I didn't have time to check the effectiveness, since I left that company).
Colleagues also suggested a solution with special characters (it seems to separate the word with <> characters), but it worked poorly.
All this applies to the models on MarianMT.
Loading BertModel from BertForMaskedLM without randomly initializing weights
Author: jmeadows17
Pull request: no

I've trained a `BertForMaskedLM` model for a few days, and I've `save_pretrained()` it.

I want to compare encoder vectors from a `BertModel`, with the embeddings from
this trained `BertForMaskedLM`.

However, simply using `BertModel.from_pretrained("path/to/new_BertForMaskedLM_model")`
warns me that "You should probably TRAIN this model....". 

It seems that `bert.pooler.dense.weight` and `bert.pooler.dense.bias` have been added and randomly initialized after the LM head was removed. I assume this means the output of this current model will be useless.

What do I need to do to compare `BertForMaskedLM` encoder embeddings (after removing the LM head) with `BertModel` encoder embeddings? 

Comments:
Hey! The model returns several outputs:

```
    last_hidden_state: torch.FloatTensor = None
    pooler_output: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
    past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
    attentions: Optional[Tuple[torch.FloatTensor]] = None
    cross_attentions: Optional[Tuple[torch.FloatTensor]] = None
```

The `pooler_output` is going to be random as the pooler weights will be randomly generated, but the rest will not be! In order to compare the two, you could compare the `hidden_states` values, which contain all intermediary values from the word embeddings up until the last transformer layer's output.

I suppose more specifically I'm after the following output:

```
model = BertModel('path/to/new_BertForMaskedLM_model')
output = model(**encoded_input)[1]
```

I've got other baselines with embeddings extracted like this. 

Do you mind explaining the difference between `hidden_states` output and `model(**encoded_input)[1]` please?

`[1]` will take the second value of the output. It may differ across models, while specifying `output.hidden_states` will always return the hidden states.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add filename to info displayed when downloading things in from_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

The progress bar used in `http_get` has a description saying "Downloading". When we download multiple files (for instance for a sharded chackpoint) that's not necessarily super informative, so this PR adds the name of the file being downloaded to the description.

cc @stas00 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you so much, Sylvain!
Can't convert Flax T5 model to PyTorch
Author: Beau-xu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.18.0
- Platform: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.6.13
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.10.2 (False)
- Tensorflow version (GPU?): 2.6.2 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.5 (cpu)
- Jax version: 0.2.17
- JaxLib version: 0.1.69
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

@patrickvonplaten @patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Failed to convert a T5 model from Flax to PyTorch

```python
import tempfile
from transformers import AutoTokenizer, FlaxT5ForConditionalGeneration, T5ForConditionalGeneration
tmp = tempfile.mkdtemp()

flax_model = FlaxT5ForConditionalGeneration.from_pretrained("t5-small")
flax_model.save_pretrained(tmp)
pt_model = T5ForConditionalGeneration.from_pretrained(tmp, from_flax=True)
```

### Expected behavior

Some weights of T5ForConditionalGeneration were not initialized from the Flax model: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']

```
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
/data/home/db72687/miniconda3/envs/decipher/lib/python3.6/site-packages/transformers/modeling_flax_pytorch_utils.py:240: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_numpy.cpp:189.)
  pt_model_dict[flax_key] = torch.from_numpy(flax_tensor)
All Flax model weights were used when initializing T5ForConditionalGeneration.

Some weights of T5ForConditionalGeneration were not initialized from the Flax model and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Comments:
Sorry, I made a mistake. 
Actually the missing weights of above the above PyTorch model converted from Flax can be correctly initialized by accepting the Flax model's weight, `fx_model.params['shared']['embedding']`.
IPEX integration in Trainer breaks with PyTorch 1.12
Author: sgugger
Pull request: no

All the tests for the IPEX integration in Trainer started to break with the latest PyTorch release. Error is:
```
ImportError: /usr/local/lib/python3.8/dist-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so: undefined symbol: _ZNK3c1010TensorImpl5sizesEv
```

cc @hshen14

Comments:
@sgugger Thanks for reporting. IPEX v1.12 will be releasing to support latest PyTorch version by next week. The team will look into the issue and let you know whether the issue is getting fixed in upcoming release. cc @jianan-gu 

@sgugger , this is Eikan from IPEX team. This is a version-mismatch issue. IPEX 1.11 is on top of PyTorch 1.11. And the upcoming release of IPEX(1.12) will resolve this issue as the latest PyTorch is 1.12. I will keep you posted as long as the IPEX is released.

Hi @sgugger,
IPEX 1.12 release is available https://intel.github.io/intel-extension-for-pytorch/1.12.0/tutorials/installation.html ;
And we also open a PR https://github.com/huggingface/transformers/pull/18072 to enhance the integration for this version mismatch issue to avoid breaking Trainer;
Thanks!
DistilBERT accuracies on the glue test set.
Author: smr97
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I need to compare my research against distilBERT as a baseline for a paper in progress. I went through your publication and found that you don't report accuracies on the glue test set and instead on the dev set. TINY BERT publication by huawei tries to reproduce your work, but the numbers are lower.

I would really appreciate some help regarding this. As far as i understand, i need to distill the student with the entire Wikipedia + bookcorpus? Is there any way to skip this step (load a model that you might have)?

Alternatively, if you have the latest submission on glue, it would really help to know the numbers.

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I want the numbers too.

You can check the model card for the evaluation results: https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
invalid multinomial distribution (with replacement=False, not enough non-negative category to sample)
Author: Muennighoff
Pull request: no

When using "sshleifer/distilbart-cnn-6-6" & do_sample the below code errors out, meanwhile the same code works for "sshleifer/distilbart-xsum-6-6". Am I missing something really obvious here? Thanks for any help!

Tranformers: 4.5.1

````
from transformers import (
    AutoModelForSeq2SeqLM, 
    AutoTokenizer
  )

model_name = "sshleifer/distilbart-cnn-6-6"
#model_name = "sshleifer/distilbart-xsum-6-6"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

text = "New York City (NYC), often simply called New York, is the most populous city in the United States"
input_ids = tokenizer.encode(text, return_tensors='pt')

sample_outputs = model.generate(input_ids, 
                                num_beams=3,
                                do_sample=True
                                )
sample_outputs
````

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

i have the same exact problem when i use `do_sample=True` can you re-open this issue?

Maybe @gante has an idea!

Hi there @Muennighoff @zeke-john 👋 

I've run the script above for both models on `v4.5.1` (and on `v4.22.dev0`) and it works with no problems -- you can see a colab [here](https://colab.research.google.com/drive/1bg7v0mxZbFxJTjj28AriYBeERsAN264E?usp=sharing).

A potential cause for errors may be GPU memory -- generation with `num_beams` is memory intensive. Let me know if you have more details about your problem :)
Report value for a step instead of epoch.
Author: zhawe01
Pull request: yes

# What does this PR do?
Report an objective function value for a step instead of epoch to optuna.

## I made this modification for the following reason:
If "eval_steps" is less than steps per epoch, there maybe warnings: `optuna/trial/_trial.py:592: UserWarning: The reported value is ignored because this ‘step’ 0 is already reported.`. This is because the epoch granularity is too coarse. So "step" are more appropriate than "epoch" here.

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. 
@sgugger @LysandreJik 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Looks good, thanks for fixing!
> 
> Can you just run `make style` on your branch to make sure the formatting check passes?

Done. @sgugger 

Thanks a lot! (test failure is already fixed on main, so merging)
CLI: reenable `pt_to_tf` test
Author: gante
Pull request: yes

# What does this PR do?

Reenables the `pt_to_tf` CLI test that was disabled a few days ago.

⚠️ Before this PR is merged, [this](https://huggingface.co/hf-internal-testing/tiny-random-gptj/discussions/1) hub PR must be merged, and CI must be rerun. This test model is also used in `tests/deepspeed/test_model_zoo.py`, not sure if the change will have implications there.

The problem was not due to code, but rather to problems in the config file of the test model. When `config.rotary_dim` is larger than `self.head_dim` (which is `config.hidden_size` divided by `config.num_attention_heads`), then we try to slice out of bounds in some tensors, causing downstream dimension-related exceptions -- e.g. [slicing here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gptj/modeling_tf_gptj.py#L229).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update index.mdx
Author: BritneyMuller
Pull request: yes

# What does this PR do?
![Screen Shot 2022-06-03 at 2 57 06 PM](https://user-images.githubusercontent.com/5594118/171920956-28640a8b-4234-4efc-b9cf-2e692c0feb9a.png)

This removes the extra space in front of the new /support image. Thank you for the suggestion @sgugger ! 

Got too excited to merge the previous image update and missed this housekeeping fix. 

Comments:
@sgugger I don't think I agree with the suggestion. This is not my area of expertise 😅, but I think HTML elements should be indented (e.g. `<img>` child element indented within the `<a>` parent element), even on a markdown file. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17548). All of your documentation changes will be reflected on that endpoint.

It wasn't indented before the first PR and was working perfectly fine.

Yes, it works in either case. I think it's usually suggested to use indentation in HTML elements for readability purposes. No strong opinion on this, though!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add GreaseLM model
Author: vblagoje
Pull request: yes

# What does this PR do?
Adds GreaseLM model. For more details, see https://github.com/snap-stanford/GreaseLM and https://arxiv.org/abs/2201.08860

For a quick check, see this [colab](https://colab.research.google.com/drive/19ePb4fZysn45wkhzAzOlcDSEWAGnSgto)
CommonSenseQA [validation](https://colab.research.google.com/drive/17Qy6X_rY3qRzetFS1znGmdS1JyP25jOO) set results. 
OpenBookQA [validation](https://colab.research.google.com/drive/1RAkGV-KBVq1FOOpLt_MnZ_z9aZlNPVW7) set results.
OpenBookQA [test](https://colab.research.google.com/drive/18SXxMPMvr98JwsqTVKjNe-Dif3XEXuqs) set results.
Results for both OpenBookQA and CommonSenseQA match the reported results from the original repo. 

## To do:
- [x] Make sure all documentation and tests are up to HF standards
- [x] Needs careful review due to ~~two~~ three soft dependencies: spacy, torch_scatter and torch_sparse

@LysandreJik @NielsRogge 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17146). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TF: remove graph mode distinction when processing boolean options
Author: gante
Pull request: yes

# What does this PR do?

Removes a very old `if` branch related to boolean options, as TF graph mode can handle both branches with no issues -- it passes core tests for the models I tried.

It also unblocks @sayakpaul in the demo he's building for the TF SegFormer (#17910), which requires setting `output_hidden_states` in graph mode.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM! All code of this type should be removed imo - it's an artifact of TF 1.x. Modern TF code is compiled by tracing and recompiled if Python flags are changed and can handle all kinds of weird Python flow control as a result.