noamaanMulla-03/transformers-issues huggingface dataset:
Fix `TFSwinSelfAttention` to have relative position index as non-trainable weight
Author: harrydrippin
Pull request: yes

# What does this PR do?

This PR fixes `TFSwinSelfAttention` to have `relative_position_index` as non-trainable weight.

## Problem

When trying to convert `SwinModel` to `TFSwinModel` by using `TFSwinModel.from_pretrained(weight_path, config, from_pt=True)`, I faced the warning below:

```
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFSwinModel:
['encoder.layers.2.blocks.0.attention.self.relative_position_index',
'encoder.layers.2.blocks.1.attention.self.relative_position_index',
'encoder.layers.2.blocks.6.attention.self.relative_position_index',
'encoder.layers.2.blocks.7.attention.self.relative_position_index', ...
```

**I checked that `SwinModel` has those keys on its weight while `TFSwinModel` hasn't.** `SwinModel` assigned this value as non-trainable weight by using `self.register_buffer`, but on `TFSwinModel` it was just assigned as class members (`self.relative_position_index = tf.reduce_sum(...)`).

## Fix

I added `relative_position_index` as non-trainable parameter by using `self.add_weight` on `build()`, so that `relative_position_index` can have proper key name on `model.weights` list. I checked the conversion that I failed is successfully done after applying this fix.

I also tried to just change `self.relative_position_index` to `tf.Variable(..., trainable=False)`, but It didn't work due to the key name. This will set key name as `relative_position_index:0`, not like `tf_swin_model/swin/encoder/layers.0/.../self/relative_position_index:0`.

## Review

This PR is related with Swin Transformer and TensorFlow.
TensorFlow: @LysandreJik

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Adding you for a final TF review before merging
Refactor `TFSwinLayer` to increase serving compatibility
Author: harrydrippin
Pull request: yes

# What does this PR do?

This PR refactors two parts of `TFSwinLayer` like below, to increase serving compatibility on multiple accelerators.

### Fix incompatible type cast on several serving architecture

On the `window_reverse` function, there is incompatible computation on the graph when restoring batch size:

```python
x = shape_list(windows)[0]
y = tf.cast(height * width / (window_size * window_size), tf.int32)
batch_size = int(x / y) # <- Here
```

This operation can cause tracing failures depending on the accelerator or compiler SDKs. I have confirmed that AWS Neuron SDK (w/ Inferentia) cannot trace this part properly.

We have `batch_size` variable on graph inside `call()`, so I modified this function to use this value by passing it as a function argument.

### Fix mixed use of member & local variables

Several parts are using window size, but some use `window_size`, and some use `self.window_size`. This can cause tracing failure because of unclear separation between compile-time constants and runtime values.

This PR applies to fix this by not using `self.window_size` on everywhere, but only determining the actual window size on `call()`.

## Review

This PR is related with Swin Transformer and TensorFlow.
R: @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[WIP] Add VQA docs
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR adds visual question answering to the pipeline tutorial (under a more general multimodal header) and the fine-tune section of the guides. It would also be nice to create a VQA Tasks video similar to the other fine-tune guides, but this is not a super high priority right now :)

## TODO

- [ ] Create fine-tune guide for VQA.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17907). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Move cache folder to huggingface/hub for consistency with hf_hub
Author: sgugger
Pull request: yes

# What does this PR do?

This PR relocates the cache to just `~/.cache/huggingface/` when no environment variable has been set. Users having pulled between #18348 and now will need to move their cache manually.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add OWL-ViT model for zero-shot object detection
Author: alaradirik
Pull request: yes

# What does this PR do?
- Adds OwlViT model for open-vocabulary object detection. Model takes in one or multiple text queries per image as input.

Original repo:
https://github.com/google-research/scenic/tree/a41d24676f64a2158bfcd7cb79b0a87673aa875b/scenic/projects/owl_vit

Test notebook:
https://colab.research.google.com/drive/1IMPWZcnlMy-tdnTDrUcOZU3oiGg-hTem?usp=sharing

@sgugger could you review my draft PR, please?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Any plan to extend it for TensorFlow version? 
There seems to be [conversion script](https://github.com/google-research/scenic/tree/a41d24676f64a2158bfcd7cb79b0a87673aa875b/scenic/projects/owl_vit#conversion-to-tensorflow) officially. 

Hi @innat. Yes, @alaradirik is already working on it! The PR is here: https://github.com/huggingface/transformers/pull/18450

You can find out which models are being implemented by searching the open issues and PRs [for example](https://github.com/huggingface/transformers/pulls?q=is%3Apr+is%3Aopen+owlvit)
importing 'LongT5Model' from 'transformers'
Author: jorgeutd
Pull request: no

Hello Team,

I am getting the following error when I am trying to import the new LongT5Model into my notebook:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_55187/343105862.py in <cell line: 3>()
      1 #from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
      2 
----> 3 from transformers import AutoTokenizer, LongT5Model
      4 
      5 tokenizer = AutoTokenizer.from_pretrained("google/longt5-tglobal-base")

ImportError: cannot import name 'LongT5Model' from 'transformers' (/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/__init__.py)

Transformers version:
4.19.4
Python version: 
3.8

Comments:
Hi,

LongT5 is only available in Transformers v4.20.

I guess it got released today. Thank you @NielsRogge you are the GOAT.

@NielsRogge do I still need to use the prefix:

if model_checkpoint in ["t5-small", "t5-base", "t5-larg", "t5-3b", "t5-11b"]:
    prefix = "summarize: "
else:
    prefix = ""

for summarization with the LongT5 model or not?

Thanks,

Jorge

@NielsRogge Also I think the model name in the long-t5 README is wrong (https://huggingface.co/google/long-t5-tglobal-large). It says 
```
tokenizer = AutoTokenizer.from_pretrained("google/longt5-tglobal-large")
model = LongT5Model.from_pretrained("google/longt5-tglobal-large")
```
But I think it should be 
```
tokenizer = AutoTokenizer.from_pretrained("google/long-t5-tglobal-large")
model = LongT5Model.from_pretrained("google/long-t5-tglobal-large")  
```


Feel free to open an issue/PR on the repo on the hub!

The code examples were fixed. Closing this issue!

@jorgeutd Hi!
Have you figured out of the question? In the official document, it said that LongT5 does not use prefix. How do we use it in different down tasks? Thanks.
`transformers-cli login` => `huggingface-cli login`
Author: julien-c
Pull request: yes


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failures are due to `ERROR tests/sagemaker/scripts/pytorch/run_glue_model_parallelism.py` trying to be run in a non-torch setup and without the appropriate decorator setup; Fine to ignore the failure for me.
Need the ability to modify PSM values for Tesseract call in LayoutLM V2/ XLM / V3 Processor
Author: kelvinAI
Pull request: no

### Feature request

There exists a need to modify PSM values while calling Tesseract during feature extraction stage with OCR enabled.
https://github.com/huggingface/transformers/blob/31ee80d55673f32c0f5d50936f371e661b74b21a/src/transformers/models/layoutlmv3/feature_extraction_layoutlmv3.py#L53



### Motivation

Changing Page Segmentation Modes(PSM) values have significant impact on the output of all LayoutLM models, depending on the type/formatting of input document. The default psm value is set to 3, and is not optimal in every situation. It is helpful if users can modify PSM values based on different document types. [PSM Reference](https://stackoverflow.com/questions/44619077/pytesseract-ocr-multiple-config-options)


### Your contribution

I've already created a PR in a branch (https://github.com/huggingface/transformers/pull/17005) for LMV2/XLM and currently using it for my own projects, but it would be better if the official repo have it so there is no need to keep maintaining/updating my own fork. Hoping to see this feature added to LayoutLMV3!

Comments:
cc @NielsRogge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to use run_glue.py with tensorboard?
Author: ayrtondenner
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

I'm running `run_glue.py` script, where I added a new task_name in `data/metrics` and `data/processors`. The training happens OK, the checkpoints are being saved, but no tfevent file is being written. Shouldn't it be written in checkpoints folder during training?

Comments:
What `--logging_dir` have you specified to the `run_glue.py` script?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm having this problem how do you fixed this?
I put the direction to the folder to --logging_dir but nothing is written there

> What `--logging_dir` have you specified to the `run_glue.py` script?

I'm having this problem how do you fixed this?
I put the direction to the folder to --logging_dir but nothing is written there
Force Alignment with Wav2Vec2 models
Author: arijitx
Pull request: no

# 🚀 Feature request

Most TTS acoustic models like FastSpeech, FastPitch requires duration of each phonemes or characters while training. In some cases there are force alignment models and aligners available for some languages, but most languages don't have one. leveraging the wav2vec model for force alignment [Forcelignment with wav2vec2](https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html) . ASR models from [Robust Speech Challenge](https://huggingface.co/models?other=robust-speech-event) can be used for this.

## Motivation

Making it easy to force align for low resource languages 

## Your contribution

I can contribute by writing a method to force align using wav2vec2 models following the above mentioned pytorch tutorial.


Comments:
I fully agree and I think it would be great if we could add a code snippet such as the one you mention to `transformers`. I think the place for now should be here: https://github.com/huggingface/transformers/tree/main/examples/research_projects/wav2vec2 - maybe we could add a `alignment.py` file there? 

More than happy to help if you want to give it a try :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support NLLB's LID model
Author: xia0nan
Pull request: no
Labels: name: New model

### Model description

Thanks for supporting NLLB and closing this issue https://github.com/huggingface/transformers/issues/18043. I'm wondering if huggingface can further support the language identification model of NLLB?
"LID (Language IDentification) model to predict the language of the input text."

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/facebookresearch/fairseq/tree/nllb#lid-model

Comments:
I figured out how to use it. No issue now.

First download model
`wget https://dl.fbaipublicfiles.com/nllb/lid/lid218e.bin`
Then use it for inference
```python
import fasttext
pretrained_lang_model = "lid218e.bin"
model = fasttext.load_model(pretrained_lang_model)
text = "これ、浅草に、行きますか"
predictions = model.predict(text, k=1) 
print(predictions)
```
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)
Author: neeraj1909
Pull request: no

## Environment info

- `transformers` version: 4.12.5
- Platform: Linux-5.10.90+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.1 (True)
- Tensorflow version (GPU?): 2.6.2 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik @vanpelt  @arfon 


## Information
I am using Captum for interpreting the attributions of the tokens in each layer using Layer-Conductance
`lc = LayerConductance(predict, model.bert.encoder.layer[i])`

Now, in the line
`layer_attributions = lc.attribute(inputs=input_ids, baselines=ref_input_ids, additional_forward_args=(attention_mask,))`
RuntimeError is generating.

A helper function to perform forward pass of the model and make predictions.
`def predict(input_ids, attention_mask=None):
    outputs, attention_weights = model(input_ids=input_ids, attention_mask=attention_mask)
    preds = torch.softmax(outputs, dim = 1)[0][1].unsqueeze(0)
    return preds`

Model I am using: "google/muril-base-cased"


#2952 

![2](https://user-images.githubusercontent.com/20766288/150690257-d4f92158-6709-4728-b8bb-e4ac995059b3.png)
![1](https://user-images.githubusercontent.com/20766288/150690261-3f31369f-3e5d-49a6-83dc-e8e9e9f311d8.png)



## Expected behavior
Code is working fine during training and prediction but raising errors while interpreting the layers with captum.

Any help would be greatly appreciated.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@neeraj1909 I am facing the same issue, did you manage to figure this out by any chance?
Fix compatibility with 1.12
Author: sgugger
Pull request: yes

# What does this PR do?

Fixes the scatter tests by installing torch_scatter wheels for PyTorch 1.12.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

We'll need a fix for all Wav2Vec2-like models it seems. Opened an issue here: https://github.com/pytorch/pytorch/issues/80569

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Should this be merged or closed @sgugger?

We're still not supporting PyTorch 1.12, so this shouldn't be closed.

Good news: we can go for `torch 1.12.1`. But FYI:

`https://pytorch-geometric.com/whl/torch-1.12.1+cpu.html` page doesn't exist, so I keep it as `1.12.0`
layoutlmv3 processor 
Author: rihabfounoun
Pull request: no

### System Info

```shell
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
There was a problem when trying to move your cache:

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1551, in <module>
    move_cache()

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1491, in move_cache
    cached_files = get_all_cached_files(cache_dir=cache_dir)

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1397, in get_all_cached_files
    for file in os.listdir(cache_dir):
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
There was a problem when trying to move your cache:

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1551, in <module>
    move_cache()

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1491, in move_cache
    cached_files = get_all_cached_files(cache_dir=cache_dir)

  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/hub.py", line 1397, in get_all_cached_files
    for file in os.listdir(cache_dir):

### Expected behavior

```shell
install the processor
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
cc @sgugger 

Is that the entire stacktrace, @founou-rihab ?
Specify en in doc-builder README example
Author: ankrgyl
Pull request: yes

# What does this PR do?

Corrects a small typo in the docs README

Fixes #18508


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
ERORR: "Missing XLA Configuration" while running the script?
Author: karndeepsingh
Pull request: no

Hi,
I was trying to train the clip model on the images and text. And using clip-Italian repository and they are using HFScripts to train the model and got the error related to torch_xla.
Please help me to remove the following error. I am trying to train it on GPU device, it seems that error is due to torch_xla which is trying to look TPU. Please help me to train it on GPU.


```
comet_ml is installed but `COMET_API_KEY` is not set.
Traceback (most recent call last):
  File "run_hybrid_clip.py", line 832, in <module>
    main()
  File "run_hybrid_clip.py", line 472, in main
    ) = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.7/site-packages/transformers/hf_argparser.py", line 214, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 101, in __init__
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 1066, in __post_init__
    and (self.device.type != "cuda")
  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/import_utils.py", line 829, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 1357, in device
    return self._setup_devices
  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/generic.py", line 49, in __get__
    cached = self.fget(obj)
  File "/opt/conda/lib/python3.7/site-packages/transformers/utils/import_utils.py", line 829, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 1299, in _setup_devices
    device = xm.xla_device()
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py", line 232, in xla_device
    devkind=devkind if devkind is not None else None)
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py", line 137, in get_xla_supported_devices
    xla_devices = _DEVICES.value
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/utils/utils.py", line 32, in value
    self._value = self._gen_fn()
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py", line 19, in <lambda>
    _DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
RuntimeError: tensorflow/compiler/xla/xla_client/computation_client.cc:273 : Missing XLA configuration

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.