noamaanMulla-03/transformers-issues huggingface dataset:
Running squad with GPT-J-6B fails due to issue in tokenizer.
Author: jojivk73
Pull request: no

### System Info

Hi,
   I am tryiing to use the run_qa.py script under examples/tensorflow/question-answering 
   Model  EleutherAI/gpt-j-6B. 
   dataset squad.
   it fails as below. 
   
![image](https://user-images.githubusercontent.com/14943401/221944528-15e0f573-1950-4d9b-9cd8-b8ec71c91e97.png)


### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install transformer from source 
2. python examples/tensorflow/question-answerin/run_qa.py with --model_name  EleutherAI/gpt-j-6B.
   ---   \
   --dataset_name squad  --do_train --do_eval 

### Expected behavior

The script to run the finetuning task

Comments:
This example does not support GPT-J out of the box since GPT-J has no CLS token (compared to BERT or XLNet). You will need to adapt the preprocessing as a result.

@sgugger Can you please elaborate.
I tired using distillbert tokenizer and it is running. 
I am not sure if that is good.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Token classification for a non-textual data
Author: vitalyshalumov
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.13.0-37-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.10.2+cu113 (True)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm looking for an implementation of an architecture that performs token classification, but the input is not an integer that represents the vocabulary but a vector of numbers.

Basically, each token in the input is represented by a vector. Each token is already an embeddings vector.

How can this be achieved?

Best,
Vitaly



### Expected behavior

Input vector of size 768 for each token. A sequence of such tokens of up to 512.

Maybe it is as simple as removing the layer
 (word_embeddings): Embedding(50265, 768, padding_idx=1)?
In any case a link to the solution would be most helpful.

Best,
Vitaly


Comments:
You should probably ask this open question on the [forums](https://discuss.huggingface.co/).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP]🌐 [i18n-KO] Translated `tasks/translation.mdx` to Korean
Author: wonhyeongseo
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니당 -->
# What does this PR do?

Partially translated the `tasks/translation.mdx` file of the documentation to Korean. I will finish off the rest by Thursday if possible. PseudoLab team members will review the quality of the translation by then.
Thank you in advance for your review. ❤️ 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

<!-- 제출 전 체크리스트로, 가짜연구소만의 체크리스트도 <details>로 감싸서 만들어두면 더 좋을 것 같아요. -->

## Who can review?
<!-- 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- @sgugger, @ArthurZucker, @eunseojo May you please review this PR? -->

Team PseudoLab, may you please review this PR?

@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Due to WSL compatibility issues causing unreachable commits, I have opened another PR on a different branch.

Closing in favor of https://github.com/huggingface/transformers/pull/22678
how to beam search
Author: Mryangkaitong
Pull request: no

The current interface is similar to 
`output=model.generate(**inputs, num_beams=4, no_repeat_ngram_size=7)`
but decoding one by one is too slow. Does beam seach support batch-by-batch decoding? 
How to write if support。

Comments:
Please use the [forums](https://discuss.huggingface.co/) to ask such questions.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Make TFForceTokensLogitsProcessor exportable as tf concrete function.
Author: mfuntowicz
Pull request: yes

Previously we were doing an inner conversion from `[[int]]` to `dict[int, int]` but `dict` is not something concrete functions like.

This PR provides a fully compatible concrete function export, removing the inner transformation to `dict` tu use `tf.Tensor` only.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21900). All of your documentation changes will be reflected on that endpoint.

Also, check CI haha

Will handle these 👍🏻 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Enable to pass two or model pretrained models to the Trainer.
Author: shamanez
Pull request: no

### Feature request

I tried a compound architecture that uses two tower models that get initialized with from_pretrained methods. 


Then I wrapped the Custom model class with the Pretrained model class. 

It works well in a single GPU.  But fails in multiple GPU settings. 


### Motivation

It would be useful to change models like DPR.  

### Your contribution

I have tested a few methods and can help to improve. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add finetuning task support for GPT-J to 4.27.0.dev
Author: jojivk73
Pull request: no

### Feature request

I have tried running fine-tuning tasks including QA, Summarization and text classification with GPT-J. As of now only QA could be made working with a minor hack to use distillbert tokenizer. Not sure if this is best. Other tasks do not work for GPT-J. Any help appreciated.
Thanks

### Motivation

Fine-tuning for LLMs (GPT-J)

### Your contribution

I need to understand the HF transformer code to make any contribution. Will submit PRs if I see possible ways to improve.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix typo
Author: Ronalmoo
Pull request: yes

# What does this PR do?

Typo on trainer.py 
This should be modified from "forword" to "forward"

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the fix!
Ernie-M for pretraining multilingual models
Author: KnutJaegersberg
Pull request: no
Labels: name: New model

name: Feature request

### Feature request

Two things that might help in that regard: 

- To train TSDAE, one needs support as class ErnieMForPreTraining, just as for Ernie
https://huggingface.co/docs/transformers/model_doc/ernie#transformers.ErnieForPreTraining

- To train cross-encoders with contrastive loss, a bit like SimCSE, one needs standard support for getting the 'attention_mask' out of the tokenizer sbert uses. Sbert just expects those. Tried to hack it in into sbert, but failed. 

### Motivation

Suspect getting Ernie-M-large for pretraining multilingual sentence embeddings will yield close to sota results.  
According to mSimCSE, we can get top multilingual embeddings just on training on their 300k dataset of english pairs, alone (worked better than cross-lingual training). 
With a stronger base model (they used xlm-roberta), sota embeddings might just lie on the streets.  
https://github.com/yaushian/mSimCSE

### Your contribution

Can't do it alone, plz help. 

Comments:
Hi @KnutJaegersberg, thanks for making this suggestion! 

Would you like to try and open a PR to add the model? We have guidance written on adding [models here](https://huggingface.co/docs/transformers/v4.27.2/en/add_new_model). As the modeling file already exists - adding this component is even easier than a whole new model. For example, see [this PR](https://github.com/huggingface/transformers/pull/21754) for adding `WhisperForAudioClassification`.

Currently in the middle of something, will try to look at it later! 

I'd love to try this out! 

Seems like this will require more than simply copy-pasting the BertForPretraining code, but actually implementing cross-attention Masked Language Modeling and Back-translation Masked Language Modeling.
tfhub.de -> tfhub.dev
Author: abhishekkrthakur
Pull request: yes


Comments:
strictly speaking, tensorboard.dev is not a part of tfhub.dev, is it?

(sorry for piggybacking on this tiny PR 😂)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Support ViT model in EncoderDecoder
Author: abhishekkrthakur
Pull request: yes
Labels: name: WIP


Comments:
The idea is to use ViT as the encoder and then a LM as the decoder for Image captioning generation, *e.g.*? @patil-suraj and I were also thinking of using `EncoderDecoder` for Speech2Text. 

At the moment, I see two solutions: -> adapt `EncoderDecoder` to be usable for all modalities not just text2text **or** create new classes, *e.g.* a `SpeechEncoderDecoder` and a `VisionEncoderDecoder` since I'm not sure `EncoderDecoder` will be able to handle all the new use-cases. *E.g.* we might end up ending way too many if-else statements that would make the code unreadable... @patil-suraj @abhi1thakur what do you think? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Unstale

Huhu,
any update at this ? :) 
a VisionEncoderDecoderModel would be a great also for models which follow in future
for example this one: [TrOCR](https://arxiv.org/abs/2109.10282)

We'll add a new `VisionEncoderDecoder` class for this actually

@patrickvonplaten nice !
Is there any way or need to contribute ? :)

Closing this as [VisionEncoderDecoder](https://huggingface.co/docs/transformers/main/en/model_doc/vision-encoder-decoder#vision-encoder-decoder-models) now exists.
Add SwiftFormer
Author: shehanmunasinghe
Pull request: no
Labels: name: New model

### Model description

'SwiftFormer' paper introduces a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations in the self-attention computation with linear element-wise multiplications.  A series of models called 'SwiftFormer' is built based on this, which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Even their small variant achieves 78.5% top-1 ImageNet1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2× faster compared to MobileViT-v2.

I would like to add this model to Huggingface.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2303.15446
Original code and weights: https://github.com/Amshaker/SwiftFormer 
Author: @Amshaker



Comments:
Make dynamic code work with offline mode
Author: sgugger
Pull request: yes

# What does this PR do?

Using dynamic code on the Hub won't work in offline mode if the model is cached. This is because of an old way of getting the commit hash I put there before we had the commit hash returned in the e-tag. Now it's very easy to get it, so this PR changes the line of code and adds a test to make sure we don't regress.

cc @VictorSanh and @leot13 since you reported the bug.

Comments:
Thank you!!!

_The documentation is not available anymore as the PR was closed or merged._
Model parallelism: Moving labels to the same device as logits for BridgeTower models
Author: shahad-mahmud
Pull request: yes

As suggested in the https://github.com/huggingface/transformers/issues/22561, moving the labels to the same device as the logits are for the BridgeTower models

@sgugger Can you please review?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[WIP] [Flax] Improving Docs 
Author: Shubhamai
Pull request: yes

# What does this PR do?

Adding relevant Jax/Flax code in `<frameworkcontent>` in Transformers Docs. 


Issues:
- Since Flax has no `Trainer` class, no code exist for `Train` section in the tasks guide ([example](https://huggingface.co/docs/transformers/tasks/token_classification#train)).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

*Not mentioning due to currently in WIP*

- Documentation: sgugger, stevhliu and MKhalusova
- Flax: sanchit-gandhi

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21924). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add GPTNeoXForSequenceClassification
Author: Asugawara
Pull request: yes

# What does this PR do?

This PR adds the GPTNeoX Seq Classification.
Would you be able to check them?

In advance, thank you for this cool OSS!

ref: https://github.com/huggingface/transformers/pull/11906

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@sgugger
@ArthurZucker
@younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Model parallelism: Moving labels to same devices as the logits are
Author: shahad-mahmud
Pull request: yes

As suggested in the https://github.com/huggingface/transformers/issues/22561, moving the labels to the same device as the logits are for the Data2Vec Text, ESM, Longformer and LongT5 models.

@sgugger Can you please review?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks a lot for your contribution!

It's a pleasure.  I would love to contribute more and expecting some guidance!
Add LayoutLMv2ForRelationExtraction
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the relation extraction head of LayoutLMv2, which was a highly requested feature as seen in #14330 #15451 #18091 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19120). All of your documentation changes will be reflected on that endpoint.

@sgugger I'm getting the following error from `make fixup`:

```
Checking all objects are properly documented.
Traceback (most recent call last):
  File "/home/niels/python_projects/transformers/utils/check_repo.py", line 788, in <module>
    check_repo_quality()
  File "/home/niels/python_projects/transformers/utils/check_repo.py", line 782, in check_repo_quality
    check_all_objects_are_documented()
  File "/home/niels/python_projects/transformers/utils/check_repo.py", line 693, in check_all_objects_are_documented
    raise Exception(
Exception: The following objects are in the public init so should be documented:
 - LayoutLMv2ForRelationExtraction
```

However, this model is added to layoutlmv2.mdx, so not sure why this error occurs.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @NielsRogge , any chance that this will ever be implemented?
Looking around the history of the PR and Iissues, it seems that there was a fair bit of interest

Hi @lamaeldo,

The reason the PR wasn't merged is because models need to output fixed size tensors, to make sure things like distributed training and ONNX export work. However LayoutLMv2ForRelationExtraction outputs lists of tensors in its current implementation, due to each example in the batch having a different amount of relations. So we would need to pad them up to a fixed size such that the model outputs fixed size tensors.

Haven't looked into that yet but if you're willing to contribute, let me know!

Btw I do have a notebook on fine-tuning this model [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutXLM).