noamaanMulla-03/transformers-issues huggingface dataset:
ESM esmfold_v1 infer_pdbs method gives TypeError
Author: maxjeblick
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.4.0-105-generic-x86_64-with-glibc2.31
- Python version: 3.9.12
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import EsmForProteinFolding

model = EsmForProteinFolding.from_pretrained("facebook/esmfold_v1").cuda()
pdbs = model.infer_pdbs(["MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"])
```
gives

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [12], line 1
----> 1 pdbs = model.infer_pdbs(["MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"])

File ~/PycharmProjects/esm/venv/lib/python3.9/site-packages/transformers/models/esm/modeling_esmfold.py:2318, in EsmForProteinFolding.infer_pdbs(self, seqs, *args, **kwargs)
   2316 def infer_pdbs(self, seqs: List[str], *args, **kwargs) -> List[str]:
   2317     """Returns the pdb (file) string from the model given an input sequence."""
-> 2318     output = self.infer(seqs, *args, **kwargs)
   2319     return self.output_to_pdb(output)

File ~/PycharmProjects/esm/venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:27, in _DecoratorContextManager.__call__.<locals>.decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File ~/PycharmProjects/esm/venv/lib/python3.9/site-packages/transformers/models/esm/modeling_esmfold.py:2280, in EsmForProteinFolding.infer(self, seqs, residx, with_mask)
   2278 if residx.ndim == 1:
   2279     residx = residx.unsqueeze(0)
-> 2280 return self.forward(
   2281     aatype,
   2282     mask,
   2283     mask_aa=with_mask is not None,
   2284     masking_pattern=with_mask,
   2285     residx=residx,
   2286 )

TypeError: forward() got an unexpected keyword argument 'mask_aa'
```

### Expected behavior

pdb will be calculated correctly.

Comments:
cc @Rocketknight1 

Hi @maxjeblick, this is caused by those methods being ported directly from `ESMFold` and not being updated to match our implementation. I'm working on a fix now! In the meantime you can use the code from our [example notebook for protein folding](https://github.com/huggingface/notebooks/blob/main/examples/protein_folding.ipynb) to convert model outputs to PDB.

@maxjeblick fixed on main now!
Fix arg names for our models
Author: Rocketknight1
Pull request: yes

Some arg names on the `infer` method for `ESMFold` don't fit our port, and this PR updates/removes them.

In future these methods will probably be moved to the tokenizer/processor but this quick fix will at least get them working for now!

Fixes #20120

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add Jukebox model (replaces #16875)
Author: ArthurZucker
Pull request: yes
Labels: name: New model

name: WIP

This is a draft pull request. 

# What does this PR do?
This PR will progressively add the [Jukebox](https://openai.com/blog/jukebox/) model to the hub.
It is linked to [#16870](https://github.com/huggingface/transformers/issues/16870). 

# Currently planned steps (WIP)

- [x]  Create template files with `transformeres-cli add-new-model-like`
- [x] `src/transformers/tokenization_jukebox.py`
- [x]  `src/transformers/test_tokenization_jukebox.py`
- [x]  `src/transformers/configuration_jukebox.py`
- [x]  `src/transformers/modeling_jukebox.py`
- [x]  `src/transformers/configuration_jukebox.py`
- [x]  `docs/source/model_doc/jukebox.rst`
- [ ] `src/transformers/tokenization_jukebox_fast.py` (will most probably use WordLevel tokenizer). Also requires to implement a converter function `class JukeboxConverter(Converter):`



Comments:
Replaces (#16875) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17826). All of your documentation changes will be reflected on that endpoint.

Okay, the `1b lyrics` and `5b lyrics` match the original code. Just need refactoring to have better variable names and wrap the sampling kwargs for easy use. 

@ArthurZucker let me know when this is ready for the final review

@ArthurZucker do you want to let me know once you want to have a final review? Let's try to not let it hang around for too long

Apart from the `kwargs` I think it is done! @patrickvonplaten  feel free to review

Slow tests are now passing, the only issue left to attend is the memory. The slow tests need a lot of RAM, and running inference with the model should also automatically send the unused `Priors` and `VQVAE` to the `cpu`. 

The documentation and models will be ready soon.

As it was previously requested, you can now instantiate `JukeboxVQVAE` and `JukeboxPrior` individually. This is convenient if people only want to use the VQVAE or generate form juste the top level prior. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17826). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17826). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17826). All of your documentation changes will be reflected on that endpoint.
OpenAI's Jukebox for music generation 
Author: ArthurZucker
Pull request: no
Labels: name: New model

### Model description

**[Jukebox](https://openai.com/blog/jukebox/)** is an autoregressive model for music generation published by OpenAI in 2020. It is based on a [hierarchical VQ-VAE](https://arxiv.org/abs/1906.00446) and Scalable Transformers (based on [Sparse Transformers](https://arxiv.org/abs/1904.10509) ) to create long music samples that can be conditioned on Genres, Artists, Timing and Lyrics. 

3 sampling strategies will be made available : 

- Ancestral sampling : tokens are generated in an autoregressive fashion and are then upsampled
- Windowed sampling :  in order to generate long sequences, samples are repeatedly produced from overlapping windows using the previous codes as context. 
- Primed sampling : a continuation of a previous audio is obtained using the VQ-VAE encoding of the audio as initial tokens for the ancestral sampling

The generated tokens are then passed through the VQ-VAE decoder to obtain the final audio. 

The Lyric conditional informations are obtained using a Lyric Transformer model. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The code is available at https://github.com/openai/jukebox, and weights are available at https://openaipublic.azureedge.net/jukebox/models/. 

Authors : 

- [prafullasd](https://github.com/prafullasd)
- [heewooj](https://github.com/heewooj)
- [jongwook](https://github.com/jongwook)
- [mcleavey](https://github.com/mcleavey)


Comments:
Converting CLIPText Model (transformers.CLIPTextModel) Embeddings Back to Text
Author: mbdzi
Pull request: no

### Feature request

Is there a method for converting CLIPText Model (transformers.CLIPTextModel) model embeddings back to text? I looked in the documentation, but could not find anything that addresses this specific query.

I am also interested in finding out the following:
- Are there tools in the Hugging Face ecosystem for calculating the weighted average of embeddings?
- Is there a way to query the CLIP Model using embeddings and not text or image inputs?


### Motivation

I would like to edit prompts mathematically. And, the easiest way to do this would be to get the vector embeddings and to effect the desired mathematical transformations on them.

### Your contribution

I have no contribution beyond my question.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.
[OWL-ViT] Make model consistent with CLIP
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves OWL-ViT by removing 3 arguments, to make the model more consistent with CLIP. 

All integration tests pass with these fixes.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20144). All of your documentation changes will be reflected on that endpoint.

The arguments `return_base_image_embeds`, `use_hidden_state` and `return_projected` are not useful for users, and the number of people that could have used them until now should be marginal (I don't think anyone is using them at all). I really hope we proceed with this PR, I would not add deprecation here as it just cleans up the code, and there are no use cases with those arguments. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20144). All of your documentation changes will be reflected on that endpoint.
how to fine tune custom dataset using coreference pretrained model 
Author: SavitaKumariPandit
Pull request: no

the pre-trained model available in hugging face hub "nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large" how to fine tuning with own custom dataset.

Comments:
Hi,

For that I'll refer to the training guide of Sentence Transformers: https://www.sbert.net/docs/training/overview.html.

> Hi,
> 
> For that I'll refer to the training guide of Sentence Transformers: https://www.sbert.net/docs/training/overview.html.

Hi @NielsRogge , I have to use my own dataset for co-reference resolution task, so above mention suggestion will work on pretrained model of "nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large". 
after  finetune I got output folder which contains  1_Pooling ,config.json, config_sentence_transformers.json,eval,modules.json,pytorch_model.bin,README.md,sentence_bert_config.json
sentencepiece.bpe.model,special_tokens_map.json, tokenizer.json, tokenizer_config.json and this folder I saved as zip file and load path of fine tune model for  prediction but when use for prediction it is not working
trocr deepspeed
Author: dwyane1023
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.10.14-1.el7.elrepo.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

Hi @stas00,


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am using huggingface integrated deepspeed to train TROCR in jupyterlab.

For both load method are the same issue.
`model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained()`
`model = VisionEncoderDecoderModel.from_pretrained()`


### Expected behavior

However, I met an issue
`AttributeError: 'VisionEncoderDecoderConfig' object has no attribute 'hidden_size'` 

Do we have any configuration to fix this?

Comments:
issue resolved by
`hidden_size = 768`
`setattr(model.config, "hidden_size", hidden_size)`

refer to this #15526
Remove Optional[PILImageResampling] typing
Author: amyeroberts
Pull request: yes

# What does this PR do?

Resolves bug if Pillow < 2.9.1 resulting from type checks. If lower versions of Pillow are used, `PILImageResampling` is an alias for the `PIL.Image` module c.f. [definition here](https://github.com/huggingface/transformers/blob/d3c05666798bd8fcc7a03564436e5100b080a5df/src/transformers/image_utils.py#L48). When defining`var: Optional[obj] = None` - specified `obj` in `Optional` can't be a module - so it fails e.g. [here](https://github.com/huggingface/transformers/blob/d3c05666798bd8fcc7a03564436e5100b080a5df/src/transformers/models/segformer/image_processing_segformer.py#L231). 

The PR also replaces any remaining `PIL.Image.Resampling` with `PILImageResampling` in the codebase. 

Another option is to define a new type. I decided to go for the fastest resolution - LMK if you think this is better. 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20172). All of your documentation changes will be reflected on that endpoint.
Roberta Gradient checkpointing  to only layers, which requires grad
Author: berkekisin
Pull request: no

### Feature request

Current gradient checkpointing in Roberta, directly uses gradient checkpointing for all layers without checking if they require grad. In our use case we only train last 3 layers of Roberta and want to use gradient checkpointing only for the last 3 layers.

### Motivation

Adding this would further decrease the memory usage on GPU.

### Your contribution

https://github.com/huggingface/transformers/blob/6268694e27f1fc0192ba24e4bec181061b4a9bf8/src/transformers/models/roberta/modeling_roberta.py#L497

at this line we can add another condition, then it would look like this:

if self.gradient_checkpointing and self.training and layer_module.requires_grad


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Code refactor.
Author: Kotmin
Pull request: yes

# What does this PR do?

Fixes # (issue)

Replacing elif -> if for readability clear code and pep principles

## Before submitting
- [ x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


 @patrickvonplaten


@sgugger





Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19328). All of your documentation changes will be reflected on that endpoint.

Huh? Changing the `elif` to `if` changes what the code does.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
where can I find document about BertPreTrainedModel?
Author: Lim-Sung-Jun
Pull request: no

hello,

I am trying to find document about BertPreTrainedModel which is used on colbertv1
I only found the source code below site ( 4.11.3 bert_modeling )
https://huggingface.co/transformers/v3.5.1/_modules/transformers/modeling_bert.html

but I didn't find any document for this in the official site for latest version
https://huggingface.co/

So, My question is **where can I find document about BertPreTrainedModel? is this removed?**

Comments:
What are you trying to do with `BertPreTrainedModel`? This is an abstract class that isn't really public facing.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Explain why loading config JSON fails
Author: noprompt
Pull request: yes

This PR improves the exception message thrown when reading configuration fails to include the information provided by the exception itself (line/column numbers, etc.). This can spare the reader of the message some trouble debugging it. See [this thread here](https://github.com/CompVis/stable-diffusion/issues/247) for background.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18964). All of your documentation changes will be reflected on that endpoint.

Could you just run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:
```
pip install -e ".[quality]"
```
And then run them with:
```
make fixup
```

@LysandreJik Yes, will do. Thank you!

@LysandreJik Sorry for the delay! I've applied the code quality changes.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP] Implement LayoutLMv2ForRelationExtraction (continues #15173)
Author: quasimik
Pull request: yes

# What does this PR do?

Continues the good work in #15173 to add LayoutLMv2ForRelationExtraction, as implemented in [Microsoft's UniLM repo](https://github.com/microsoft/unilm/blob/152193af4b295ae39cf0c2a492da3ee5cc5abe29/layoutlmft/layoutlmft/models/layoutlmv2/modeling_layoutlmv2.py#L895-L937)

Tests are not written yet, and I might need help with that.

@NielsRogge

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18943). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi,

Thanks for your work. However, we don't want to add LayoutLMv2ForRelationExtraction with the design that the Microsoft authors created (as the model returns lists, rather than fixed size tensors). The latter is necessary for the model to work on a distributed environment, and for things like ONNX. See comments at #19120

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Can't use decoder_inputs_embeds argument on MBartForConditionalGeneration
Author: hibiki12y
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.23
- Python version: 3.9.12
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

maybe not necessary

### Expected behavior

https://github.com/huggingface/transformers/pull/13800

Looks same problem at MBartForConditionalGeneration.foward

It is written as:

```python
if decoder_input_ids is None:
    decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id)
```

So, looks need edit to:

```python
if decoder_input_ids is None and decoder_inputs_embeds is None:
    decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id)
```

Comments:
Maybe of interest to @ArthurZucker :)

Having a look right now! Thanks for finding this 🤗