noamaanMulla-03/transformers-issues huggingface dataset:
cannot import name 'OwlViTImageProcessor' from 'transformers'
Author: mazhai
Pull request: no

### System Info

transformer version:4.29.2
platform: macOS Ventura
cpu: apple M2 Max
python versin: 3.10

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import requests
from PIL import Image
import torch

from transformers import OwlViTProcessor, OwlViTForObjectDetection, OwlViTImageProcessor

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = [["a photo of a cat", "a photo of a dog"]]
inputs = processor(text=texts, images=image, return_tensors="pt")
outputs = model(**inputs)
```

### Expected behavior


exception: 
```
ImportError: cannot import name 'OwlViTImageProcessor' from 'transformers' (/Users/tt/opt/anaconda3/envs/test_transfromer/lib/python3.10/site-packages/transformers/__init__.py)
```

Comments:
It is definitely in the lbirary so I would suggest re-installing `transformers` and double-checking you are running your code in the right environment.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

When I install pytorch or tensorflow using micromamba, each insists on downgrading transformers back to 4.17.  However, this version throws the error: ImportError: cannot import name 'OwlViTProcessor' from 'transformers'. However, you need pytorch or tensorflow installed to run the code.

Hi @kirstin-sandbox, if the compatibility issue is on an old version of transformers, there isn't much we can do unfortunately. We can only iterate and build on the latest. If you need a newer version, I'd open an issue in the micromamba repo. 

> However, you need pytorch or tensorflow installed to run the code.

Which code? 

Installing pytorch or tensorflow should be compatible with running transformers. It also isn't necessary for importing the owl vit processor.
Update `dtype_byte_size` to handle torch.float8_e4m3fn/float8_e5m2 types
Author: mgoin
Pull request: yes

# What does this PR do?

Currently using the new `torch.float8_e4m3fn` dtype will cause an error because `dtype_byte_size()` doesn't know where to find the number of bits in the dtype string.

Code to reproduce:
```python
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("echarlaix/tiny-random-mistral")
model.lm_head.weight = torch.nn.Parameter(model.lm_head.weight.to(torch.float8_e4m3fn))
model.save_pretrained("test")
```

Error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/michael/venvs/test/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2557, in save_pretrained
    shards, index = shard_checkpoint(state_dict, max_shard_size=max_shard_size, weights_name=weights_name)
  File "/home/michael/venvs/test/lib/python3.10/site-packages/transformers/modeling_utils.py", line 381, in shard_checkpoint
    weight_size = weight.numel() * dtype_byte_size(weight.dtype)
  File "/home/michael/venvs/test/lib/python3.10/site-packages/transformers/modeling_utils.py", line 328, in dtype_byte_size
    raise ValueError(f"`dtype` is not a valid dtype: {dtype}.")
ValueError: `dtype` is not a valid dtype: torch.float8_e4m3fn.
```

We can fix this by changing the regex from `[^\d](\d+)$` to `[^\d](\d+)_?` so we match on every set of numbers following an alphabetical character. This can match multiple groups, but we can always look at the first group to get the first number which is significant. See the picture below for a demonstration of the matches on all numbered pytorch dtypes:
<img width="802" alt="Screenshot 2024-04-25 at 12 27 07 PM" src="https://github.com/huggingface/transformers/assets/3195154/867548ce-dcbc-4233-a19a-6c59ee986d3f">

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Tagging @sgugger since they touched this function last.


Comments:
Thanks! This is great for fp8

Sure @amyeroberts, I added a test specifically for `dtype_byte_size` over most of the torch dtypes. Let me know if you have something else in mind, thanks!

For the quality checks, running `make fixup` should resolve
Remove skipping logic now that set_epoch exists
Author: muellerzr
Pull request: yes

# What does this PR do?


I'm fairly confident that this logic provided is no longer needed now that Accelerate `DataLoaders` have a `set_epoch` capability and as a result we don't need to manually skip the batches like done here for random states. 

Should fix https://github.com/huggingface/accelerate/issues/2709 and fix https://github.com/huggingface/transformers/issues/30500

I've verified the slow `tests/trainer` **all pass** with this change on single *and* multi GPU, so it seems safe. 

(We have tests for resuming training at certain points and asserting that results are identical)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30501). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'm not super familiar with the dataloader class. Am I correct in understanding the `set_epoch` will effectively do the same as the logic here: restore the state to the last trained epoch?

Correct, it'll restore the random state to where it is at a certain point.

So doing `dataloader.set_epoch(2)` will give you the same iteration data as the third epoch in the series.
listing train_dataloader sampler throws out of memory error
Author: gioaca00
Pull request: no
Labels: name: trainer

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.29.3
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@muellerz @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Problem arises when I run ```trainer.train(resume_from_checkpoint=True)``` from the last checkpoint of a run that ran for more than 24 hours. The bug is in this line: [https://github.com/huggingface/transformers/blob/20081c743ee2ce31d178f2182c7466c3313adcd2/src/transformers/trainer.py#L2156](url). When listing the sampler, the gpu goes out of memory. Whatever that line is trying to accomplish, it shouldn't give an error. Perhaps it's an OOM error. If I comment out that line and try to load the same checkpoint as before, the training is restarted correctly.

### Expected behavior

When I run ```trainer.train(resume_from_checkpoint=True)``` from whichever previous checkpoint, it should continue to train and not throwing an error. Line 2156 of trainer.py should be modified so that it doesn't list the sampler.

Comments:
Oh that's a big ol' bug we can fix now with accelerate I think. Thanks!
SegGptImageProcessor broken for 2D prompt masks that are not PIL images
Author: EduardoPach
Pull request: no
Labels: name: Vision

### System Info

- `transformers` version: 4.39.0.dev0
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.7
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The bug happens at this [line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/seggpt/image_processing_seggpt.py#L351) because we expect 3-dimensional input, however, prompt masks can be 2D (semantic map) or 3D (in case a palette was already applied). This, however, is just a problem when passing `prompt_masks` as something different than a `PIL.Image`

```python
from transformers import AutoImageProcessor
import numpy as np
model_id = "BAAI/seggpt-vit-large"
image_processor = AutoImageProcessor.from_pretrained(model_id)
img_2d = np.random.rand(18,18)
_ = image_processor(images=None, prompt_masks=img_2d, return_tensors="pt")
```

### Expected behavior

It should work for every valid image input type rather than just for `PIL.Image` 

Comments:
Fix GroundingDINO, DPR after BERT SDPA update
Author: amyeroberts
Pull request: yes

# What does this PR do?

GroundingDINO and DPR models broke after the merging in of #28802 as they use BERT as a backbone / encoder. SDPA is now selected as the attention implementation for BERT by default, but wasn't supported by these models. 

For GroundingDINO, we make sure to pass the current model's config implementation when making the model. 
For DPR, as the only attention is from BERT, we say it now supports SDPA. 

Fixes failing tests on main e.g. this run: https://app.circleci.com/pipelines/github/huggingface/transformers/91014/workflows/dfdf4b73-b123-4c08-bbc4-bb6214737293/jobs/1188921



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30506). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PatchTST and PatchTSMixer categorical features and exogenous variables
Author: chrisconstant
Pull request: no
Labels: name: Feature request

name: Time Series

### Feature request

Include categorical features and exogenous variables as input for the PatchTST and PatchTSMixer timeseries foundation models

### Motivation

Categorical features and exogenous variables are key components in timeseries modelling

### Your contribution

-

Comments:
@kashif 

@chrisconst2 so due to the nature of patching the only potential covariates that can be included with PatchTST and I believe PatchTSMixer are going to be static features and static real-valued covariates... I believe the method is not too flexible in being able to accommodate these extra bells and whistle... what kind of exogenous variables were you thinking of using?

@kashif  There are many exogenous variables can improve the perfomace in stock / sales forecasting.
For stock forecasting, the typical exogenous variables  are fundamental indicator : year-on-year (YoY) growth, profit margin, earnings per share (EPS), price-to-earnings (PE) ratio, and profit after tax (PAT), etc ...
For sales forecasting, could be : discount rate,  with gift,  gift value , balabala ...

@eromoe of course, however, this method doesn't have the inductive bias to incorporate covariates (especially temporal ones) in a straight-forward fashion. I have a variant of patch-tst called lag-tst that doesn't have this issue... https://github.com/awslabs/gluonts/tree/dev/src/gluonts/torch/model/lag_tst
BUG: Unexpected behaviour by the server.
Author: sanjay-nit
Pull request: no

### System Info

Hi, I'm running in issue while installing custom packages in HuggingFace Inference Endpoints(Custom handler)

I deployed on T4 machines and it was working well but when I restarted few times it throws error.
Yes, I can see the warning but is there any better way of installing custom dependencies on machine. eg: tesseract-engine etc.

Below code was working fine since last few weeks but now the same code is throwing error.
`subprocess.run("apt install -y pciutils", shell=True, check=True)`


Error: 
![image](https://github.com/huggingface/transformers/assets/103114249/2155a9d0-38b5-4965-bf32-4629b3c1e6bc)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`subprocess.run("apt install -y pciutils", shell=True, check=True)`

### Expected behavior

It is should install the custom packages.

Comments:
Hi @sanjay-nit, thanks for opening an issue! 

The reproduction line if for installing the `pciutils` library, and not related to transformers. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
OSError: Can't load config for 't5-large'. Make sure that: //// Which version will support this?
Author: daehuikim
Pull request: no

### System Info

- `transformers` version: 3.2.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.3.1 (False)

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am just following instructions given in the README of control prefix repo (https://github.com/jordiclive/ControlPrefixes?tab=readme-ov-file)

Wheneer I try "Usage" part, I encounter same error like below.

1. Original setting
```
OSError: Can't load config for 't5-large'. Make sure that:

- 't5-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-large' is the correct path to a directory containing a config.json file
```

2. Changing model name as in hf hub.
```
OSError: Can't load config for 'google-t5/t5-large'. Make sure that:

- 'google-t5/t5-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'google-t5/t5-large' is the correct path to a directory containing a config.json file
```




### Expected behavior

AutoConfig.pretrained does not have argument like "local_files only".
So, If there is a version conflict, I want to know which version will support to reproduce result of control prefix repo

Thanks for reading my issue!

Comments:
Hi @daehuikim, thanks for raising an issue! 

v3.20 is a very old version of transformers ~3 years. I'd suggest installing the most recent version `pip install -U transformers`

The version compatibility is something that the https://github.com/jordiclive/ControlPrefixes repo should provide. I'd suggest opening an issue there to get the necessary transformers version if the latest doesn't work

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
cannot load model back due to  [does not appear to have a file named config.json]
Author: yanan1116
Pull request: no

### System Info

vllm version: 0.4.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I fine-tuned the mistral-7b-v0.2 model using the trainer of huggingface https://huggingface.co/docs/trl/v0.8.6/trainer
the training worked well and finally it saved the model, as bellow:
`adapter_config.json  adapter_model.safetensors  checkpoint-16  checkpoint-24  checkpoint-8  README.md  special_tokens_map.json  tokenizer_config.json  tokenizer.json  tokenizer.model  training_args.bin`

However, when I try to load it back via vllm, it caused error: 

> does not appear to have a file named config.json

```
from langchain_community.llms import VLLM
llm = VLLM(model="path_to_the_model", 
       trust_remote_code=True,  # mandatory for hf models
       max_new_tokens=64,
       temperature=0,
       # tensor_parallel_size=... # for distributed inference
)
llm.invoke("what is the captital city of ontario ?")
```
however, when I load it via 
`AutoModelForCausalLM.from_pretrained`, everthing is fine.

any advice ?


### Expected behavior

It should load it back .

Comments:
sft via lora. 

should I download https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/blob/main/config.json into the model saved path ?

i also test the code in https://docs.vllm.ai/en/latest/models/lora.html

```
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model=LOCAL_PATH, enable_lora=True)
```
where LOCAL_PATH stores:

> adapter_config.json  
adapter_model.safetensors  
README.md  
special_tokens_map.json  
tokenizer_config.json  
tokenizer.json 
tokenizer.model 
 trainer_state.json  
training_args.bin

also,  `model =   AutoModelForCausalLM.from_pretrained(LOCAL_PATH)` works well.



So did this snippet work? -->
```python
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model=LOCAL_PATH, enable_lora=True)
```

Maybe pass the `enable_lora=True` kwarg to the langchain alternative. Another alternative is to merge the lora weights and get a "base" model back, copy the base config, and reload in langchain. But tbh, this is more of a langchain/vllm issue rather than the transformers library itself.

> So did this snippet work? -->
> 
> ```python
> from vllm import LLM, SamplingParams
> from vllm.lora.request import LoRARequest
> 
> llm = LLM(model=LOCAL_PATH, enable_lora=True)
> ```
> 
> Maybe pass the `enable_lora=True` kwarg to the langchain alternative. Another alternative is to merge the lora weights and get a "base" model back, copy the base config, and reload in langchain. But tbh, this is more of a langchain/vllm issue rather than the transformers library itself.

this does not work neither. yes, seems that it is a vllm issue.
Bugfix: Add the missing "no label" class in ConditionalDetr Object Detection Head
Author: rootAvish
Pull request: yes

# What does this PR do?

Adds a "no label" class to the object detection head in ConditionalDetr, per the docstring of this class we pass in num_classes equal to our original number of classes, however doing that makes the formulation wrong and it uses up one of the classes of the original dataset (the one with the max label id value) as the negative label, since the +1 is not added here.

This line of code looks like it's copied verbatim from modeling_detr with the comment included but it doesn't actually have the + 1 for whatever reason.

Comments:
My bad, I didn't realise the loss variant used here is a sigmoid. I was having problems with a model I had trained where the class with the max id was the only class not getting leaned despite being the majority class of my classifier, saw the comment in the source code and thought the issue is that of the missing class label.