noamaanMulla-03/transformers-issues huggingface dataset:
Support Streaming to Other Locations Besides STDOUT
Author: sam-h-bean
Pull request: no

### Feature request

I would like to build a token streaming model that sends the tokens to a web socket or SSE connection. Today I would need to put a redirect from stdout to the other location which is a pain. Instead I would like to receive a raw python generator from the TextStreamer object that I can iterate over in any way I need.

### Motivation

I'd like to emulate something like https://github.com/hyperonym/basaran but in native HF code.

### Your contribution

TBD

Comments:
cc @gante 

@sam-h-bean an iterator class was merged yesterday, but I haven't communicated about it :)

You can check its implementation [here](https://github.com/huggingface/transformers/blob/fc5b7419d4c8121d8f1fa915504bcc353422559e/src/transformers/generation/streamers.py#L125). This would be what you are looking for, correct?

EDIT: communicated [here](https://twitter.com/joao_gante/status/1643330507093196800)

@gante Is there going to be an option for using this with the pipelines API? I would like to incorporate this feature into langchain but that currently only supports the pipeline API.

@sam-h-bean yes, it is in the works! :D 

@gante What about dynamic batching combined with streaming? If I wanted to support dynamic batching for an LLM because I expected a high amount of throughput but I wanted to stream tokens back to each client individually how would I accomplish that?

@sam-h-bean for now only the [text-generation-inference](https://github.com/huggingface/text-generation-inference) supports it. 

I'd like to add it to `transformers` sometime in the future, but it definitely won't happen in the next months.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[wip: testing doc-builder]
Author: xenova
Pull request: yes

# DO NOT MERGE
Testing https://github.com/huggingface/doc-builder/pull/373



Comments:
Unfortunately trying to use a custom commit hash just skips the doc-building (since it can't find it). Will wait for @mishig25 to update his branch :) 

Testing continued at https://github.com/huggingface/transformers/pull/23625
Fix tensor device while attention_mask is not None
Author: zspo
Pull request: yes

# What does this PR do?

The PR while fix the tensor device while attention_mask is not None, the tensor like `torch.tensor(torch.finfo(attn_weights.dtype).min)`.

1. I don't use the `os.environ[“CUDA_VISIBLE_DEVICES”]` because i want to load other models in the same script.
2. Use torch.cuda.set_device(6) while other gpu device has already occupied.
3. The model is loaded in device `cuda:6`, but the new tensor will be load in `cuda:0` while attention_mask is not None.

```
{
    'input_ids': tensor([[    1, 29871, 30919]], device='cuda:6'), 
    'attention_mask': tensor([[1, 1, 1]], device='cuda:6')
}

input_ids device:  cuda:6
model device:  cuda:6

 File "/data1/env/miniconda3/envs/llmdev/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 231, inforward
    attn_weights = torch.max(attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min))
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:6 and cuda:0!
```

```
print('attn_weights device', torch.tensor(torch.finfo(attn_weights.dtype).min).device)
attn_weights device cuda:0
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada and @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix accelerate logger bug
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a slow test that requires accelerate t[hat is currently failing](https://github.com/huggingface/transformers/actions/runs/5035387610/jobs/9030918234) with the following error:

```bash
RuntimeError: You must initialize the accelerate state by calling either `PartialState()` or `Accelerator()` before using the logging utility.
```

I suspect this comes from https://github.com/huggingface/accelerate/pull/1446 
The fix seems to be to first initialize a dummy accelerator. However I couldn't reproduce the issue with a simpler snippet. It seems to appear only when a model is created together with CPU offloading + multi GPU. 

I tried this snippet:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bigscience/bloom-1b7"

device_map = {
    "transformer.word_embeddings": 0,
    "transformer.word_embeddings_layernorm": 0,
    "lm_head": 0,
    "transformer.h.0": "cpu",
    "transformer.h.1": "cpu",
    "transformer.h.2": 0,
    "transformer.h.3": 0,
    "transformer.h.4": 0,
    "transformer.h.5": 0,
    "transformer.h.6": 0,
    "transformer.h.7": 0,
    "transformer.h.8": 0,
    "transformer.h.9": 1,
    "transformer.h.10": 0,
    "transformer.h.11": 1,
    "transformer.h.12": 0,
    "transformer.h.13": 0,
    "transformer.h.14": 1,
    "transformer.h.15": 0,
    "transformer.h.16": 0,
    "transformer.h.17": 1,
    "transformer.h.18": 1,
    "transformer.h.19": 0,
    "transformer.h.20": 1,
    "transformer.h.21": 1,
    "transformer.h.22": 0,
    "transformer.h.23": 0,
    "transformer.ln_f": 1,
}

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map=device_map,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

input_text = "hello"
encoded_input = tokenizer(input_text, return_tensors="pt")

# Check the exactness of the results
output_parallel = model.generate(input_ids=encoded_input["input_ids"].to(0), max_new_tokens=10)
```
But it didn't raised any error, strangely, the error is only raised when the test is run.

I would appreciate any insight @muellerzr @sgugger 🙏 

Comments:
I'll try looking into it more, however also note that for logging you should use the `PartialState` not accelerator :)

_The documentation is not available anymore as the PR was closed or merged._
Prediction code snippet for graphormer on graph classification tasks.
Author: techthiyanes
Pull request: no

### System Info

Kindly share the code snippet of prediction for graphormer model.

### Who can help?

cc @clefourrier

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1) Train the model for ogl data.
2) Using trainer.predict to get the prediction dataset results.
3) Unable to write common prediction code for validation dataset without trainer.predict
4) This requires trainer.train to get completed every time.
5) Kindly share the snippet similar to normal text classification and other tasks.

### Expected behavior

Prediction code snippet for graphormer on graph classification tasks.

Comments:
Hi, could you share the logs that you got at step 3?

> Hi, could you share the logs that you got at step 3?


Now I'm able to write out prediction code independently.

Code Snippet :

#  For validation, Kindly opt only one data alone from the dataset.
val_ds1 =  val_ds.select(list(range(0,1)))
#  Trainer prediction
preds = trainer.predict(val_ds1)
# Output : PredictionOutput(predictions=(array([[ 3.615392, -2.313876]], dtype=float32),

# Independent prediction code

inputs = dict()
dataset_processed['train'][0] = val_ds1[0]
import torch
inputs['input_nodes'] = torch.tensor([dataset_processed['train'][0]['input_nodes']])
inputs['input_edges'] = torch.tensor([dataset_processed['train'][0]['input_edges']])
inputs['attn_bias'] = torch.tensor([dataset_processed['train'][0]['attn_bias']])
inputs['in_degree'] = torch.tensor([dataset_processed['train'][0]['in_degree']])
inputs['out_degree'] = torch.tensor([dataset_processed['train'][0]['out_degree']])
inputs['spatial_pos'] = torch.tensor([dataset_processed['train'][0]['spatial_pos']])
inputs['attn_edge_type'] = torch.tensor([dataset_processed['train'][0]['attn_edge_type']])


from transformers import GraphormerForGraphClassification

model_checkpoint = "/content/graph-classification/checkpoint-1029" # pre-trained model from which to fine-tune

model = GraphormerForGraphClassification.from_pretrained(
    model_checkpoint, 
    num_classes=2,
    ignore_mismatched_sizes = True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
)
with torch.no_grad():
    logits = model(**inputs).logits

predicted_class_id = logits.argmax().item()
predicted_class_id,logits

#  Output : (0, tensor([[ 3.6154, -2.3139]]))

Works as expected. Hence closing this issue. Thanks a lot.
Mobilenet_v1 Dropout probability
Author: andreysher
Pull request: no

Hi @amyeroberts [here](https://github.com/huggingface/transformers/pull/17799) is the pull request with some interesting bug. Mobilenetv2 made from tensorflow checkpoint, but in Tensorflow dropout prob is the fraction of **non zero** values, but in PyTorch this is fraction of **zero** values. It should be **1 - prob** for PyTorch.

Load Mobilenetv1 model from 'google/mobilenet_v1_1.0_224'

Dropout prob should be 1 - tf_prob

Comments:
@andreysher Thanks for raising this issue. The dropout rate `p` or `rate` is defined the same for TensorFlow and PyTorch layers. 

From the [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout): 
> The Dropout layer randomly sets input units to 0 with a frequency of `rate` at each step during training time

From the [PyTorch documentation](https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html): 
> During training, randomly zeroes some of the elements of the input tensor with probability `p` 

Is there something specific about the original MobileNet implementation or checkpoints which means this doesn't apply? 


Thanks for this clarification! But i don't understand why default dropout_prob in the PR is 0.999? If i load model by 
```
from transformers import AutoImageProcessor, MobileNetV1ForImageClassification

model = MobileNetV1ForImageClassification.from_pretrained("google/mobilenet_v1_1.0_224")
```
i get model with dropout p = 0.999. This is unexpected. Could you enplane why such value is presented in pretrained model?

@andreysher Ah, OK, I think I see what the issue is. The reason the dropout value p=0.999 is because it's set in the [model config](https://huggingface.co/google/mobilenet_v1_1.0_224/blob/dd9bc45ff57d9492e00d48547587baf03f0cdade/config.json#L5). This is a mistake as, as you had identified, the probability in the original mobilenet repo is the [keep probability](https://github.com/tensorflow/models/blob/ad32e81e31232675319d5572b78bc196216fd52e/research/slim/nets/mobilenet_v1.py#L306) i.e. `(1 - p)`. I've opened PRs on the hub to update the model configs:

- https://huggingface.co/google/mobilenet_v1_0.75_192/discussions/2
- https://huggingface.co/google/mobilenet_v2_1.0_224/discussions/3
- https://huggingface.co/google/mobilenet_v1_1.0_224/discussions/2
- https://huggingface.co/google/mobilenet_v2_0.35_96/discussions/2
- https://huggingface.co/google/mobilenet_v2_1.4_224/discussions/2
- https://huggingface.co/google/mobilenet_v2_0.75_160/discussions/2

Thanks for flagging and the time take to explain. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
no transformers version 4.29.0.dev0
Author: skye95git
Pull request: no

https://github.com/huggingface/transformers/blob/bfb3925fcbbb4fda83e023448e36e4d6c6f16a4c/examples/pytorch/language-modeling/run_mlm.py#L56

I try to run script `run_mlm.py`, meet an error: ImportError: This example requires a source install from HuggingFace Transformers (see `https://huggingface.co/transformers/installation.html#installing-from-source`), but the version found is 4.28.0.
Check out https://huggingface.co/transformers/examples.html for the examples corresponding to other versions of HuggingFace Transformers. 
There seems no transformers version 4.29.0.dev0.

Comments:
`pip install git+https://github.com/huggingface/transformers`

The above will install the git/head version which is 4.29.0.dev0.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add callbacks method to the trainer which are called when the loop starts and when skipping steps ends
Author: bruno-hays
Pull request: no

### Feature request

Add callbacks method in the training loop when the loop starts and when the steps skipping end.

### Motivation

When using an iterable dataset, all preprocessing is done on the fly when a batch is requested by the dataloader. As a result, when resuming training from a checkpoint, steps skipping can be extremely long because all the preprocessing steps are done for nothing.

If we have callback(s) that signal when the step skipping starts and ends, we could for instance set an environment variable that we can use in all our processing to signal to not do anything with the current batch.

The callback on_step_begin is only called once the first useful batch is loaded, it is then too late to signal to actually perform the processings.

I'm also open to propostions if you know another way of rapidly skipping batches when using an iterable dataset

### Your contribution

I could contribute to a PR if needed

Comments:
cc @sgugger 

It sounds like an interesting addition, but the callbacks don't have access to the training dataset, so I'm not sure how you would use that for your use case.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Debug example code for MegaForCausalLM
Author: Tylersuard
Pull request: yes

set ignore_mismatched_sizes=True in model loading code for MegaForCausalLM so that the example code runs without errors.

# What does this PR do?

Fixes # 22974


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.  https://github.com/huggingface/transformers/issues/22974
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts I ran "make style" in the main Transformers directory and then pushed the changes, and every test now fails, including the ones that passed previously.  What am I doing wrong?

@Tylersuard Hmmmm..... OK, I'm not exactly sure what happened - I suspect there might be a mismatch in installed package versions. Here's how I would try to resolve:

First undo the most recent changes added to the unrelated files: 
* Undo the last two commits: `git revert --hard HEAD~2`
* Push these changes to the PR: `git push -f`

Then get your branch in sync with `main`: 
* Get the latest version of main: `git checkout main && git fetch upstream main && git rebase upstream/main`
* Install latest formatting settings `pip install -e ".[quality]"`
* Rebase main onto this branch `git checkout patch-1 && git rebase upstream/main`
* Push these changes to the PR (you'll have to force): `git push --force`
* Make any style changes `make style` 
* Commit changes made (should just be to the modeling_mega.py file): `git add src/transformers/models/mega/modeling_mega.py && git commit -m "Fix up" && git push`

@amyeroberts Very clear instructions, thank you!

@amyeroberts Ok, all done!  I do not have write access, so I can't merge the PR
Muellerzr fix deepspeed
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR fixes the slow tests by avoiding a recursion issue with `self.n_gpus`. It has `self.n_gpu` first check to see if we've assigned/spawned `self._n_gpu`, and if not then call setup_devices. This let's us maintain the clean `ParallelMode.DISTRIBUTED` check, without needing the complex check block in `setup_devices`.

I've confirmed DeepSpeed tests pass.

If we'd rather not do this, then the logic at https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L1732-L1735 would need to be repeated (which can be done, just a bit messy)

Fixes # (issue)

Failing deepspeed tests on nightly


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23657). All of your documentation changes will be reflected on that endpoint.
Add PerSAM
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the PerSAM model.

Question: when you do:

```
from transformers import PerSamModel

model = PerSamModel.from_pretrained("facebook/sam-vit-huge")
```

you get this warning:

```
You are using a model of type sam to instantiate a model of type persam. This is not supported for all configurations of models and can yield errors.
```

was wondering whether we could suppress this warning. PerSAM uses the exact same weights as the original SAM model, just modifies the forward pass with 2 additional arguments. Currently the model_type is set to "persam" in `PerSamConfig`.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ok, will close this PR in favor of modifying `modeling_sam.py`.
WhisperForAudioClassification RuntimeError tensor size doesn't match
Author: xiao1ongbao
Pull request: no

### System Info


- `transformers` version: 4.28.0
- Platform: Linux-5.4.190-107.353.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Multi-GPU on the same machine

### Who can help?

@sanchit-gandhi finetuning WhisperForAudioClassification model and getting this error. The dataset and script works fine when switching to the Wav2Vec2 model.

  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jovyan/.local/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1739, in forward
    encoder_outputs = self.encoder(
  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jovyan/.local/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 823, in forward
    hidden_states = inputs_embeds + embed_pos
RuntimeError: The size of tensor a (750) must match the size of tensor b (1500) at non-singleton dimension 1

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Followed this tutorial, and switched Wav2vec2 to WhisperForAudioClassification model.
Using a dataset with 100K+ wav files.
Training fails with the above error.

### Expected behavior

It would train with no errors.

Comments:
Hey @xiao1ongbao! I'm hoping that your fine-tuning run was successful! Let me know if you encounter any issues - more than happy to help here 🤗

Hey! How did you solve that?

Do you have a reproducible code snippet to trigger the error @lnfin? I didn't encounter this in my experiments!