noamaanMulla-03/transformers-issues huggingface dataset:
Perturb Hidden-State in Encoder-Decoder Models
Author: vin-nag
Pull request: no

Hi All,

I'm fairly new at using huggingface - so I apologize if this is answered in the documentation. I've looked around and don't think this has been asked before. I'm trying to perturb and use the hidden-state of an encoder-decoder model on the summarization task. 

More specifically, I'd like to 

1. Get a fixed-length hidden state for a given input after passing it to an encoder model, 
2. perturb it, and 
3. pass it to the decoder model to get an output. 

1 and 2 seem straightforward. To get the hidden state I use:
` hidden_states = model.base_model.encoder(inputs).last_hidden_state`
and to get a fixed length embedding I'm getting the last element of this list as per this [discussion](https://github.com/huggingface/transformers/issues/1950). Perturbing this would be as simple as adding noise to a tensor.

As for the third part, I'm having difficulty using this perturbed hidden-state in the decoder to generate an output. I've looked through the code and it seems a lot of the steps are abstracted out to accommodate many kinds of models. 

From what I understand, we can modify the model.generate() method so that the perturbation is added. However, this doesn't necessarily work for me since I wanted to use this hidden-state for other purposes before passing it to the decoder.

The other approach would be to create a separate function that takes as input the hidden-state and uses the second part of the code from model.generate() to produce outputs.

Before I proceed to implement this, I was wondering if there is a simpler way using existing code to do this. 






Comments:
Hi @vin-nag 

If you want to directly pass `hidden_states` then you could do it this way

```python
model = EncoderDecoderModel.from_pretrained("google/roberta2roberta_L-24_gigaword")
tok = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_gigaword")

article = """australian shares closed down #.# percent monday
following a weak lead from the united states and
lower commodity prices , dealers said ."""

enc = tok(article, return_tensors="pt")
hidden_states = model.encoder(**enc, return_dict=True)

# perturb the last_hidden_state
hidden_states.last_hidden_state = perturb(hidden_states.last_hidden_state)

gen_ids = model.generate(input_ids=None, encoder_outputs=hidden_states, attention_mask=enc["attention_mask"])
tok.batch_decode(gen_ids)
```

@patil-suraj Thank you so much!

Hi @patil-suraj, thanks for the script. How can I access the `loss` after perturbation?
How to use run_glue.py with tensorboard?
Author: Daromog
Pull request: no
Labels: name: bug

### System Info

I'm writting a folder path in --logging_dir but nothing is written there .
I've tried with 

--logging_dir foldername
--logging_dir pathtofolder

But nothing works

@LysandreJik 
@sgugger

### Who can help?

@sgugger
@LysandreJik 


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

    --logging_dir foldername
    --logging_dir pathtofolder

    --logging_strategy steps \
    --logging_first_step True \
    --logging_steps 5 \

### Expected behavior

Nothing is saved in the folder

Comments:
Try using `--report_to tensorboard`
Add checks for some workflow jobs
Author: ydshieh
Pull request: yes

# What does this PR do?

We have encountered errors
```bash
stderr: nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
```
(due to auto update) several times. In such cases, the reports failed to send to slack channels. We were not aware of this issue on push CI for a few days.

This PR checks the setup job and also adds a check on the CI runners. If such jobs fail, it could still send the report containing some information.

**We should also disable the auto update (for some packages)**

**I will add the same check to scheduled CI and past CI (if the changes are approved)**

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[bloom] Add alibi cache for fixed inputs
Author: stas00
Pull request: no

### Feature request

Add alibi cache for BLOOM

### Motivation

Many training scenarios involve fixed input length. In which case re-creating the deterministic alibi tensor that is the same on each forward is a waste. The other similar scenario is when one groups inputs into bins with padding to the fixed length of the bin.

I propose we add a small cache - even of 1 value to speed up these scenarios.

So a 1-value cache would be that we save the tensor in the `BloomModel` object and rebuild it only if tensor's length needs to change.

@younesbelkada 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BLOOM - modifying slow tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

- changed the non passing tests to fp32
- reduced sequence length
- remove padding test

All these matters have been discussed on Slack but mainly:

1- Generations tests were not passing because the linear layers does not give the same results between torch 1.11 and torch 1.12
2- batched generation can be flaky sometimes in half precision mode, this should be expected. Therefore we reduce the sequence length of the generated output
3- One should **always** use `padding_side=left` when doing batched generations

cc @ydshieh @patrickvonplaten


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17963). All of your documentation changes will be reflected on that endpoint.

Hi, @younesbelkada 

Could you explain a bit more on ` One should always use padding_side=left when doing batched generations`?
And what are examples of test failures when using `padding=right` ? I couldn't find you mentioning this on Slack.

Thanks!


Hi @ydshieh !
From the internal discussions here is a summary of why one should always use `padding_side=left` (cc @patrickvonplaten ):
- Imagine: `["hello my name is", "hey <pad> <pad> <pad>"]`
For the first input the correct token will be sampled from "is" - however for the second input, generate would incorrectly sample from `"<pad>"` where as it should sample from `"hey"`. Making sure everything is batched on the left circumvents this problem !

@younesbelkada - IMO we should not expect the generation to be flaky ever, why is this the case here?

Hi @younesbelkada : I have 3 questions 🙏 

- with fp16:
  - do we get stable results in a specific torch version (i.e. the same result across many runs)
- after changing to fp32 (without reducing the seq length)
  - do we get the same results across torch 1.11 and 1.12?
  - do we get stable results in a specific torch version (i.e. the same result across many runs)

Hi @ydshieh !
After merging this PR: https://github.com/huggingface/transformers/pull/17866 the slow tests are now passing. Our conclusion is that:
1- In half precision mode we might not get the same results across batched generation and it should be expected 
2- This behavior is observed ONLY on small models !

I'm still a bit confused by this PR - generations are not flaky for pretrained models normally and it's a bit weird to me that all this test does is modifying slow generation tests

Closing as it has been fixed by https://github.com/huggingface/transformers/pull/18344
Deberta V2: Fix critical trace warnings to allow ONNX export
Author: iiLaurens
Pull request: yes

# What does this PR do?

This PR fixes some untraceable functions used in Deberta V2. Specifically, `math.sqrt`, `np.arange`, `np.tile` and `np.where` were replaced with their torch equivalents. I also applied some type conversions to make sure that the types are compatible with ONNX ops (opsset 15 was the focus). The remaining trace warnings that I did not solve seem to concern configuration items, which should stay constant for any given model.

Fixes #18237

## Who can review?

@LysandreJik


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @BigBird01 for knowledge, @michaelbenayoun 

@michaelbenayoun I resolved all the comments. Could you verify and merge?
Examples: add Bloom support for token classification
Author: stefan-it
Pull request: yes

Hi,

this PR adds support for fine-tuning Bloom models for token classification tasks.

After trying out the current example, I got an error message from the tokenizer part, that `add_prefix_space=True` needs to be set. So with this PR, Bloom can be used in token classification example for PyTorch (there's no support for TensorFlow/FLAX yet).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @younesbelkada 

Hi @younesbelkada ,

sure no problem, here are the steps (fresh environment):

```bash
git clone https://github.com/huggingface/transformers.git
cd transformers/
pip3 install -e .
pip3 install seqeval evaluate
cd examples/pytorch/token-classification
python3 run_ner.py   --model_name_or_path bigscience/bloom-560m   --dataset_name conll2003   --output_dir ./output-bloom-560m   --do_train   --do_eval   --do_predict
```

Then the following error message is thrown:

```bash
Traceback (most recent call last):
  File "run_ner.py", line 630, in <module>
    main()
  File "run_ner.py", line 464, in main
    train_dataset = train_dataset.map(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2387, in map
    return self._map_single(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2775, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "run_ner.py", line 422, in tokenize_and_align_labels
    tokenized_inputs = tokenizer(
  File "/workspace/transformers/src/transformers/tokenization_utils_base.py", line 2475, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/workspace/transformers/src/transformers/tokenization_utils_base.py", line 2561, in _call_one
    return self.batch_encode_plus(
  File "/workspace/transformers/src/transformers/tokenization_utils_base.py", line 2752, in batch_encode_plus
    return self._batch_encode_plus(
  File "/workspace/transformers/src/transformers/models/bloom/tokenization_bloom_fast.py", line 140, in _batch_encode_plus
    raise Exception(
Exception: You need to instantiate BloomTokenizerFast with add_prefix_space=True to use it with pretokenized inputs.
```
Fix Yolos ONNX export test
Author: ydshieh
Pull request: yes

# What does this PR do?

YOLOS has issue on ONNX exporting on CUDA. Let's skip it, just like [this](https://github.com/ultralytics/yolov5/pull/8378).

**Question**: we can enable this if there is a way to run the export non-dynamically for this model.

current job failure
```
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_143_yolos_default
(line 318)  AssertionError: yolos, default -> Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu! (when checking argument for argument index in method wrapper__index_select)
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_144_yolos_object_detection
(line 318)  AssertionError: yolos, object-detection -> Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu! (when checking argument for argument index in method wrapper__index_select)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Getting error following the official docs for T5 translation fine-tuning: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'new_zeros'
Author: gorkemozkaya
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I follow the TF steps on [huggingface.co/docs/transformers/tasks/translation](https://huggingface.co/docs/transformers/tasks/translation), I get the error `'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'new_zeros'`.  I created a Colab notebook that reproduces the issue

https://github.com/gorkemozkaya/Data-Science-Notes/blob/master/reproducing_bugs/Error_with_the_translation_fine_tuning_example.ipynb


### Expected behavior

Getting the data pipeline in the TF-dataset form without getting an error

Comments:
I noticed the issue is due to using the PT model instead of the TF, but the documentation still needs to be fixed, because it is using the `model`  for creating the `data_collator` before the model is actually loaded, for both TF and PT. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

https://github.com/huggingface/transformers/pull/18169#issuecomment-1186713812 solves the issue
[bnb] Minor modifications
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addressing the request from https://github.com/huggingface/transformers/pull/17901 to refactor the documentation a little bit and fixes small details in the bnb PR. 

- Fixing documentation
- Added useful troubleshooting document for more efficient debugging
- Updated the `Dockerfile` with the correct `bitsandbytes` version

cc @stas00 @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

You could also read the rendered doc (not GitHub page) below

https://moon-ci-docs.huggingface.co/docs/transformers/pr_18631/en/perf_train_gpu_one#efficient-training-on-a-single-gpu
(toward the end)

@ydshieh do you think we should keep the version `0.31.8` on the Dockerfile since we know it's the version that works 🤔 

> @ydshieh do you think we should keep the version `0.31.8` on the Dockerfile since we know it's the version that works 🤔

So far let's not add this. If the newer versions constantly break things, we can ping a particular versions.

Great works for me!

Great thanks! 
Do you think that we can merge it for now? I am especially interested in seeing if the changes in the `Dockerfile` would change anything (theoretically no, but we are using `bitsandbytes==0.31.5` on the Dockerfile and the latest verison is the `0.31.8`). For the rest since it's related to documentation I think we can always re-iterate.
Perhaps I can open a separate PR just to change the Dockerfile?

If you feel it's complete and want to merge it, go for it. We can always improve it later.

Thanks a lot!
Just went through a final pass, looks good to me!

@younesbelkada, I have just realized that this PR added to the wrong perf doc. This new feature is inference only and thus ideally should go into the inference doc and not training. Probably the many-gpu one. https://github.com/huggingface/transformers/blob/main/docs/source/en/perf_infer_gpu_many.mdx

Thanks.

No problem at all and sorry for the inconvenience, will re-open a PR for that!

I have addressed a PR at: https://github.com/huggingface/transformers/pull/18671 !
No module named 'evaluate'
Author: skye95git
Pull request: no

https://github.com/huggingface/transformers/blob/c99e984657b64dd8f19de74405bbf13763ab4f2b/examples/pytorch/language-modeling/run_mlm.py#L35

When I pre-train RoBerta from scratch using the latest version `run_mlm.py`, there is an error :
```
import evaluate
ModuleNotFoundError: No module named 'evaluate'
```

Comments:
Hi! You need to do `pip install evaluate`. #18666 will also add it to each of the `examples` internal `requirements.txt` file, so it will be installed when you do `pip install -r requirements.txt`

> Hi! You need to do `pip install evaluate`. #18666 will also add it to each of the `examples` internal `requirements.txt` file, so it will be installed when you do `pip install -r requirements.txt`

Thanks! It work.
_tf_available for customized built tensorflow
Author: kevint324
Pull request: no
Labels: name: Feature request

### System Info

n/a

### Who can help?

@Rocketknight1 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction



```
  File "virtualenv_mlu/lib/python3.8/site-packages/transformers/pipelines/base.py", line 212, in infer_framework_load_model
    raise RuntimeError(
RuntimeError: At least one of TensorFlow 2.0 or PyTorch should be installed. To install TensorFlow 2.0, read the instructions at https://www.tensorflow.org/install/ To install PyTorch, read the instructions at https://pytorch.or
```

### Expected behavior

https://github.com/huggingface/transformers/blob/02b176c4ce14340d26d42825523f406959c6c202/src/transformers/utils/import_utils.py#L63L75

I built a tensorflow-xxu for our in house accelerator and tried to run the transformer example.
I got the RuntimeError indicates tf is not available.
Currently the _tf_available has a hard-coded candidates list.
I'm not suire if extending the existing list is a good idea.
Maybe adding some runtime flexibility would be better?

Thanks
Kevin

Comments:
Hi @kevint324 I think it (extending the list ) is fine if you would work with a specific `transformers` version. But it would be a bit tedious if you want to use newer versions constantly.

cc @Rocketknight1 for the idea regarding `adding some runtime flexibility`.

Changed the tag to `Feature request` instead :-)

Hi @kevint324, I filed a PR that might resolve this, but I want to check with other maintainers that it's okay before I merge it. In the meantime, can you try it out? Just run `pip install git+https://github.com/huggingface/transformers.git@allow_force_tf_availability`, then set the environment variable `FORCE_TF_AVAILABLE=1` before running your code, and it should skip those checks now.

Yes, it works.
Thanks for the quick fix.
Allow users to force TF availability
Author: Rocketknight1
Pull request: yes

We have a user report that with custom Tensorflow builds and package names that  `_tf_available` can return `False` even if `import tensorflow` succeeds, because the user's package name isn't in the [allowed list](https://github.com/huggingface/transformers/blob/02b176c4ce14340d26d42825523f406959c6c202/src/transformers/utils/import_utils.py#L63L75). 

This is quite niche, so I don't want to do anything that could affect other users and workflows, but I added a `FORCE_TF_AVAILABLE` envvar that will skip version checks and just make sure TF is treated as available. @sgugger WDYT, or is there a better solution?

Fixed #18642 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ah, additional nit: I'd document this somewhere.

@LysandreJik I think a lot of these envvars aren't documented anywhere - I can't see any documentation for USE_TF or USE_TORCH! Maybe we should make a separate docs PR with a list of envvars that `transformers` uses?

That would be fantastic :) 

Thanks for your contribution, merging!
Is run_clip.py an example of fine-tune or an example of training a vision-text model from scratch？
Author: gongshaojie12
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

Hi @patil-suraj I am a newer to CLIP, and while browsing [run_clip.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/run_clip.py), I have a question, Is `run_clip.py` an example of fine-tune or an example of training a vision-text model from scratch?  Is it possible to fine-tune CLIP on my own dataset using `run_clip.py`？

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python run_clip.py

### Expected behavior

fine-tune

Comments:
Hey @gongshaojie12, is the README available [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) answering your questions?

In the [code sample](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text#train-the-model), you'll see it's using @ydshieh's "coco_dataset_script" as a dataset; feel free to replace the dataset here by a similar dataset of yours to use the script on your own data.

Hi!

The script is mostly for finetuning. But it's also possible to train from scratch - you just need to create a model, save it, and use it for the argument `model_name_or_path` - if this is what you like to do.

Hi, @LysandreJik @ydshieh  Thank you very much for your reply, maybe the first sentence in the README (`The following example showcases how to train a CLIP-like vision-text dual encoder model using a pre-trained vision and text encoder.`) bothered me. This sentence makes me think that `run_clip.py` is training a CLIP-like model from scratch, rather than fine-tuning the existing CLIP model. If there is something wrong with my understanding, please criticize and correct me, thank you!

The doc mentions `using a pre-trained vision and text encoder.`, so I think there is no ambiguity here. It doesn't necessary finetune the original clip checkpoints thought. You can use any text and image encoders.

Close this issue now. Don't hesitate if you have further question, @gongshaojie12