noamaanMulla-03/transformers-issues huggingface dataset:
[examples/flax] clip style image-text training example
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds an example script to train CLIP style vision-text dual encoder models using pre-trained vision and text encoder.
It supports `CLIP` and `ViT` as image encoder and `BERT` and ROBERTa` (or any other text encoder in flax) as text encoder.

This example uses`torchvision` and torch `Dataloader` for faster pre-processing and data loading. This is important as data loading and processing can become a bottleneck on TPU. This will be revised later to use `datasets`.

Comments:
And think you might have to run `make style` once

Merging!

hi @patil-suraj i tried to run your script(run_hybrid_clip.py)

after few step i got error

```
it/s/home/acul/env/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:699: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  tensor = as_tensor(value)
Epoch ... (1/40):   0%|                                                | 0/40 [02:43<?, ?it/s]
Traceback (most recent call last):                           | 39/462 [02:43<06:48,  1.03it/s]
  File "run_hybrid_clip.py", line 556, in <module>
    main()
  File "run_hybrid_clip.py", line 502, in main
    state, train_metric = p_train_step(state, batch)
  File "/home/acul/env/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 183, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/home/acul/env/lib/python3.8/site-packages/jax/_src/api.py", line 1623, in f_pmapped
    for arg in args: _check_arg(arg)
  File "/home/acul/env/lib/python3.8/site-packages/jax/_src/api.py", line 2296, in _check_arg
    raise TypeError(f"Argument '{arg}' of type {type(arg)} is not a valid JAX type.")
jax._src.traceback_util.UnfilteredStackTrace: TypeError: Argument '[[list([1, 1, 1, 1, 
.
.
.
.
  list([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])]]' of type <class 'numpy.ndarray'> is not a valid JAX type.

```
i use [bert](https://huggingface.co/indobenchmark/indobert-base-p1) and clip as model vision and text

any idea how to fix

Thank you for reporting! 
Could you please open an issue and post the command that you used?

sure 

opening here:
https://github.com/huggingface/transformers/issues/12502
Expand text-generation pipeline support for other causal models e.g., BigBirdForCausalLM
Author: AliOskooeiTR
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Tried using the text generation pipeline (TextGenerationPipeline) with BigBirdForCausalLM but seems like the pipeline currently only supports a limited number of models. Is there a reason for this? Is there a workaround short of implementing the pipeline myself? Thank you.


Comments:
create LxmertModelIntegrationTest Pytorch
Author: sadakmed
Pull request: yes

# What does this PR do?

this pr fix issue #9951 as it implements an integration test for  LXMERT

@LysandreJik


Comments:
Hello! It seems this is simply passing the test?

> Hello! It seems this is simply passing the test?

I made it to claim the issue and work on it. 

@LysandreJik  Lxmert requires `visual_feats` and `visual_pos` could I change the `model.config.visual_feats_dim` to a smaller value like 5 or 10. 
Edit: or we could use a `seed` and generate a random tensor with original `visual_feats_dim`

@LysandreJik  in this I used `np.random.seed` to fix the `visual_feats` and `visual_pos`,  otherwise we could load lxmertmodel by lxmertconfig while changing `visual_feat_dim` to something manageable. 

What do you suggest?

@LysandreJik  is there something I can help with here?

> Could you try to replace the `np.random.rand` by the `ids_tensor`
well the `visual_feats` is torch.float will the `ids_tensor` returns int32.


@LysandreJik  I think another alternative, is to make the `model.config.visual_feat_dim` smaller then we can have a fixed `visual_feats` 


what do you think?


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik  I did raise an issue to have a context manager that will fix a seed #10143, do you think it will be usefull here?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello @sadakmed, sorry for taking a while to merge this. Do you mind rebasing on `master`, and running `make fixup` at the root of your clone? There's an issue with the code quality.

Will merge this right after.

Thanks a lot for your contribution @sadakmed!
implementing tflxmertmodel integration test
Author: sadakmed
Pull request: yes

# What does this PR do?
this PR implement an integration test for TFLxmertmodel as requested in #9954  

@LysandreJik 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @LysandreJik  what about this one 

Sure! It seems there is no branch linked to this PR, however. The changes are still visible here https://github.com/huggingface/transformers/pull/10052/files but I am unable to reopen.
[Flax] Dataset streaming example
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR adds an example of how to use dataset streaming.

@lhoestq - I would be super happy if you could take a look! I've left some comments that should help understand my thought process. I'd be very happy about some Feedback :-) 

The scripts runs as expected at the moment - after your feedback, I'll make it nicer (add better defaults) & add the eval coed as well :-)

Comments:
The training speed of this script compared to `run_mlm_flax` is sadly reduced by ~50%. @lhoestq - I think the "batchify" function that is called before every training step is probably the reason for this.
BERT pretraining: [SEP] vs. Segment Embeddings?
Author: velixo
Pull request: no

I’m confused about the differences between the intent of the [SEP] tokens and Segment Embeddings applied to the input of BERT during pretraining.

As far as I’ve understood, the [SEP] tokens are inserted between sentence A and B to enable the model’s ability to distinguish between the two sentences for BERTs Next-Sentence Prediction pretraining-task. Similarly, the Segment Embeddings are added to the input embeddings to alter the input, creating another opportunity for the model to learn that sentence A and B are distinct things.

However, these seem to be facilitating the same purpose. Why can’t BERT be trained on only Segment Embeddings, omitting [SEP] tokens? What additional information do [SEP] tokens conceptually provide, that the Segment Embeddings don’t?

Furthermore, [SEP] tokens aren’t used directly anyways. NSP is trained on the [CLS] embeddings, which I understand to sort of represent an embedding of sentence continuity.

Comments:
From the BERT paper: "We differentiate the sentences in two ways. First, we separate them with a special token ([SEP]). Second, we add a learned embedding to every token indicating whether it belongs to sentence A or sentence B."

Deep learning, as you may now, is a lot of experimenting, and in this case, it was a design choice. I guess you could try to omit the [SEP] token, perhaps it doesn't add much information to the model. Or omit the token type embeddings, and check whether the results are significantly different. 

To give another example, people are experimenting with all kinds of position encodings (include absolute ones, as in BERT, relatives ones, as in T5, sinusoidal ones, as in the original Transformer, now there are rotary embeddings, as in the new RoFormer paper)... 

So the question you're asking is a genuine research question :)   

Thank you for the quick answer, good to know! I was suspecting it might be something along these lines :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Trainer.predict using customized model.predict function?
Author: iamlockelightning
Pull request: no

I am using the [Trainer](https://huggingface.co/transformers/main_classes/trainer.html) to train a sentence-bert model with triplet-loss. Then I want to do some inference. How to call Trainer.predict using custom model.predict function?

I use `model.forward()` to calculate loss in training stage. But I want to use a customized `model.predict()` to calculate prediction results based on `model.forward()` (e.g., model.forward() -> embedding -> other method to calculate prediction instead of the loss function)

I saw the `prediction_step()` function just called `outputs = model(**inputs)` to get `(loss, logits, labels)`

Is there any good method to do that?

Comments:
Cant you just subclass the Trainer class and write your own `predict`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
RuntimeError: The size of tensor a (716) must match the size of tensor b (512) at non-singleton dimension 1
Author: lisa563
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0.dev0
- Platform: windows 7
- Python version: 3.8.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [√] the official example scripts: (give details below)
`python run_ner.py --model_name_or_path nlpaueb/legal-bert-base-uncased --train_file ***.json --validation_file ***.json --output_dir /tmp/*** --do_train --do_eval`

* [ ] my own modified scripts: (give details below)

The tasks I am working on is: NER
* [ ] an official GLUE/SQUaD task: (give the name)
* [√] my own task or dataset: (give details below)

## Error
`File "run_ner.py", line 504, in <module>
    main()
  File "run_ner.py", line 446, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\trainer.py", line
1240, in train
    tr_loss += self.training_step(model, inputs)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\trainer.py", line
1635, in training_step
    loss = self.compute_loss(model, inputs)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\trainer.py", line
1667, in compute_loss
    outputs = model(**inputs)
  File "D:\Tool\Install\Python\lib\site-packages\torch\nn\modules\module.py", li
ne 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\models\bert\modeli
ng_bert.py", line 1679, in forward
    outputs = self.bert(
  File "D:\Tool\Install\Python\lib\site-packages\torch\nn\modules\module.py", li
ne 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\models\bert\modeli
ng_bert.py", line 964, in forward
    embedding_output = self.embeddings(
  File "D:\Tool\Install\Python\lib\site-packages\torch\nn\modules\module.py", li
ne 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\Tool\Install\Python\lib\site-packages\transformers\models\bert\modeli
ng_bert.py", line 207, in forward
    embeddings += position_embeddings
RuntimeError: The size of tensor a (716) must match the size of tensor b (512) at non-singleton dimension 1
  5%|██▏                                      | 12/231 [03:07<56:54, 15.59s/i
t]`



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to get back the identified words from LayoutLMForTokenClassification?
Author: karandua2016
Pull request: no

I am using LayoutLMForTokenClassification as described [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb). In the end, the tutorial shows an annotated image with identified classes for various tokens.
How can I get back the original words as well to be annotated along with the labels?

I tried to read the words with tokenizer.decode(input_ids).split(" ") but the tokenizer broke words into multiple tokens which it wasn't supposed to. So, I have more words/outputs/boxes that I am supposed to have.



Comments:
Hi,

A solution for this can be the following (taken from my [Fine-tuning BERT for NER notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb)):

![image](https://user-images.githubusercontent.com/48327001/119956504-72b47100-bfa1-11eb-949a-955ede29584a.png)

In my notebook for `LayoutLMForTokenClassification`, only the label for the first word piece of each word matters. In HuggingFace Transformers, a tokenizer takes an additional parameter called `return_offsets_mapping` which can be set to `True` to return the (char_start, char_end) for each token.

You can use this to determine whether a token is the first wordpiece of a word, or not. As we are only interested in the label of the first wordpiece, you can assign its label to be the label for the entire word.

Do you understand?



I do. Thanks. I tried doing this by referring your BERT code. But I am getting this error, unfortunately. 

![image](https://user-images.githubusercontent.com/18511921/120136599-26ab3b80-c1f0-11eb-9478-cba341955c6a.png)

Apologies if I messed up. This is the first time that I am working with transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
'error': 'Model Matthieu/stsb-xlm-r-multilingual is currently loading'
Author: Matthieu-Tinycoaching
Pull request: no

Hello,

I have pushed on model hub (https://huggingface.co/Matthieu/stsb-xlm-r-multilingual) a pretrained sentence transformer model (https://huggingface.co/sentence-transformers/stsb-xlm-r-multilingual).

However, when trying to get prediction via th API_URL I stil got the following error:
`{'error': 'Model Matthieu/stsb-xlm-r-multilingual is currently loading', 'estimated_time': 44.49033436}`

How could I deal with this problem?

Thanks!



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Provides an option to select the parallel mode of the Trainer.
Author: hijkzzz
Pull request: no

# 🚀 Feature request

Provides an option to select the parallel mode of the Trainer.

## Motivation

For multiple GPUs, Trainer uses `nn.DataParallel` for parallel computing by default, however, this approach results in a large memory occupation for the first GPU. Please provide an API to switch to `nn.parallel.DistributedDataParallel`. Also, for the `Trainer.predict()` function is there an option to turn off parallel computing?

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

@sgugger 

Comments:
This is already implemented, it just depends on how you launch your training script. To use distributed data parallel, you have to launch it with `torch.distributed.launch`.

> This is already implemented, it just depends on how you launch your training script. To use distributed data parallel, you have to launch it with `torch.distributed.launch`.

Hi~
How to do for jupyter?
Also what should I do if I use DataParallel for train and only want to use one GPU for predict?
As the `DataParallel` requires droplast, which is not allowed in predict phase.
Thanks


You can't do this directly in jupyter, you have to launch a script using the pytorch utilities (it's not a Trainer limitation, it's a PyTorch one). You can completely predict in parallel with the `Trainer`, it will complete the last batch to make it the same size as the others and then truncate the predictions.

> You can't do this directly in jupyter, you have to launch a script using the pytorch utilities (it's not a Trainer limitation, it's a PyTorch one). You can completely predict in parallel with the `Trainer`, it will complete the last batch to make it the same size as the others and then truncate the predictions.

But we can't truncate predictions because it's a contest or a client's demand and we need all the test results we can get.

As a demo, #11833
https://github.com/huggingface/transformers/issues/11833

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
mutil gpu errors
Author: Soulscb
Pull request: no

i want to use multi gpus to train,but it erros;

model = nn.DataParallel(model)
model = model.cuda()
model.train_model(train_df, eval_data=eval_df)
torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'train_model'

so how can i to use multi gpus

Comments:
Is this related to `transformers`? 

yes    this is simpletransformers and i find transformers multi gpus is hard to  run 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Flax MLM example script has PyTorch dependency
Author: cakiki
Pull request: no

**Script in question:** [run_mlm_flax.py](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py)

```bash
Traceback (most recent call last):
  File "./run_mlm_flax.py", line 319, in <module>
    f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
  File "/home/christopher/transformers/src/transformers/file_utils.py", line 1641, in wrapper
    raise ImportError(f"Method `{func.__name__}` requires PyTorch.")
ImportError: Method `device` requires PyTorch.
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: '4.9.0.dev0' (editable pip install from `git` source)
- Platform: TPU v3-8 (Operating System: Ubuntu 20.04.2 LTS, Kernel: Linux 5.4.0-1043-gcp, Architecture: x86-64)
- Python version: Python 3.8.10 [GCC 9.4.0] on linux
- PyTorch version (GPU?): '1.9.0+cu102' (extras['all'])
- Tensorflow version (GPU?): '2.5.0' (extras['all'])
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Yes

### Who can help

@patrickvonplaten and @patil-suraj 

## Information

Model I am using: roBERTa

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set up a Flax-based Huggingface environment (`$ pip install -e ".[flax]"`) according to step 4 [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#tpu-vm)
2. Follow [the tutorial](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#how-to-use-the-hub-for-collaboration) on training roBERTa on Alemannic OSCAR.


## Expected behavior
 The script should run with no error. The absence of PyTorch should have no bearing on a Flax script.

Comments:
Our Model works well in the local ! But when we upload that it doesnt work showing irrelevent performance ! 
Author: sami10644
Pull request: no

We tried every possible way . We followed the uploading steps perfectly . Deleting and re uploading  are done several times . It performs perfectly in the local ! When we download after uploading it  it shows this mysterious  problem  . 

Comments:
Do you have a link to your model?


Yes ! can you tell something about tokenizer ?
Fix order of state and input in Flax Quickstart README
Author: navjotts
Pull request: yes

# What does this PR do?
Correct the order of state and input for `flax.linen.apply()` in Flax Quickstart README

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@patrickvonplaten @patil-suraj 

Comments:
Note: if the above correction is valid, then we ll need to update [this image](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/lm_flax_inference.png) as well