noamaanMulla-03/transformers-issues huggingface dataset:
Issues with Training VisionEncoderDecoder with Seq2SeqTrainer
Author: PrithivirajDamodaran
Pull request: no

Hello Team, 

I am trying to code up a VisionEncoderDecoder with VIT + BERT and finetune it with `Seq2SeqTrainer.` (Presuming the support exists).

```python
from transformers import VisionEncoderDecoderModel
from transformers import ViTFeatureExtractor, AutoTokenizer
import torch 

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224-in21k')
vitbert = VisionEncoderDecoderModel.from_encoder_decoder_pretrained('google/vit-base-patch16-224-in21k', 'bert-base-uncased')
vitbert.to(device)
print("Model loaded")
```
VITModel accepts `pixel_values`  but the VisionEncoderDecoder model apparently fails to recognize the preprocessed inputs with image encodings as `pixel_values` and `labels`. So I changed the `pixel_values` into `input_ids` for the model to accept? (not sure if I am in the right direction) .

**The Pre-Process snippet** 
```python

input_encodings = feature_extractor(images=inputs, return_tensors="pt")
input_encodings["input_ids"] = input_encodings["pixel_values"]
```

**Feature casting snippet**
```python
from datasets import Features, Array3D, Sequence, Value
features = Features({
  "input_ids": Array3D(dtype="float32", shape=(3, 224, 224)),
  "labels": Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
})
preprocessed_train_ds = tokenized_train_datasets.map(ds_preprocess_function, batched=True, features=features)
preprocessed_val_ds = tokenized_eval_datasets.map(ds_preprocess_function, batched=True, features=features)
```

But `Trainer.train` throws the below error.

```python
ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['labels']
```

Not sure what I am missing here. 

Please advice.

Comments:
Hi,

Did you take a look at my [notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb)?

I just tested, everything seems to work fine on my end. You don't need to rename the `pixel_values` to `input_ids`.



> Hi,
> 
> 
> 
> Did you take a look at my [notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb)?
> 
> 
> 
> I just tested, everything seems to work fine on my end. You don't need to rename the `pixel_values` to `input_ids`.
> 
> 
> 
> 

Thanks a lot will look at. 

All good now ! 

I supplied the decoder tokeniser instead of the encoder feature extractor  in the tokeniser parameter while setting up the seq2seq trainer. 

Thanks
How to load a fine-tuned model and do predictions?
Author: terenceau1
Pull request: no

I have fine-tuned a Named entity recognition BERT model using the example here
https://github.com/huggingface/transformers/tree/master/examples/pytorch/token-classification

The command I used is:
 python3 run_ner.py --config_name (The bert model I chose) --tokenizer_name (The bert model I chose) --model_name_or_path (The bert model I chose) --train_file (My train file) --(my validation file)  --text_column_name Words --label_column_name Tags --output_dir output --do_train --do_eval 




I successfully obtained the output, which is  config.json and model.h5

With this, how can I load my model (the model.h5 above) and make a prediction, on an example sentence such as "I visited Github today"?


For the tutorial in the link above, the code has the "do_predict" option, but I do not know how to ask the "run_ner.py" code to load my fine-tuned model, and I want to see the predictions (instead of the code giving me a metrics of accuracy = xx%, precision = xx%.


Thanks.

Comments:
Hi, please use the [forum](https://discuss.huggingface.co/) to post such general questions, we use issues for bug reports and feature requests. Thank you!

You can check out [this post](https://discuss.huggingface.co/t/decoding-the-predicted-output-array-in-distilbertbase-uncased-model-for-ner/10673/2?u=nielsr) I wrote on the forum about performing inference with NER models.

> Hi, please use the [forum](https://discuss.huggingface.co/) to post such general questions, we use issues for bug reports and feature requests. Thank you!

Apologies, I am quite new here. Please close my post if necessary.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
In pre-training a model how can I resume from last saved model?
Author: puraminy
Pull request: no

Using flax example for training a T5 model from scratch, I am training a model using:

```
thon run_t5_mlm_flax.py \
	--output_dir="/content/drive/MyDrive/Pouramini/" \
	--cache_dir="/content/drive/MyDrive/cache/" \
	--preprocessing_num_workers="20" \
  --model_type="t5" \
	--config_name="/content/drive/MyDrive/Pouramini/" \
	--tokenizer_name="/content/drive/MyDrive/Pouramini/" \
	--dataset_name="oscar" \
	--dataset_config_name="unshuffled_deduplicated_fa" \
	--max_seq_length="256" \
	--per_device_train_batch_size="32" \
	--per_device_eval_batch_size="32" \
	--adafactor \
	--learning_rate="0.005" \
	--weight_decay="0.001" \
	--warmup_steps="2000" \
	--logging_steps="500" \
	--save_steps="5000" \
	--eval_steps="50000" \
  --resume_from_checkpoint=True
```

Sometimes, the training interrupts and I must run the command again. But while I have specified `resume_from_checkpoint` it seems it trains from scratch, because the loss it shows is higher than the loss of the last saved model. What did I miss in the code above?
I thought to add:

```
  --model_name_or_path="/content/drive/MyDrive/Pouramini/" \
```
Is it necessary?

Comments:
cc @sgugger 

It's Flax, so it's more up to @patil-suraj aisle than mine :-)

`resume_from_checkpoint` is not yet correctly implemented in flax examples.  You could set the `--model_name_or_path` to the last saved checkpoint, this will load the model from that checkpoint, but note that the optimizer state is not saved.
For a complete resume, we need to load both model and optimizer states from the last checkpoint. 

For now, you could find how to do this in [this t5 example ](https://github.com/gsarti/t5-flax-gcp/blob/34def61c98097224b4d9bca6c1e0832cd015f06a/run_t5_mlm_flax.py#L405)

@patil-suraj  Thank you, I followed the example and added save and restore functions. However, I resumed a model (with 30000 steps) by just adding `--model_name_or_path` and if I interrupt my current running it will again save a model (with another 30000) without saving the optimizer.  I would like to know if the training was okay? is it a serious stage? I mean should I train the model again from scratch or I can still rely on these saved models?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Benchmark] Tokenizers as Collate functions vs normal in loop tokenizing
Author: sachinruk
Pull request: no

Apologies if this is more of a bug than it is a benchmark, just wanted to check if it was normal for tokenizer as collate function to run twice as slow. [Here's the colab notebook](https://colab.research.google.com/drive/1dqnM3pWIZgUMTrk42RJrvby6-ZZCHNRO?usp=sharing).

Comments:
Maybe @sgugger has an idea.

Please explain to us what the problem is, the Colab notebook you link is not public.

Sorry @sgugger, should be public now but basically the problem is that using a collate function slows down dataloader:
```python
sentence_dl = DataLoader(
    sentence_ds,
    BATCH_SIZE,
    num_workers=NUM_WORKERS,
    shuffle=False,
    drop_last=False,
    pin_memory=True,
)
# fast
for batch in tqdm(sentence_dl):
    x = collate_fn(batch)
```

```python
sentence_dl = DataLoader(
    sentence_ds,
    BATCH_SIZE,
    num_workers=NUM_WORKERS,
    shuffle=False,
    drop_last=False,
    pin_memory=True,
    collate_fn=collate_fn,
)
# slow
for batch in tqdm(sentence_dl):
    continue
```

Where sentence_ds is just a `Dataset` created from the sentences (strings) in "snli" dataset and collate function is:
```python
tokenizer = AutoTokenizer.from_pretrained(LANGUAGE_MODEL)

class CollateFn:
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
        
    def __call__(self, x):
        return self.tokenizer(
            x, 
            max_length=MAX_TEXT_LENGTH, 
            truncation=True, 
            padding="max_length", 
            return_tensors="pt"
        )
    
collate_fn = CollateFn(tokenizer)
```

Understood. Yes, it's very likely that using the multiprocessing on the dataloders side slows down the multiprocessing of the "fast" tokenizer.

FYI, for best speed, I would recommend passing 1,000 samples at once to the tokenizer and preprocessing once and for all beforehand.
can one run squad using gpt2?
Author: David-Levinthal
Pull request: no
Labels: name: wontfix

looking through the new notes discussing GPT-2 I do not understand how one might run a squad fine tuning on a pretrained gpt-2 model
Any assistance would be greatly appreciated

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


2 years late, but can anyone figure it out?
Integration.py
Author: cs-ark
Pull request: no

# 🚀 Feature request

CometCallback in integration is overriding my experiment object declared in optimizer.

## Motivation

reporting logs to comet through cometcallback is overriding experiment objects already declared before the TrainerAPI is invoked. Is there any way to make space for cometcallback to get the experiment object instead of overriding it. This way, we can not only use Optimizers of comet_ml but also use important metrics like "log_confusion_matrix".

If it's not viable, please let me know.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Pinging @dsblank :)

cc @sgugger 

I think we have solved this issue here:

https://github.com/comet-ml/issue-tracking/issues/434

And we are working on an internal refactor to allow more flexibility on the transformer side.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ModuleNotFoundError: No module named 'transformers.models.fnet.configuration_fnet
Author: fractaldna22
Pull request: no

```py
model_lang = SentenceTransformer('clip-ViT-B-32-multilingual-v1').cuda()

def enc_text(txt):
  if multilang:
    emb = model_lang.encode([txt], convert_to_tensor=True, show_progress_bar=False)
  else:
    emb = model_clip.encode_text(clip.tokenize(txt).cuda())
  return emb.detach().clone()

```

```
Downloading: 100%
690/690 [00:00<00:00, 26.1kB/s]
[...]
Downloading: 100%
1.57M/1.57M [00:01<00:00, 1.13MB/s]
```
```
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-6-325284ff9354> in <module>()
----> 1 model_lang = SentenceTransformer('clip-ViT-B-32-multilingual-v1').cuda()
      2 
      3 def enc_text(txt):
      4   if multilang:
      5     emb = model_lang.encode([txt], convert_to_tensor=True, show_progress_bar=False)

14 frames
/usr/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'transformers.models.fnet.configuration_fnet'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
```

Comments:
It's very possibly linked to an outdated version of `transformers`: FNet was only released a few weeks ago, as part of the v4.11 release.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
add Tuple as possible type hint for EvalPredictions label_ids
Author: ameasure
Pull request: yes

# What does this PR do?
This adds Tuple as a type hint for the label_ids attribute in EvalPredictions. It is a valid type that label_ids can have.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@stas00


Comments:
If that's the case, then for consistency the following 2 classes should have the proposed changed as well.

@sgugger, what do you think?
Pre-trained electra-large model doesn't converge when fine-tuned on SST-2
Author: shanybarhom
Pull request: no
Labels: name: Need more information

name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ELECTRA 

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SST-2


## To reproduce

Steps to reproduce the behavior:

1. Load the large pre-trained ELECTRA model using ElectraModel.from_pretrained('google/electra-large-discriminator', output_hidden_states=True)
2. fine-tune it on SST-2 using a simple binary classification head (linear, ReLU, linear, Sigmoid) on top of the [CLS] hidden state with BCEWithLogitsLoss and AdamW for 3-4 epochs.
3. Model is stuck on ~55% accuracy during all training and the loss increases

Important: When I do the same with 'google/electra-base-discriminator', and 'google/electra-small-discriminator' I'm getting an accuracy of ~93% and 89% (respectively) on the first epoch.

Hyperparameters: 
batch_size: 16
lr: 0.000005
adam_epsilon: 0.0000001
max_len: 32

## Expected behavior

I expected that the fine-tuned electra-large model will outperform the base and small electra model and have better results than ~50% accuracy

## Environment info
   
- `transformers` version: 2.9.0
- Platform: Linux-5.3.0-1017-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



Comments:
This is a general question and does not sound like a bug. With these large models it is sometimes hard to find a good set of hyperparameters to get the model to converge well. The same is true, and reported, for ALBERT. I don't think this is a bug.

@shanybarhom A good start would be to use the hyper-parameters mentioned in the [ELECTRA](https://arxiv.org/abs/2003.10555) paper :) Just refer to table 7.

So your batch size, adam epsilon and epochs are very different compared to the ELECTRA parameters.

Thanks, @stefan-it I've tried to use the same hyper-parameters as mentioned in the ELECTRA paper (lr=0.00005, batch_size=32, adam_epsilon=0.000001, epochs=3), but the electra-large model still doesn't converge (accuracy of ~50%). 

@BramVanroy  I thought it is a bug since the electra-base and the electra-small converge quite quickly (90% accuracy after the first epoch) with the same code and data, while the large model is stuck on ~50% during the training, so it felt like a bug, but of course, it may not.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am facing the same problem with ELECTRA-large. I really appreciate if there is any further direction for this.
Fix dummy objects for quantization
Author: sgugger
Pull request: yes

# What does this PR do?

The dummy objects introduced recently are actually not generated and maintained by the dummy scripts, because the `is_xxx_available` test they rely on contains an _ in the xxx. This PR fixes that.
Another fix concerns the models `ForNextSentencePrediction` which were not included in the list yet (a better fix will use the model mapping names from the auto mapping one day when I have more time :-) )

This is blocking the doc building for the new frontend so merging as soon as it's green.

Comments:
Thanks for taking care of it!
Is the index of the vocabulary in Tokenizer the same as the index of WordEmbedding?
Author: wmathor
Pull request: no

I have a word vector A, which has dimension `[1, 768]`. I compute the similarity between vector A and all words in `bert.embeddings.word_embeddings.weight` and get the most similar index is `idx`. I use the following code to see what token the `idx` is

```python
token = tokenizer.decode([idx])
```

But when I look at the index of the token in Tokenizer using the following code, I find that it is not equal to the `idx`

```python
new_idx = tokenizer(token)['input_ids'][1]
# new_idx != idx
```

Comments:
Fatal error in event_loop.c
Author: Crabzmatic
Pull request: no

## Environment info

- `transformers` version: 4.12.5
- Platform: Windows-10-10.0.22504-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## To reproduce

Steps to reproduce the behavior:

1. run any script using `datasets` 

```
Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
at 0x7FFD34C74380: aws_backtrace_print
at 0x7FFD34C63560: aws_fatal_assert
at 0x7FFD34B65F10: aws_event_loop_wait_for_stop_completion
at 0x7FFD34C71470: aws_ref_count_release
at 0x7FFD34B63D80: aws_server_bootstrap_set_alpn_callback
at 0x7FFD34C71470: aws_ref_count_release
at 0x7FFD34B63760: aws_client_bootstrap_release
at 0x7FFD4C7F76F0: Aws::Crt::Io::ClientBootstrap::~ClientBootstrap
at 0x7FFD34DFEB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFE024D36C0: _sys_nerr
at 0x7FFE0249FFA0: execute_onexit_table
at 0x7FFE0249FFA0: execute_onexit_table
at 0x7FFD34DFEB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFD34DFEB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFE04A9EDC0: RtlActivateActivationContextUnsafeFast
at 0x7FFE04AF2310: LdrShutdownProcess
at 0x7FFE04AF2240: RtlExitUserProcess
at 0x7FFE03C8E080: ExitProcess
at 0x7FFE0249E040: exit
at 0x7FFE0249E040: exit
at 0x7FF69C8C1160: OPENSSL_Applink
at 0x7FFE03C86AA0: BaseThreadInitThunk
at 0x7FFE04AD1EB0: RtlUserThreadStart

```

Even this two lines produce this error:

```
from datasets import load_dataset
dataset = load_dataset('wikiann', 'en')
```

```
Downloading and preparing dataset wikiann/en (download: 223.17 MiB, generated: 8.88 MiB, post-processed: Unknown size, total: 232.05 MiB) to [...]
100%|██████████| 3/3 [00:00<00:00, 498.27it/s]
Dataset wikiann downloaded and prepared to [...]. Subsequent calls will reuse this data.
Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
```

Also note that `D:\bld\aws-c-io_1633633258269\work\source\` is not a path on my PC.

## Expected behavior

I would expect no fatal errors.


Comments:
Moved to `datasets` repo: https://github.com/huggingface/datasets/issues/3310
Allow user to choose DDP backends: nccl or gloo (#13441)
Author: BruceDai003
Pull request: yes


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Improve `add-new-pipeline` docs a bit
Author: stancld
Pull request: yes

# What does this PR do?

This PR slightly improve syntaxs in code snippets in the `add-new-pipeline.rst` file.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
AttributeError: 'MarianTokenizer' object has no attribute 'prepare_translation_batch'
Author: sdhzlxm
Pull request: no
Labels: name: marian

Problem 1:
When I load tokenizer from local directory.  And use it according MarianMT tutorial. There will be an error occurs. 
AttributeError: 'MarianTokenizer' object has no attribute 'prepare_translation_batch'

Problem 2:
When I downloads models and tokenizers from the web, such as running 'tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-zh-en")'. There will be some errors.
RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(818) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 

Comments:
Which tutorial? its called `prepare_seq2seq_batch` now.

"prepare_seq2seq_batch" works. Thanks a lot.

I follow this tutorial, https://huggingface.co/transformers/model_doc/marian.html

Where can I find the new user manual for MarianMT model? Thank you.


https://huggingface.co/transformers/master/model_doc/marian.html

Have they change it again and add a maximum length?

I am getting: `AttributeError: 'MarianTokenizer' object has no attribute 'prepare_seq2seq_batch'`

I changed it to `prepare_translation_batch` and it works

It fails again... and changing to `prepare_seq2seq_batch` throws the deprecation warning...