noamaanMulla-03/transformers-issues huggingface dataset:
When will ELECTRA pretraining from  scratch will be available?
Author: 008karan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
Is pre training from scratch for electra available? couldnt find it. 
Thanks!


Comments:
Working on it as we speak :).

I'd say it will be out in a few weeks at most.

Is Albert pretraining from scratch is available? @LysandreJik 


@LysandreJik do you think the update will be available by the end of the month ? Maybe it has been postponed due to the recent addition of the Trainer and the refactor of the language_modeling script ?

It has been postponed a bit due to the recent addition of the Trainer and the TPU work on it, but I definitely aim to have it out earlier than by the end of the month :)

Was looking for Albert pre-training from scratch but I think there is support for Bert, Roberta and distillbert only as of now. 
@LysandreJik can you guide how can I do Albert pretraining from scratch?

@LysandreJik Is the code for pretraining Electra from scratch available now?


> @LysandreJik Is the code for pretraining Electra from scratch available now?

Not yet. There's a PR about it;
https://github.com/huggingface/transformers/pull/4656

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Are there any updates to this, or plans to release the ELECTRA pre-training from scratch feature soon?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hello. 
Are there any updates?

Curious as well.

The development of the ELECTRA pretraining from scratch is in a stale state with no plans to work on it further, see https://github.com/huggingface/transformers/pull/4656#issuecomment-711082850

See https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004 by @richarddwang for a PyTorch implementation of the ELECTRA pretraining.
Trainer skips training when continuing training with model.from_pretrained()
Author: elenif
Pull request: no

I'm fine-tuning the gpt2 model using TFTrainer.

As I have to save some computational power, I needed to part the data set and train it one after another.

The first training (from TFGPT2LMHeadModel.from_pretrained('gpt2')) works well. After storing the model in a folder under a different name, and reloading it again to continue training, the trainer basically skips the training and stores the old model in the new model folder.

I don't understand the reason for this behavior.

transformers: 4.5.0
tensorflow:2.4.1

@Rocketknight1 

Training code:
```

def train(file, file_id, batch_size, num_epochs):
    tokenizer, vocab_size = load_tokenizer()
    max_seq_length = 1000

    args = TFTrainingArguments(output_dir='out_gpt',
                               num_train_epochs=num_epochs,
                               do_train=True,
                               per_device_train_batch_size=batch_size,
                               gradient_accumulation_steps=2,
                               max_grad_norm=1.0)

    data = prepare_pre_training_data_set(file, tokenizer, max_seq_length)
    print("Datasets loaded...")

    with args.strategy.scope():
        model = TFGPT2LMHeadModel.from_pretrained('gpt2_trained_' + str(file_id)) 

        optimizer = tf.keras.optimizers.Adam(lr=0.0005)
        cat_loss = tf.losses.CategoricalCrossentropy()
        model.compile(optimizer=optimizer, loss=cat_loss)

    trainer = TFTrainer(model=model,
                        train_dataset=data,
                        args=args)

    trainer.train()

    trainer.save_model("gpt2_trained_+str(file_id+1)")
```

Comments:
I'm really sorry to may have bothered you with that. Turns out, it works when using a different GPU :) 

It struck me when I tested the same code with less data on my local machine, the trainer logged epochs (which it did not before), so I tested it on several devices. I was just a bit confused to be honest, since there were no errors/warnings thrown neither by tensorflow nor the transformers library even when using trainer (debug=True).

Hey! Don't worry about it - TFTrainer is currently not very maintained, and we're looking at switching away from it to a pure Keras framework, so some bits of it can be quite confusing right now. Don't be shy about letting us know if you run into other issues, especially if they look like they might be bugs at our end!
Plug a custom tokenizer into PreTrainedTokenizer
Author: DhruvilKarani
Pull request: no

Is there a clean way to use a customer tokenizer trained from the tokenizers library in a PretrainedTokenizer interface?


```
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file(config_file)

```

The config file is generated using tokenizer.save() method.

I want to use this tokenizer in a PretrainedTokenizer/PretrainedTokenizerFast class.

The closest thing I found in the existing issues is [this](https://github.com/huggingface/tokenizers/issues/259#issuecomment-625905930)

When I tried the above solution using a PretrainedTokenizer class, 

```
class CustomTokenizer(PreTrainedTokenizer):
    def __init__(
            self,
            vocab_file=vocab_file,
            merges_file=merges_file,
            bos_token="
",
            eos_token="
",
            sep_token="
",
            cls_token="
",
            unk_token="<unk>",
            pad_token="<pad>",
            mask_token="<mask>",
            **kwargs
        ):
        super().__init__(
            tokenizer,
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            **kwargs,
        )
```

I got an exception

```

  File /path/custom_tokenizer.py", line 37, in __init__
    super().__init__(
TypeError: __init__() takes 1 positional argument but 2 were given
```

Is there a solution/workaround?

Comments:
Would this [documentation](https://huggingface.co/transformers/fast_tokenizers.html) help you out?

This is exactly what I was looking for! No idea how I missed it. Thank you :)
Fix broken link
Author: WybeKoper
Pull request: yes

# What does this PR do?

Fixes a broken link in model_doc/pegasus

Link was pointing to: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_pegasus_xsum.sh

File had been moved to: https://github.com/huggingface/transformers/blob/master/examples/research_projects/seq2seq-distillation/finetune_pegasus_xsum.sh

Fixes # (issue)
[#9257](https://github.com/huggingface/transformers/issues/9257)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone

Comments:
Wrong num_label configuration in Fine Tuning NER when model_name_or_path is specified
Author: sankaran45
Pull request: no

I have a pre-trained model that i have cached locally and i am passing model_name_or_path.

When i pass do_train, it results in the following code executing causing a second model reload. Unfortunately, this codepath ignores the config causing default num_labels to be passed leading to pytorch assert.

If last_checkpoint didn't detect it explicitly (at the beginning of main) wouldn't it be better to not reload unless explicitly asked by the user ?

 if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Confusion
Author: Sankalp1233
Pull request: no
Labels: name: Migration

Like you mention below: I am doing a question answering task I was using BERT originally so can
I convert from BERT to this?
To download and use any of the pretrained models on your given task, you just need to use those three lines of codes (PyTorch version):

>>> from transformers import AutoTokenizer, AutoModel

>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> model = AutoModel.from_pretrained("bert-base-uncased")

>>> inputs = tokenizer("Hello world!", return_tensors="pt")
>>> outputs = model(**inputs)
or for TensorFlow:

>>> from transformers import AutoTokenizer, TFAutoModel

>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> model = TFAutoModel.from_pretrained("bert-base-uncased")

>>> inputs = tokenizer("Hello world!", return_tensors="tf")
>>> outputs = model(**inputs)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Cannot get test logits after training for TFSequenceClassifier on TF 2
Author: dr-smgad
Pull request: no

- `transformers` version: 4.4.2
- Platform: Linux-5.4.17-2036.102.0.2.el7uek.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: (True)
- Using distributed or parallel set-up in script?: (True)

After fitting the model, I am trying to get the logits of the test dataset to calculate the confusion matrix as follows:
`y_pred = tf.nn.softmax(bert_model.predict(test_dataset))`
`y_pred_argmax = tf.math.argmax(y_pred, axis=1)`

I am getting this error:
`ValueError: Attempt to convert a value (TFSequenceClassifierOutput(loss=None, logits=None, hidden_states=None, attentions=None)) with an unsupported type (<class 'transformers.modeling_tf_outputs.TFSequenceClassifierOutput'>) to a Tensor.`

When I try to print the predict() output I get:
`TFSequenceClassifierOutput(loss=None, logits=None, hidden_states=None, attentions=None)`

Note that using the `evaluate()` method works fine
`loss, accuracy = bert_model.evaluate(test_dataset)`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Tokenizer becomes very slow after adding new tokens
Author: shauli-ravfogel
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik

## Information

Hi,

When I am trying to add a large number (50k) new tokens to BERT's tokenizer, the tokenizer becomes very slow, taking 29 seconds to tokenize a single short sentence.

## To reproduce

```
from transformers import BertTokenizer
import time

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

new_tokens = ["aaa"+str(i) for i in range(50000)]
tokenizer.add_tokens(new_tokens) # takes some time

sentence = "a short sentence."
start = time.time()
tokenizer.tokenize(sentence)
print(time.time() - start)

> 29.049
```




Comments:
Hey @shauli-ravfogel, this should have been fixed on `master`. Can you try installing from source and let me know if it works better?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Wav2vec2 Training Loss not decreasing
Author: gauravgund
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.4.0
- Platform: Google Colab
- Python version: python 3.6

@patrickvonplaten




Models:

wav2vec2

I am following the recent implementation of wav2vec2 for fine-tuning:
https://huggingface.co/blog/fine-tune-wav2vec2-english

Settings:
Pretrained model: "facebook/wav2vec2-base-960h",
gradient_checkpointing=True,
ctc_loss_reduction="mean",
pad_token_id=processor.tokenizer.pad_token_id
attention_dropout=0.1,
hidden_dropout=0.1,
feat_proj_dropout=0.0,
mask_time_prob=0.05,
layerdrop=0.1,
gradient_checkpointing=True,
ctc_loss_reduction="mean",
group_by_length=True,
per_device_train_batch_size=32,
evaluation_strategy="steps",
num_train_epochs=1500,
fp16=True,
save_steps=400, #this would mean every 400 steps model gets saved which also means Google drive gets full
eval_steps=400,
logging_steps=400,
learning_rate=0.0005,
warmup_steps=500,
save_total_limit=2,

Issue:

Step | Training Loss | Validation Loss | Wer | Runtime | Samples Per Second
-- | -- | -- | -- | -- | --
400 | 5.063200 | 4.566135 | 1.000000 | 0.715900 | 6.984000
800 | 5.115200 | 4.514411 | 1.000000 | 0.732400 | 6.827000
1200 | 5.119200 | 4.485986 | 1.000000 | 0.724300 | 6.903000

The training loss is marginally decreasing and WER is still 1. What can be done to improve and faster training with better accuracy.

I also tried with a higher learning rate but training loss was still very poor, it seems the model is not converging.

Comments:
It seems that your training epochs is set to 1500. Set it up to 5 for a quick trial

It is giving training loss of more than a 100 in that case!

Is your training finished or is still running? If it is the second, just try again with less number of epochs for example

This is the output for 5 epochs:


TrainOutput(global_step=10, training_loss=93.45169677734376, metrics={'train_runtime': 48.9011, 'train_samples_per_second': 0.204, 'total_flos': 2.6027104384512e+16, 'epoch': 5.0, 'init_mem_cpu_alloc_delta': 348007, 'init_mem_gpu_alloc_delta': 377847808, 'init_mem_cpu_peaked_delta': 18306, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 706705, 'train_mem_gpu_alloc_delta': 1120621568, 'train_mem_cpu_peaked_delta': 161498645, 'train_mem_gpu_peaked_delta': 7221921792})

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to resume_from_checkpoint for Seq2SeqTrainer of EncoderDecoderLM
Author: agi-templar
Pull request: no

Hi transformers developers:

First thanks to your great work!

I noticed there was a resume from checkpoint method for the Seq2SeqTrainer when the LM was BART-like out-of-the-box Seq2Seq LMs, like this:

https://github.com/huggingface/transformers/blob/fd338abdeba25cb40b27650ba2203ac6789d2776/examples/seq2seq/run_summarization.py#L529

I wonder how can I use it for a [EncoderDecoderLM](https://huggingface.co/transformers/model_doc/encoderdecoder.html), where the whole model has two parts (encoder and decoder). Should I load the checkpoint for encoder and decoder separately, or is there a nice way we can do that in one line like BART?

Thanks in advance!



Comments:
Hi @DapangLiu 

The `resume_from_checkpoint` should work for any `PreTrainedModel` class. Even though `EncoderDecoder` model is initialized using two sepearte models when save it using `save_pretarined` it's saved as a single model and can be loaded using `from_pretrained`.

Make sure that your checkpoint directory contains the saved model file, the saved optimizer and scheduler (if you also want to resume that) and it should work.

Solved my problem. Thanks @patil-suraj !
Problem with data download
Author: chatzich
Pull request: no

Hello can I ask which is the directory that the downloaded stuff is stored? I am  trying to bundle these data into a docker image and every time the image is built transformers is downloading the 440M data from the beggining

Comments:
@chatzich your question seems similar to this - #2323. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Update training.rst
Author: TomorrowIsAnOtherDay
Pull request: yes

# What does this PR do?
fix a typo in tutorial


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Create embeddings vectors for the context parameter of QuestionAnsweringPipeline for reusability.
Author: talhaanwarch
Pull request: no

Create embeddings vectors for the context parameter of QuestionAnsweringPipeline for reusability.
**Scinario**
For each time we pass question and context to QuestionAnsweringPipeline, the context vector is created. Is there a way to create this context for once and just pass the question to save time and make inference quicker. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Not very good answers
Author: Sankalp1233
Pull request: no

When I try to feed in this context from a long txt file lets say below:
[New_Spark_Questions.txt](https://github.com/huggingface/transformers/files/6273496/New_Spark_Questions.txt)
and I feed in a question from that txt file: Would police or the FBI ever be able to access DNA or other information collected?
it is not giving me a very good answer: I get Answer: 'help speed up the progress of autism research' with score 0.4306815266609192. How do we see all the cores available, how does the model decide which answer is the best?



Comments:
The text you are providing is probably too long for the model. Most Transformer models accept a sequence length of 512 tokens.

Which model did you use?

I just used the general q-a pipeline:
from google.colab import files
uploaded = files.upload() 
filename = "New_Spark_Questions.txt"
new_file = uploaded[filename].decode("utf-8")
!pip3 install sentencepiece
!pip3 install git+https://github.com/huggingface/transformers
question = "Why should I take part in SPARK?" 
from transformers import pipeline
qa = pipeline("question-answering")
answer = qa(question=question, context=new_file)
print(f"Question: {question}")
print(f"Answer: '{answer['answer']}' with score {answer['score']}")
How does the q-a pipeline decide the score?
Also how do we use a model like Bert, XLNet, etc. on a q-a pipeline
Does the input in the q-a pipeline have to be a dictionary?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Return output probabilities with Generate function 
Author: tshrjn
Pull request: no

# 🚀 Feature request

Output the token probabilities along with the tokens when generating sequence.

## Motivation

For understanding model confidence, this is quite useful. 
Also, for abstractive QA with long contexts, one needs to use doc-strides to take into account the contexts & then choose the best answer according to the probability of the generated text.


## Your contribution

I can try submitting a PR for non-beam decoding, but guidance would be appreciated.
Also, are there any existing solutions to this issue? If so, what & where?

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/7654 . But yes, it seems like many people are asking for this feature and it should be quite straight-forward to implement it... -> feel free to give a shot :-) 

Hi. When can we expect this to be released? 😊

Next week Tuesday or Wednesday :-) 

Hey! Can anyone point me to the API/code location of this feature? Sorry if I have missed something. :) Thank you!

@moqingyan https://huggingface.co/transformers/internal/generation_utils.html

> @moqingyan https://huggingface.co/transformers/internal/generation_utils.html

Thank you!

My problem was: I set the configuration `output_scores` to `True` in the `generate` function and failed obtain the scores in the returned results. After I struggled in the libraries for hours, I finally figured out I also need to set `return_dict_in_generate` to `True` to obtain the attention values from the `generate` function. :) 

I think this behavior is non-intuitive, as I have specified I need the scores in the output already. But anyway I figured this out. Hope this comment is helpful to anyone who runs into this issue. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.