noamaanMulla-03/transformers-issues huggingface dataset:
Trainer Evaluates at every step
Author: Megh-Thakkar
Pull request: no

Hi, thanks for the amazing and easy to use library. While using the Trainer with Training Arguments, the trainer is evaluating at every step, instead of eval_steps.
Version: 4.3.0
```python
training_args = TrainingArguments(output_dir='outputs', per_device_train_batch_size=1,per_device_eval_batch_size=2,
                                        evaluation_strategy='steps', do_eval=True,do_train=True, eval_steps=6)
```
Am I passing any wrong combination of arguments? None of the training_args are getting manually changed at any place in the code.

Thank you very much.

Comments:
It's hard to know without seeing your code. This combination of arguments should evaluate every 6 steps.
Also how do you know it's evaluation every step instead of every 6 steps?

Hi, thank you for the reply. I have actually overridden the 'evaluate' function of the trainer, and have certain print statements inside the function plus a tqdm progress bar as well. This function is executed after every training step.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Exporting transformers models in ONNX format
Author: chetanambi
Pull request: no

I am trying to convert transformer model to ONNX by referring the article **[here](https://huggingface.co/transformers/serialization.html)** but running into below error. Can you please guide me if this is not the correct way of doing it.

Code I am using in Colab:
```
!git clone https://github.com/huggingface/transformers.git
%cd transformers
!pip install .
%cd src/transformers
!python3 convert_graph_to_onnx.py --framework pt --model bert-base-cased bert-base-cased.onnx
```

Traceback (most recent call last):
  File "convert_graph_to_onnx.py", line 22, in <module>
    from .file_utils import ModelOutput, is_tf_available, is_torch_available
ModuleNotFoundError: No module named '__main__.file_utils'; '__main__' is not a package

Comments:
Hi!
Try this one, but do it from an empty folder:
`python3 -m transformers.convert_graph_to_onnx --framework pt --model bert-base-cased bert-base-cased.onnx`

@Denovitz Thanks a lot. I was able to convert the BERT model to ONNX successfully. Do you have a sample code of how the converted ONNX model can be used further for inferences? I am able to use ONNX for TF, Keras, Sklearn, Xgboost and other models but stuck with transformer model. Appreciate any inputs. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Bug in RAG Sequence generate 
Author: krishanudb
Pull request: no

IMHO there is a bug in the RAG Sequence model, in the generate function.
The shapes mismatch all the time. I looked into the code and found the issue in the following loop. 

https://github.com/huggingface/transformers/blob/85395e4901f87b880f364bcd6424fe37da94574b/src/transformers/models/rag/modeling_rag.py#L936

Kindly let me know if there is indeed a bug or is it just my code problem.
Thanks.

Comments:
Hi @krishanudb 

Could you post a code snippet so we can reproduce the issue? Please post your env info, short code snippet, and stack trace if possible when reporting bugs. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Remove Token from Vocab?
Author: BigSalmon2
Pull request: no

Is there a way I can remove a token from vocab.json?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Quick poll] Give your opinion on the future of 🤗 transformers: 40k edition!
Author: LysandreJik
Pull request: no
Labels: name: Discussion

Thanks to all of you, Transformers just passed 40k :star2:  this week!

Our libraries have always been about the community and we need your input to define the direction of the next 40k stars.

If you have a couple of minutes and want to participate in shaping the future of the library, please share your thoughts: https://forms.gle/FackvXzWJBWQz2WY8

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗

Comments:
Just did, thanks a lot @LysandreJik, the form is super quick to fill-in and interesting!
Everyone, we're waiting for you

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Conversion of BPE tokenizer for Marian models
Author: SaricVr
Pull request: no
Labels: name: Fast Tokenizers

Hello,

I was searching for the pt-en model of Marian and noticed that it has not been converted for the huggingface library apparently because it uses a BPE tokenizer. Is it possible to convert BPE-based models to be used in huggingface somehow?

Thank you

Comments:
cc'ing @n1t0 on this in case he didn't see it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Some model use serve previous version can not do inference in web api.
Author: svjack
Pull request: no

the model serve in
https://huggingface.co/wptoux/albert-chinese-large-qa
can not do inference by click “compute” button.
because it use transformers 3.0.2 but can not
properly load in current version.
I think model online server should consider its implement transformer version.

Comments:
The error here seems to be because there's a dissociated tokenizer and model. The tokenizer should be BERT while the model should be ALBERT.

The configuration should reflect this by having a `"tokenizer_class": "BertTokenizer"`.

Pinging @wptoux

An example can be seen with PhoBERT having the model set as RoBERTa and the tokenizer as `PhobertTokenizer`: https://huggingface.co/vinai/phobert-base/blob/main/config.json

> The error here seems to be because there's a dissociated tokenizer and model. The tokenizer should be BERT while the model should be ALBERT.
> 
> The configuration should reflect this by having a `"tokenizer_class": "BertTokenizer"`.
> 
> Pinging @wptoux
> 
> An example can be seen with PhoBERT having the model set as RoBERTa and the tokenizer as `PhobertTokenizer`: https://huggingface.co/vinai/phobert-base/blob/main/config.json

This library has improved a lot since I release this model, I will update it.

Glad to hear it @wptoux! Thank you!

I have fixed the problem, and the web api is working now.

Here is an test example

Context: 李白（701年—762年12月） ，字太白，号青莲居士，又号“谪仙人”，唐代伟大的浪漫主义诗人，被后人誉为“诗仙”，与杜甫并称为“李杜”，为了与另两位诗人李商隐与杜牧即“小李杜”区别，杜甫与李白又合称“大李杜”。北京大学教授李志敏评价：“李白之诗呼吸宇宙，出乎道；杜甫之诗德参天地，源于儒，皆至天人合一境界，故能出神入化。

Question: 如何评价李白的诗

Answer: 李白之诗呼吸宇宙，出乎道

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[docs] use `versionadded`, `versionchanged` and `deprecated` directive
Author: jeffydc
Pull request: no

# 🚀 Feature request

## Documentation
Use `.. versionadded::`, `.. versionchanged::` and `.. deprecated::` directive, so that user knows which features are added / changed / deprecated in which version and they can navigate the docs easily without changing the docs from version to version.

Ref: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionadded


## Motivation
To be able to know (without going to github) which features are introduced / changed / deprecated / improved in which version just from the docs.

Since `transfomers` is widely used in production, this slight change to the docs can help users see bird eye view for the features of the library.

Let me know what do you think


Comments:
Cool idea! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
urgent please help on memory issue during save 
Author: ghost
Pull request: no

Hi
I am getting very large memory usage during saving model/evaluation of T5, resulting in job kill, this is very urgent as I lose access to train the models please help 
thanks

Comments:
Hi @juliahane 

It would be hard to answer without knowing the details 
Could you post the command that you are using, env info, which T5 model, training details etc ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix init
Author: patelrajnath
Pull request: yes

# What does this PR do?
"RobertaPreTrainedModel" is missing in models' __init__.py. It is needed, in case we need to create a subclass of the same like "RobertaForTokenClassification".

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 Model Cards: @julien-c
 -->


Comments:
Yes, let's wait for #9446 to be merged please :-)

@LysandreJik yes, I'm not sure regarding the CI errors, as I'm not that much into it. I ran the tests in Pycharm, they passed. Can we check the Logs if we could find some clue there?


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Apache Hadoop (HDFS) File Loading from_pretrained
Author: mfahadyousaf
Pull request: no

# 🚀 Feature request

Loading configuration file for **transformers.AutoConfig** and **transformers.AutoModelForSequenceClassification** using the function **from_pretrained** by giving the HDFS file path

## Motivation
In case of file that is not locally available, the library utilizes the **get_from_cache** function inside **transformers/file_utils.py** file to try to download the model from the remote resource. But, in case of no ETag being present in the header of the response returned, an OSError("Distant resource does not have an ETag, we won't be able to reliably ensure reproducibility.") exception is raised. In case of HDFS, this ETag validation shouldn't be considered as a mandatory requirement but as an optional one. Either another mechanism should be used to ensure the reliability of the resource or this ETag check should be made optional.
Please see below the code-snippet screenshot of the aforementioned file.
![image](https://user-images.githubusercontent.com/44267622/103738038-262dc080-5015-11eb-98f4-047be7b81bdd.png)


Additional information.
Apache Hadoop Version: 2.7.7, rc1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Flax] Bugfixes in `run_mlm_flax.py`
Author: TevenLeScao
Pull request: yes

# What does this PR do?

This PR fixes a few bugs I have observed when using `run_mlm_flax.py`:
- As discussed with @mfuntowicz , `jnp.split` is a lot slower than `np.split` on the first iteration, outright hanging in my tests on simplewiki (~20MB). As this operation doesn't need to be traced. we can use `np.split` instead.
- When using a HF `datasets`, the text column was also passed to the model as input, causing a bug. The PR removes the text column in `dataset.map` to avoid this.
- Finally, using `warmup_steps = 0` (as is default) causes the Flax optimizer to output NaNs. We use 1 as a minimum value for the same warmup-less behaviour.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Some Models do not support gradient checkpointing
Author: jingtaozhan
Pull request: no

Thanks for this wonderful library. 

I found some models do not support gradient_checkpointing, which I believe is a very important feature. For example,

ElectraModel: ElectraConfig has no gradient_checkpointing option but ElectraModel will use gradient_checkpointing if config.gradient_checkpointing = True
DistillBERT: DistillBertConfig has no gradient_checkpointing option and DistillBertModel does not support gradient_checkpointing.

I assume all transformer-based models should be able to support gradient_checkpointing.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Allow to provide specific params in WandbCallback
Author: raphael0202
Pull request: no

# 🚀 Feature request

It would be nice to be able to track additional params in wandb when using the Trainer interface. For example, I need to track down how many layers were frozen in each experiment. I'm currently using a custom WandCallback class to do this.

## Your contribution

I can work on a PR.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Sorry I just noticed this issue.
You can actually already do it. After your run has been created, you can do `wandb.config['frozen_layers'] = 3`
We need to add a way to let you also create a run first (instead of `Trainer`) and then let `Trainer` adds automatically the extra configuration parameters.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
It seem do not support convert multilabel classification model  to onnx  ?
Author: MrRace
Pull request: no

I have fine tune my multilabel classification model(pytorch)，and try to use the `from transformers.convert_graph_to_onnx import convert` to convert the pytorch model to onnx model. As the `pipeline_name` is pre-defined, and it seems not  be suited to multilabel classification， therefor I try use `pipeline_name="sentiment-analysis"`, however when reload the onnx model, its prediction result seems wrong. Could you tell how should i do to get the right result？ Thanks a lot!


Comments:
Hi @MrRace, 

thanks for reporting the issue. 

What model are you using as multilabels classification support?

> Hi @MrRace,
> 
> thanks for reporting the issue.
> 
> What model are you using as multilabels classification support?

Thanks for your reply. I use BERT to do multilabels classification. And want to export the fine tuned model to onnx. 

Looking at it!

> Looking at it!

Looking forward to your reply!

Hi @mfuntowicz  Is there any suggestion for it?



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
@ehsan-soe I fixed the problem by truncating incomplete batches. So if there are 2001 examples and my batch size = 2, then I truncate the last example and train on the first 2000. This has fixed it for me both with and without distributed. My load_and_cache function now looks like this
Author: ChaooMa
Pull request: no
Labels: name: wontfix

@ehsan-soe I fixed the problem by truncating incomplete batches. So if there are 2001 examples and my batch size = 2, then I truncate the last example and train on the first 2000. This has fixed it for me both with and without distributed. My load_and_cache function now looks like this
```
def load_and_cache_examples(args, tokenizer, evaluate=False, fpath=None):
    if fpath:
        dataset = TextDataset(tokenizer, args, fpath)
    else:
        dataset = TextDataset(tokenizer, args, args.eval_data_path if evaluate else args.train_data_path)

    # Ignore incomplete batches
    # If you don't do this, you'll get an error at the end of training
    n = len(dataset) % args.per_gpu_train_batch_size
    if n != 0:
        dataset.examples = dataset.examples[:-n]
    return dataset
```

_Originally posted by @isabelcachola in https://github.com/huggingface/transformers/issues/1220#issuecomment-557237248_

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Issues with finetune_trainer.py on multiple gpus
Author: rabeehk
Pull request: no
Labels: name: wontfix

## Environment info     
- `transformers` version:3.5.1
- Platform: google cloud
- Python version: 3.7
- PyTorch version (GPU?): yes 1.6
- Tensorflow version (GPU?): - 
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: - 

### Who can help
 Trainer: @sgugger
 Text Generation: @patrickvonplaten @TevenLeScao
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information
Hi
I am trying to train finetune_trainer.py on multiple gpus, one issue I got is 

1) Runtime error: Input,ouptut,and indices must be on the current device, 
Looking into the codes, in training_args.py when you set device for the n_gpu > 0 on line 401, this should be changed from cuda:0 to cuda to me.

2) The accuracy on multiple gpus does not match on single gpu and this is much lower, any idea on this.

Thank you. 



Comments:
Hi there. There is little anyone can do to help without knowing the actual command you are running.


Hi
I am running
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_trainer.py
once
with multiple gpu machines once with 1 machine,
I have adapted it for my usecase, this is hard for me to provide the exact
command, as I need to share the whole codebase.
My question is though more general I observe performance differences if you
run this code on 1 dataset with multiple gpu/gpu, any thoughts on this?
thanks
Rabeeh

On Mon, Nov 23, 2020 at 1:16 AM Sylvain Gugger <notifications@github.com>
wrote:

> Hi there. There is little anyone can do to help without knowing the actual
> command you are running.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8718#issuecomment-731871920>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCHAF5CQRJP2LDI2XQDSRGSUPANCNFSM4T6X5WFQ>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.