noamaanMulla-03/transformers-issues huggingface dataset:
TPU CI testing
Author: zcain117
Pull request: yes

Run TPU CI testing using CircleCI.

I sent a guide in Slack with steps needed on the owner side for CircleCI and Google Cloud to make this work (setting env vars, creating GKE cluster, and populating the dataset).



Comments:
@LysandreJik I'm wondering if CircleCI does not run the pending changes to `.circleci/config.yml` if the changes came from a PR from a forked repo.

When testing on my private repo, I used a branch on the main repo and CircleCI did include the pending changes when running. I've seen lots of differences between branches on main repo and forked repo PRs when it comes to CircleCI and Github Actions.

The action item might be to remake this PR as a branch on the repo rather than a forked PR if you'd like to see the `job` run before submit.




> @LysandreJik I'm wondering if CircleCI does not run the pending changes to `.circleci/config.yml` if the changes came from a PR from a forked repo.
> 
> When testing on my private repo, I used a branch on the main repo and CircleCI did include the pending changes when running. I've seen lots of differences between branches on main repo and forked repo PRs when it comes to CircleCI and Github Actions.
> 
> The action item might be to remake this PR as a branch on the repo rather than a forked PR if you'd like to see the `job` run before submit.

I think the forked repo is irrelevant. I was able to run new CircleCI changes in a similar PR for PyTorch Lightning: https://github.com/PyTorchLightning/pytorch-lightning/pull/2486


Looks like this rebase went wrong - recreated the change in https://github.com/huggingface/transformers/pull/6158
Applying hugging face transformer in sequence labeling problem
Author: Michael95-m
Pull request: no

Hello.. Thanks for your great framework. Btw, what I'd like to know is that can I apply these hugging face transformer models in **sequence labeling** problems like **part of speech tagging** and **word segmentation**(because I see only **ner** model in example folder). If I can, **how** can I do that?? Can I get some helps like **example scripts** about how to apply these transformers for sequence labeling problems??

Comments:
you just use the ner code for POS tagging problem, the only difference is the set of target classes
When using "transformers.WarmUp" with tensorflow 2.0.0, warming up restart in each epoch!
Author: Wchenguang
Pull request: no

When using "transformers.WarmUp" with tensorflow 2.0.0, warming up restart in each epoch!
That is because, in keras "Callbacks.on_batch_begin(self, batch, logs)", batch start from zero in each epoch when using "fit" method.

Comments:
Hello!

Can you give a code that better explain your issue and a way to reproduce it please? Thanks :) 

Sorry! I check again and it works well!
Close it!
Initializing XLMRobertaTokenizer using pretrained tokenizer expects serialized vocab
Author: aoxolotl
Pull request: no

Hi,
I am training an XLMRoberta model from scratch on Hindi. I am using a sentencepiece tokenizer trained exclusively on monolingual data following the steps mentioned in the [tokenizers repository](https://github.com/huggingface/tokenizers/tree/704cf3fdd2f607ead58a561b892b510b49c301db/bindings/python#using-the-provided-tokenizers). This results in the creation of `vocab.json` and `merges.txt`.
However when I try to initialize the tokenizer using `XLMRobertaTokenizer.from_pretrained` I get an error saying
```assumed 'models/sentencepiece' was a path, a model identifier, or url to a directory containing vocabulary files named ['sentencepiece.bpe.model'] but couldn't find such vocabulary files at this path or url. ``` 
I am assuming this is a serialized file based on [huggingface.co model](https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-base-sentencepiece.bpe.model) but don't know how to serialize my vocab.json file. I have already tried using `pickle` and `numpy`
Versions used:
transformers: 2.9.1
tokenizers: 0.7.0

Comments:
Hi! The XLM-R tokenizer only accepts SentencePiece files, which cannot be created yet with the `tokenizers` library (soon!). You should use the official SentencePiece library for that.
Usage of Pytorch Native AMP in place of apex (Pytorch 1.6) in Trainer
Author: prajjwal1
Pull request: no

# 🚀 Feature request
It would be nice to remove the Apex dependency for `fp16` training and use the native [Pytorch's AMP methods](https://github.com/pytorch/pytorch/releases) in the `Trainer` method. Pytorch recommends Apex users to switch to it's native implementation, even [Apex does it so](https://github.com/NVIDIA/apex/issues/818). Moreover, it will eliminate the need for users to build apex by themselves. 

## Your contribution
I am happy to submit a PR if you think it would be a good addition. Please let me know.


Comments:
I had a query, Pytorch examples show loss being calculated as :
```
with autocast():
    output = model(input)
    loss = loss_fn(output, target)
scaler.scale(loss).backward()
```
But in all `SequenceClassification` and other models, loss is calculated in the `forward pass`. We can use `@autocast` decorator on the forward pass as the docs suggest, but this introduce so many changes for one feature. Maybe, there's a workaround. Does computing loss in `autocast` scope affect the loss itself when `backward` is called upon it ?

Hi there,

Note that we won't pin the version of PyTorch to 1.6 minimum, so the use of native mixed precision will have to be controlled by a test on the pytorch version (basically use native mixed precision when the version allows it and use apex otherwise). 

Otherwise, I don't think the loss being computed inside the model should be a problem, the line would probably be
```
with autocast():
    outputs = model(**inputs)
    loss = outputs[0]
```
inside Trainer but I haven't run tests yet.

You're welcome to try to work on a PR with this, otherwise it is on my TODO for when I have time (hopefully in the next few weeks).

Hi Sylvain,

I've opened up a PR. I know that pinning of version won't be done. To address, I've addressed this issue the same way as we handle `scheduler.get_lr()`.
Add Pytorch Native AMP support in Trainer
Author: prajjwal1
Pull request: yes

Pytorch 1.6 introduces Native AMP support. This eliminates the need to build and install Apex and provides improvements over problems highlighted in [Apex #818](https://github.com/NVIDIA/apex/issues/818) and flexibility. This is the recommended way to use AMP.

With this PR, Trainer will automatically use Pytorch's native AMP if 1.6 version is installed, otherwise, it will use Apex.

This PR will close [#6115](https://github.com/huggingface/transformers/issues/6115). 

Comments:
parse arguments from dict
Author: patil-suraj
Pull request: yes

This PR adds parse_dict method to HfArgumentParser to allow parsing arguments from dict

@julien-c 
As suggested by you here #4791, I've added a simple unit test to check if the dataclass returned by `parse_dict` is same as manually initialised one.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=h1) Report
> Merging [#4869](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9f5d5a531d769d07403f59661884e254f8420afe&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4869/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4869      +/-   ##
==========================================
+ Coverage   76.55%   76.57%   +0.01%     
==========================================
  Files         128      128              
  Lines       21502    21510       +8     
==========================================
+ Hits        16461    16471      +10     
+ Misses       5041     5039       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/hf\_argparser.py](https://codecov.io/gh/huggingface/transformers/pull/4869/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9oZl9hcmdwYXJzZXIucHk=) | `69.23% <100.00%> (+2.96%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4869/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.12% <0.00%> (-0.48%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4869/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.27% <0.00%> (+0.41%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4869/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.94%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=footer). Last update [9f5d5a5...4461f41](https://codecov.io/gh/huggingface/transformers/pull/4869?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @LysandreJik , what do you think about this ? If it's not really necessary, I will close the PR. Thanks!
[Benchmark]
Author: julio3361
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!


Comments:
`j`

Y

Grgj6r

Yes mi bro

Yes mi bro 

K

J
pass lowercase to fast tokenizer
Author: boy2000-007man
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/tokenization_gpt2.py#L338-L343
only passes four parameters without `lowercase` to
https://github.com/huggingface/tokenizers/blob/704cf3fdd2f607ead58a561b892b510b49c301db/bindings/python/tokenizers/implementations/byte_level_bpe.py#L15

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
do_lower_case strips accents!
Author: avacaondata
Pull request: no
Labels: name: wontfix

# 🐛 Bug

When calling BertTokenizer with do_lower_case=True, the tokenizer gets rid of accents, which is a misleading behavior not indicated in the name of the parameter. We suggest that you create another parameter which indicates whether or not to strip accents, separated from do_lower_case! This also happens in AutoTokenizer. For some languages, like spanish, this is crucial (hacia is not the same as hacía). Moreover, it's set to True by default. 
https://github.com/huggingface/transformers/blob/2184f87003c18ad8a172ecab9a821626522cf8e7/src/transformers/tokenization_bert.py#L346


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Well - is this realy a bug or just an improvement of the documentation?

In my opinion it's a bug as it's misleading. If you want to do lower that doesn't mean you want to strip accents too. Those are two separate actions which the user should decide separately.
Hidden State Embedding-Transformers
Author: DaniMlk
Pull request: no

Hi everybody, I want to use Bert model to get the embedding for a sentence after I fine-tuned it with raw texts. I was wondering if is that possible or not and anybody can help me with that? 

Comments:
Hi! Did you have a look at [this older issue](https://github.com/huggingface/transformers/issues/1950)?

> Hi! Did you have a look at [this older issue](https://github.com/huggingface/transformers/issues/1950)?

Yes I did, but my concern is that if I want to fine-tune it on my raw text data (language model with LM head) then how should I use it for sentence embedding? Can I just remove the LM head of it?

I think this is not necessary since, according to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm), you can access every hidden states of your model without removing the LM head.

> I think this is not necessary since, according to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm), you can access every hidden state of your model without removing the LM head.

Yes, thank you for your reply, I figured it out the language model network can be split into two parts which are 
```
self.bert = BertModel(config)
self.cls = BertOnlyMLMHead(config)
```
then I just need to get the output from self.bert if I want to access the hidden states.
t
Author: owowobread
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
bert embedding make OOM in albert
Author: urekalion
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help


## Details
hi, i made item collaborative filtering model by albert model.
i set vocab size to 600k, it made OOM. 
(after initialized model, memory used 3 GB. but During initialization, it takes 11 GB)

i tracked source code and i found this: 
albert model initialize albert word embedding layer after initializing bert word embedding layer.
embedding layer is initialized twice. so it made OOM. 
- bert word embedding : vocab size x hidden layer size(4048)
- albert word embedding : vocab size x embedding size(128) => bert word embedding free

is there any problem if i fix AlbertEmbeddings to nn.Module?

thanks

`
class AlbertEmbeddings(BertEmbeddings):
    """
    Construct the embeddings from word, position and token_type embeddings.
    """

    def __init__(self, config):
        super().__init__(config)

        self.word_embeddings = nn.Embedding(config.vocab_size, config.embedding_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.embedding_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.embedding_size)
        self.LayerNorm = torch.nn.LayerNorm(config.embedding_size, eps=config.layer_norm_eps)

`
`
class BertEmbeddings(nn.Module):
    """Construct the embeddings from word, position and token_type embeddings.
    """

    def __init__(self, config):
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Typo in GPT2 documentation 
Author: ELotfi
Pull request: no
Labels: name: wontfix

# 🐛 Bug

In the GPT2 documentation [page](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel), in the parameters of both LMHead and DoubleHeads model, 'inputs_embeds' is registered as 'input_embeds' which leads to an error upon implementation.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
OSError: Model name 'transfo-xl-wt103' was not found in tokenizers model name list (transfo-xl-wt103). We assumed 'transfo-xl-wt103' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.bin', 'vocab.txt'] but couldn't find such vocabulary files at this path or url.
Author: Amling2017
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0 
- Platform: ubantu
- Python version:3.6
- PyTorch version (GPU?):yes
- Tensorflow version (GPU?):yes
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tokenizer cannot load form model on disk
Author: makaveli10
Pull request: no

I wanted to load model saved on disk but it keeps on throwing this error

    File "train.py", line 1, in <module>
     import config
    File "/media/saurabh/D/code/bert_imdb_sentiment/src/config.py", line 14, in <module>
     do_lower_case=True
     File "/home/saurabh/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 393, in from_pretrained
     return cls._from_pretrained(*inputs, **kwargs)
     File "/home/saurabh/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 496, in _from_pretrained
     lst(cls.vocab_files_names.values()),
    OSError: Model name '.../input/bert_based_uncased/' was not found in tokenizers model name list 
    (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-
    multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased,   
    bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-
    uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-
    squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-
    dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-
    cased). We assumed '..input//bert_based_uncased/' was a path, a model identifier, or url to a 
    directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at 
    this path or url.

Comments:
@makaveli10 how to solve it ?
Request for hosting model files in a Virtual Hosted-Style S3 buckets
Author: rjsaito
Pull request: no
Labels: name: wontfix

Is there any plans for the s3 buckets currently hosting the model files to migrate from the current "Path-Style Request" format to a "Virtual Hosted-Style Request" format?

Path-Style URLs follow the following format (s3.amazonaws.com/* OR s3.Region.amazonaws.com/*). For example, today the model config file for 'bert_uncased_L-2_H-128_A-2' is accessed via the Path-Style URL:

https://s3.amazonaws.com/models.huggingface.co/bert/google/bert_uncased_L-2_H-128_A-2/config.json

According to AWS, they will be deprecating Path-Style requests (though there will be legacy support) - but one major reason for the migration to the Virtual Hosted-Style URL (which takes the form bucket-name.s3.amazonaws.com or bucket-name.s3.Region.amazonaws.com) is for security reasons (e.g. if companies/organizations need to whitelist sites in their servers to utilize transformer models, the virtual hosted style will reduce the "blast radius" in cases of security breaches).

More details on "Path" vs "Virtual-Hosted" style requests:

https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#path-style-access

Comments:
Hi @rjsaito we are actually moving to serving all our files from the cloudfront powered cdn.huggingface.co

Awesome! Do you have a current ETA when this change would be in place?

It's already in place for model weights.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.