noamaanMulla-03/transformers-issues huggingface dataset:
cannot import name 'TFBertForSequenceClassification'
Author: samarthsarin
Pull request: no

I am unable to import TFBertForSequenceClassification.

from transformers import TFBertForSequenceClassification shows an error of cannot import name 'TFBertForSequenceClassification'

Comments:
Hi! The TensorFlow components are only available when you have TF2 installed on your system. Could you please check that you have it in the environment in which you're running your code?

It worked. Thanks
Sequence length more than 512
Author: nayakt
Pull request: no
Labels: name: wontfix

Hi,

My dataset has sequence with more than 512 words and when use wordpieces sequence length goes beyond 512. How to handle this issue with BERT ?

Regards
Tapas

Comments:
https://github.com/google-research/bert/issues/27#issuecomment-435265194

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
cannot import name 'XLNetForMultipleChoice' but python can import
Author: yyHaker
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert):

Language I am using the model on (English):

when I use the following command to run run_multiple_choice.py, like:
'''
python examples/run_multiple_choice.py --model_type bert --task_name race --model_name_or_path bert_large --do_train --do_eval --do_lower_case --data_dir $RACE_DIR --learning_rate 5e-5 --num_train_epochs 3 --max_seq_length 80 --output_dir models_bert/race_base --per_gpu_eval_batch_size=16 --per_gpu_train_batch_size=16 --gradient_accumulation_steps 2 --overwrite_output
'''
it gives the error information:
![image](https://user-images.githubusercontent.com/18585628/65219953-479b2e00-daec-11e9-9ecf-fd85bc8e5e34.png)
but when I in my python environment to import the package , it has no problem!
![image](https://user-images.githubusercontent.com/18585628/65220000-626da280-daec-11e9-935b-4cd951474b39.png)
what's wrong with the run_multiple_choice.py?
## Environment

* OS:
* Python version: 3.6.2
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? yes
* Distributed of parallel setup ? 
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
I found maybe the current code is not consistent with the pip package pytorch_transformers, so when use the pip package it does't work, but when just run the code without the pip package, it can work, but you need change some path to make the code work correctly!

Hi, I believe this was fixed with @VictorSanh's commit ae50ad9
PreTrainedTokenizer.from_pretrained should be more general
Author: matt-gardner
Pull request: no
Labels: name: wontfix

I'm trying to implement a general interface to any of these transformer models in AllenNLP.  I would love to be able to do something like `PreTrainedTokenizer.from_pretrained(model_name)`, and have this work for any model name across any of your implemented models.  It looks like what needs to happen for this is to detect which underlying model is being requested, and pass off to that class's `_from_pretrained` method.  Does this make sense?  In particular, I think the thing that needs to change is here: https://github.com/huggingface/pytorch-transformers/blob/067923d3267325f525f4e46f357360c191ba562e/pytorch_transformers/tokenization_utils.py#L149-L151

It could be changed to something like:

```python
    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs):
        if 'bert' in pretrained_model_name_or_path:
            # import BertTokenizer here to avoid circular dependencies
            return BertTokenizer._from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
        # other cases here
        # default to existing behavior if we can't detect the model class
        return cls._from_pretrained(*inputs, **kwargs)
```

If this looks right to you, I can put together an initial PR for this.

It would be super helpful if there were similar functionality for models, too, but I haven't gotten far enough yet to worry about how exactly that would work =).

Comments:
Yes, this is a nice idea, I was thinking about implementing something like this for another reason (simplifying the task of maintaining `torch.hub` configuration files).

Regarding the library architecture, I think it's better to make a new (very simple) class, something like `AutoTokenizer` in a new file `tokenizer_auto.py` deriving from `PreTrainedTokenizer` (no need to avoid circular dependencies in this case).

Then the idea would be to make a new file `modeling_auto.py` as well with something like `AutoModel`, pretty much like `AutoTokenizer`, and `AutoModelForSequenceClassification`, `AutoModelForQuestionAnswering` that would encapsulate standard architectures on top of each model.

Maybe the `AutoXXX` is not the best name, also thought about `GenericXXX` or `UniversalXXX` but they convey meanings that could be misleading.

`{Generic,Universal,Auto}` Tokenizer and Model interfaces would be awesome (I'm also highly interested in that, as I'm currently working on Flair to add support for all six architectures) :heart: 

Ok, I've hacked something together for an internal hackathon this week.  I'll see if I can pick this up the way you suggest next week, if no one else gets to it first.  I also don't know much about your PR requirements, so if someone who's more familiar with this repo wants to pick it up, I wouldn't complain =).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to restore a training?
Author: bingyupiaoyao
Pull request: no
Labels: name: wontfix

For example, I use "run_glue.py" to train a model and stop at Epoch 30, and how to restore the training process from Epoch 30?

Comments:
You will have to modify the provided example to save/reload the model, optimizer and scheduler states.

I've updated the scheduler classes in #872 so that we can save/reload the schedulers with the standard PyTorch serialization practice:
```
torch.save(schedule.state_dict(), FILE_NAME)           # save
schedule.load_state_dict(torch.load(FILE_NAME))    # reload
```

Thanks for the response~

@thomwolf - I happen to need save/resume training for `run_glue.py`; I'm willing to implement this and make a PR if I can get feedback about the overall approach.

It looks like I would want to save:

- `global_step`
- `step` (from this one could presumably skip to the right place in the `epoch_iterator` of `train`)
- optimizer, model, scheduler (these look like they should be trivially amenable to `torch.save`)
- random seed, torch's random seed, numpy's random seed

Based on the current set up, I'm not actually sure what the right way to preserve the `train_sampler` and `train_dataloader` states are (without also serializing them too, which seems like a waste, but is by far the easiest way to handle it if they are so amenable), since their states can't be recreated with the above information. In my case, punting on these is an acceptable option, as well as being wasteful and saving to disk.

Thoughts?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update README.md
Author: Santosh-Gupta
Pull request: yes

Lines 183 - 200, fixed indentation. Line 198, replaced `tokenizer_class` with `BertTokenizer`, since `tokenizer_class` is not defined in the loop it belongs to.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=h1) Report
> Merging [#1369](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa?src=pr&el=desc) will **increase** coverage by `0.92%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1369/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1369      +/-   ##
==========================================
+ Coverage   83.76%   84.69%   +0.92%     
==========================================
  Files          84       84              
  Lines       12596    12596              
==========================================
+ Hits        10551    10668     +117     
+ Misses       2045     1928     -117
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1369/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `72.14% <0%> (+0.89%)` | :arrow_up: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1369/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `71.22% <0%> (+5.75%)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1369/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `95% <0%> (+7.5%)` | :arrow_up: |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1369/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `76.92% <0%> (+66.43%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=footer). Last update [ae50ad9...d1176d5](https://codecov.io/gh/huggingface/transformers/pull/1369?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks for updating the README!
Chunking Long Documents for Classification Tasks
Author: anassalamah
Pull request: no

## 🚀 Feature

A way to process long documents for downstream classification tasks. One approach is to chunk long sequences with a specific stride similar to what is done in the run_squad example.

## Motivation

For classification tasks using datasets that are on average longer than 512 tokens, I believe it would improve performance.

## Additional context

https://github.com/google-research/bert/issues/27#issuecomment-435265194

Comments:
I'm not sure that I understand. As you say, you can see it implemented in the run_squad example. What else would you like? 

Hello Bram,

I mean I want to apply it with a sequence classification task like BertForSequenceClassification, for example, versus what is being done in squad.

I don't think it should be too hard but I'm not exactly sure how a long document that is being chunked gets trained. Do we ignore the fact that these are chunks of the same document and just treat them as independent docs? Or do we do some sort of trick to join the tokens/embeddings with the first chunk? 

How would this be implemented for sequence classification?

I quickly glared over the `convert_examples_to_features` function, and it seems that given some stride different parts are used as input. So, yes, as far as I can see they are treated as independent docs.

https://github.com/huggingface/transformers/blob/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa/examples/utils_squad.py#L189-L397



isn't there a way to deal with long documents without ignoring the fact that the chunks represent the same doc? 

Maybe something along the lines of https://finetune.indico.io/chunk.html?highlight=long or https://explosion.ai/blog/spacy-pytorch-transformers#batching

After a first look, I don't see how `spacy-pytorch-transformers` does anything special rather than processing a document sentence-per-sentence. `finetune`'s approach might be what you after (taking the mean over all the slided windows), but as always: "a mean is just a mean", so the question remains how representative it is of the whole document. I am not saying that slicing is _better_ by any means, but averaging can distort "real" values greatly.

Yeah I see your point. I'm starting to think that maybe trying out chunking with a couple of different strides and maybe at inference time taking a voting approach would be a better option.

In any case, thank you for your feedback!

I agree that that might be the more efficient approach. No worries, thanks for the interesting question. If you think it's okay the question, please close it so it's easy to keep track of all open issues.

Hi, just to let you know that there is an option to manage strides in the `encode_plus` method. It handles special tokens and returns the overflowing elements in the `overflowing_tokens` field of the returned dictionary.
Using pytorch-transformer to reimplement the "Attention is all you need" paper
Author: roholazandie
Pull request: no

## ❓ Questions & Help
I use this repo for a long time but I realized even though the name is PyTorch transformers I can't find an easy way to re-implement the original paper of "Attention is all you need" with pretrained model. Can someone help me?

Comments:
Hi, this repository's objective is mainly to host **pretrained** models, not really to build a model from scratch.

You could use some of this library's components though, like multi-headed attention, to help you in your endeavor.
run_tf_glue.py breaks when changing to a glue dataset different from mrpc
Author: philipp-eisen
Pull request: no

## 🐛 Bug - run_tf_glue.py breaks when changing to a glue dataset different from mrpc

<!-- Important information -->

[run_tf_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py) breaks when changing to a glue dataset different from `mrpc`, where the features are not called `'sentence1'` and `'sentence2'`. That happens because of the hard coded accesses in the tensor_dict https://github.com/huggingface/transformers/blob/ca559826c4188be8713e46f191ddf5f379c196e7/transformers/data/processors/glue.py#L83

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SST-2
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py#L11 
2. Change `mrpc` to `sst-2`
3. 💥BOOM! broken

## Expected behavior
* [ ] Handle all glue datasets from `tensorflow_datasets` correctly

P.S.: A colleague and I are currently working on a fix and will submit a PR for this issue in the next couple of minutes.


Comments:
Fixed with #1355
Built-in pretrained models location
Author: kaunghtetsan275
Pull request: no

My laptop was run out of disk space while loading built-in pre-trained model.

Now BertForTokenClassification.from_pretrained("bert-base-cased") gives me RuntimeError: unexpected EOF, expected 5896093 more bytes. The file might be corrupted.

Where can I find that incomplete model and delete it so I can download the model from the start again?

Comments:
I've found it. It's a binary file. It's in ~/.cache/torch/transformers
Fix syntax typo in README.md
Author: dnahurnyi
Pull request: yes

![image](https://user-images.githubusercontent.com/27808442/65991735-8343d980-e496-11e9-90b4-bfbd61d02de0.png)


Comments:
Thanks :)

Your welcome, you are doing a great job!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=h1) Report
> Merging [#1396](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5c3b32d44d0164aaa9b91405f48e53cf53a82b35?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1396/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1396   +/-   ##
=======================================
  Coverage   84.69%   84.69%           
=======================================
  Files          84       84           
  Lines       12596    12596           
=======================================
  Hits        10668    10668           
  Misses       1928     1928
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=footer). Last update [5c3b32d...6b92911](https://codecov.io/gh/huggingface/transformers/pull/1396?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix fp16 masking in PoolerEndLogits
Author: slayton58
Pull request: yes

Necessary to run xlnet squad fine-tuning with `--fp16 --fp16_opt_level="O2"`, otherwise loss is immediately `NaN` and fine-tuning cannot proceed.

Similar to #1249 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=h1) Report
> Merging [#1284](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2dc8cb87341223e86220516951bb4ad84f880b4a?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1284/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1284      +/-   ##
==========================================
- Coverage   84.69%   84.68%   -0.01%     
==========================================
  Files          84       84              
  Lines       12596    12598       +2     
==========================================
+ Hits        10668    10669       +1     
- Misses       1928     1929       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1284/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `92.44% <66.66%> (-0.25%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=footer). Last update [2dc8cb8...c50783e](https://codecov.io/gh/huggingface/transformers/pull/1284?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM @slayton58 thanks
XLNet tokenizer returns empty list instead of string for some indexes
Author: andrey999333
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

XLNet

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Create tokenizer: tokenizer = XLNetTokenizer.from_pretrained(path)
2. Try to decode index 4: tokenizer.decode(4)
3. You get empty list [] although expected is a string

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

Decode method of the tokenizer should return a string and not a list.

## Environment

* OS:
* Python version: 3.7
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0


Comments:
Hi,
`tokenizer.decode()` expect a sequence of ids as indicated in the doc/docstring: https://huggingface.co/pytorch-transformers/main_classes/tokenizer.html#pytorch_transformers.PreTrainedTokenizer.decode

well, as far as i see, the tokenizer can accept both - sequence or single index. For example:
`tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')`
`tokenizer.decode(3)`
produce `'<cls>'` as well as `tokenizer.decode([3])`.

And the problem happens in both cases:
`tokenizer.decode([4])` produces empty string. And, what is even more strange, if index 4 is in the sequence, the result is a list instead of a string. For example:

`tokenizer.decode([35,109])` gives `'I new'`, but `tokenizer.decode([35,109,4])` generates list instead of a string `['I new']`

ok, it looks like 4 is an index of a separation token and it turns one sequence into sequence of sequences each of which will be decoded into string. So if index 4 presents in the sequence, the result will be list of strings. This behavior is quite unintuitive. If it was created purposely, should be properly documented. I've got very strange bug in my code that was hard to track, since index 4 was predicted very rarely and out of 500 predictions 1 was buggy.  

Thanks for the bug report. Indeed this was an unwanted behavior. Fixed on master now.