noamaanMulla-03/transformers-issues huggingface dataset:
Support multiprocessing when loading pretrained weights
Author: rmrao
Pull request: no
Labels: name: wontfix

## 🐛 Bug

So this is an issue that probably won't crop up for *too* many people, but there's a synchronization issue in loading pretrained weights if doing so in a multiprocess setting if they are not present in the cache.

For context, I'm trying to use `torch.distributed.launch` and doing so inside a fresh docker container which doesn't have cached weights. When doing this, each process looks for the weights in the cache and starts downloading them. They then all try to copy the files to the same place. I suppose `shutil.copyfileobj` is not thread-safe, because this leads to a corrupted weight file.

A simple, easy solution would be to add a check _after_ the file is downloaded as well. So you could wrap [these lines in `pytorch_transformers/file_utils.py`](https://github.com/huggingface/pytorch-transformers/blob/df9d6effae43e92761eb92540bc45fac846789ee/pytorch_transformers/file_utils.py#L252-L262) in a second condition like this:

```python 
if not os.path.exists(cache_path):
    # Download File
    if not os.path.exists(cache_path):  # second check for multiprocessing
       # Copy to cache_path
```

A better solution might be to detect the multiprocessing and only download the file once? I think `torch.distributed` could help here, but it would probably be hard to handle all the possible use cases.

Comments:
Hi! Indeed, you have to be careful when downloading the models in a multiprocessing manner so that you do not download them several times. 

You can see how we do it in our examples (like this [run_glue example)](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L429-L438), where we manage it with the `barriers` that come with `torch.distributed`.

Yup, found that right before you commented 😄 

Is there a reasonable way to include this within the download script itself? Or a place in the README to mention this?

If not, feel free to close.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
transformers module doesn't work with torch compiled on Cuda 10.0?
Author: ehsan-soe
Pull request: no

Hi,

I have been using this repo for long time and it was working okay. Very recently it gave me ```ModuleNotFoundError: No module named 'transformers'``` error and I had to reinstall it from source.
It seems like during installation my torch version (or the Cuda it is compiled on) got updated.
I reinstall my PyTorch with cudatoolkit=10.0 since I have Cuda 10.0 and I cannot update it now.
But I realized doing that, I am not able to import transformers again.

Any idea what should I do?


Comments:
Solved! closing.
Show pretrained model and config file download address directly in README.md & doc
Author: Sunnycheey
Pull request: no

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
Show download address of pretrained model address directly in markdown file.
## Motivation
Since my server cannot directly go to aws server, and it's not configed to proxy too. Intuitively i need to download pretrained model in my computer and then upload it to my server. The question is I need to checkout source code for download address. It's a really bad experience.
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Comments:
I feel that this would clutter the README, leading to a bad experience for 99.99% of users. But you can always submit a PR and see what the maintainers think.

Just make a PR.
Retaining unknown token behaver consistency in tokenizer for BERT and XLNET
Author: ziliwang
Pull request: yes

I found the BERT tokenizer and XLNET tokenizer behave difference.
for example, `"His name is 燚"`
In BERT, it's tokenized as:` ['his', 'name', 'is', '[UNK]'],`
but in XLNET, it's tokenized as:` ['▁His', '▁name', '▁is', '▁', '燚']`
The difference may adverse to the uniform training frame for BERT and XLNET, and trouble the length-based processing.

Comments:
The implementation of grad clipping is not correct when gradient accumulation is enabled
Author: yangyiben
Pull request: no

torch.nn.utils.clip_grad_norm_ should be applied before optimizer.step() not after each backward



## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This should be fixed in https://github.com/huggingface/transformers/pull/1580

yeah @yangyiben please let me know if that merge fixes it

#1580 is now merged
[CLS] & [SEP] tokens missing in documentation
Author: hawkeoni
Pull request: no

https://github.com/huggingface/transformers/blob/fd97761c5a977fd22df789d2851cf57c7c9c0930/transformers/modeling_bert.py#L1017-L1023
In this example of bert for token classification input sentence is encoded, but [CLS] & [SEP] tokens are not added. Is this intentional or just a typo?
Do I need to add [CLS] & [SEP] tokens when I fine tune base bert for sequence classification or token classification?

Comments:
@hawkeoni  [CLS] and [SEP] tokens are added automatically as long as you use the tokenizer,  ```BertTokenizer```

@enzoampil It doesn't seem to work.
The following code
```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
sentence = "Hello there, General Kenobi!"
print(tokenizer.encode(sentence))
print(tokenizer.cls_token_id, tokenizer.sep_token_id)
```
produces the next output: 

[7592, 2045, 1010, 2236, 6358, 16429, 2072, 999]
101 102

As you can see, cls and sep tokens are not in the list.

@hawkeoni please try ```print(tokenizer.encode(sentence, add_special_tokens=True))```

@enzoampil I'm sorry, but you're missing the point that the documentation is plainly wrong and misleading. That's why I asked whether the tokens should be added at all.

@hawkeoni Apologies, yes I did miss your point.

Is this intentional or just a typo? **Looks like a typo since special tokens weren't added. Setting ```add_special_tokens=True``` should make this correct (will add this in).**

Do I need to add [CLS] & [SEP] tokens when I fine tune base bert for sequence classification or token classification? **Yes, I believe this is currently handled by ```load_and_cache_examples``` in the sample training scripts (e.g. ```run_ner.py```)**

@enzoampil Thanks for your answer! If you plan on fixing this typo, please, fix it everywhere, so this issue never occurs again.
You can find it with 
```bash
grep -iR "input_ids = torch.tensor(tokenizer.encode(" .
```


@hawkeoni thanks for the bash script reco. Ended up using it :)

Thank you both for that.
Please note that the special tokens will be added by default from now on (already on master and in the coming release).
When I uesd gpt2, I got a error
Author: ruidongtd
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
`  File "test.py", line 92, in <module>
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 282, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 411, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 122, in __init__
    self.encoder = json.load(open(vocab_file, encoding="utf-8"))
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 295508 (char 295507)`



Comments:
We had a temporary issue with our `gpt2` model, should be fixed now.

I reinstalled `transformers` (`pip install transformers`), and the error remains.

> We had a temporary issue with our `gpt2` model, should be fixed now.
I reinstalled transformers (pip install transformers), and the error remains.


> We had a temporary issue with our `gpt2` model, should be fixed now.

??

use `force_download` option
where can i assign step in function lr_lambda of Class WramupLinearSchedule?
Author: lsy641
Pull request: no
Labels: name: wontfix

As i know LambdaLR's get_lr function recieve last_epoch as parameter, so where does it get the step of current training?


class WarmupLinearSchedule(LambdaLR):
    """ Linear warmup and then linear decay.
        Linearly increases learning rate from 0 to 1 over `warmup_steps` training steps.
        Linearly decreases learning rate from 1. to 0. over remaining `t_total - warmup_steps` steps.
    """
    def __init__(self, optimizer, warmup_steps, t_total, last_epoch=-1):
        self.warmup_steps = warmup_steps
        self.t_total = t_total
        super(WarmupLinearSchedule, self).__init__(optimizer, self.lr_lambda, last_epoch=last_epoch)

    def lr_lambda(self, step):
        if step < self.warmup_steps:
            return float(step) / float(max(1, self.warmup_steps))
        return max(0.0, float(self.t_total - step) / float(max(1.0, self.t_total - self.warmup_steps)))

Comments:
Hi, to use a scheduler you have to tell it when to perform an optimization step, as detailed on the [pytorch documentation](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate). It increases the step by one every time you call `scheduler.step()`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Threads running on evaluation?
Author: vivekam101
Pull request: no

Hi Transformers :)

Thanks for the wonderful repo. One quick question. I was running examples/test_examples.py and came to see logs printing multiple times while running in cpu. Is there threads spawned ? 
snippet from examples/test_examples.py
**
        testargs = ["run_squad.py",
                    "--train_file=./tests_samples/SQUAD/dev-v2.0-small.json",
                    "--predict_file=./tests_samples/SQUAD/dev-v2.0-small.json",
                    "--model_name=bert-base-uncased",
                    "--output_dir=./tests_samples/temp_dir",
                    "--max_steps=10",
                    "--warmup_steps=2",
                    "--do_train",
                    "--do_eval",
                    "--version_2_with_negative",
                    "--learning_rate=2e-4",
                    "--per_gpu_train_batch_size=2",
                    "--per_gpu_eval_batch_size=1",
                    "--overwrite_output_dir",
                    "--seed=42"]
***

Result where i see multiple logs prints
-----
11/05/2019 07:11:33 - INFO - utils_squad -   *** Example ***
*** Example ***
*** Example ***
11/05/2019 07:11:33 - INFO - utils_squad -   unique_id: 1000000013
unique_id: 1000000013
unique_id: 1000000013
11/05/2019 07:11:33 - INFO - utils_squad -   example_index: 13
example_index: 13
example_index: 13
11/05/2019 07:11:33 - INFO - utils_squad -   doc_span_index: 0
doc_span_index: 0
doc_span_index: 0


Comments:
*attaching complete log
[test_log.txt](https://github.com/huggingface/transformers/files/3806959/test_log.txt)


Sorry. guys. logger has been called multiple times hence the issue closing the issue.
--cache_dir argument in run_lm_finetuning.py not used at all
Author: mpavlovic
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT-2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: run_lm_finetuning.py

The tasks I am working on is:
* [ ] my own task or dataset: Language model finetuning on custom dataset from human resources domain

## To Reproduce

Steps to reproduce the behavior:

1. Clone the repo
2. Navigate to transformers/examples directory
3. Prepare custom train and test datasets (.txt files)
4. Create ./cache directory
3. Run the following command in terminal (with replaced custom_ arguments):
```
python run_lm_finetuning.py \
   --output_dir=<custom_output_dir_path> \
   --model_type=gpt2 \
   --model_name_or_path=gpt2 \
   --do_train \
   --train_data_file=<custom_train_data_file> \
   --do_eval \
   --eval_data_file=<custom_eval_data_file> \
   --per_gpu_eval_batch_size=1   \
   --per_gpu_train_batch_size=1   \
   --save_total_limit=2 \
   --num_train_epochs=1 \
   --cache_dir=./cache
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

When the model is downloaded from S3, it is stored to default cache directory in `<user_home>/.cache/transformers/` directory, instead to `./cache`, as specified in `--cache_dir` argument. Seems like `--cache_dir` argument isn't used in `.from_pretrained()` methods in lines 472, 473 and 477 in the run_lm_finetuning.py script.

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.6
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Fixing mode in evaluate during training
Author: oneraghavan
Pull request: yes

This fixs the error of not passing mode while using the option of evaluate_during_training option.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=h1) Report
> Merging [#1709](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8a62835577a2a93642546858b21372e43c1a1ff8?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1709/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1709   +/-   ##
=======================================
  Coverage   85.14%   85.14%           
=======================================
  Files          94       94           
  Lines       13920    13920           
=======================================
  Hits        11852    11852           
  Misses       2068     2068
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=footer). Last update [8a62835...e5b1048](https://codecov.io/gh/huggingface/transformers/pull/1709?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok thanks!
Retaining unknown token behaver consistency in tokenizer for BERT and XLNET
Author: ziliwang
Pull request: yes

I found the BERT tokenizer and XLNET tokenizer behave difference.
for example, `"His name is 燚"`
In BERT, it's tokenized as: `['his', 'name', 'is', '[UNK]']`,
but in XLNET, it's tokenized as: `['▁His', '▁name', '▁is', '▁', '燚']`
The difference may adverse to the uniform training frame for BERT and XLNET, and trouble the length-based processing.

Comments:
Hi, thanks for this but this was the expected behavior for the Bert tokenizer.
TFBertForSequenceClassification.from_pretrained ERROR
Author: dkicenan
Pull request: no

### code:
model = TFBertForSequenceClassification.from_pretrained('./bert-base-cased-tf_model.h5')

### Error message:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 212, in from_pretrained
    **kwargs
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/configuration_utils.py", line 154, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/configuration_utils.py", line 186, in from_json_file
    text = reader.read()
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

### Environment
py:3.6
tf:2.0.0

### 
I haved download "bert-base-cased-tf_model.h5" "bert-base-cased-vocab.txt" "bert-base-cased-config.json" in the same dir. when I load model "from_pretrained", I got error above. Please help me what 's wrong and how to fix it. Thanks!









Comments:
Just give the path to your folder instead of the model so the loading method can also find the configuration files.

You can read more [here](https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained)

@thomwolf   Thank you for your help! I can load the pretrained model now.
About distilled the SQuAD?
Author: renxingkai
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Thank you for your excellent work. I want to know that have you released the distilled model code on SQuAD dataset? And how to set the max sequence length on teacher model and student model? Are they the same length?

Comments:
i have the same question.  how to deal with the max sequence length between teacher model and student model

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Large Memory Layers
Author: Enumaris
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Implement models with Large Memory Layers from this paper: https://arxiv.org/pdf/1907.05242.pdf

## Motivation

These models seem very promising. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
request for a Bert_base uncase model.bin file
Author: PJRahul
Pull request: no

## 🚀 Feature

currently we r using BERT_Large uncased which slows down our application we like to use BERT_Base uncased model but BERT_base uncased model does not contain bin file.could  you please let me know where I get model.bin file.

Comments:
As with any model hosted on our S3, you can do as follows to load one of the checkpoints:

```py
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased")
```

You can find the list of pre-trained models in [the documentation](https://huggingface.co/transformers/pretrained_models.html).

can we use bert-based-uncased model for QA(question-answer) if yes then how

because model.predict(doc,q) giving error( **BertModel has no attribute predict**)




The usage u need to reference on examples/run_squad.py and u will know
everything on that code

On Mon, Nov 4, 2019 at 02:53 rhl2k <notifications@github.com> wrote:

> can we use bert-based-uncased model for QA(question-answer) if yes then how
>
> because model.predict(doc,q) giving error( *BertModel has no attribute
> predict*)
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/1687?email_source=notifications&email_token=AIEAE4FNAI7STEJQG4J6GYDQR4M2XA5CNFSM4JHYQUQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC52AAA#issuecomment-549167104>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4GZOEPRXAX6NLWSAX3QR4M2XANCNFSM4JHYQUQQ>
> .
>


You can find the documentation [here](https://huggingface.co/transformers/). The [quickstart](https://huggingface.co/transformers/quickstart.html) may be especially useful for you. As @pohanchi said, looking at the examples can also help in understanding the usage.