noamaanMulla-03/transformers-issues huggingface dataset:
Does this project have this function ?
Author: SeekPoint
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->


can we use this project to calculate the probability that a input text as a real/resonable sentence base on the corpus we trained

Comments:
https://github.com/huggingface/transformers/issues/2311

@frankniujc it is helpful
but maybe a better way is take the all tokens  in a whole, not prediction the next tokens

The probability of a sentence P(s0s1s2s3s4...sn) = P(s1|s0) * P(s2|s0s1) * P(s3|s0s1s2) * ... * P(sn|s0s1s2...sn-1)

So you can do something like this
```Python
def sentence_probability(sent):
    bos = tokenizer.encode('
')
    tokens = tokenizer.encode(sent)
    tokens = bos + tokens
    input_ids = torch.tensor(tokens).unsqueeze(0).to('cuda')

    sent_probs = []

    for i, next_word in enumerate(tokens[1:]):
        next_word_logits = model(input_ids[:,:i+1])[0][0, -1].detach()
        next_word_prob = F.log_softmax(next_word_logits, dim=0)[next_word].item()

        sent_probs.append(next_word_prob)

    return sum(sent_probs)
```

@loveJasmine Have a look at [`lm-scorer`](https://github.com/simonepri/lm-scorer).

It is a tiny wrapper around `transformers` I wrote that allows you to get sentences probabilities using models that support it (only GPT2 models are implemented at the time of writing).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to make my own dataset to use BART summarization?
Author: laetokang
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hello I'm trying to use BART summarization model. I have a dataset that take the form of dataframe which has two columns 'document' and 'summary'. 

Q1.
I read this Readme.md.
> line.this should make a directory called cnn_dm/ with files like test.source. To use your own data, copy that files format. Each article to be summarized is on its own 

I don't understand this sentences well. So I just saw the cnn datasets(train.source, train.target, test.source, test.target). **How can i distinguish between each documents?**

Q2. 
**How can i change my own datasets into the cnn files format?**



<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
keras
Author: jeffreyjohnens
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
How can I pinpoints Logs directory to Google Drive while finetuning GPT-2 model, which helps in visualizing data via tensorboard?
Author: hmdgit
Pull request: no
Labels: name: wontfix


Hi,

I am facing an issue in retrieving logs, while fine-tuning GPT-2 model by using [Google Colab](https://colab.research.google.com/github/interactive-fiction-class/interactive-fiction-class.github.io/blob/master/homeworks/language-model/hw4_transformer.ipynb).

As the finetunining takes several hours, Google Colab halts the running process at certain point in time, even there are remaining epochs left behind. 

In that case, I can be able to successfully continue my finetuning by including a parameter "should_continue" while running the script using [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).

However, my logs vanished, and I could not be to retrieve data of tensorbaord, which has been run with these commands

```
%load_ext tensorboard
%tensorboard --logdir=runs
```

**Is there a way to pin point my logs to the Google Drive, so that I can retrieve and visualize them with tensor-board at any point in time?**


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Allow do_lower_case regardless of do_basic_tokenize
Author: BramVanroy
Pull request: no
Labels: name: Help wanted

name: wontfix

It would make sense to me that you could set `do_lower_case=True` even when `do_basic_tokenize=False`. If your input has already been tokenized (but not lower-cased), you still want to lowercase it. As the code is currently written, that does not seem possible as [the BasicTokenizer is responsible for the lowercasing](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization.py#L101-L105):

        if do_basic_tokenize:
          self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case,
                                                never_split=never_split)
        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
        self.max_len = max_len if max_len is not None else int(1e12)

I would then propose to add lowercase to its own method or to add a do_lower_case to WordpieceTokenizer.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf Please re-open. If I have the time, I can try to work on this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Bump so I don't forget about this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Deserialize BERT Sequence Lassifier Quantized Model & Inferencing Issue
Author: suryapa1
Pull request: no
Labels: name: wontfix

Dynamic Quantization:  Loading Quantized Model issue here:

Currently am looking at below collab link for quantization, where all Linear layers are quantized in corresponding BERTForSequence Classification.

Step:1,
Serialized Quantized Model

Step 2:
Deserialized Using below code:

Screenshot#1:
Linear layers of the encoder are converted to DynamicQuantizedLinear Layers just after conversion
Once loaded serialized quantized model for future use couldn't show up DynamicQuantizedLinear layers rather Linear layers for the query, key, values as shown in screenshots.


Also, inference these deserialized models shows error predictions as shown here, so someone guide me how to deserialize quantized models as it is taking hell lot of time to figure out the same.


https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb?authuser=1#scrollTo=dUJ1NGinLAa1


## Information
<img width="922" alt="Screenshot 2020-04-08 at 3 50 23 PM" src="https://user-images.githubusercontent.com/6042186/78773790-5c201480-79b1-11ea-930a-aff268c97394.png">
<img width="681" alt="Screenshot 2020-04-08 at 3 49 08 PM" src="https://user-images.githubusercontent.com/6042186/78773823-6a6e3080-79b1-11ea-88ef-406889880e83.png">



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can not set different token and model dir in `run_glue.py`
Author: Lapis-Hong
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
the original code as follows, when i use args.tokenizer_name different from model_name_or_path, it still call for the model_name_or_path config file.
tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        cache_dir=args.cache_dir if args.cache_dir else None,
)

It would be solved  by adding config params as follows:
tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        config=config,
        cache_dir=args.cache_dir if args.cache_dir else None,
   )



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix the example command for SQuAD
Author: kaniblu
Pull request: yes

Issue #4549: added the missing argument `--do_lower_case` for reproducing the intended results.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=h1) Report
> Merging [#4561](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4561/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4561      +/-   ##
==========================================
- Coverage   77.87%   77.86%   -0.02%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16016    16013       -3     
- Misses       4550     4553       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4561/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4561/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.50% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4561/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=footer). Last update [a34a989...b44c742](https://codecov.io/gh/huggingface/transformers/pull/4561?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Can you add the do_lower_case flag for all other instances in that README that use bert uncased? Thanks!

Sure. Presumably for the tf version, the script will handle the casings automatically.

From the code it would seem that way, but I am not sure actually. 

cc @thomwolf @LysandreJik: `do_lower_case` was missing in the commands to run squad with bert-base-uncased. Is this flag also necessary in the Tensorflow version? It's not present in the code, so I would assume not.

The other changes LGTM!

Closed by #4245 (we still need to investigate why the lowercasing is not properly populated by the model's config)
Improve ONNX logging
Author: mfuntowicz
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=h1) Report
> Merging [#4999](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9931f817b75ecb2c8bb08b6e9d4cbec4b0933935&el=desc) will **increase** coverage by `0.31%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4999/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4999      +/-   ##
==========================================
+ Coverage   76.89%   77.20%   +0.31%     
==========================================
  Files         128      128              
  Lines       21854    21854              
==========================================
+ Hits        16804    16872      +68     
+ Misses       5050     4982      -68     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/4999/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `75.85% <0.00%> (+19.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=footer). Last update [9931f81...a7a5c18](https://codecov.io/gh/huggingface/transformers/pull/4999?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Cannot import transformers due to issue with signal.py (SIGKILL)
Author: scottleith
Pull request: no

# 🐛 Bug

Can't import transformers, get a problem with signal.py. I see that an issue was recently fixed... should I install a nightly build, or...?

>>> import transformers
2020-06-15 05:40:54.098560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\15194\anaconda3\envs\ai_env\lib\site-packages\transformers\__init__.py", line 371, in <module>
    from .benchmark import PyTorchBenchmark, PyTorchBenchmarkArguments
  File "C:\Users\15194\anaconda3\envs\ai_env\lib\site-packages\transformers\benchmark\__init__.py", line 10, in <module>
    from .benchmark import PyTorchBenchmark
  File "C:\Users\15194\anaconda3\envs\ai_env\lib\site-packages\transformers\benchmark\benchmark.py", line 32, in <module>
    from .benchmark_utils import Benchmark, Memory, measure_peak_memory_cpu, start_memory_tracing, stop_memory_tracing
  File "C:\Users\15194\anaconda3\envs\ai_env\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 19, in <module>
    from signal import SIGKILL
**ImportError: cannot import name 'SIGKILL' from 'signal' (C:\Users\15194\anaconda3\envs\ai_env\lib\signal.py)**

## Expected behavior

The package to import.

## Environment info
- `transformers` version: 2.11
- Platform: Windows 10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0, GPU = yes ( torch.cuda.is_available == True).
- Tensorflow version (GPU?): 2.2.0; GPU = yes.
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.


Comments:
Just had to reinstall. I installed pytorch AFTER transformers, and I cloned the transformers repository. Just uninstalled and reinstalled with pip install transformers and the issue went away.
Remove unused arguments in Multiple Choice example
Author: sgugger
Pull request: yes

In the dataset preparation, the arguments `pad_token_segment_id`, `pad_on_left`, `pad_token` and `mask_padding_with_zero` are inferred from the tokenizer to be sent to `convert_examples_to_features` which then does not use them (since `tokenizer.encode_plus` does all of this using the tokenizer state).

This PR cleans that up (and removes the TODO).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=h1) Report
> Merging [#4853](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/37be3786cf1de9d21233f543c231866e68954998&el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4853/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4853      +/-   ##
==========================================
+ Coverage   76.40%   76.54%   +0.14%     
==========================================
  Files         128      128              
  Lines       21533    21533              
==========================================
+ Hits        16452    16483      +31     
+ Misses       5081     5050      -31     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4853/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.15% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4853/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.00% <0.00%> (+10.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=footer). Last update [37be378...9e1b14a](https://codecov.io/gh/huggingface/transformers/pull/4853?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Also, should the `DataProcessor` in this file simply use the one in `transformers`?

LGTM
got an unexpected keyword argument 'cache_dir'
Author: chiyuzhang94
Pull request: no

I used the following code to run the job: `
export GLUE_DIR=./data

python3 run_classifier.py \
  --task_name COLA \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/ \
  --bert_model bert-large-uncased \
  --max_seq_length 20 \
  --train_batch_size 10 \
  --learning_rate 2e-5 \
  --num_train_epochs 2.0 \
  --output_dir ./output`

Then, I got the output:
`01/11/2019 02:02:55 - INFO - __main__ -   device: cpu n_gpu: 0, distributed training: False, 16-bits training: False
01/11/2019 02:02:56 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /Users/chiyuzhang/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
01/11/2019 02:02:56 - INFO - pytorch_pretrained_bert.modeling -   loading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gz from cache at /Users/chiyuzhang/.pytorch_pretrained_bert/214d4777e8e3eb234563136cd3a49f6bc34131de836848454373fa43f10adc5e.abfbb80ee795a608acbf35c7bf2d2d58574df3887cdd94b355fc67e03fddba05
01/11/2019 02:02:56 - INFO - pytorch_pretrained_bert.modeling -   extracting archive file /Users/chiyuzhang/.pytorch_pretrained_bert/214d4777e8e3eb234563136cd3a49f6bc34131de836848454373fa43f10adc5e.abfbb80ee795a608acbf35c7bf2d2d58574df3887cdd94b355fc67e03fddba05 to temp dir /var/folders/j0/_kd2ppm53wnb6pjypwy3gc_00000gn/T/tmpynwe_15z
01/11/2019 02:03:06 - INFO - pytorch_pretrained_bert.modeling -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "max_position_embeddings": 512,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

Traceback (most recent call last):
  File "run_classifier.py", line 619, in <module>
    main()
  File "run_classifier.py", line 455, in main
    num_labels = 2)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 502, in from_pretrained
    model = cls(config, *inputs, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'cache_dir'`

Could you please help me to fix this problem?

Comments:
You should update to the latest version of `pytorch_pretrained_bert`(`pip install pytorch_pretrained_bert --upgrade`)