noamaanMulla-03/transformers-issues huggingface dataset:
Create README.md
Author: Moumeneb1
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=h1) Report
> Merging [#5259](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d12ceb48bad126768e44d2bd958fa7638abd0f16&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5259/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5259      +/-   ##
==========================================
- Coverage   79.10%   79.07%   -0.04%     
==========================================
  Files         138      138              
  Lines       24073    24073              
==========================================
- Hits        19043    19035       -8     
- Misses       5030     5038       +8     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5259/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `88.05% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5259/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.89% <0.00%> (-0.89%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=footer). Last update [d12ceb4...39fef05](https://codecov.io/gh/huggingface/transformers/pull/5259?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing 🤗 

[model page](https://huggingface.co/moumeneb1/flaubert-base-cased-ecology_crisis)
Fix convert_graph_to_onnx script
Author: n1t0
Pull request: yes

- Remove all references to `args` in methods, using arguments instead. This lets us use the `convert` method directly by importing it in another script.
- Check that the wanted framework is installed before creating the pipeline, otherwise, it might fail to instantiate it.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=h1) Report
> Merging [#5230](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9022ef021a56db975d25c7108cbd19d0dd399174&el=desc) will **increase** coverage by `0.89%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5230/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5230      +/-   ##
==========================================
+ Coverage   77.08%   77.98%   +0.89%     
==========================================
  Files         138      138              
  Lines       23841    23841              
==========================================
+ Hits        18379    18592     +213     
+ Misses       5462     5249     -213     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5230/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.00% <0.00%> (+0.29%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5230/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.90% <0.00%> (+1.38%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5230/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `94.92% <0.00%> (+75.00%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=footer). Last update [9022ef0...833fb6d](https://codecov.io/gh/huggingface/transformers/pull/5230?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Run error when run PPLM in example!
Author: huanghonggit
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
![image](https://user-images.githubusercontent.com/49581245/84684669-94247480-af6b-11ea-993b-a2eed85ac805.png)
  
And after I run other model also report error like this. 

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Is it possible to mimic trim_batch using new tokenizer strategies?
Author: sshleifer
Pull request: no
Labels: name: Core: Tokenization

I am trying to replace the old workflow of
calling batch_encode_plus to make tensors of shape
`(n_examples, model_max_length)` and then calling `trim_batch` to reduce padding computation, with the new tokenizers kwargs.
Is this possible?
The following code does not seem to truncate inputs longer than 512 (the second assert breaks).




Attempt:
```python
from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
kw = dict(max_length=512, 
          pad_to_max_length=True, padding=True, return_tensors='pt', truncation='only_first')
batch = tokenizer(['tiny sentence 1', 'tiny_sentence2'],**kw)
assert batch.input_ids.shape[1] == 7, batch.input_ids.shape[1]
input_ids, mask = trim_batch(**batch, pad_token_id=tokenizer.pad_token_id)
assert input_ids.shape[1] == 7, batch.input_ids.shape[1]

batch_overflow = tokenizer(['tiny sentence 1'*1000, 'tiny_sentence2'], **kw)


assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]
```

Traceback:
```python
assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]

AssertionError: 3002
```

Help much appreciated, @mfuntowicz @thomwolf 

Comments:
Hi @sshleifer you should read the detailed description on the tokenizers refactoring PR https://github.com/huggingface/transformers/pull/4510#issue-421650163

Until it's added in the doc (will be soon), it's required reading for all core contributors of `transformers`.

Thanks. I read that, and am still somewhat confused about why I pass `truncation=True` and get entries that are longer than `tokenizer.max_model_length`. The PR description says:

![image](https://user-images.githubusercontent.com/6045025/85311364-56e84700-b483-11ea-83f9-ee11a615170f.png)

Here is a simplified example:

```python
from transformers import BartTokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
assert tokenizer.model_max_length == 1024

# tokenizer.batch_encode_plus returns ids shaped (2, 1024)
batch_sentences = ['tiny sentence 1'*1000, 'tiny_sentence2']
ids = tokenizer.batch_encode_plus(batch_sentences, pad_to_max_length=True, max_length=tokenizer.model_max_length,
                                  truncation=True, return_tensors='pt').input_ids
assert ids.shape[1] <= tokenizer.model_max_length, ids.shape[1]

# tokenizer.__call__ returns ids shaped (2, 3002)
ids = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors='pt',
                max_length=tokenizer.model_max_length, ).input_ids
assert ids.shape[1] <= tokenizer.model_max_length, ids.shape[1]
```

I'll take a look
Remove links for all docs
Author: sgugger
Pull request: yes

Now that someone has done an awesome version selector for the docs, there's no need to list all versions in the README.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=h1) Report
> Merging [#5280](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0e1fce3c0129d05b65a83cdd89e8eadded553f2e&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5280/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5280      +/-   ##
==========================================
- Coverage   79.11%   79.08%   -0.03%     
==========================================
  Files         138      138              
  Lines       24080    24080              
==========================================
- Hits        19050    19043       -7     
- Misses       5030     5037       +7     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5280/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.51% <0.00%> (-1.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5280/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `88.05% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5280/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.62% <0.00%> (-0.15%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=footer). Last update [0e1fce3...49327ba](https://codecov.io/gh/huggingface/transformers/pull/5280?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
BertTokenizerFast does not support `pad_to_max_length` argument
Author: jarednielsen
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

The fast tokenizer has different behavior from the normal tokenizer.

```python
from transformers import BertTokenizer, BertTokenizerFast

BertTokenizer.from_pretrained("bert-base-uncased").encode("hello world", max_length=128, pad_to_max_length="right")
# succeeds
BertTokenizerFast.from_pretrained("bert-base-uncased").encode("hello world", max_length=128, pad_to_max_length="right")
*** TypeError: enable_padding() got an unexpected keyword argument 'max_length'
```

## Environment info
     
- `transformers` version: 2.11.0
- `tokenizers` version: 0.8.0rc3
- Platform: Ubuntu 18.04
- Python version: 3.7


Comments:
Hi @jarednielsen, if you installed from source then padding is handled in a different way. You'll need to use the newly added `padding` argument. According to the docs 

`padding` (:obj:`Union[bool, str]`, `optional`, defaults to :obj:`False`):
            Activate and control padding. Accepts the following values:
    
            * `True` or `'longest'`: pad to the longest sequence in the batch (or no padding if only a single sequence if provided),
            * `'max_length'`: pad to a max length specified in `max_length` or to the max acceptable input length for the model if no length is provided (`max_length=None`)
            * `False` or `'do_not_pad'` (default): No padding (i.e. can output batch with sequences of uneven lengths)


Yes, this works on master (both the old and new tokenizer API) and should work in the new release that will be out very soon.

Thank you for the quick response! Reading https://github.com/huggingface/transformers/pull/4510 makes it much clearer.

Yes, we even have a nice tutorial on the new tokenizer API now thanks to the amazing @sgugger:
https://huggingface.co/transformers/master/preprocessing.html
Tokenizer batch_encode_plus unexpected behavior 
Author: vrdn-23
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
bert-base-multilingual-cased

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
A script to tokenize sentences into inputs for the model

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Standard text that is used for intent and domain classification

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[  101, 61694, 10133, 15127, 11324, 10124, 14268,   102]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[61694, 10133, 15127, 11324, 10124, 14268,     0,     0]])
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[    0, 33600,    31,   759,  9351,    83,  3362,     2]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[33600,    31,   759,  9351,    83,  3362,     1,     1]])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

When running batch_encode_plus for a single example without adding special tokens and pad_to_max_length set to True, I would not expect any pad tokens. I did not see any mention in the documentation as to why this behavior is the expected norm. 
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: MacOS
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Yes, this was fixed on master today with https://github.com/huggingface/transformers/pull/5252
Model with fastest inference?
Author: tqdo
Pull request: no

Just curious whether there is any benchmark for inference speed of models in the transformers library? I am interested in the question-answer task but I think a benchmark on any tasks that BERT can do should be good. Thank you.

Comments:
@tqdo  There is extensive info and a spreadsheet [here](https://huggingface.co/transformers/benchmarks.html?highlight=benchmark).

Thanks a lot
AttributeError: module 'tensorflow' has no attribute 'repeat'
Author: whwhwwhh
Pull request: no
Labels: name: TensorFlow

I tried to run the pipeline task 'summarization', but get a error with "module **'tensorflow' has no attribute 'repeat' "** Does anyone encountered the same problem? How to fix it?

**my installed tensorflow == 2.0.0**

error messages:
/home/ww/anaconda3/envs/environment_name/lib/python3.6/site-packages/transformers/pipelines.py", line 1446, in __call__
    inputs["input_ids"], attention_mask=inputs["attention_mask"], **generate_kwargs,
  File "/home/ww/anaconda3/envs/environment_name/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 747, in generate
    tf.repeat(tf.expand_dims(tf.range(batch_size), -1), repeats=num_beams * effective_batch_mult, axis=1),
AttributeError: module 'tensorflow' has no attribute 'repeat'


Comments:
@LysandreJik @jplu 

It seems that Tensorflow does not include `repeat`  in 2.0 and 2.0.1 (see https://github.com/tensorflow/tensorflow/issues/38839). Perhaps best to have 2.1 as a min requirement?

Indeed, TF < 2.1 doesn't have the `tf.repeat()` function. Put TF >= 2.1 as min requirement looks to be a good solution to me.

@LysandreJik are you ok with this?

Yeah I checked tensorflow API and upgrade my tf yo 2.2 it works now


On 25 Jun 2020, at 4:50 pm, Julien Plu <notifications@github.com> wrote:

﻿

Indeed, TF < 2.1 doesn't have the tf.repeat() function. Put TF >= 2.1 as min requirement looks to be a good solution to me.

@LysandreJik<https://github.com/LysandreJik> are you ok with this?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/5218#issuecomment-649272046>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKTGDOKKTAPDE6SIYCKGTO3RYLXTNANCNFSM4OF2P2IQ>.


Sure, I'm okay with this!
After reading the tutorial I can use the BertModel to extract word embedding but how to use it extract the sentence embedding?
Author: leopardv10
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
You need to do some sort of pooling technique to use the output of the bertmodel for sentence embedding. I found this resource to be extremely helpful in discussing the different types of pooling techniques to use with a bert model: https://github.com/hanxiao/bert-as-service#q-what-are-the-available-pooling-strategies

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why do I run example/run_ner.py with no output
Author: Vvegetables
Pull request: no
Labels: name: Need more information

name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
### This is the log I ran. Any questions? Please help me

![image](https://user-images.githubusercontent.com/9248572/75306701-c1e29200-5884-11ea-9ffe-07c02375413e.png)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Please don't post a screenshot. Copy-and-paste your code or output instead.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@Vvegetables , does this problem still exist?

it's fine，thanks


发自我的iPhone

------------------ Original ------------------
From: Stefan Schweter <notifications@github.com&gt;
Date: 周日,4月 26,2020 20:55
To: huggingface/transformers <transformers@noreply.github.com&gt;
Cc: Vvegetables <625736074@qq.com&gt;, Mention <mention@noreply.github.com&gt;
Subject: Re: [huggingface/transformers] Why do I run example/run_ner.py with no output (#3025)





 
@Vvegetables , does this problem still exist?
 
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.