noamaanMulla-03/transformers-issues huggingface dataset:
ViT modeling file is missing drop path present in PyTorch image models
Author: muqeeth
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.10.1+cu113 (True)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@rwightman @NielsRogge

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Vision transformer modeling file from Pytorch image models uses [drop path ](https://github.com/rwightman/pytorch-image-models/blob/324a4e58b6b0365d16dcf8f93739be8f74cd7d37/timm/models/vision_transformer.py#L234) after attention and feedforward layers. It is missing in the current Huggingface modeling_vit.py file. We found that on VTAB benchmark, having drop path significantly boost the performance of ViTB model. 
2. The epsilon value of layer norm used in pretrained ViTConfig is 1e-12 whereas vision_transformer.py file in  pytorch-image-models uses epsilon value of 1e-6. Because of this, google/vit-base-patch16-224 checkpoint performs slightly different with modeling_vit.py from Huggingface and vision_transformer.py from pytorch-image-models in the test mode. 

### Expected behavior

1. Given that Drop path is shown to perform well on VTAB benchmark , it can be added to current modeling file of Huggingface. 
2. Epsilon value in layer norm of the transformer can be made consistent to pytorch-image-models. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Problem at using CLIPFeatureExtractor from transformers.models.clip.feature_extraction_clip
Author: heyzude
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Ubuntu
- Python version: 3.7
- PyTorch version (GPU?): 1.7.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Models:

CLIPFeatureExtractor (which is used to provide feature to CLIPModel)

## Information

When I use CLIPFeatureExtractor(), (https://huggingface.co/transformers/v4.6.0/_modules/transformers/models/clip/feature_extraction_clip.html#CLIPFeatureExtractor) at the __call__ function, after images pass through resize, center_crop, and normalize, it is given to `BatchFeature` of `feature_extraction_utils` and then problem happens.

The problem happens inside `convert_to_tensors` function, inside the for loop starting from line 166, the code tries to change value to tensor, but sometimes the type of value is a list or something else, so it goes to except statement. So I had to write an extra snippet for the case when the value is list.


Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

`clip_feature_extractor = CLIPFeatureExtractor()`
`# images are list of images in numpy array or some other types such as PIL or tensor`
`clip_feature_extractor(images=images, return_tensors='pt')`

Comments:
Hi,

> images are list of images in numpy array or some other types such as PIL or tensor
clip_feature_extractor(images=images, return_tensors='pt')

If you pass a list of images, make sure they are all of the same type.

Yes I did passed a list of images whose images are all in same type, but the problem happens

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Can you elaborate, or provide a minimal reproducer?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm also having this issue.  It only works if I pass in PIL images.  It does not work on tensors _or_ lists of tensors.
PreTrainedTokenizerFast with tokenizer object is acting on original tokenizer object
Author: YBooks
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

- To reproduce this error, we can create a tokenizer and try to wrap it in the PreTrainedTokenizerFast
```
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, trainers
data = [
    "My first sentence",
    "My second sentence",
    "My third sentence is a bit longer",
    "My fourth sentence is longer than the third one"
]

tokenizer = Tokenizer(models.WordLevel(unk_token="<unk>"))
trainer = trainers.WordLevelTrainer(vocab_size=10, special_tokens=["<unk>", "<pad>"])
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
tokenizer.train_from_iterator(data, trainer=trainer)

tokenizer.enable_padding(pad_token="<pad>", pad_id=tokenizer.token_to_id("<pad>"))
tokenizer.enable_truncation(max_length=5)
print(tokenizer.encode(data[-1]).ids, tokenizer.padding)
```
This gives an output with len 5 and an explicit padding object

- In the other hand if we load our tokenizer in the PreTrainedTokenizerFast class and print the same thing like before.
```
from transformers import PreTrainedTokenizerFast

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
fast_tokenizer(data)

print(tokenizer.encode(data[-1]).ids, tokenizer.padding)
```

This gives an output with len > 5 and None in padding

### Expected behavior

The expected behavior should be the same with tokenizer before loading it in the PreTrainedTokenizerFast wrapper. It should not impact the padding and the truncation part

Comments:
cc @SaulLu 

Hi @YBooks

Thank you very much for the detailed issue :hugs: ! 

I see that you have already proposed a fix that has been merged and that solves the problem you are pointing out. If you are happy with it, is it ok if we close this issue?

Hey @SaulLu 
Yes sure. My pleasure

@YBooks ,  @SaulLu , @sgugger can we reopen this issue, since https://github.com/huggingface/transformers/pull/18408 creates another one ?
Calculating Confidence score for Question Answering Models
Author: UmerTariq1
Pull request: no
Labels: name: wontfix

For QA task (extractive QA) the pipeline provides/returns 4 values. 
1. a probability score
2.  start index
3. end index
4. extracted answer
(https://huggingface.co/transformers/main_classes/pipelines.html#transformers.QuestionAnsweringPipeline)

But If I am using the model class directly   (not using pipeline) like the code below then i am unable to find the probability score :

`  
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForQuestionAnswering.from_pretrained(MODEL_NAME,return_dict=True)
_encoding = tokenizer.encode_plus(question, docText, return_tensors="pt",max_length=4096)
input_ids = encoding["input_ids"]
attention_mask = encoding["attention_mask"]

start_scores, end_scores = model(input_ids, attention_mask=attention_mask).values()
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
answer_tokens = all_tokens[torch.argmax(start_scores) :torch.argmax(end_scores)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))_
`
This answer object has only 3 values. extracted answer, start score and end score.I cant find any way or function to calculate the probability score. I want it because I want to sort multiple answers according to their probability/confidence score. 

This is a duplicate of Issue #5768 but that issue has been marked close without any answer.

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

model_name = "distilbert-base-uncased-finetuned-sst-2-english"

model = AutoModelForSequenceClassification.from_pretrained(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)


train_encoding = tokenizer(X_train, truncation=True, padding=True, max_length=512, return_tensors="pt")


# Training the model with PyTorch


with torch.no_grad():

    outputs = model(**train_encoding)
    
    # Normalize logits and spans 
    start = F.softmax(outputs.start_logits, dim=1)
    end = F.softmax(outputs.end_logits, dim=1)


    # Getting the start and end index of the tensor to retrieve the answer
    start_index = torch.argmax((start), dim=1)
    end_index = torch.argmax((end), dim=1)



# Computing score

outer = np.matmul(np.expand_dims(start, -1), np.expand_dims(end, 1))
max_answer_len = 512
candidates = np.tril(np.triu(outer), max_answer_len - 1)
idx_sorts = [np.argmax(candidates[i].flatten()) for i in range(len(candidates))]
scores = [candidates[[i], start_indices[i], end_indices[i]][0] for i in range(len(candidates))]


# Extracting answer
tokenizer.decode(train_encoding["input_ids"][start_index:end_index], skip_special_tokens=True)
Check self-hosted runners are online
Author: ydshieh
Pull request: yes

# What does this PR do?

#18905 checks if the docker could be launched inside the runners.

However, the runners could be offline due to some unknown reasons, and we are not aware of this problem (job hangs forever) so far.

This PR adds a check for runner being online or offline.

However, it might happen that a runner becomes offline in the middle of a workflow run. This situation is not easy to deal with, and we still need to prevent such situation. Therefore, a new scheduled (per hour) workflow is created to check runner availability.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Added type hints for TFConvBertModel
Author: kishore-s-15
Pull request: yes

Based on Issue #16059 

I have added type hints for the [TFConvBertModel](https://huggingface.co/docs/transformers/model_doc/convbert#transformers.TFConvBertModel).

@Rocketknight1 Could you kindly check if this is fine?

Thanks in advance.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @Rocketknight1.
fix: create a copy for tokenizer object
Author: YBooks
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

To not have the object in the PreTrainedTokenizerFast and not impact its padding/truncating attribute we can just have a deep copy of the object 
Fixes # ([18406](https://github.com/huggingface/transformers/issues/18406))


## Who can review?

@LysandreJik @sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@YBooks , this PR creates another issue since deepcopy of tokenizer object is not possible when a Custom Pretokenizer is used. 

```python 
tokenizer.pre_tokenizer = Custom()
```

See open issue in tokenizers here: https://github.com/huggingface/tokenizers/issues/581

I suggest using another serialization/loading scheme instead of copy.

Would like to open a PR with a fix?

Yes, I will give it a try.
Add type hints for TF MPNet models
Author: kishore-s-15
Pull request: yes

Based on Issue https://github.com/huggingface/transformers/issues/16059

I have added type hints for the all the Tensorflow MPNet models.

@Rocketknight1 Could you kindly check if this is fine?

Thanks in advance.

Comments:
Thanks @Rocketknight1 😊.

_The documentation is not available anymore as the PR was closed or merged._
Added type hints to ResNetForImageClassification
Author: kishore-s-15
Pull request: yes

Based on Issue #16059.

While looking into Issue #16059, I found that the `ResNetForImageClassification` model's type hints are inconsistent with the [docs](https://huggingface.co/docs/transformers/model_doc/resnet#transformers.ResNetForImageClassification.forward). Modified it to be consistent with the docs.

@Rocketknight1 Could you kindly check if this is fine?

This is my First PR for the Transformers library. Thanks in advance.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @Rocketknight1.
Added Type hints for VIT MAE
Author: kishore-s-15
Pull request: yes

Based on Issue #16059 

While looking through the codebase, I found that the ViTMAE model doesn't had the type hints as suggested in Issue #16059. Added type hints for [ViTMAEModel](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEModel) and [ViTMAEForPreTraining](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEForPreTraining) models.

@Rocketknight1 Could you kindly check if this is fine?

Thanks in advance.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @Rocketknight1.
how to customize the encoder_output when using the generate function in BART?
Author: xjw-star
Pull request: no

For instance, I would like to concatenate two different last encoder hidden states from two different text. How can I achieve it using existing generate function?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Trainer Bug
Author: rich-junwang
Pull request: no
Labels: name: bug

### System Info

I might be wrong, but looks this line here: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_seq2seq.py#L176 introduces a bug. _gen_kwargs may not be an attribute for an instance. 


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

If we have a trainer that directly calls the function `prediction_step`, we'll have this error

### Expected behavior

no error is thrown when we call `prediction_step` function

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Improve vision models docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

Improves the docs of several vision models:
* for `xxxForMaskedImageModeling` models, add a link to the `run_mim.py` script
* for `ViTMAEForPreTraining`, add a link to the `run_mae.py` script
* for ViT, add a tip about interpolation of pre-trained position embeddings (in order to fine-tune on higher resolution images)
* add figures for ViT and BEiT

Comments:
_The documentation is not available anymore as the PR was closed or merged._
model.generate() has the same speed on CPU and GPU
Author: tomdzh
Pull request: no

Hi, I find that model.generate() of BART and T5 has roughly the same running speed when running on CPU and GPU. Why doesn't GPU give faster speed? Thanks!

## Environment info
- `transformers` version: 4.1.1
- Python version: 3.6
- PyTorch version (GPU?): 1.3.1
- Using GPU in script?: yes

### Who can help
 TextGeneration: @TevenLeScao 
 Bart: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): BART and T5
## To reproduce
```python
import time
from transformers import BartTokenizer, BartForConditionalGeneration

device = 'cpu'

# change to GPU
# device = 'cuda:0'

text_to_summarize = "My friends are cool but they eat too many carbs."

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')

inputs = tokenizer(text_to_summarize, return_tensors='pt')

inputs = inputs['input_ids'].to(device)
model = model.to(device)

start = time.time()
summary_ids = model.generate(inputs)
print("Time spent (s): ", time.time() - start)
```
## Expected behavior
I expected running on GPU should give me much faster speed. But running on GPU gave me roughly the same speed as CPU, both around 0.3s in this case.


Comments:
Just realize that I used a single input... Issued closed

Thanks, your post helped me so much!
I'm using BloomModel on AWS Lambda Function but Lambda doesn't support GPU. 
So I write the code like that:
device = 'cpu'
#topic variable is already given
prompt = f' About {Topic} is what I think:  '
inputs = tokenizer(prompt, return_tensors='pt')

inputs = inputs['input_ids'].to(device)
model = model.to(device)
sample = model.generate(inputs, max_length=100, temperature=0.9, repetition_penalty = 2.0)
output = tokenizer.decode(sample[0], truncate_before_pattern=[r"\n\n^#", "^'''", "\n\n\n"])
Beit postprocessing
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds a post-processing method to BeiTFeatureExtractor for semantic segmentation.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._