Zero-values in ViT attention mask (pixel mask)
Author: martinaianaro99
Pull request: no

### System Info


- `transformers` version: 4.33.2
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The code loads a binary matrix from a file, converts it to a PyTorch tensor with values ​​0 and 1, and assigns it to pixel_mask, which is then added to a dictionary called encoding.

part of code of ViltDataset class:
```python
                pixel_mask_filename = "/content/drive/MyDrive/VILT/Pixel_Masks/" + str(pid) + ".npy"
                pixel_mask_external = np.load(pixel_mask_filename).astype(int)
                pixel_mask_external = torch.tensor(pixel_mask_external, dtype=torch.long)
                pixel_mask_external = torch.where(pixel_mask_external > 0, torch.tensor(1), torch.tensor(0))
                pixel_mask=pixel_mask_external
                encoding["pixel_mask"]=pixel_mask
```
Processor:
```python
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")
encoding = self.processor(image, masked_sentence, padding="max_length", max_length=40, truncation=True, return_tensors="pt")
```
Pixel mask are extracted in this method:
```python
def collate_fn(batch):
    batch = [item for item in batch if item is not None]
    input_ids = [item['input_ids'] for item in batch]
    attention_mask = [item['attention_mask'] for item in batch]
    token_type_ids = [item['token_type_ids'] for item in batch]
    labels = [item['labels'] for item in batch]
    pixel_values = [item['pixel_values'] for item in batch]
    pixel_mask = [item['pixel_mask'] for item in batch]



    # create new batch
    collated_batch = {}
    collated_batch['input_ids'] = torch.stack(input_ids)
    collated_batch['attention_mask'] = torch.stack(attention_mask)
    collated_batch['token_type_ids'] = torch.stack(token_type_ids)
    collated_batch['labels'] = torch.stack(labels)
    collated_batch['pixel_values'] = torch.stack(pixel_values)
    collated_batch['pixel_mask'] = torch.stack(pixel_mask)
    return collated_batch
```
### Expected behavior

I assign to variable pixel mask an array of values 0 or 1.
Error is:
```python
RuntimeError: Input and output sizes should be greater than 0, but got input (H: 12, W: 12) output (H: 0, W: 0).
```
If I assign to pixel mask other values like 1 or 2, it works.

Complete error:
```python
----> 4 train_loss, time_t, memory_t = train_model(model, device, train_dataloader, val_dataloader,learning_rate=LEARNING_RATE, weight_decay=weight_decay,  num_epochs=2)

10 frames
[/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py](https://localhost:8080/#) in interpolate(input, size, scale_factor, mode, align_corners, recompute_scale_factor, antialias)
   3957         if antialias:
   3958             return torch._C._nn._upsample_bilinear2d_aa(input, output_size, align_corners, scale_factors)
-> 3959         return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
   3960     if input.dim() == 5 and mode == "trilinear":
   3961         assert align_corners is not None

RuntimeError: Input and output sizes should be greater than 0, but got input (H: 12, W: 12) output (H: 0, W: 0)
```

Comments:
Hey, the error does not seem to indicate and issue with padding but rather an issue with the `upsampling` function, and thus the shape of the pixel mask. I am not sure this is a  bug in transformers as you are using your custom code. 
I don't have access to the full traceback so can't help you further. Can you share the full traceback? 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.