Error projecting concatenated Fourier Features.
Author: sr-ndai
Pull request: no

### System Info

- `transformers` version: 4.27.4
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers.models.perceiver.modeling_perceiver import PerceiverImagePreprocessor

input_preprocessor = PerceiverImagePreprocessor(
    config, 
    prep_type="conv",
    spatial_downsample=4,
    temporal_downsample=1,
    position_encoding_type="fourier",
    in_channels=4,
    out_channels=256,
    conv2d_use_batchnorm=True,
    concat_or_add_pos="concat",
    project_pos_dim=128,
    fourier_position_encoding_kwargs = dict(
        num_bands=64,
        max_resolution=[25,50], # 4x downsample include
    )
    )

test = torch.randn(1,4,100,200)
inputs, modality_sizes, inputs_without_pos = input_preprocessor(test)
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 23
     21 test = torch.randn(1,4,100,200)
     22 # preprocessor outputs a tuple
---> 23 inputs, modality_sizes, inputs_without_pos = input_preprocessor(test)
     25 print(inputs.shape)
     26 print(inputs_without_pos.shape)

torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

transformers/models/perceiver/modeling_perceiver.py:3226), in PerceiverImagePreprocessor.forward(self, inputs, pos, network_input_is_1d)
   3223     else:
   3224         raise ValueError("Unsupported data format for conv1x1.")
-> 3226 inputs, inputs_without_pos = self._build_network_inputs(inputs, network_input_is_1d)
   3227 modality_sizes = None  # Size for each modality, only needed for multimodal
   3229 return inputs, modality_sizes, inputs_without_pos

transformers/models/perceiver/modeling_perceiver.py:3169), in PerceiverImagePreprocessor._build_network_inputs(self, inputs, network_input_is_1d)
   3166     pos_enc = self.position_embeddings(index_dims, batch_size, device=inputs.device, dtype=inputs.dtype)
   3168 # Optionally project them to a target dimension.
-> 3169 pos_enc = self.positions_projection(pos_enc)
   3171 if not network_input_is_1d:
   3172     # Reshape pos to match the input feature shape
   3173     # if the network takes non-1D inputs
   3174     sh = inputs.shape

torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

torch/nn/modules/linear.py:114), in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -> Tensor:
--> 114     return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1176x258 and 256x128)

### Expected behavior

Linear projection input number of features should equal the number of positional features.

When concatenating Fourier Features the expected number of positional features is: (2 * num_bands * num_dims) + 2.
The build_position_encoding() function takes as input the out_channels which is used to define the input number of features for linear projection. PerceiverImagePreprocessor incorrectly passes in the embedding dimension to out_channels for positional encoding projection.

Possible Fix:
Use the positional encoding class method output_size() to pull number of input features for projection within build_position_encoding() function.

positions_projection = nn.Linear(output_pos_enc.output_size(), project_pos_dim) if project_pos_dim > 0 else nn.Identity()

Comments:
@sr-ndai Thanks for raising this issue and detailed reproduction. 

Could you share the config or checkpoint being used in this example? 

@amyeroberts No problem, here is the configuration I used:
```python
from transformers import PerceiverConfig

config = PerceiverConfig(
    num_latents = 128,
    d_latents=256,
    num_blocks=6,
    qk_channels = 256,
    num_self_attends_per_block=2,
    num_self_attention_heads=8,
    num_cross_attention_heads=8,
    self_attention_widening_factor=2,
    cross_attention_widening_factor=1,
    hidden_act='gelu_new',
    attention_probs_dropout_prob=0.1,
    use_query_residual=True,
    num_labels = 128 # For decoder
)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.