wav2vec2 model: error trying to do inference
Labels: triaged, bug

### 🐛 Describe the bug

I tried to do torch.compile() on the second most downloaded model of facebook on huggingface: https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self

I took the example of usage they provide and just added the torch.compile() step.

I also tried with an another custom audio sample earlier, and strangely, it ran, but very slowly compared to the original model...
But it had the same prediction result

### Error logs

Here is the warnings and the traceback:

```
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Found cached dataset librispeech_asr_dummy (/home/noetits/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc)
It is strongly recommended to pass the ``sampling_rate`` argument to this function. Failing to do so can result in silent errors that might be hard to debug.
/home/noetits/miniconda3/envs/.../lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
  warnings.warn(message, UserWarning)
/home/noetits/miniconda3/envs/.../lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
  warnings.warn(message, UserWarning)
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In [1], line 17
     14 input_values = processor(ds[0]["audio"]["array"], return_tensors="pt", padding="longest").input_values
     16 # retrieve logits
---> 17 logits = model(input_values).logits
     19 # take argmax and decode
     20 predicted_ids = torch.argmax(logits, dim=-1)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/torch/nn/modules/module.py:1480, in Module._call_impl(self, *args, **kwargs)
   1475 # If we don't have any hooks, we want to skip the rest of the logic in
   1476 # this function, and just call forward.
   1477 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1478         or _global_backward_pre_hooks or _global_backward_hooks
   1479         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1480     return forward_call(*args, **kwargs)
   1481 # Do not call functions when jit is used
   1482 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/.../lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:80, in OptimizedModule.forward(self, *args, **kwargs)
     79 def forward(self, *args, **kwargs):
---> 80     return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:209, in _TorchDynamoContext.__call__.<locals>._fn(*args, **kwargs)
    207 dynamic_ctx.__enter__()
    208 try:
--> 209     return fn(*args, **kwargs)
    210 finally:
    211     set_eval_frame(prior)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:1742, in Wav2Vec2ForCTC.forward(self, input_values, attention_mask, output_attentions, output_hidden_states, return_dict, labels)
   1732 r"""
   1733 labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*):
   1734     Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
   (...)
   1737     config.vocab_size - 1]`.
   1738 """
   1740 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> 1742 outputs = self.wav2vec2(
   1743     input_values,
   1744     attention_mask=attention_mask,
   1745     output_attentions=output_attentions,
   1746     output_hidden_states=output_hidden_states,
   1747     return_dict=return_dict,
   1748 )
   1750 hidden_states = outputs[0]
   1751 hidden_states = self.dropout(hidden_states)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/torch/nn/modules/module.py:1480, in Module._call_impl(self, *args, **kwargs)
   1475 # If we don't have any hooks, we want to skip the rest of the logic in
   1476 # this function, and just call forward.
   1477 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1478         or _global_backward_pre_hooks or _global_backward_hooks
   1479         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1480     return forward_call(*args, **kwargs)
   1481 # Do not call functions when jit is used
   1482 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/.../lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:1308, in Wav2Vec2Model.forward(self, input_values, attention_mask, mask_time_indices, output_attentions, output_hidden_states, return_dict)
   1304         hidden_states[mask_feature_indices] = 0
   1306     return hidden_states
-> 1308 @add_start_docstrings_to_model_forward(WAV_2_VEC_2_INPUTS_DOCSTRING)
   1309 @add_code_sample_docstrings(
   1310     processor_class=_PROCESSOR_FOR_DOC,
   1311     checkpoint=_CHECKPOINT_FOR_DOC,
   1312     output_type=Wav2Vec2BaseModelOutput,
   1313     config_class=_CONFIG_FOR_DOC,
   1314     modality="audio",
   1315     expected_output=_EXPECTED_OUTPUT_SHAPE,
   1316 )
   1317 def forward(
   1318     self,
   1319     input_values,
   1320     attention_mask=None,
   1321     mask_time_indices=None,
   1322     output_attentions=None,
   1323     output_hidden_states=None,
   1324     return_dict=None,
   1325 ):
   1326     output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
   1327     output_hidden_states = (
   1328         output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
   1329     )

File ~/miniconda3/envs/.../lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:209, in _TorchDynamoContext.__call__.<locals>._fn(*args, **kwargs)
    207 dynamic_ctx.__enter__()
    208 try:
--> 209     return fn(*args, **kwargs)
    210 finally:
    211     set_eval_frame(prior)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:2107, in aot_module_simplified.<locals>.forward(*runtime_args)
   2105 full_args.extend(params_flat)
   2106 full_args.extend(runtime_args)
-> 2107 return compiled_fn(full_args)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:811, in make_boxed_func.<locals>.g(args)
    810 def g(args):
--> 811     return f(*args)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:1687, in aot_dispatch_autograd.<locals>.debug_compiled_function(*args)
   1681     elif not can_require_grad:
   1682         assert not a.requires_grad, format_guard_bug_msg(
   1683             aot_config,
   1684             f"{describe_input(i, aot_config)} would not require grad"
   1685         )
-> 1687 return compiled_function(*args)

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:1551, in aot_dispatch_autograd.<locals>.compiled_function(*args)
   1548 else:
   1549     args_with_synthetic_bases = args
-> 1551 all_outs = CompiledFunction.apply(*args_with_synthetic_bases)
   1552 if CompiledFunction.num_aliasing_metadata_outs > 0:
   1553     outs = all_outs[:-CompiledFunction.num_aliasing_metadata_outs]

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:1455, in aot_dispatch_autograd.<locals>.CompiledFunction.forward(ctx, *deduped_flat_tensor_args)
   1447 @staticmethod
   1448 def forward(ctx, *deduped_flat_tensor_args):
   1449 
   (...)
   1453     # - Note that in the synthetic bases case, mutated_inputs will correspond to an updated version
   1454     #   of the original view, and not the synthetic base
-> 1455     fw_outs = call_func_with_args(
   1456         CompiledFunction.compiled_fw, deduped_flat_tensor_args, disable_amp=disable_amp
   1457     )
   1459     num_non_aliased_outs = CompiledFunction.num_non_aliased_outs
   1460     num_aliasing_metadata_outs = CompiledFunction.num_aliasing_metadata_outs

File ~/miniconda3/envs/.../lib/python3.10/site-packages/functorch/_src/aot_autograd.py:836, in call_func_with_args(f, args, steal_args, disable_amp)
    834 try:
    835     if hasattr(f, "_boxed_call"):
--> 836         out = normalize_as_list(f(args))
    837     else:
    838         # TODO: Please remove soon
    839         # https://github.com/pytorch/pytorch/pull/83137#issuecomment-1211320670
    840         warnings.warn(
    841             "Your compiler for AOTAutograd is returning a a function that doesn't take boxed arguments. "
    842             "Please wrap it with functorch.compile.make_boxed_func or handle the boxed arguments yourself. "
    843             "See https://github.com/pytorch/pytorch/pull/83137#issuecomment-1211320670 for rationale."
    844         )

File /tmp/torchinductor_noetits/dh/cdhukqo6kp2i5wvu22fwo63insaj3b7qi4fq4a6rr52eycvrif32.py:1270, in call(args)
   1268 del primals_33
   1269 buf1 = aten.convolution(buf0, primals_1, primals_2, (5,), (0,), (1,), False, (0,), 1)
-> 1270 assert_size_stride(buf1, (1, 512, 14879), (7618048, 14879, 1))
   1271 del primals_2
   1272 buf2 = empty_strided((1, 14879, 1), (14879, 1, 14879), device='cpu', dtype=torch.float32)

AssertionError: expected size 512==512, stride 1==14879 at dim=1
```

### Minified repro

I didn't try the minifier, but with the huggingface example, if you add just the `torch.compile()`, I assume you can reproduce the problem

Versions:
Working on a WSL UBUNTU 22.04 LTS

torch                   1.14.0.dev20221204+cpu
torchaudio              0.14.0.dev20221204+cpu
torchvision             0.15.0.dev20221204+cpu

```
 from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
 from datasets import load_dataset
 import torch
 
 # load model and processor
 processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
 model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
 model=torch.compile(model)
     
 # load dummy dataset and read soundfiles
 ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
 
 # tokenize
 input_values = processor(ds[0]["audio"]["array"], return_tensors="pt", padding="longest").input_values
 
 # retrieve logits
 logits = model(input_values).logits
 
 # take argmax and decode
 predicted_ids = torch.argmax(logits, dim=-1)
 transcription = processor.batch_decode(predicted_ids)
```