FX tracing of HubertForSequenceClassification fails with TypeError: 'HFProxy' object cannot be interpreted as an integer
Author: pbelevich
Pull request: no
Labels: name: bug

### System Info

```shell
current main branch, 4.20.0.dev0
```


### Who can help?

@mich

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import inspect

import transformers.utils.fx as fx
from transformers import *

model = HubertForSequenceClassification(HubertConfig())

input_names = {'input_values'}  # model.dummy_inputs.keys() - doesn't work properly for HubertForSequenceClassification
sig = inspect.signature(model.forward)
concrete_args = {p.name: p.default for p in sig.parameters.values() if p.name not in input_names}

hf_tracer = fx.HFTracer()

hf_tracer.trace(model, concrete_args=concrete_args)
```
```
Traceback (most recent call last):
  File "/Users/pbelevich/PycharmProjects/PiPPy/test/hf_test6.py", line 14, in <module>
    hf_tracer.trace(model, concrete_args=concrete_args)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/utils/fx.py", line 924, in trace
    self.graph = super().trace(root, concrete_args=concrete_args)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 587, in trace
    self.create_node('output', 'output', (self.create_arg(fn(*args)),), {},
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/models/hubert/modeling_hubert.py", line 1296, in forward
    outputs = self.hubert(
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 577, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/utils/fx.py", line 881, in call_module
    return super().call_module(m, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 372, in call_module
    return forward(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 573, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/models/hubert/modeling_hubert.py", line 1064, in forward
    hidden_states = self._mask_hidden_states(hidden_states, mask_time_indices=mask_time_indices)
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/models/hubert/modeling_hubert.py", line 988, in _mask_hidden_states
    mask_time_indices = _compute_mask_indices(
  File "/Users/pbelevich/miniconda3/envs/PiPPy/lib/python3.9/site-packages/transformers/models/hubert/modeling_hubert.py", line 132, in _compute_mask_indices
    else [sequence_length for _ in range(batch_size)]
TypeError: 'HFProxy' object cannot be interpreted as an integer
```

### Expected behavior

```shell
No error
```


Comments:
Hi, this seems related to the model being in training mode, hence calling `transformers.models.hubert.modeling_hubert._compute_mask_indices`. The reason this is not caugh by the tests is because we are testing models in eval mode...

Anyways, if you just need for inference purposes, doing `model.eval()` should solve your issue.

A fix for this might not be very easy because the function in question uses numpy... I will try to rewrite it in PyTorch and see how it goes.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.