Unable so save quantized model for mobile using Speech2Text
Author: stefan-falk
Pull request: no

## Environment info

- Platform: Ubuntu 20.04
- Python version: 3.9
- PyTorch version (GPU?): 1.10.0 (yes)

### Who can help

@patrickvonplaten @anton-l 

## Information

I am trying to save a quantized model for speech recognition. Nothing fancy, I'm just trying to explore 🤗 for this topic hoping I can get some models for mobile out of it.

```python
model_path = "facebook/s2t-small-librispeech-asr"
labels = None

# Initialize the model
model = Speech2TextForConditionalGeneration.from_pretrained(model_path)

model = model.eval()
model.gradient_checkpointing_disable()

# Apply quantization / script / optimize for mobile
quantized_model = torch.quantization.quantize_dynamic(model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8)
scripted_model = torch.jit.script(quantized_model)
```

but it appears that there are some nodes which cannot be scripted as the error suggests:

```none
  File "/home/sfalk/miniconda3/envs/speech/lib/python3.9/site-packages/torch/jit/frontend.py", line 702, in build_Call
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/sfalk/miniconda3/envs/speech/lib/python3.9/site-packages/torch/jit/frontend.py", line 702, in <listcomp>
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/sfalk/miniconda3/envs/speech/lib/python3.9/site-packages/torch/jit/frontend.py", line 286, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported:
  File "/home/sfalk/miniconda3/envs/speech/lib/python3.9/site-packages/transformers/modeling_utils.py", line 987
        activations".
        """
        return any(hasattr(m, "gradient_checkpointing") and m.gradient_checkpointing for m in self.modules())
                  ~ <--- HERE
```

## Expected behavior

Well, it would be great if it simply worked but unfortunately there's not really a tutorial or an example out there. I've already raised this issue in https://github.com/huggingface/transformers/issues/14523 - it's not 100% related but I hope that there will be some examples on how we can make these model ready for mobile.


Comments:
We don't fully support torch quantization in `transformers`. Did you try out the optimum library instead: https://github.com/huggingface/optimum . Also gently pinging @michaelbenayoun @echarlaix @lewtun here

Hi @stefan-falk! As @patrickvonplaten mentioned, we don't support native torch quantization. However we're working on enabling ONNX export for all speech models, so that they can be supported by [optimum](https://github.com/huggingface/optimum) in quantization mode. Will keep you updated on that :) 

Hi and thanks for the reply! If you could update me on that matter that would be great, thank you 👍 

If I may I'd like to ask this kind of directly: So.. I just want to be able to train (basically) any speech recognition model from scratch and use it on mobile. I've tried out the Wav2Vec2 Android demo app, which works fine, but the problem here is that training from scratch is really non-trival. I am not sure what I can do here if I am aiming at other languages than English.

My question here is basically: Is there any guideline which I can follow in order to accomplish this or am I making this too complicated and should look into something else? 😆 I realize that there is a cross-language Wav2Vec model but this would be too large for mobile deployment I assume.

Speech-recognition from scratch is really not easy. It also defeats a bit the purpose of Wav2Vec2 which is a *pretrained* speech recognition model that can be *fine-tuned* extremely easily. 

I strongly recommend fine-tuning pretrained speech recognition models instead of training from scratch. We have a lot of pretrained checkpoints on the Hub in different sizes for different tasks that I would strongly recommend to leverage:

- Wav2Vec2: https://huggingface.co/models?other=wav2vec2 : A bunch of wav2vec2 models for speech-recognition and others
- Robust Wav2Vec2: https://huggingface.co/models?arxiv=arxiv:2104.01027 - wav2vec2's robust version. This should work well on more "real-world" data
- XLS-R: https://huggingface.co/models?other=xls_r_pretrained: very good pretrained checkpoints for multi-lingual
- SEW: https://huggingface.co/models?other=sew & https://huggingface.co/models?other=sew-d for low-resource speech recognition
- WavLM: https://huggingface.co/models?other=walm Microsoft's SOTA speech model
- UniSpeech: https://huggingface.co/models?other=unispeech and https://huggingface.co/models?other=unispeech_sat for speaker related tasks 

@patrickvonplaten thank you, this is very helpful. I'll follow you advice and try to stay on the Wav2Vec route then. 

Thanks a lot :)

DistilHubert: https://huggingface.co/ntu-spml/distilhubert could also be interesting for mobile 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.