Wav2vec2 quantization dimention error

username: mi_gupta
cooked: <p>i have been trying to quantise the WAV2VEC2_ASR_BASE_960H model using static_fx mode but getting the example input size error. not sure what size to give (sample input size is (1,54400) and sampling rate is 16k) . also can anybody help me to get the steps to static quantize the model</p>
<pre><code class="lang-auto">import torch
import torchaudio
bundle = torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H
model = bundle.get_model()
from torch.quantization import quantize_fx
import copy
m = copy.deepcopy(model)
m.eval()
qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs=torch.rand(1,54400))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8ed4eed25825d71925380fd687e33933d6759a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8ed4eed25825d71925380fd687e33933d6759a4" title="torch_Screenshot 2024-02-05 104324"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8ed4eed25825d71925380fd687e33933d6759a4.png" alt="torch_Screenshot 2024-02-05 104324" data-base62-sha1="o6oE2d348XmW1Q3u0hj4axhdG84" width="690" height="373" data-dominant-color="383939"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch_Screenshot 2024-02-05 104324</span><span class="informations">1137×615 37.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: HDCharles
cooked: <p>This doesn’t look like an example input size error, it looks like symbolic tracing error. If you don’t do quantization but do normal fx tracing does your model work?</p>
<p><a href="https://pytorch.org/docs/stable/fx.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/fx.html</a></p>

username: mi_gupta
cooked: <p>i tried with</p>
<pre><code class="lang-auto">from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from torch.quantization import quantize_fx
model_f = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

# test = m_torch.encoder
# test = Transformer()

# from torch.quantization import quantize_fx
# m = copy.deepcopy(m_torch.encoder)

m = copy.deepcopy(model_f)


m.eval()
qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
# Prepare
model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs = torch.randn(768,512))
# Calibrate - Use representative (validation) data.
with torch.inference_mode():
  for _ in range(10):
    x = torch.randn(768,512)
    model_prepared(x)
# quantize
model_quantized = quantize_fx.convert_fx(model_prepared)

</code></pre>
<p>getting error</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-7-a2e234fa7675&gt; in &lt;cell line: 17&gt;()
     15 qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
     16 # Prepare
---&gt; 17 model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs = torch.randn(768,512))
     18 # Calibrate - Use representative (validation) data.
     19 with torch.inference_mode():

10 frames
/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py in _get_feature_vector_attention_mask(self, feature_vector_length, attention_mask, add_adapter)
   1151         batch_size = attention_mask.shape[0]
   1152 
-&gt; 1153         attention_mask = torch.zeros(
   1154             (batch_size, feature_vector_length), dtype=attention_mask.dtype, device=attention_mask.device
   1155         )

TypeError: zeros() received an invalid combination of arguments - got (tuple, device=Attribute, dtype=Attribute), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
</code></pre>

username: mi_gupta
cooked: <aside class="quote no-group" data-username="HDCharles" data-post="2" data-topic="196633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hdcharles/48/38408_2.png" class="avatar"> HDCharles:</div>
<blockquote>
<p>normal fx tracing does your model work?</p>
<p><a href="https://pytorch.org/docs/stable/fx.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/fx.html </a></p>
</blockquote>
</aside>
<p>for normal tracing also getting same error</p>
<pre><code class="lang-auto">from torch.fx import symbolic_trace
# Symbolic tracing frontend - captures the semantics of the module
symbolic_traced : torch.fx.GraphModule = symbolic_trace(model)
</code></pre>

username: HDCharles
cooked: <p>in that case its not a quantization issue, its a tracing issue, you can either use eager mode quantization (to remove the need for traceability) or get this model to be traceable. I’m not an expert but i think the tracing can’t do different things based on the input data so if you hardcode dtype and device in that line of the model, it may get past that point.</p>
<p>attention_mask = torch.zeros(<br>
(batch_size, feature_vector_length), dtype=&lt;torch.fp32_or_whatever&gt;, device=&lt;“cpu” probably&gt;<br>
)</p>