Dynamic Quantization on ALBERT (pytorch) 
Author: Rachnas
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi,
Thank you for providing great documentation on quantization:
https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html
I am trying similar steps on Albert Pytorch model, converted "albert-base-v1" to quantized one by applying dynamic quantization on linear layers. At inference stage (with quantized model), I get following error:
w = ( self.dense.weight.t()
         .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
          .to(context_layer.dtype)
AttributeError: 'function' object has no attribute 't'
Any pointers about how to solve this error ?  


Comments:
I know this doesn't directly answer the question, but I have been playing around with quantization of BERT and everything is good until I want to load the model into my notebook.  The size of the model is inflated back to over 400 MB from under 200 MB, and the accuracy takes a huge hit.  I noticed this when I tried to load the quantized model in the notebook of the pytorch tutorial as well.  Have you been able to successful load in and use a quantized model in the first place?

I tested `albert-base-v1` as well, since I can't get `albert-base-v2` to work (created an issue), and I can confirm that I am getting the same error.  When `outputs = quantized_model(input_ids, labels=labels)` is run, the error occurs.

@ElektrikSpark , I can evaluate using quantized bert model as shown in the documentation. Accuracy is low as compared to original Bert.  After saving quantized model, I tried loading it from command line, it is not working for me. 
With Albert, quantization step is not completing.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any solution to this error?

Same issue here.

I found issue while loading the quantized bert model, accuracy score decreases significantly. Does this mean, we can't use quantized BERT for production? I am not sure then why this [tutorial](https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html) was provided

I was able to solve this issue by using this

`model = torch.quantization.quantize_dynamic(
                    big_model, {torch.nn.Bilinear}, dtype=torch.qint8
                    )
`

Notice I used Bilinear instead of Linear, now dont ask me why, I just saw someone do something similar while quantizing GPT2 model

for those still looking for a workaround solution to this issue: you may try following changes to AlbertAttention.forward()

        ...
        # Should find a better way to do this
        # w = (
        #     self.dense.weight.t()
        #     .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
        #     .to(context_layer.dtype)
        # )
        # b = self.dense.bias.to(context_layer.dtype)
        #
        # note that dequantize() is required as quantized tensor with dtype.qint8 cannot be converted to
        # dtype.float32 by calling .to(context_layer.dtype).
        #
        # Different from self.dense.weight(), self.dense.bias() returns regular tensor not quantized tensor
        w = (
            (self.dense.weight().t().dequantize() if callable(self.dense.weight) else self.dense.weight.t())
            .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
            .to(context_layer.dtype)
        )
        b = (self.dense.bias() if callable(self.dense.bias) else self.dense.bias) \
            .to(context_layer.dtype)


I ran into the same problem，like this:
    model_pt_quantized(input_ids=model_inputs["input_ids"], token_type_ids=model_inputs["token_type_ids"], attention_mask=model_inputs["attention_mask"])
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 563, in forward
    output_hidden_states=output_hidden_states,
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 346, in forward
    output_hidden_states,
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 299, in forward
    layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 277, in forward
    attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 251, in forward
    self.dense.weight.t()
AttributeError: 'function' object has no attribute 't'
Does pytorch support dynamic albert quantization now？