Saving PruneBERT notebook fails to run on torch > 1.5
Author: lewtun
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@VictorSanh 

## Information

The Saving PruneBERT [notebook](https://github.com/huggingface/transformers/blob/b11386e158e86e62d4041eabd86d044cd1695737/examples/movement-pruning/Saving_PruneBERT.ipynb) from the _examples/movement-pruning/_ directory is not compatible with PyTorch > 1.5 because Torchbind is used for `_packed_params` in v1.6 and higher (see PR [here](https://github.com/pytorch/pytorch/pull/34140)).

In particular, cell 4 of the notebook

```python
# Elementary representation: we decompose the quantized tensors into (scale, zero_point, int_repr).
# See https://pytorch.org/docs/stable/quantization.html

# We further leverage the fact that int_repr is sparse matrix to optimize the storage: we decompose int_repr into
# its CSR representation (data, indptr, indices).

elementary_qtz_st = {}
for name, param in qtz_st.items():
    if param.is_quantized:
        print("Decompose quantization for", name)
        # We need to extract the scale, the zero_point and the int_repr for the quantized tensor and modules
        scale = param.q_scale()                                # torch.tensor(1,) - float32
        zero_point = param.q_zero_point()                      # torch.tensor(1,) - int32
        elementary_qtz_st[f"{name}.scale"] = scale
        elementary_qtz_st[f"{name}.zero_point"] = zero_point

        # We assume the int_repr is sparse and compute its CSR representation
        # Only the FCs in the encoder are actually sparse
        int_repr = param.int_repr()                         # torch.tensor(nb_rows, nb_columns) - int8
        int_repr_cs = sparse.csr_matrix(int_repr)           # scipy.sparse.csr.csr_matrix

        elementary_qtz_st[f"{name}.int_repr.data"] = int_repr_cs.data                  # np.array int8
        elementary_qtz_st[f"{name}.int_repr.indptr"] = int_repr_cs.indptr              # np.array int32
        assert max(int_repr_cs.indices) < 65535 # If not, we shall fall back to int32
        elementary_qtz_st[f"{name}.int_repr.indices"] = np.uint16(int_repr_cs.indices) # np.array uint16
        elementary_qtz_st[f"{name}.int_repr.shape"] = int_repr_cs.shape                # tuple(int, int)
    else:
        elementary_qtz_st[name] = param
```
fails with the following error

```
AttributeError                            Traceback (most recent call last)
<ipython-input-14-1266eb0d5085> in <module>
      9 #     if isinstance(param, tuple):
     10 #         param = param[0]
---> 11     if "dtype" not in name and param.is_quantized:
     12         print("Decompose quantization for", name)
     13         # We need to extract the scale, the zero_point and the int_repr for the quantized tensor and modules

AttributeError: 'tuple' object has no attribute 'is_quantized'
```

This is because in torch >= 1.6, the `layer_name.weight` and `layer_name.bias` tensors have been bundled as a tuple of the form `(weight, bias)` in `param`.

A simple fix I tried was to pick out the weight tensor directly by checking for a tuple in the for loop:

```
elementary_qtz_st = {}
for name, param in qtz_st.items():
    if isintstance(param, tuple):
        param = param[0]
    if param.is_quantized:
        print("Decompose quantization for", name)
```

but this produces mismatch between the keys of `qtz_st` and `elementary_qtz_st` because we append the `.scale` and `.zero_point` attributes to `_packed_params` and lose the bias term.

I'm currently trying to find a proper fix, but thought I should report this in the meantime.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Clone `transformers` and follow the steps to install the `movement-pruning` example
2. Upgrade torch to v1.6 with `pip install torch==1.6`
3. Try to run the `Saving_PruneBERT.ipynb` notebook

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The `Saving_PruneBERT.ipynb` notebook runs end-to-end without errors.

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Thanks for reporting that @lewtun!
Feel free to open a PR when you have a working solution for higher versions of PyTorch!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.