How to access the parameters of the uppermost layer of the HuggingFace Transformers via ".modules()"?
Author: h56cho
Pull request: no

Hello,
I would like to apply the function `module.PyroSample()` to the parameters that pertains to the 24th layer (the uppermost layer) of the `RobertaForMultipleChoice` pre-trained model (`roberta-large`). How should I fix the loop below so that I only fix the parameters that are from the 24th layer? Currently, the loop applies `module.PyroSample()` to every parameter except the `_dummy_param`.
Thank you,

```python
for m in model_RobertaForMultipleChoice.modules():
     for name, value in list(m.named_parameters(recurse=False)):
         if name != "_dummy_param":
                    setattr(m, name, module.PyroSample(prior=dist.Normal(0, 1)
                                              .expand(value.shape)
                                              .to_event(value.dim()))) 

Comments:
Hello,
I'd hate to keep bother you on this, but could you quickly tell me how I should adjust `model_RobertaForMultipleChoice.modules()` to only return those parameters that pertains to the uppermost layer of `roberta-large` pre-trained model?

I know how to convert the HuggingFace Transformers into Pyro models now - for this, I just need to add a dummy parameter after converting the Transformer into a Pyro model, and then define `svi` and do `svi.step()` for training. But the problem I am facing now is that the Transformer, when converted into a Pyro model, causes memory leak and my EC2 instance can't handle the amount of analysis needed for the Pyro model. So now I am trying to convert only the 24th layer of the `roberta-large` model into a Pyro layer, and see if my instance can handle it at that capacity.

How should I adjust the statement `model_RobertaForMultipleChoice.modules()` to only return those parameters that pertains to the uppermost layer (24th layer) of the `roberta-large` pre-trained model? Thank you,

I think you can leverage the variable `m._pyro_name`. For example running this code:
```python
from transformers import RobertaForMultipleChoice, RobertaConfig
import pyro.distributions as dist
import pyro.nn.module as module

# get the pre-trained HuggingFace RobertaForMultipleChoice and resize the token embeddings
# after adding the special token
model_RobertaForMultipleChoice = RobertaForMultipleChoice(RobertaConfig())

# convert the HuggingFace model into a pyro model
module.to_pyro_module_(model_RobertaForMultipleChoice)

for m in model_RobertaForMultipleChoice.modules():
    for name, value in list(m.named_parameters(recurse=False)):
        if "roberta.encoder.layer.11" in m._pyro_name:
            print(f"Set weights for: {m._pyro_name}")
            setattr(m, name, module.PyroSample(prior=dist.Normal(0, 1).expand(value.shape).to_event(value.dim())))
```

would only set the weights of the 11th layer to a `PyroSample`.
Running the code above gives the following output:
```
Set weights for: roberta.encoder.layer.11.attention.self.query
Set weights for: roberta.encoder.layer.11.attention.self.query
Set weights for: roberta.encoder.layer.11.attention.self.key
Set weights for: roberta.encoder.layer.11.attention.self.key
Set weights for: roberta.encoder.layer.11.attention.self.value
Set weights for: roberta.encoder.layer.11.attention.self.value
Set weights for: roberta.encoder.layer.11.attention.output.dense
Set weights for: roberta.encoder.layer.11.attention.output.dense
Set weights for: roberta.encoder.layer.11.attention.output.LayerNorm
Set weights for: roberta.encoder.layer.11.attention.output.LayerNorm
Set weights for: roberta.encoder.layer.11.intermediate.dense
Set weights for: roberta.encoder.layer.11.intermediate.dense
Set weights for: roberta.encoder.layer.11.output.dense
Set weights for: roberta.encoder.layer.11.output.dense
Set weights for: roberta.encoder.layer.11.output.LayerNorm
Set weights for: roberta.encoder.layer.11.output.LayerNorm
```

Let me know if this works for you. Also, if you manage to get a Bayesian Neural Network using `transformers` it would be amazing if you can share some code or add a notebook :-)