Module registering None parameters

username: jwillette
cooked: <p>I just want to make sure that I am understanding this part correctly. I see in the MultiheadAttention module (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention</a>) that if the $q, k, v$ tensors are all the same dimension, then the model calls <code>self.register_parameter(&lt;name&gt;, None)</code> which is the used later in the <code>nn.functional.multi_head_attention_forward</code> function.</p>
<p>If the registered parameters are <code>None</code> (or like a few lines before that <code>nn.Parameter(torch.empty(&lt;size&gt;))</code>) then that would mean that they are there, but they have no effect on the calculation, more like just a placeholder so they exist. Is this correct?</p>

username: albanD
cooked: <p>This means that they are defined but don’t have values.<br>
This is useful to make sure that user code won’t try to use this attribute (as it is already reserved to be a Parameter) leading to weird behavior later.</p>
<p>Note that <code>torch.empty(&lt;size&gt;)</code> actually returns a Tensor, but it contains uninitialized memory (so can have any values in it).</p>

username: jwillette
cooked: <p>So then do these <code>None</code> parameters just have a null effect on the forward pass? like an identity matrix or something? Or is the calculation just kind of skipped in the forward pass?</p>

username: albanD
cooked: <p>It depends on how the forward function is implemented. If you try to access them, they will be None.<br>
For example, the linear layer has this condition in the forward pass <a href="https://github.com/pytorch/pytorch/blob/7aa605ed92dbb726a2f285246369b0a17972ba12/torch/nn/functional.py#L1594-L1595">here</a> where it checks if the bias is None or not.</p>

username: jetcai1900
cooked: <p>COuld you elaborate more on what register_parameter(“a”, None) means? Does this mean “a” would not be used? Or just a is initialized to be none? Thanks.</p>

username: albanD
cooked: <p>It means it is initialized to None.<br>
That allows you to do checks like <code> if self.a is None</code> instead of having to check if the attribute “a” exists every time.</p>

username: jetcai1900
cooked: <p>What is the advantage of using self.a instead of using the attribute “a”? I feel they are the same?</p>

username: albanD
cooked: <p>Yes they are very similar. The main advantages here are:</p>
<ul>
<li>Users can’t set anything but a Parameter on <code>a</code>. So the attribute named is “reserved” for a Parameter</li>
<li>I would argue that it is nicer to read</li>
</ul>

username: jetcai1900
cooked: <p>I see. Thanks a lot for your response. Could I ask a last question? Why after setting the “weight” parameter in BatchNorm1d to none, the BatchNorm1d cannot work in pytorch 1.5.0? But in pytorch 0.1.12, setting the “weight” parameter to none does not affect the usage of BatchNorm1d? Thanks so much.</p>

username: albanD
cooked: <p>Most likely due to some implementation details and refactors.<br>
You should use <code>affine=False</code> if you don’t want the linear transformation after the batchnorm.</p>

username: jetcai1900
cooked: <p>I see. Thanks a lot for your help!</p>