Defining a module parameter as a function of other module parameters

username: Andrew_Cirincione
cooked: <p>Hello! I am currently working on a custom module inside my pytorch model. This module creates a gaussian kernel that is convolved across the models input, while the parameters of the gaussian kernel are to be learned. There is to be 256 x 256 gaussian kernels generated. My <strong>init</strong> function looks like this:</p>
<pre><code class="lang-auto">    def __init__(self, channel_num = 512, size = 32, ksizeDN = 12):
        super().__init__()
        self.channel_num = channel_num
        self.size = size
        self.ksizeDN = ksizeDN

        self.thetaD = torch.nn.Parameter(uniform.Uniform(0, torch.pi).sample([self.channel_num, self.channel_num]),
                                    requires_grad=True)

        self.p = torch.nn.Parameter(uniform.Uniform(2, 6).sample([self.channel_num, self.channel_num]),
                                    requires_grad=True)

        self.sig = torch.nn.Parameter(uniform.Uniform(2, 6).sample([self.channel_num, self.channel_num]),
                                      requires_grad=True)

        self.a = torch.nn.Parameter(
            torch.abs(torch.randn(self.channel_num, self.channel_num, requires_grad=True)))
        #self.nU = torch.nn.Parameter(torch.abs(torch.randn(1, self.channel_num, 1, 1, requires_grad=True)))

        self.gaussian_bank = torch.nn.Parameter(torch.zeros(self.channel_num, self.channel_num, self.ksizeDN * 2 + 1,
                                                            self.ksizeDN * 2 + 1), requires_grad=False)
        self.x = torch.linspace(-self.ksizeDN, self.ksizeDN, self.ksizeDN * 2 + 1)
        self.y = torch.linspace(-self.ksizeDN, self.ksizeDN, self.ksizeDN * 2 + 1)
        self.xv, self.yv = torch.meshgrid(self.x, self.y)

        for i in range(self.channel_num):
            for u in range(self.channel_num):
                self.gaussian_bank[i, u, :, :] = self.get_gaussian(i, u)

</code></pre>
<p>The first four parameters are variables in the gaussian kernel equation (self.get_gaussian), which is called when constructing the ‘gaussian bank’ in the for loop. The gaussian bank is 256 x 256 x 25 x 25. For reference, the gaussian kernel equation is defined as:</p>
<pre><code class="lang-auto">    def get_gaussian(self, cc, oc):

        xrot = (self.xv * torch.cos(self.thetaD[cc, oc]) + self.yv * torch.sin(self.thetaD[cc, oc]))
        yrot = (-self.xv * torch.sin(self.thetaD[cc, oc]) + self.yv * torch.cos(self.thetaD[cc, oc]))
        g_kernel = torch.tensor((abs(self.a[cc, oc]) /
                                 (2 * torch.pi * self.p[cc, oc] * self.sig[cc, oc])) * \
                                torch.exp(-0.5 * ((((xrot) ** 2) / self.p[cc, oc] ** 2) +
                                                  (((yrot) ** 2) / self.sig[cc, oc] ** 2))))

        return g_kernel
</code></pre>
<p>After running the training on this, I notice that even though the parameters are changing, the ‘gaussian bank’ parameter never changes. In fact, all the parameters (thetaD, a, sig and p) all seem to go to zero over multiple epochs, which does not seem right. How can I define the gaussian bank parameter so that the model updates the gaussian bank whenever the parameters are adjusted? I am a bit new to pytorch, so apologies for any confusion - I am willing to answer anything I missed and I am very appreciative for any help!</p>

username: KFrank
cooked: <p>Hi Andrew!</p>
<aside class="quote no-group" data-username="Andrew_Cirincione" data-post="1" data-topic="148274" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrew_cirincione/48/46169_2.png" class="avatar"> Andrew_Cirincione:</div>
<blockquote>
<p>This module creates a gaussian kernel that is convolved across the models input, while the parameters of the gaussian kernel are to be learned.</p>
</blockquote>
</aside>
<p>The parameters that are to be learned should be the <code>Parameter</code>s  of<br>
your <code>Module</code>.  <code>gaussian_bank</code> should just be a regular variable – not<br>
a <code>Parameter</code> – because it is not being “learned” (optimized) directly.</p>
<p>(I assume that <code>gaussian_bank</code> is what you are referring to as your<br>
“gaussian kernel.”)</p>
<p>I don’t think I would even have <code>gaussian_bank</code> be a “property” of your<br>
<code>Module</code>.  (That is, I would not have it be <code>self.gaussian_bank</code>, but rather<br>
just a “free” variable in your <code>Module</code>’s <code>forward()</code> method.)</p>
<aside class="quote no-group">
<blockquote>
<p>After running the training on this, I notice that even though the parameters are changing, the ‘gaussian bank’ parameter never changes. In fact, all the parameters (thetaD, a, sig and p) all seem to go to zero over multiple epochs, which does not seem right.</p>
</blockquote>
</aside>
<p>Something seems fishy here.</p>
<p>You say that <code>thetaD</code>, <code>a</code>, <code>sig</code>, and <code>p</code> – the parameters from which you<br>
derive <code>gaussian_bank</code> – are changing, specifically, they “go to zero.”<br>
However, <code>gaussian_bank</code> “never changes.”</p>
<p>Looking superficially at your code, I do see:</p>
<pre data-code-wrap="python"><code class="lang-python">self.gaussian_bank[i, u, :, :] = self.get_gaussian(i, u)
</code></pre>
<p>So you are modifying <code>self.gaussian_bank</code> (inplace, because you are<br>
indexing into it).  Therefore, unless <code>get_gaussian()</code> is returning a constant<br>
for some reason, <code>gaussian_bank</code> really should be changing, reflecting the<br>
changes in its underlying parameters (such as <code>thetaD</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Andrew_Cirincione
cooked: <p>Great, it looks like this might be leading me in the right direction. One other thing I was wondering - it seems like weight decay may not be appropriate to use on these parameters as it seems to punish them too much, which is possibly why these parameters kept shooting down to zero.</p>
<p>Do you know any way that I can apply weight decay to specific modules and not to others? For example, I have a resnet backend that I want to apply weight decay to, but I dont want to apply it in this module. Thanks for the help so far!</p>

username: KFrank
cooked: <p>Hi Andrew!</p>
<aside class="quote no-group" data-username="Andrew_Cirincione" data-post="3" data-topic="148274" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrew_cirincione/48/46169_2.png" class="avatar"> Andrew_Cirincione:</div>
<blockquote>
<p>it seems like weight decay may not be appropriate to use on these parameters</p>
</blockquote>
</aside>
<p>My intuition would lead me to agree with you on this.  The way I look at<br>
it, there is a lot of redundancy in the weights of a typical network.  There<br>
are lots of individual scalar parameters, many of which can trade off against<br>
one another – weight A and weight B might both be able to play the same<br>
role as one another, so you don’t need both (but you don’t know ahead of<br>
time which you could leave out).</p>
<p>Or weight A and weight B could compensate for one another so that<br>
weight A could run off to <code>+inf</code> while weight B runs off to <code>-inf</code>.  Weight<br>
decay (and other regularization techniques) serve to “tamp down” this<br>
redundancy.</p>
<p>But your gaussian-kernel parameters don’t seem to me to be like this.<br>
Each one has a well-defined meaning, they don’t really overlap with<br>
one another, and you can’t leave one out without restricting the structure<br>
of your gaussian kernel.</p>
<p>So not applying weight decay to your gaussian-kernel parameters makes<br>
sense to me.  (You could also just apply much weaker weight decay to them<br>
than to your “regular” network parameters, but I would lean towards no<br>
weight decay at all.)</p>
<aside class="quote no-group">
<blockquote>
<p>Do you know any way that I can apply weight decay to specific modules and not to others?</p>
</blockquote>
</aside>
<p>Yes.  Use the <a href="https://pytorch.org/docs/1.11/optim.html#per-parameter-options" rel="noopener nofollow ugc">parameter-group</a> feature of pytorch’s <code>Optimizer</code>s.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Andrew_Cirincione
cooked: <p>Awesome, thanks for all the help!</p>