Setting custom paramter in my own model

username: Karthik_Ganesan
cooked: <p>I have a custom implementation of Conv2D and I want to add a custom value to this model. Essentially, I want to set this value at the model level and then have access to it within MyConv2D. I tried to do this by adding a new parameter to the model definition and then passing that value my layer, like this:</p>
<pre><code class="lang-auto">class custom_model(nn.Module):
  def __init__(self):
    super(custom_model, self).__init__()
    self.custom_value = torch.nn.Parameter(torch.zeros(1), requires_grad=False)
    self.conv_1 = MyConv2d((self.custom_value.item()), 1, 4, 5, stride=1, padding=0)
    ...
</code></pre>
<p>Then when I load the model, I first add this new parameter to my state_dict:</p>
<pre><code class="lang-auto">copied_state_dict = torch.load(PATH, map_location=device)
copied_state_dict['custom_value'] = torch.tensor([1])
custom_net.load_state_dict(copied_state_dict)
</code></pre>
<p>Not only does this seem like a really inelegant solution but it also doesn’t work! Because I initialize <code>custom_value</code> with 0, it always stays zero. And that’s the point where I’ve exhausted my knowledge of PyTorch!</p>
<p>So is there a nice way for me to set a property (parameter or otherwise) at a model level and have access to it in my own layer code? I am not planning to train this model. I just load weights from a pre-trained model and only run inference.</p>
<ol>
<li>I am OK to have it as a parameter. If I can directly access it within the <code>MyConv2D</code> layer, that’s all I need.</li>
<li>If I can use some other property of a model (I’ve seen mention of a buffer but not sure if that is a better solution here). But I would like an easy way to set this property before I run the model, as I need to change this property for each run.</li>
</ol>
<p>Thanks!</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Karthik_Ganesan" data-post="1" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/48/34393_2.png" class="avatar"> Karthik_Ganesan:</div>
<blockquote>
<p>So is there a nice way for me to set a property (parameter or otherwise) at a model level and have access to it in my own layer code?</p>
</blockquote>
</aside>
<p>Question: Why not pass the <code>custom_value</code> while calling the <code>self.conv_1</code> in <code>custom_model</code>’s <code>forward()</code>? This way, either you can set the <code>custom_value</code> at each run before <code>forward()</code> call or directly parameterize the <code>forward()</code> with <code>custom_value</code>.</p>
<aside class="quote no-group" data-username="Karthik_Ganesan" data-post="1" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/48/34393_2.png" class="avatar"> Karthik_Ganesan:</div>
<blockquote>
<p>I am not planning to train this model.</p>
</blockquote>
</aside>
<p>Then <code>custom_value</code> is not expected to get gradients?<br>
If so, it doesn’t need to be <code>nn.Parameter()</code>.</p>
<aside class="quote no-group" data-username="Karthik_Ganesan" data-post="1" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/48/34393_2.png" class="avatar"> Karthik_Ganesan:</div>
<blockquote>
<p>I initialize <code>custom_value</code> with 0, it always stays zero.</p>
</blockquote>
</aside>
<p>By the time you are trying to set <code>custom_value</code>, <code>self.conv_1</code> have been already created with its old value. (i.e., 0)</p>

username: Karthik_Ganesan
cooked: <aside class="quote no-group" data-username="InnovArul" data-post="2" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Why not pass the <code>custom_value</code> while calling the <code>self.conv_1</code> in <code>custom_model</code>’s <code>forward()</code>? This way, either you can set the <code>custom_value</code> at each run before <code>forward()</code> call or directly parameterize the <code>forward()</code> with <code>custom_value</code>.</p>
</blockquote>
</aside>
<p>Yes, I can set it as a parameter directly, but then I would have to do this for each layer right? That’s why i was seeing if there is a way to set something at the model level that I can access within each layer.</p>
<aside class="quote no-group" data-username="InnovArul" data-post="2" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Then <code>custom_value</code> is not expected to get gradients?<br>
If so, it doesn’t need to be <code>nn.Parameter()</code>.</p>
</blockquote>
</aside>
<p>No, like I said in my post, I was just using parameter since I can set it easily using state_dict(). Again, it would be ideal if it is something I can set about the model before I run it. But doesn’t have to be parameter no. As long as I can set it easily.</p>
<aside class="quote no-group" data-username="InnovArul" data-post="2" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>By the time you are trying to set <code>custom_value</code>, <code>self.conv_1</code> have been already created with its old value. (i.e., 0)</p>
</blockquote>
</aside>
<p>Yes, so the question is is there something else I could use instead of custom_value here to pass to my MyConv2D layer. So one thing you mentioned was:</p>
<aside class="quote no-group" data-username="InnovArul" data-post="2" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>directly parameterize the <code>forward()</code> with <code>custom_value</code>.</p>
</blockquote>
</aside>
<p>Do you mean, just pass an additional parameter to my <code>MyConv2D.forward()</code>? I can do that too but it would not be anything associated with my model then right? If there isn’t a way to do this, that’s probably what I’ll end up doing.</p>

username: InnovArul
cooked: <p>I am not sure of your usecase and why do you take this route of modifying with <code>state_dict()</code>.</p>
<p>There are two things that you could try:</p>
<ol>
<li></li>
</ol>
<aside class="quote no-group" data-username="Karthik_Ganesan" data-post="1" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/48/34393_2.png" class="avatar"> Karthik_Ganesan:</div>
<blockquote>
<pre><code class="lang-auto">    self.conv_1 = MyConv2d((self.custom_value.item()), 1, 4, 5, stride=1, padding=0)
</code></pre>
</blockquote>
</aside>
<p>Try passing <code>self.custom_value</code> without <code>.item()</code>. i.e., pass the reference of the object so that if you change the values, it gets reflected in <code>self.conv_1</code>.</p>
<ol start="2">
<li>Instead of dealing with <code>state_dict</code>, why not create a method to set <code>custom_value</code> dynamically?<br>
and set it as <code>self.custom_value[0] = &lt;new val&gt;</code> without creating new tensor.</li>
</ol>

username: Karthik_Ganesan
cooked: <aside class="quote no-group" data-username="InnovArul" data-post="4" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Try passing <code>self.custom_value</code> without <code>.item()</code>. i.e., pass the reference of the object so that if you change the values, it gets reflected in <code>self.conv_1</code>.</p>
</blockquote>
</aside>
<p>I tried that already but unfortunately, it then passes it as a paramter to my layer so then I need to also add <code>MyConv2D.custom_value</code> as well, which would just be a copy of the same <code>custom_value</code> in each layer, which is why I tried to only pass the value instead of the parameter.</p>
<aside class="quote no-group" data-username="InnovArul" data-post="4" data-topic="137262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Instead of dealing with <code>state_dict</code>, why not create a method to set <code>custom_value</code> dynamically?<br>
and set it as <code>self.custom_value[0] = &lt;new val&gt;</code> without creating new tensor.</p>
</blockquote>
</aside>
<p>Could you please elaborate on this a bit? so this method would just return the value I want to set? Where should I call this line? Before I run my network? And for this, would it be OK if I made custom_value just a tensor and not a parameter (since it doesn’t need grad anyway).</p>

username: InnovArul
cooked: <p>I meant something like this:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/InnovArul/7a3133925df1091aee462148204bb9c9">
  <header class="source">

      <a href="https://gist.github.com/InnovArul/7a3133925df1091aee462148204bb9c9" target="_blank" rel="noopener">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/7a3133925df1091aee462148204bb9c9" target="_blank" rel="noopener">https://gist.github.com/InnovArul/7a3133925df1091aee462148204bb9c9</a></h4>

  <h5>custom_val.py</h5>
  <pre><code class="Python">import torch, torch.nn as nn

class LowlevelModule(nn.Module):
    def __init__(self, custom_val):
        super().__init__()
        self.custom_val = custom_val

    def print_custom_val(self):
        print(self.custom_val.item())
</code></pre>
    This file has been truncated. <a href="https://gist.github.com/InnovArul/7a3133925df1091aee462148204bb9c9" target="_blank" rel="noopener">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Karthik_Ganesan
cooked: <p>Thank you, I think this will work for me.  ̶B̶u̶t̶ ̶i̶s̶ ̶t̶h̶e̶r̶e̶ ̶a̶ ̶n̶i̶c̶e̶r̶ ̶w̶a̶y̶ ̶t̶o̶ ̶s̶e̶t̶ ̶<code>̶m̶.̶c̶u̶s̶t̶o̶m̶_̶v̶a̶l̶[̶0̶]̶</code>̶ ̶h̶e̶r̶e̶?̶ ̶I̶ ̶c̶a̶n̶ ̶k̶e̶e̶p̶ ̶m̶y̶ ̶c̶u̶s̶t̶o̶m̶_̶v̶a̶l̶u̶e̶ ̶a̶s̶ ̶t̶h̶e̶ ̶f̶i̶r̶s̶t̶ ̶e̶l̶e̶m̶e̶n̶t̶ ̶b̶u̶t̶ ̶i̶s̶ ̶t̶h̶e̶r̶e̶ ̶a̶ ̶n̶i̶c̶e̶r̶ ̶w̶a̶y̶ ̶t̶o̶ ̶s̶e̶t̶ ̶t̶h̶i̶s̶ ̶s̶o̶ ̶t̶h̶a̶t̶ ̶i̶f̶ ̶<code>̶c̶u̶s̶t̶o̶m̶_̶v̶a̶l̶u̶e̶</code>̶ ̶i̶s̶n̶’̶t̶ ̶t̶h̶e̶ ̶f̶i̶r̶s̶t̶ ̶e̶l̶e̶m̶e̶n̶t̶,̶ ̶i̶t̶ ̶w̶o̶u̶l̶d̶ ̶s̶t̶i̶l̶l̶ ̶w̶o̶r̶k̶?̶</p>
<p>EDIT: My mistake. I thought this was to access to the first element of the model, instead of just a way to access the tensor value!</p>