Dynamic parameter declaration in forward function

ypxie: <p>In the current pytorch example, all the parameters have to be pre-defined in the class <strong>init</strong> function or use existing nn.Module, like nn.Linear.  However, this requires us to compute the parameters size correctly for the whole graph, which can be quite tedious and error prone when the model become complex.</p>
<p>Declare the parameters in the forward function seems to be a solution because the intermediate results are already known at that point, but the thing is this might make the parameters be declared every time we run the forward function.</p>
<p>Tensorflow solve this problem by proving a <code>tf.get_variable</code> function.<br>
can we also add something like <code>Module.get_parameter</code>, which create a new parameter or return an existing one if it already exists in the Module.parameters?</p>
<pre><code class="lang-auto">def conv_relu(input):
    # Create variable named "weights", its shape can be infered from input.
    kernel_shape = (64, input.size()[0],3,3)
    weights = nn.Module.get_parameter("weights", kernel_shape,
        initializer='uniform')
    conv = F.conv2d(input, weights)
    return F.relu(conv)
</code></pre>
<p><a href="https://www.tensorflow.org/versions/r1.0/how_tos/variable_scope/" class="onebox" target="_blank" rel="nofollow noopener">https://www.tensorflow.org/versions/r1.0/how_tos/variable_scope/</a></p>

apaszke: <p>You can create parameters in the forward function too. Just guard them with an <code>if</code> to prevent reassigning at every iteration:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('weight', None)

    def forward(self, input):
        if self.weight is None:
            self.weight = nn.Parameter(torch.randn(input.size()))
        return self.weight @ input
</code></pre>

ypxie: <p>In this way, model.cuda() which is usually called before forward() might not work properly. if one more if_else is added to check use_cuda,  the code can be unnecessarily long.</p>

apaszke: <p>Good point. This would be better then:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('weight', None)

    def reset_parameters(self, input):
        self.weight = nn.Parameter(input.new(input.size()).normal_(0, 1))    

    def forward(self, input):
        if self.weight is None:
            self.reset_parameters(input)
        return self.weight @ input
</code></pre>
<p><code>input.new</code> will create a new tensor of the same type as input, and it will be placed on the same GPU as input.</p>

ypxie: <p>Thanks for your reply, but I didn’t get why would this help? It seems to me register_parameter just register a None parameter to the parameters list.  How could model.cuda() affect a None parameter?</p>

ypxie: <p>How about the following?</p>
<pre><code class="lang-auto">def cuda(self, device_id=None):
        """Moves all model parameters and buffers to the GPU.

        Arguments:
            device_id (int, optional): if specified, all parameters will be
                copied to that device
        """
        self._cuda = Ture
        self._device_id = device_id
        return self._apply(lambda t: t.cuda(device_id))

def cpu(self, device_id=None):
        """Moves all model parameters and buffers to the CPU."""
        self._cuda = False
        return self._apply(lambda t: t.cpu())

def get_parameter(self, name, shape):
       if not name in self._parameters.keys():
            self._parameters[name] = nn.Parameter(torch.randn(shape))
       var = self._parameters[name]      
       fn = lambda t: t.cuda(self._device_id) if self._cuda else lambda t: t.cpu()
       var.data = fn(var.data)
       if var.grad is not None: 
           var.grad.data = fn(var.grad.data)
       return var

</code></pre>

apaszke: <p><code>model.cuda()</code> won’t affect it, unless it has be reassigned. However, if you call <code>model.cuda()</code> and then forward a CUDA input, <code>input.new</code> will allocate a CUDA tensor, so the types will always match. I find that solution simpler and more robust than what you proposed. Doesn’t it work for you?</p>

apaszke: <p>Also, don’t fiddle with internal fields. It’s not a good idea. They’re subject to change without notice.</p>

ypxie: <p>I just realized <code>input.new(shape)</code> shape is not necessarily equal to input.size(), that makes sense.<br>
Thanks.</p>

Ben_Usman: <p>I wonder how we are supposed to use registered variables in optimizers (e.g. SGD). <code>.parameters()</code> call does not return <code>None</code> parameters, so if you want to register parameter it in advance, you can create optimizer only after first forward pass?</p>

Amir_Rosenfeld: <p><a class="mention" href="/u/ypxie">@ypxie</a>: Note you have: <code>self.cuda = Ture</code> instead of   <code>True</code></p>

ypxie: <p>you can do a forward pass in the last line of <strong>init</strong> funciton.</p>

quoniammm: <p>Hello，I am not very good at python now.I am confused at it.What is the meaning of <code>@ input</code>?</p>

quoniammm: <p>I also have another question.If I use <code>register_parameter</code> to register a parameter.This parameter will update automatically when the training step is run.<br>
If I only want to have a parameter that is just dynamic and don’t need to be updated.How can i implement it.For example:</p>
<pre><code class="lang-auto">self.title_conv = nn.Sequential(
            nn.Conv1d(),
            nn.ReLU(),
# the kernel_size is changed because the input's length of conv layer is Changeable.
# Therefore, kernel_size was computed in forward function.
# Then i want pass it to maxpool layer.How can I implement it?
            nn.MaxPool1d(kernel_size=)
        )
</code></pre>

karlTUM: <p>I also have the same problem, since I would like to adapt different kernel sizes according to different input data sizes. Did you solve your problem?</p>

Dave_Kielpinski: <p>Here’s another way that seems to work. Like your example with <code>register_parameter</code>, a forward pass is required before initializing the optimizer. Do you see any clear advantages or disadvantages?</p>
<pre><code class="lang-auto">class DynamicLinear(nn.Module):
    def __init__(self, output_dim):
        super(DynamicLinear, self).__init__()
        self.output_dim = output_dim

    def forward(self, inputs):
        if not hasattr(self, '_linear'):
            input_dim = inputs.shape[-1]
            self._linear = nn.Linear(input_dim, self.output_dim)
        return self._linear(inputs)</code></pre>

OrNot: <p>won’t work properly .    same issue about  model.cuda()</p>

jinfagang: <p>How to set the bias?</p>
<p>I mean F.conv2d using default bias or not? how to make nn.Conv2d dyamic weights in forward and also same effect as calling F.conv2d without bais?</p>

ilanfri: <p>Apologies for reviving this, I’m looking to do precisely this but it seems this no longer works. It seems from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_parameter" rel="noopener nofollow ugc">current documentation for <code>register_parameter()</code></a> that for its second argument, " If <code>None</code> , the parameter is <strong>not</strong> included in the module’s <code>state_dict</code>.".</p>
<p>I think the above therefore results in the parameter not being found in the .parameters() method when an object is instantiated from this class.</p>
<p>Instead of using <code>None</code> as the second argument to <code>register_parameter()</code> I could use <code>nn.Parameter()</code>, but then then not sure how to check if a parameter is uninitialised, so that I can write the correct <code>if</code> statement in the <code>forward()</code> method to avoid resetting the parameter at each iteration.</p>
<p>What is the up-to-date way to do this?</p>
<p>(Apologies if I am mistaken and have the wrong end of the stick here).</p>