Dynamic parameter declaration in forward function

ypxie: Declare the parameters in the forward function seems to be a solution because the intermediate results are already known at that point, but the thing is this might make the parameters be declared every time we run the forward function. <SCODE>def conv_relu(input):
    # Create variable named "weights", its shape can be infered from input.
    kernel_shape = (64, input.size()[0],3,3)
    weights = nn.Module.get_parameter("weights", kernel_shape,
        initializer='uniform')
    conv = F.conv2d(input, weights)
    return F.relu(conv)
<ECODE>

apaszke: <SCODE>class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('weight', None)

    def forward(self, input):
        if self.weight is None:
            self.weight = nn.Parameter(torch.randn(input.size()))
        return self.weight @ input
<ECODE>

ypxie: In this way, model.cuda() which is usually called before forward() might not work properly. if one more if_else is added to check use_cuda,  the code can be unnecessarily long.

apaszke: Good point. This would be better then: <SCODE>class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('weight', None)

    def reset_parameters(self, input):
        self.weight = nn.Parameter(input.new(input.size()).normal_(0, 1))    

    def forward(self, input):
        if self.weight is None:
            self.reset_parameters(input)
        return self.weight @ input
<ECODE>

ypxie: Thanks for your reply, but I didn’t get why would this help? It seems to me register_parameter just register a None parameter to the parameters list.  How could model.cuda() affect a None parameter?

ypxie: How about the following? <SCODE>def cuda(self, device_id=None):
        """Moves all model parameters and buffers to the GPU.

        Arguments:
            device_id (int, optional): if specified, all parameters will be
                copied to that device
        """
        self._cuda = Ture
        self._device_id = device_id
        return self._apply(lambda t: t.cuda(device_id))

def cpu(self, device_id=None):
        """Moves all model parameters and buffers to the CPU."""
        self._cuda = False
        return self._apply(lambda t: t.cpu())

def get_parameter(self, name, shape):
       if not name in self._parameters.keys():
            self._parameters[name] = nn.Parameter(torch.randn(shape))
       var = self._parameters[name]      
       fn = lambda t: t.cuda(self._device_id) if self._cuda else lambda t: t.cpu()
       var.data = fn(var.data)
       if var.grad is not None: 
           var.grad.data = fn(var.grad.data)
       return var

<ECODE>

apaszke: 

apaszke: Also, don’t fiddle with internal fields. It’s not a good idea. They’re subject to change without notice.

ypxie: 

Ben_Usman: 

Amir_Rosenfeld: 

ypxie: 

quoniammm: 

quoniammm: <SCODE>self.title_conv = nn.Sequential(
            nn.Conv1d(),
            nn.ReLU(),
# the kernel_size is changed because the input's length of conv layer is Changeable.
# Therefore, kernel_size was computed in forward function.
# Then i want pass it to maxpool layer.How can I implement it?
            nn.MaxPool1d(kernel_size=)
        )
<ECODE>

karlTUM: I also have the same problem, since I would like to adapt different kernel sizes according to different input data sizes. Did you solve your problem?

Dave_Kielpinski: <SCODE>class DynamicLinear(nn.Module):
    def __init__(self, output_dim):
        super(DynamicLinear, self).__init__()
        self.output_dim = output_dim

    def forward(self, inputs):
        if not hasattr(self, '_linear'):
            input_dim = inputs.shape[-1]
            self._linear = nn.Linear(input_dim, self.output_dim)
        return self._linear(inputs)<ECODE>

OrNot: won’t work properly .    same issue about  model.cuda()

jinfagang: How to set the bias? I mean F.conv2d using default bias or not? how to make nn.Conv2d dyamic weights in forward and also same effect as calling F.conv2d without bais?

ilanfri: I think the above therefore results in the parameter not being found in the .parameters() method when an object is instantiated from this class. What is the up-to-date way to do this? (Apologies if I am mistaken and have the wrong end of the stick here).