PyTorch - Passing Hyperparameters to backprop of autograd.Function

Max_Unhold: <p>Hello,<br>
I am writing a custom nn.Module class (as a layer) that calls an Autograd function. I want to write a custom backward function. The following example is from <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">PyTorch - Extending</a>.</p>
<p>My problem is, that I want to <em>control a hyperparameter used in the backward</em> of the <code>LinearFunction(Function)</code> <em>from outside</em> - from the <code>nn.Module class</code>. In the nn.Module class, we can use Function.apply in order to pass hyperparameters<br>
from the <code>nn.Module</code> to the forward method of the <code>LinearFunction(Function)</code>. But the backward only takes the outputs.</p>
<p>How do I pass the values from the <code>nn.Module</code> to the <code>LinearFunction</code>?</p>
<pre><code class="lang-auto">class LinearFunction(Function):
    @staticmethod
    # bias is an optional argument for the forward method
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output
    @staticmethod
    def backward(ctx, grad_output):
        
        #--------------------------------------------------------------
        # I want to use some Hyperparameter in the backward of this Function
        # but the backward function only takes grad_output!
        #--------------------------------------------------------------
        
        input, weight, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0)
        return grad_input, grad_weight, grad_bias

some_hyperparameter = 0.1

class Linear(nn.Module):
    def __init__(self, input_features, output_features,some_hyperparameter,bias=True):
        super(Linear, self).__init__()
        self.input_features  = input_features
        self.output_features = output_features
        self.weight = nn.Parameter(torch.empty(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.empty(output_features))
        else:
            self.register_parameter('bias', None)

        nn.init.uniform_(self.weight, -0.1, 0.1)
        if self.bias is not None:
            nn.init.uniform_(self.bias, -0.1, 0.1)
            
        self.some_hyperparameter = some_hyperparameter

    def forward(self, input):
        #--------------------------------------------------------------
        # to do: pass the hyperparameter to LinearFunction.backward (?)
        #--------------------------------------------------------------
        return LinearFunction.apply(input, self.weight, self.bias)

    def extra_repr(self):
        return 'input_features={}, output_features={}, bias={}'.format(
            self.input_features, self.output_features, self.bias is not None
        )
</code></pre>

tom: <p>The typical way to do this (i.e. PyTorch uses it internally, too) is to pass it to the forward and store it as a ctx attribute (<code>ctx.my_hyperparameter = ...</code>) and read it as <code>ctx.my_hyperparameter</code>in the backward. Only input/output tensors want to go through saved tensors, so you should be safe here.</p>
<p>Best regards</p>
<p>Thomas</p>

Max_Unhold: <p>Could you also tell me how to pass those parameters<br>
to the <strong>forward function</strong> without having to calculate the backprop for them?</p>
<p>When I pass <strong>5</strong> parameters to the forward function including <strong>2</strong> that are non-trainable hyperparameters, Pytorch asks me to return <strong>5</strong> gradients.</p>

tom: <p>Return None as gradient.</p>