Need help for custom loss backward() function

Aman_Pathak: <p>How should I implement my very own custom loss function.<br>
There are two ways to implement using a custom loss class and writing function with <code>init</code> and <code>forward</code>.<br>
I had the code already written for TensorFlow in class, therefore I chose to go along.<br>
For PyTorch, I tried explored the below relevant discussions, <em>but could not figure how to design the backwards function for same.</em></p>
<ul>
<li><a href="https://discuss.pytorch.org/t/solved-what-is-the-correct-way-to-implement-custom-loss-function/3568/4">Post_1</a></li>
<li><a href="https://discuss.pytorch.org/t/build-your-own-loss-function-in-pytorch/235/18">Post_2</a></li>
<li><a href="https://discuss.pytorch.org/t/write-custom-loss-function/3369/2">Post_3</a></li>
</ul>
<p>Custom Loss:</p>
<pre><code class="lang-auto">class MSE_ITL_RegLoss(torch.nn.Module):
    def __init__(self, latent_code, prior, regularization_factor, kernel, batch_size):
        super (MSE_ITL_RegLoss).__init__()
        
        self.latent_code = latent_code         
        self.regularization_factor = regularization_factor
        self.prior = prior
        self.kernel = kernel
        self.batch_size = batch_size
        
    
    def evaluate_ITL(self, first_pdf, second_pdf):       
        error = first_pdf - second_pdf
        pairwise_error = torch.reshape(torch.unsqueeze(error ,1) - error, (-1,))
        temp = torch.exp(-(np.sqrt(2) * self.kernel) * torch.square(pairwise_error_t))
        return torch.sum(temp)/self.batch_size**2
        
    
    def cs_div (self):
        p_x = self.evaluate_ITL(self.latent_code, self.latent_code)
        q_x = self.evaluate_ITL(self.prior,self.prior)
        p_x__q_x = self.evaluate_ITL(self.latent_code, self.prior) 
        return p_x__q_x / torch.sqrt(p_x * q_x)
        
        
    def forward(self, y_true, y_pred):       
        mse = torch.mean(torch.square(y_true_t-y_pred_t))
        cs_div = self.cs_div()       
        final_loss = mse + self.regularization_factor * cs_div
        return final_loss
</code></pre>
<p>Model:</p>
<pre><code class="lang-auto">class modelMSE_CSD (nn.Module):
    def __init__(self, input_size, bottleneck_dim):
        super(modelMSE_CSD, self).__init__()

        self.bottleneck_dim = bottleneck_dim
        self.input_size = input_size
        
        self.encoder = nn.Sequential(
            nn.Linear(self.input_size, 800),
            nn.ReLU(),
            nn.Linear(800,400),
            nn.ReLU(),
            nn.Linear(400, self.bottleneck_dim)
        )
        
        self.decoder = nn.Sequential(
            nn.Linear(self.bottleneck_dim, 400),
            nn.ReLU(),
            nn.Linear(400,800),
            nn.ReLU(),
            nn.Linear(800, self.input_size),
            nn.Sigmoid()
        )
        
    def forward(self,x):
        e_out = self.encoder(x)
        d_out = self.decoder(e_out)
        return e_out, d_out
</code></pre>
<p>Training Loop:</p>
<pre><code class="lang-auto">losses = []
for epoch in range(EPOCHS):
    running_loss = 0.0
    
    for batch_num, (data, target) in enumerate(dataloader):
        
        data = Variable(data, requires_grad = True)
        target = Variable(target, requires_grad = True)

        model_csd.train()
        latent_code, decoded_layer = model_csd(data)
        
        loss = MSE_ITL_RegLoss(latent_code=latent_code, prior=PRIOR_DIST, 
                               regularization_factor=REGULARIZER, kernel=KERNEL, batch_size=BATCH_SIZE)

        optimizer.zero_grad()
        **loss.backward()**
        optimizer.step()
        
        running_loss += loss.item() * data.size[0]
    losses.append(running_loss/len(dataset))
    print("Epoch: ",epoch,"Loss :", running_loss/len(dataset))```



Please suggest if the other method of declaring the custom loss is more easier.
Any links/resources would be appreciated.</code></pre>

my3bikaht: <p>You don’t need to define backward method if your custom loss class is inherited from nn.Module, it’s already there.</p>

Aman_Pathak: <p>Hi <a class="mention" href="/u/my3bikaht">@my3bikaht</a> , thank you for your response.</p>
<p>While running the training loop (code mentioned above), I get the error that <code>AttributeError: 'MSE_ITL_RegLoss' object has no attribute 'backward'</code></p>
<p>This is the error traceback:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-13-fc852af04b41&gt; in &lt;module&gt;
     15 
     16         optimizer.zero_grad()
---&gt; 17         loss.backward()
     18         optimizer.step()
     19 

c:\...\torch\nn\modules\module.py in __getattr__(self, name)
   1176                 return modules[name]
   1177         raise AttributeError("'{}' object has no attribute '{}'".format(
-&gt; 1178             type(self).__name__, name))
   1179 
   1180     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -&gt; None:

AttributeError: 'MSE_ITL_RegLoss' object has no attribute 'backward'
</code></pre>
<p>Is there anything that can be done?</p>

ptrblck: <p>You are creating an object called <code>loss</code> from the <code>MSE_ITL_RegLoss</code> class:</p>
<pre><code class="lang-python">loss = MSE_ITL_RegLoss(latent_code=latent_code, prior=PRIOR_DIST, 
                               regularization_factor=REGULARIZER, kernel=KERNEL, batch_size=BATCH_SIZE)
</code></pre>
<p>but are never actually calculating the <code>final_loss</code> computed in the <code>forward</code>.<br>
Based on the class definition you would need to call this object (same as calling a model with inputs) to get the loss:</p>
<pre><code class="lang-python">loss = MSE_ITL_RegLoss(latent_code=latent_code, prior=PRIOR_DIST, 
                               regularization_factor=REGULARIZER, kernel=KERNEL, batch_size=BATCH_SIZE)
final_loss = loss(y_true, y_pred)
final_loss.backward()
</code></pre>