Custom loss function not decreasing or changing

username: sanaa
cooked: <p>I have defined a custom loss function but the loss function is not decreasing, not even changing.<br>
my loss function aims to minimize the inverse of gap statistic  which is used to evaluate the cluster formed from my embeddings.<br>
this is a toy code:</p>
<pre><code class="lang-auto">def get_data():
    Xlist = []
    for i in range(6):
        X, _ = make_blobs(n_samples=1000, n_features=2)
        Xlist.append(X)
    dat = np.stack(np.array(Xlist), axis=2)
    dat = torch.from_numpy(np.reshape(dat,(-1))).float()
    return dat

def custom_loss_function(gap):
    loss = torch.sum([1./g for g in gap])
    return loss

def calculate_Wk(X):
    Sum_of_squared_distances = []
    for k in [10, 50, 150, 300, 500]:
        optics = OPTICS(min_samples=k).fit(X)
        Sum_of_squared_distances.append(sum(optics.core_distances_))
    sum_of_squared_dist = np.log(np.array(Sum_of_squared_distances))
    normalized_wcd = sum_of_squared_dist - np.max(sum_of_squared_dist)
    return normalized_wcd

def calculate_gap(output,x, Wks, s_prime):
    list_of_G = []
    Wks = torch.tensor(Wks)
    output = torch.tensor(output)
    x = np.reshape(x, (1000, 2, 6))
    for i, weight in enumerate(output):
        Wk = calculate_Wk(weight * x[:,:,i])
        Wk = torch.tensor(Wk)
        G = Wks - Wk
        optimum = 0
        for i in range(0, len(G)):

            if(G[i] &gt;= G[i+1] + s_prime[i+1]):
                optimum = i+1
                break

        if optimum == 0:
            optimum = 1
        list_of_G.append(G[optimum])
    return list_of_G

def get_reference_data():
    e = np.finfo(float).eps
    simulated_Wk = np.zeros((20, 5)) + e 
    simulated_sk = np.zeros((20, 5)) + e
    
    for i in range(20):
        
        temp_wk = []
        temp_sk = []
        
        X = np.random.uniform(0, 1, size=(1000, 2))
        simulated_Wk[i] = calculate_Wk(X)
    Wks = np.mean(simulated_Wk + e, axis=0)
    sks = np.std(simulated_Wk + e, axis=0) * np.sqrt(1 + 1/20)
    
    return Wks, sks

Wks, s_prime = get_reference_data()
in_features = 12000
out_features = 6
model = nn.Sequential(nn.Linear(in_features, out_features))
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
x = get_data()
for n in range(10):
    output = model(x)
    #This is the part where i calculate the gap statistic that i want to minimize.
    list_of_gaps = calculate_gap(output, x, Wks, s_prime)
    loss = custom_loss_func(list_of_gaps)
    loss = Variable(loss, requires_grad = True)
    loss.backward()
    optimizer.step()
</code></pre>
<p>The loss is not even changing, my model isnt learning anything. Can you help me out with this?</p>

username: AlphaBetaGamma96
cooked: <p>You’ve missed the <code>return</code> statement within your loss function.</p>
<pre><code class="lang-auto">def custom_loss_function(gap):
  loss = torch.sum([1./g for g in gap])
</code></pre>
<p>Try your code again but with,</p>
<pre><code class="lang-auto">def custom_loss_function(gap):
  return torch.sum([1./g for g in gap])
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Also, another potential problem could be that you’re detaching the output of your model with</p>
<pre><code class="lang-auto">output1 = output.detach().numpy()
</code></pre>
<p>This will break the gradients within the model and probably explains why your model isn’t learning! The gradients are zero!</p>
<p>Also, remember to clear the gradient cache of your parameters (via <code>optimizer.zero_grad()</code>) otherwise your gradients will acculumate from all epochs! You’ll want to have something like this within your code!</p>
<pre><code class="lang-auto">for epoch in range(epochs):
  #lines to calculate loss
  optimizer.zero_grad() #clear current gradient values!
  loss = custom_loss_function(list_of_gaps) #calculate loss
  loss.backward() #calculate gradients
  optimizer.step() #update parameters
</code></pre>

username: sanaa
cooked: <p>Hi, This was a typo in this code, i am returning the loss.</p>

username: AlphaBetaGamma96
cooked: <p>What about my 2nd comment? Using the <code>detach</code> function will kill any gradients in your network which is most likely the explanation as to why it’s not learning.</p>
<p>Also could you indent your code by wrapping it in three backticks ``` , it makes it easier for people to read/copy! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sanaa
cooked: <p>i tried removing the detach statement, my loss is still not decreasing. its constant.</p>

username: AlphaBetaGamma96
cooked: <p>Would you mind sharing how <code>calculate_gap</code> is done? There might be a line in there which is causing your gradient to be zero.</p>

username: sanaa
cooked: <p>Ive updated the code now. please check. thanks. this is all im doing.</p>

username: AlphaBetaGamma96
cooked: <p>After having a brief look through, it seems you’re swapping between <code>torch</code> and <code>numpy</code>, when moving back and forth between the library would break the gradient of any intermediate computations, no? I’d suggest trying to remove all dependencies on <code>numpy</code>  and purely use <code>torch</code> operations so autograd can track the operations.</p>
<p>Also, you do use the gradient of your input data at all (i.e. <code>x</code>) ? If you do, make sure to enable grad for that data! You can add <code>x.requires_grad_()</code> before your loop.</p>

username: sanaa
cooked: <p>so im using scikit learn OPTICS to calculate clusters. that requires the input x to be in numpy. i did do requires_grad() like you said, but i have to detach before i send it to calculate gap or it gives me</p>
<pre><code class="lang-auto">RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.</code></pre>

username: sanaa
cooked: <p>im doing this now. im detaching x but im also adding requires_grad=True for the loss.</p>
<pre><code class="lang-auto">in_features = 12000
out_features = 6
model = nn.Sequential(nn.Linear(in_features, out_features))
optimizer = torch.optim.SGD(model.parameters(), lr=0.000001)
print(model)
x = get_data()
for n in range(10):
    x.requires_grad_() 
    output = model(x)
    output1 = output.detach()
    x = x.detach()
    list_of_gaps = calculate_gap(output1, x, Wks, s_prime)
    loss = my_loss(list_of_gaps)
    loss = Variable(loss, requires_grad = True)
    print('LOSS', loss)
    loss.backward()
    optimizer.step()</code></pre>

username: AlphaBetaGamma96
cooked: <p>If you’re using <code>scikit-learn</code>, perhaps try using <code>skorch</code>? It’s a PyTorch version of scikit-learn that wraps around it. (github repo: <a href="https://github.com/skorch-dev/skorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - skorch-dev/skorch: A scikit-learn compatible neural network library that wraps PyTorch</a>)</p>
<p>The main issue is that the outputs of your model are being detached, so they have no connection to your model weights, and therefore as your loss is dependent on <code>output</code> and <code>x</code> (both of which are detached), your <code>loss</code> will have no gradient with respect to your model parameters! Which is why it’s not decreasing!</p>
<p>You’ll need to calculate your <code>loss</code> value without using the <code>detach()</code> method at all. Also, you don’t need the <code>loss = Variable(loss, requires_grad=True)</code> line, I think!</p>

username: sanaa
cooked: <p>thanks, let me try this out. ill get back to you. thanks for the help!</p>

username: AlphaBetaGamma96
cooked: <p>Not a problem, and good luck! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sanaa
cooked: <p>just checked skorch out, they dont have clustering algorithms implemented…</p>

username: sanaa
cooked: <p>i willl try and create a dummy function using torch to see if my loss is decreasing. if it is, i can go ahead and implement in torch. i want to know if that really is what is causing the issue</p>

username: sanaa
cooked: <p>I have completely removed gap calculation and im doing a dummy mean to get the G, which i pass to the loss function now. all my variables are requires_grad True. but loss is still constant. Can you maybe try running the code as well?</p>
<pre><code class="lang-auto">def calculate_gap(output,x, Wks, s_prime):
    list_of_G = []
    Wks = torch.tensor(Wks)
    output = torch.tensor(output)
    x = torch.reshape(x, (1000, 2, 6))
    print(output)
    for i, weight in enumerate(output):
        Wk = weight * x[:,:,i]
        G = torch.mean(Wk)
   return G

def my_loss(g):
    loss = torch.square(1./g)
    return loss

in_features = 12000
out_features = 6
model = nn.Sequential(nn.Linear(in_features, out_features))
optimizer = torch.optim.SGD(model.parameters(), lr=0.000001)
print(model)
x = get_data()
for n in range(10):
    x.requires_grad_(True) 
    output = model(x)
    g = calculate_gap(output, x, Wks, s_prime)
    print(g.requires_grad)
    loss = my_loss(g)
    print(loss.requires_grad)
    loss = Variable(loss, requires_grad = True)
    print('LOSS', loss)
    loss.backward()
    optimizer.step()
</code></pre>