Loss isn't decreasing - unable to overfit on purpose

username: JohnCwok
cooked: <p>Hello,</p>
<p>I am interested in using RNNs for musicology and need to predict several thing at each “time” step : melody, timing, and harmony. I decided to start with a plain old LSTM before moving on to more advanced concepts. Roughly, the architecture looks like this :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4292de45980e9daaee57a518d71ec1393da8f2f9.png" alt="first_architecture" data-base62-sha1="9uWaXHPbCHjA6kOIxNVfSIGRCRj" width="519" height="411"></p>
<p>The three outputs predictions are fed to their respective loss functions which are then added together for training.<br>
Upon testing, I realized that I am unable to purposely overfit the model by feeding it only one training example : the loss never converges to 0.<br>
I verified carefully the code, taken separately, each of these outputs can be overfitted, but when I add the three losses together I run into trouble.<br>
This means that there is a problem with the design of my network, but I don’t really understand where. Do you have any idea what is going on ?</p>

username: Mazhar_Shaikh
cooked: <p>Hi John, Could you please post the code that defines your model?</p>

username: JohnCwok
cooked: <p>Sure :</p>
<pre><code class="lang-auto">class musicLSTM(nn.Module):
    def __init__(self,input_size,output_size,hidden_size,deepness):
        super().__init__()
        self.deepness = deepness
        self.input_size = input_size
        self.output_size = output_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size,num_layers = deepness, batch_first = True)
        self.output_layer = nn.Linear(hidden_size,output_size)
        self.tanh = nn.Tanh()
        self.softmax = nn.LogSoftmax(dim = 2)
    
    def forward(self, notes, timing, harmony, hidden = None):
        if hidden == None:
            hidden = (torch.zeros(self.deepness,1,self.hidden_size),torch.zeros(self.deepness,1,self.hidden_size))
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        else:
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        return out, hidden 
</code></pre>
<p>and the training is done with this function :</p>
<pre><code class="lang-auto">def train_rnn(model,notes,utils,chords):
    start = time.time()
    criterion = nn.NLLLoss()
    timing_criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters())
    n_iters = 1000
    print_every = 1
    plot_every = 2
    all_losses = []
    total_loss = 0
    
    for iter in range(1,n_iters+1):
        optimizer.zero_grad()       
        loss = 0
        output, hidden = model(notes, timing, harmony)
        
        for i in range(5):
            ln = criterion(output[0][i][0:130].unsqueeze(0), targetTensor(notes[0][i+1][0:130]))
            lt = timing_criterion(output[0][i][130:135], timing[0][i+1])
            lh = criterion(output[0][i][135:246].unsqueeze(0), targetTensor(harmony[0][i+1][0:111]))
            loss += ln + lt + lh
        loss.backward()
        optimizer.step()    
        total_loss += loss.item() / notes.size(1)
        if iter % print_every == 0:
             print('%s (%d %d%%) %.2f' % (time_since(start), iter, iter / n_iters * 100, loss))
        if iter % plot_every == 0:
            all_losses.append(total_loss / plot_every)
            total_loss = 0
    plt.figure()
    plt.plot(all_losses)
</code></pre>
<p>Defined that way, the training actually only happens on the first 5 notes of the song defined by <code>notes</code>,<code>timing</code> and <code>harmony</code>.  Notes has the dimension <code>(1, melody_length, 130)</code>, timing <code>(1,melody_length, 5)</code> and harmony <code>(1, melody_length, 111)</code>. 130 and 111 being the possible pitches and chords available.</p>

username: JohnCwok
cooked: <p>I found the problem : I applied softmax to the full output, ignoring the fact that part of the ouput is not supposed to be a probability density which destroyed any attempt of training it.<br>
EDIT : even if I correct this, the loss still won’t to zero even though my training sample basically consists of three notes fed over and over … I don’t know what is happening.</p>

username: Abhimanyu_Dikshit
cooked: <p>Hi John,</p>
<p>I am new to pytorch. I ran into similar troubles while attemting to train an lstm based language model. Have you been able to solve this? Might I suggest using 3 separate linear projection layers to project each of your targets into 3 different spaces before taking a softmax and calculating error?</p>