Model is not updating

Austin_Jianxiao_Li: <p>Hi! I am a beginner in deep learning. I have implemented the following model. The problem I am facing now is that the weights and bias are not being updated. When I plot the losses, the losses are periodic(looks the same every epoch). Can anyone please help me see what is causing this? Thank you!</p>
<p>class MyModel(torch.nn.Module):</p>
<pre><code>def __init__(self):

    """

    In the constructor we instantiate two nn.Linear modules and assign them as

    member variables.

    """

    super(MyModel, self).__init__()

    

    # Pooling layer

    self.pool = torch.nn.MaxPool1d(20, stride=2)

    self.pool_avg = torch.nn.AvgPool1d(127)

    # Time(left) convolution

    self.time1 = torch.nn.Conv1d(in_channels=1, out_channels=64, kernel_size=16, stride=1, padding=17)

    self.time2 = torch.nn.Conv1d(in_channels=64, out_channels=128, kernel_size=16, stride=1, padding=17)

    self.time3 = torch.nn.Conv1d(in_channels=128, out_channels=256, kernel_size=16, stride=1, padding=17)

    # Frequency(right) convolution

    self.freq1 = torch.nn.Conv1d(in_channels=1, out_channels=64, kernel_size=16, stride=1, padding=17)

    self.freq2 = torch.nn.Conv1d(in_channels=64, out_channels=128, kernel_size=16, stride=1, padding=17)

    self.freq3 = torch.nn.Conv1d(in_channels=128, out_channels=256, kernel_size=16, stride=1, padding=17)

    # Fully conencted layer

    self.linear1 = torch.nn.Linear(512, 256)

    self.linear2 = torch.nn.Linear(256, 128)

    self.linear3 = torch.nn.Linear(128, 64)

    # Fianl layer

    self.final = torch.nn.Softmax(dim=1)

def forward(self, time_domian, freq_domain, clean_result):

    """

    In the forward function we accept a Tensor of input data and we must return

    a Tensor of output data. We can use Modules defined in the constructor as

    well as arbitrary operators on Tensors.

    """

    # input dimension

    time_domian = time_domian.unsqueeze(1)

    freq_domain = freq_domain.unsqueeze(1)

    # Time(left) convolution

    # print(f"len time domain: {time_domian.shape}")

    time1_out = self.time1(time_domian)

    time1_out = self.pool(time1_out)

    

    # print(f"len time1_out: {time1_out.shape}")

    time2_out = self.time2(time1_out)

    time2_out = self.pool(time2_out)

    # print(f"len time2_out: {time2_out.shape}")

    time3_out = self.time3(time2_out)

    time3_out = self.pool(time3_out)

    # print(f"len time3_out: {time3_out.shape}")

    # Frequency(right) convolution

    freq1_out = self.freq1(freq_domain)

    freq1_out = self.pool(freq1_out)

    freq2_out = self.freq2(freq1_out)

    freq2_out = self.pool(freq2_out)

    freq3_out = self.freq3(freq2_out)

    freq3_out = self.pool(freq3_out)

    # print(f"len freq3_out: {freq3_out.shape}")

    # Connection

    conv_out = torch.cat((time3_out, freq3_out), dim=1)

    conv_out_ave = torch.squeeze(self.pool_avg(conv_out))

                    

    # print(f"len conv_out: {conv_out.shape}")

    # print(f"len conv_out_ave: {conv_out_ave.shape}")

    # Fully conencted layer

    fc1_out = self.linear1(conv_out_ave).clamp(min=0)    # relu

    # print(f"len fc1_out: {fc1_out.shape}")

    fc2_out = self.linear2(fc1_out).clamp(min=0)    # relu

    # print(f"len fc2_out: {fc2_out.shape}")

    fc3_out = self.linear3(fc2_out).clamp(min=0)    # relu

    #print(f"len fc3_out: {fc3_out.shape}")

    # print(fc3_out)

    # Final layer

    final_out =  torch.max(self.final(fc3_out), dim=1)[0]

    # print(f"len final_out: {final_out.shape}")

    # print(final_out)

    return final_out</code></pre>

ptrblck: <p>The <code>torch.max</code> operation in:</p>
<pre><code class="lang-python">final_out =  torch.max(self.final(fc3_out), dim=1)[0]
</code></pre>
<p>will allow the gradient to pass to the max. value only and will set all other values to zero, so I’m not sure if that’s really what you want.<br>
If you are dealing with e.g. a multi-class classification, pass the raw logits to <code>nn.CrossEntropyLoss</code> instead of using <code>torch.max</code> or applying softmax.</p>

Austin_Jianxiao_Li: <p>Do you mean passing “fc3_out” directly to loss_fn?</p>
<p>Also, the code below is how I am performing forward and backward propagation. I wonder if this part of the code is causing the gradient not updating:</p>
<pre><code>    y_pred = model(time, freq, clean_result=False)

    # convert to 1 &amp; -1

    for i in range(len(y_pred)):

        if y_pred[i] &gt;= 0.5:

            y_pred[i] = 1

        else: 

            y_pred[i] = 0

    # Compute loss

    loss = loss_fn(y_pred, y)

    # Zero gradients, perform a backward pass, and update the weights.

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()</code></pre>

dhyey: <p>what is loss function you are using?</p>
<p>the loop in which you are changing probablities to prediction is causing the problem!!!</p>

Austin_Jianxiao_Li: <aside class="quote no-group quote-modified" data-username="dhyey" data-post="4" data-topic="101314">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9de0a6/48.png" class="avatar"> dhyey:</div>
<blockquote>
<p>what loss function you are using</p>
</blockquote>
</aside>
<p>Can you please elaborate on “the loop in which you are changing probablities to prediction is causing the problem”? Thank you!</p>
<p>I am using the following loss function:</p>
<pre><code class="lang-auto">loss_fn = torch.nn.BCELoss()

# define learning rate and optimizer

learning_rate = 1e-4

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>

dhyey: <p>so yo need to remove</p>
<pre><code class="lang-auto">    for i in range(len(y_pred)):

        if y_pred[i] &gt;= 0.5:

            y_pred[i] = 1

        else: 

            y_pred[i] = 0
</code></pre>
<p>because by doing this you are converting probablities to labels but bceLoss expects probablities in “y_pred”</p>
<p>also for this you need to apply sigmoid function to last layer output!!<br>
and last layer should be a linear layer that transforms data to n*1 dimensions</p>

dhyey: <p>also are you doing binary classification or multilabel??<br>
if binary your last layer should be linear layer for 128 X 64 mapping to 64 X 1<br>
if multi class :<br>
loss_function = CrossEntropyLoss<br>
remove the layer which is doing max(self.final)<br>
and make sure tht y  contains  encoded label according to target and dtype of y to be as torch.Long</p>