Same output for all input value

pedram1: <p>Hi there</p>
<p>I have tained an 1D Autoencoder NN, which you can find the its details below:</p>
<p>Autoencoder(<br>
(encoder): Sequential(<br>
(0): Conv1d(1, 5, kernel_size=(5,), stride=(2,))<br>
(1): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(2): ReLU(inplace=True)<br>
(3): Conv1d(5, 10, kernel_size=(5,), stride=(2,))<br>
(4): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(5): ReLU(inplace=True)<br>
(6): Conv1d(10, 15, kernel_size=(5,), stride=(2,))<br>
(7): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(8): ReLU(inplace=True)<br>
(9): Conv1d(15, 20, kernel_size=(4,), stride=(1,))<br>
(10): ReLU(inplace=True)<br>
)<br>
(decoder): Sequential(<br>
(0): ConvTranspose1d(20, 15, kernel_size=(1,), stride=(4,))<br>
(1): ReLU(inplace=True)<br>
(2): ConvTranspose1d(15, 10, kernel_size=(2,), stride=(4,))<br>
(3): ReLU(inplace=True)<br>
(4): ConvTranspose1d(10, 5, kernel_size=(9,), stride=(2,))<br>
(5): ReLU(inplace=True)<br>
(6): ConvTranspose1d(5, 1, kernel_size=(10,), stride=(2,))<br>
(7): ReLU(inplace=True)<br>
)<br>
)</p>
<p>my loss decrease like 1/f function however, when I feed my NN with some data, I get almost same output for different input. Has any one had any similar experience before?<br>
I should mention that my training sample was 65536 signal each has length of 94 points.<br>
When I said feeding my data I meant  I did this below:</p>
<p>for i in range(65536) :<br>
out_put[i] = model(data_pixel2[i].unsqueeze(0))</p>
<p>which here data_pixel2 is my input.<br>
Appreciate any comments.</p>

ptrblck: <p>Could you try to remove the last <code>ReLU</code> from your <code>decoder</code> and rerun the code again?<br>
I’ve seen similar issues, where the last non-linearity was messing with the training procedure.</p>

pedram1: <p>thank you Patrick however it became worse than what it was before. Don’t you think if there would be a problem with the way that I normalized my input data? below you can see how I did that:</p>
<p>new_data = np.empty((256,256,94))<br>
for i in range (256) :<br>
for j in range (256) :<br>
for k in range (94) :</p>
<pre><code>  new_data[i,j,k] = ((data[i,j,k])-(mean))/(max-min)
</code></pre>
<p>data is a tensor that I stored my signal there and new data was used as input of my autoencoder</p>
<p>Regards</p>

ptrblck: <p>How did you calculate <code>mean</code>, <code>min</code>, and <code>max</code>?<br>
Usually you would normalize the input signals using the channel stats, but your use case might be different.</p>

pedram1: <p>thank you for your reply.<br>
I did it in the below way:</p>
<p>mean = np.mean(data)</p>
<p>max = np.amax(data)</p>
<p>min = np.amin(data)<br>
new_data = np.empty((256,256,94))<br>
for i in range (256):<br>
for j in range (256):<br>
for k in range (94):<br>
new_data[i,j,k] = (data[i,j,k]-mean)/(max-min)</p>
<p>where data is my input that has the shape of (256,256,94), its a tiff file.</p>

ptrblck: <p>Since you are calculating the global statistics, you could simply use:</p>
<pre><code class="lang-python">data = (data - mean) / (max - min)
</code></pre>
<p>Did you play around with some hyperparameters, e.g. the learning rate and checked, if this would change the behavior?<br>
If that doesn’t help, I would recommend to try to overfit your model on a small dataset samples, e.g. just 10 samples, and make sure your model predicts these samples perfectly.<br>
If that doesn’t work, there might be another error in the code I haven’t found yet.</p>

pedram1: <p>Yes I did, things didn’t changed significantly even at some sets of hyper parameter it became worse.<br>
Sure, I’m going to try to overfit my model with just a few example.<br>
would u please take a look at my training section to see if I am feeding my NN correctly?</p>
<p>learning_rate = 1e-5</p>
<p>num_epochs = 360</p>
<p>model = Autoencoder().cuda()</p>
<p>criterion = nn.MSELoss()</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,</p>
<p>weight_decay=1e-5)</p>
<p><span class="hashtag">#training</span> loop<br>
loss = 0<br>
epoch_num = 0<br>
error = []<br>
for epoch in range(num_epochs):<br>
for data in train_loader:<br>
img = data<br>
img = Variable(img).cuda()</p>
<pre><code>output = model (img)
loss = criterion (output,img)

optimizer.zero_grad()
loss.backward()
optimizer.step()
error.append(loss.item())


if epoch%10 == 9:
  epoch_num += epoch_num
  plt.plot(error)


  print ('\r Train Epoch : {}/{} \tLoss : {:.4f}'.format (epoch+1,num_epochs,loss/32))
</code></pre>
<p>model_save_name = ‘Autoencoder 09 Jan 2020 2nd with new normalzation 2nd trial.pt’<br>
path = F"/content/drive/My Drive/{model_save_name}"<br>
torch.save(model.state_dict(), path)<br>
plt.plot(error)</p>
<p>I have to recall you thay my input size is a tensor of (65536,1,94) shape, which 65536 is number of signals 1 is number of channel and 94 is length of each signal.</p>
<p>Kind Regards,</p>

ptrblck: <p>I cannot see any obvious mistakes, besides the usage of the deprecated <code>Variable</code>, so you can just remove it.</p>
<p>Did you overfit the small sample successfully?</p>

pedram1: <p>Thank you for your time<br>
Actually I tried that one as well, however got similar results. same output for all different input values,</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

pedram1: <p>I generated some artificial data to see if anything changes,<br>
although generated data was quite different from my actual data, surprisingly I got the same out out, quite similar to what I have had with actual data <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=9" title=":open_mouth:" class="emoji" alt=":open_mouth:"></p>

shamoons: <p>Did you get a resolution?</p>