akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot filter warnings

Duncan_Blythe1: <p>When I train a jit-script RNN I get the warning:</p>
<p><code>Warning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (_cudnn_impl at /pytorch/aten/src/ATen/native/cudnn/RNN.cpp:1266)</code></p>
<p>If I try to filter the warning with <code>warnings.filterwarnings('ignore')</code> or something like that, the warning persists. Is this intended behaviour?</p>

Duncan_Blythe1: <p>Anyone know anything about this? I’m going crazy with huge warning messages in my logs.</p>
Custom shuffle function with DistributedSampler

Rakshit_Kothari: <p>I have a dataset class which reads samples.<br>
Sample attributes: <code>(domain, image, label)</code></p>
<p>In each batch, I want to enforce that every single domain is represented. For example, 8 domains would warrant a batch size of <code>8K</code> with <code>K</code> samples/domain.</p>
<p>I have written a custom shuffle function which sorts the entire pool of images and rearranges the index such that each batch will have <code>K</code> samples from each domain.</p>
<p>This sorting occurs as <code>train_loader.dataset.sort('my_custom_shuffler')</code> at every reset point (like an epoch or some other logic). To ensure that train_loader does not jumble my list again, <code>train_loader.shuffle</code> has been set of False.</p>
<p>In my application, DistributedSampler(shuffle=False) has been defined and assigned to <code>train_loader</code> before the train looping starts. Will my manual sorting affect DistributedSampler (DS)? Does DS assign samples to each rank in the very beginning and maintain that order? If yes, then manual shuffling should not affect DS operation.</p>

mrshenli: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/glaringlee">@glaringlee</a> for <code>DataLoader</code> and <code>DataSampler</code></p>
PyTorch installs version 1.7.1 instead of 1.8.0

stalhabukhari: <p>Hi, I am trying to install PyTorch <strong>1.8.0</strong> via conda, using the command provided on the main page:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b85e0d441e3038e9db2bf3a16cb832bed254947.png" data-download-href="https://discuss.pytorch.org/uploads/default/5b85e0d441e3038e9db2bf3a16cb832bed254947" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b85e0d441e3038e9db2bf3a16cb832bed254947_2_690x249.png" alt="image" data-base62-sha1="d3EhwmYulKkUpE8VAIlPCu9W78X" width="690" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b85e0d441e3038e9db2bf3a16cb832bed254947_2_690x249.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b85e0d441e3038e9db2bf3a16cb832bed254947_2_1035x373.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b85e0d441e3038e9db2bf3a16cb832bed254947.png 2x" data-dominant-color="F7D6D0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1303×471 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, when I get the command to run, I see the following prompt:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fe9b8690c62682802f70c8ca8360386fd3bedb0.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fe9b8690c62682802f70c8ca8360386fd3bedb0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fe9b8690c62682802f70c8ca8360386fd3bedb0.png" alt="image" data-base62-sha1="4yjxNtTk4YsPP2gK9jqenJP5j56" width="684" height="500" data-dominant-color="831083"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1094×799 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can notice that it is installing version <strong>1.7.1</strong> instead. Is this expected?</p>

ptrblck: <p>No, that’s not expected and I guess you might need to update your conda base environment so that the latest version can be found.</p>

stalhabukhari: <p>I updated conda from <strong>4.7.12</strong> to <strong>4.9.2</strong>, and it worked. Thank you so much!</p>

TinNhutNguyen: <p>Hi,<br>
I check anaconda version through anaconda prompt and it seems to be up to date but my PyTorch is still 1.7.1 (checking by using jupyter notebook import torch <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a60f72305e57023baba852806a323aea8ddd413.jpeg" alt="2021-03-24_1-33-13" data-base62-sha1="fb4mgT5sPydNStCoW4qJvNGr5S3" width="563" height="181"> print(torch.<strong>version</strong>)). Any help, please?</p>
Custom dataloader using Python Multiprocessing and num_workers in torch dataloader

Ajinkya_Ambatwar: <p>Hi, I have a custom dataloader. I have explicitly used python’s <code>multiprocessing</code> to parallelize data preprocessing in my custom dataloader. I am using 8 workers(<code>num_threads</code>) in <code>multiprocessing</code> in my dataLoader.<br>
I wanted to know, how will that affect my <code>torch.utils.data.DataLoader</code> call?<br>
Will the <code>num_workers</code> argument be set to 8? Or can I leave it at 0?<br>
My custom Loader looks like this</p>
<pre><code class="lang-auto">def processParams(params):
        &lt;some operations on params&gt;
        return params


def processParamsParallel(params, pool):
      results = pool.map(processParams, params)
      return results

class DataLoader(object):
      def __init__(self, params, maxId):
             self.params = params
             self.id = 0
             self.maxId = maxId
             self.pool = Pool(processes=8)
      def __iter__(self):
             while self.id&lt;=self.maxId:
                 if self.id==self.maxId:
                       self.id = 0
                 results = processParamsParallel(self.params, self.pool)
                 self.id+=1
                 yield results
</code></pre>
<p>It’s a very rough example of what I am trying to do.<br>
Now in the torch call</p>
<pre><code class="lang-auto">dl = DataLoader(params, 50)
dl_torch = torch.utils.data.Dataloader(dl, num_workers = &lt;what_here?&gt;, prefetch_factor = &lt;what_here?&gt;)
</code></pre>
<p>On a similar note, how will <code>prefetch_factor</code> be affected given that the <code>num_workers</code> are not set by torch call but by the custom dataLoader itself?</p>
<p>From torch.utils.data</p>
<p>prefetch_factor - Number of samples loaded in advance by each worker. <code>2</code> means there will be a total of 2 * num_workers samples prefetched across all workers. (default: <code>2</code> )</p>
<p>Thank you in Advance!</p>
Looking for someone to build a Pointer Network in PyTorch

torch_user_92: <p>Hi how’s it going?</p>
<p>I’m looking for someone who can build a Pointer Network (attention based) for me in PyTorch for a problem that I have.</p>
<p>Please respond if interested!</p>
<p>Will discuss compensation via private message.</p>

Henry_P: <p>Hello <a class="mention" href="/u/torch_user_92">@torch_user_92</a><br>
Hope you are well and safe.</p>
<p>I would be glad to work with you on your requirement and would like to have a detailed discussion regarding the same.</p>
<p>Please add me on skype: live:.cid.dc5377d1a462c5b4 so that we can have a chat regarding your requirement.</p>
<p>You can also mail me on <a href="mailto:henry@cisinlabs.com">henry@cisinlabs.com</a></p>
<p>Hoping to listen from you soon.</p>
<p>Best Regards,<br>
Henry</p>

Canopus: <p><a class="mention" href="/u/torch_user_92">@torch_user_92</a>, did you get what you were looking for?</p>
<p>Any update would be appreciated!</p>
<p>Regards</p>

torch_user_92: <p>Hi I did not. The previous response Henry_P has no knowledge of PyTorch. Would you be able to help me with this?</p>

Canopus: <p>Thanks for keep me posted.</p>
<p>Can we have a discovery call sometime this week?</p>
<p>If yes please give me your best time window, I will send you a meeting invite. We can move our conversation to email, <a href="mailto:sandeep.thakur@canopusinfosystems.com">sandeep.thakur@canopusinfosystems.com</a>.</p>
<p>Chat soon</p>

Canopus: <p><a class="mention" href="/u/torch_user_92">@torch_user_92</a>, I hope you had a great weekend.</p>
<p>Are you still looking for help?</p>
<p>Speak to you soon, cheers</p>

Canopus: <p>Just floating on the top of your inbox. Is the requirement still alive?</p>
<p>Chat soon!</p>
Confusion with input shapes to BCEWithLogitsLoss()

Kaustubh_Kulkarni: <p>I accidentally input a tensor of shape (batch_size, 1, 2, 2) into the BCEWithLogitsLoss() function and gave the expected outputs with the same shape as well. This gave me only one value as loss for each batch whereas I would expect 4 different values as loss for each batch (since it is 1, 2, 2). I wanted to know what torch does internally that I get such an output.</p>
<p>Below is the code I used to test:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
</code></pre>
<pre><code class="lang-auto">a = torch.ones(10, 1, 2, 2)
b = torch.randn(10, 1, 2, 2)
</code></pre>
<pre><code class="lang-auto">t_loss = torch.binary_cross_entropy_with_logits_loss(b, a)
</code></pre>
<p>This <code>t_loss</code> gave me an output loss of shape <code>(10, 1, 2, 2)</code>.</p>
<pre><code class="lang-auto">lfunc = nn.BCEWithlogitsLoss()
n_loss = lfunc(b, a)
f_loss = F.binary_cross_entropy_with_logits_loss(b, a)
</code></pre>
<p>Here, <code>n_loss</code> and <code>f_loss</code> gave me the same output consisting of only one number.</p>
<p>I was wondering how PyTorch is internally handling this kind of situation.</p>

shivammehta007: <p>If you see the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">documentation</a> for the implementation it has 2 modes in reduction ‘mean’ and ‘sum’ which just specifies the mean and sums of all the (10, 1, 2, 2) values default is mean .</p>
<pre><code class="lang-auto">a = torch.ones(10, 1, 2, 2)
b = torch.randn(10, 1, 2, 2)

torch.binary_cross_entropy_with_logits(a, b)
# shape (10, 1, 2, 2)
</code></pre>
<p>to demonstrate consider this demo code</p>
<pre><code class="lang-auto">F.binary_cross_entropy_with_logits(a, b, reduction='mean')
#&gt; tensor(1.2240)
F.binary_cross_entropy_with_logits(a, b, reduction='sum')
#&gt; tensor(48.9613)
</code></pre>
<p>which is equivalent to</p>
<pre><code class="lang-auto">torch.mean(torch.binary_cross_entropy_with_logits(a, b))
#&gt; tensor(1.2240)
torch.sum(torch.binary_cross_entropy_with_logits(a, b))
#&gt; tensor(48.9613)
</code></pre>
<p>and <strong>why</strong> we use one single return value is because of gradient calculation and we can backpropagate using <code>loss.backward()</code> only on a single value without explicitly writing gradient tensor for a tensor loss.</p>

Kaustubh_Kulkarni: <p>Thank you for your response!</p>
Generate sequential data with GAN

Yolkandwhite: <p>Hello everyone.</p>
<p>I have question about sequential data.</p>
<p>What I’m trying to do is to generate sequential image(specifically series of MRI images) using GAN.</p>
<p>and I am unsure that If I put slices of shuffled image, it will give an out put of one person’s MRI series.</p>
<p>I have 3D MRI data which is 2d images stacked.</p>
<p>Is it possible to generate an series of an image instance?</p>
<p>and If it is, how should I put the ground truth MRI data?.</p>
<p>one person per mini-batch? or just shuffle all the data?</p>
Returns Nonetype

Tornike: <p>Hello, Created custom dataset that crops faces from images.</p>
<p>This is my getitem function which is working great.</p>
<blockquote>
<p>def <strong>getitem</strong>(self,index):<br>
images=glob.glob(self.path1)<br>
img_path=os.path.join(self.path1,self.df.iloc[index,2][0].split(‘/’,1)[1])<br>
image=cv.imread(img_path,cv.IMREAD_GRAYSCALE)<br>
y_label=torch.tensor(int(self.df.iloc[index,3]))<br>
try:<br>
image=cv.resize(image,(80,80))<br>
image=torch.tensor(image)<br>
except:<br>
pass</p>
</blockquote>
<p>Then splitting dataset into train_set, test_set and creating Dataloaders.</p>
<pre><code class="lang-auto">train_set,test_set=torch.utils.data.random_split(dataset,[49862,len(dataset) -49862])
train_loader=DataLoader(train_set,batch_size=10,shuffle=True)
test_loader=DataLoader(test_set,batch_size=10,shuffle=False)

</code></pre>
<p>And When i iterate through train_loader it  gives me an error.</p>
<pre><code class="lang-auto">  for b,(X_train,y_train) in enumerate(train_loader):

</code></pre>
<blockquote>
<p>AttributeError: ‘NoneType’ object has no attribute ‘float’</p>
</blockquote>
<p>But when i iterate through train_set, it works perfectly.</p>
<p>Any help??</p>

ptrblck: <p>I guess the <code>pass</code> statement might create the issue.<br>
In the case the <code>cv.resize</code> or <code>torch.tensor</code> operation is failing for any reason, you would <code>pass</code> this sample and might return <code>None</code> without raising any error (and thus the <code>DataLoader</code> will fail).<br>
Could you raise a <code>RuntimeError</code> with a proper error message and check, why one of these two operations is failing?</p>

Tornike: <p>Yes, the problem was that pass returned None, so instead of pass i just created np.ones of size 32,32 and that solved.</p>
MNIST normalization and torchvision's Normalize

PersonalFinance004: <p>I want to normalize the MNIST dataset.<br>
Here is how I calculate mean and standard-deviation:</p>
<pre><code>transform=tv.transforms.Compose([tv.transforms.ToTensor()])
train_dataset = tv.datasets.MNIST('../data', train=True, download=True, transform=transform)
mean = torch.mean(torch.Tensor.float(train_dataset.data))
std = torch.std(torch.Tensor.float(train_dataset.data))
</code></pre>
<p>If I manually normalize the data like this:</p>
<blockquote>
<p>train_dataset.data = (train_dataset.data - mean) / std<br>
test_dataset.data = (test_dataset.data - mean) / std</p>
</blockquote>
<p>I get decent accuracy (~0.978), though not better than without normalization (~0.9796).</p>
<p>However, if I use the Normalize transform with the same mean and std:</p>
<blockquote>
<p>transform=tv.transforms.Compose([tv.transforms.ToTensor(), tv.transforms.Normalize(mean, std)])<br>
train_dataset = tv.datasets.MNIST(‘…/data’, train=True, download=True, transform=transform)<br>
test_dataset = tv.datasets.MNIST(‘…/data’, train=False, download=True, transform=transform)</p>
</blockquote>
<p>I get very low accuracy (0.135). Why is that, and how should I use Normalize instead?</p>
<p>Second question: I also tried (manual) pixel-wise normalization:</p>
<blockquote>
<p>px_mean = torch.mean(torch.Tensor.float(train_dataset.data), dim=0)<br>
px_std = torch.std(torch.Tensor.float(train_dataset.data), dim=0)+1e-10<br>
train_dataset.data = (train_dataset.data-px_mean)/px_std<br>
test_dataset.data = (test_dataset.data-px_mean)/px_std</p>
</blockquote>
<p>but this, too, gives me very low accuracy (~0.135). Am I doing it wrong, and if so, how to do it correctly?</p>

ptrblck: <p>The internal <code>.data</code> will store the raw dataset in <code>uint8</code> with values in the range <code>[0, 255]</code>.<br>
The mean of these values (transformed to <code>FloatTensor</code>s) would thus be <code>33.3184</code>.<br>
Normalizing the raw data with these values would thus work.<br>
However, since <code>ToTensor()</code> already normalizes the tensors to the range <code>[0, 1]</code>, the <code>mean</code> and <code>std</code> in <code>transforms.Normalize</code> should also be in this range (divided by 255.).</p>

PersonalFinance004: <p>I see. I tried</p>
<pre><code class="lang-auto">transform = torchvision.transforms.Compose([
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize(mean=mean/255, std=std/255)])
</code></pre>
<p>with</p>
<pre><code class="lang-auto">mean = 33.3184
std = 78.5675
</code></pre>
<p>but this gives me lower accuracy (<code>0.88</code>) than ToTensor() on its own (<code>0.98</code>). Is that expected for MNIST?</p>

ptrblck: <p>It depends on your model and overall training, i.e.:</p>
<ul>
<li>is this result giving the training, evaluation, or final test accuracy</li>
<li>how reproducible is this result (i.e. are you always seeing the difference in accuracy using different seeds)</li>
<li>how much hyper-parameter turning did you apply (learning rate, model architecture, etc.) or did you use a “static” setup?</li>
</ul>
RuntimeError:shape '[4, 98304]' is invalid for input of size 113216

iesnek01: <p>Hi, I am learning to train a basic nn model for image classification, the error happened when I was trying to feed in image data into the model. I understand that I should input correct size of image data. My image data is 128*256 with 3 channels. What I don’t understand is where does the number 113216 come from? I checked all related parameters or image meta data, but didn’t find a clue. Here is my code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5462e39747ef371f6ae96bb04bbd58135c2ffeed.png" data-download-href="https://discuss.pytorch.org/uploads/default/5462e39747ef371f6ae96bb04bbd58135c2ffeed" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5462e39747ef371f6ae96bb04bbd58135c2ffeed.png" alt="image" data-base62-sha1="c2vYwP8Po07CuJSH7va71Adl5il" width="529" height="500" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">854×806 10.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks for your help!</p>

Dwight_Foster: <p>The 113216 is flattened size of x. So the product of all of the dimensions. Can you print out the shape of x before you try and reshape it like this</p>
<pre><code class="lang-auto">x = self.pool(F.relu(self.conv2(x)))
print(x.shape)
</code></pre>
<p>Then just type in those dimensions instead of the ones you have.</p>
Generate 2 D data

sunny1: <p>Hi, i want to generate the u(N,2) data to feed the NN and also want to save the rx and rt as x_space and t_time to again feed in to get the differential of u wrt to rt and rx.</p>
<p>rx = torch.tensor((np.arange(0,1,.01).reshape(100,1)),  requires_grad=True, dtype=torch.float32)<br>
rt = torch.tensor((np.arange(0,30,.3).reshape(100,1)), requires_grad=True,  dtype = torch.float32)</p>
<p>def differential(u_pred,x,t):</p>
<pre><code>    z = torch.ones(u_pred.shape).requires_grad_(True)
    dx = torch.autograd.grad(u_pred, x, grad_outputs=z, create_graph=True)[0]
    dt = torch.autograd.grad(u_pred, t, grad_outputs=z, create_graph=True)[0]
    dxx = torch.autograd.grad(dx, z, grad_outputs=torch.ones(dx.shape), create_graph=True)[0]
    f = dxx-dt
    f = pn.MSE(f , torch.zeros(len(f)))
    
    return f
</code></pre>
<p>Is it a better way to do this so that i can save u, t and x to feed in?</p>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [6, 1, 3, 5], but got 3-dimensional input of size [10, 32, 32] instead

Tornike: <p>Hello, trying to train model on 32x32 images. but  got dimensional error.</p>
<p>Here is my Module.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d250fe0427b435940cbf3d90dfdf21d456d2b2c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d250fe0427b435940cbf3d90dfdf21d456d2b2c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d250fe0427b435940cbf3d90dfdf21d456d2b2c1_2_690x387.png" alt="image" data-base62-sha1="u0xNdIAzfk2Xp697njZNesVQFb3" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d250fe0427b435940cbf3d90dfdf21d456d2b2c1_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d250fe0427b435940cbf3d90dfdf21d456d2b2c1_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d250fe0427b435940cbf3d90dfdf21d456d2b2c1.png 2x" data-dominant-color="E0E1E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here is an error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c_2_690x387.png" alt="image" data-base62-sha1="ywugLfyXfuGCGauwFApyARyISdK" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1f6119d0e64c255ef2524a5dbb9dc0c7c8ced4c.png 2x" data-dominant-color="E2E3E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i just don’t get where is that 4th input?</p>

Dwight_Foster: <p>The fourth dimension is the number of channels. Currently you only have batch size and x and y. You need to add the number of channels. To just do it simply in your forward function you can do this</p>
<pre><code class="lang-auto">def forward(self, x):
     x = x.unsqueeze(1)</code></pre>

Tornike: <p>Finally, thank you. I thought that color channels were the first input in self.conv1</p>
Why does the neural network predict only positive values, when targets in the training dataset contain negative values as well?

Capo_Mestre: <p>Hello!<br>
As you can see predicted curves are zeroed at the places where target-curves have negative values.</p>
<p>So the question:</p>
<p>Why does the neural network predict (orange) only positive values, when targets (blue) in the training dataset contain negative values as well? What could possible reasons be?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34f2e1e3e1202aa387ec85277efa42166652a88a.png" alt="Figure 2021-03-22 192131" data-base62-sha1="7ypcuPKgxpZr5Qj2xm25CPJdPbc" width="374" height="251"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d7a359cee3b084bc4fd907acc0e2876ebbcf0cc.png" alt="Figure 2021-03-22 192149" data-base62-sha1="hU1w0DaFg9khaw683rv6hIFzcCM" width="374" height="251"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bc0fe676f4fc0cea9b4c1ceec0660bbe4b4958f.png" alt="Figure 2021-03-22 192206" data-base62-sha1="fnezKlN9IVLlltzzdRsUuWKWFj1" width="374" height="251"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84d3440dca813bef8bf27b04de95d17695331ecb.png" alt="Untitled" data-base62-sha1="iX1EZ86c42d5K5MnhHqhOSeMlZh" width="374" height="251"></p>

Dwight_Foster: <p>Can you provide some code?</p>

patrickwilliams3: <p>Do you have a ReLU layer at the end that would zero out any negative values?</p>

Capo_Mestre: <p>I am happy, but the code is huge, and I dont have any idea where this behaviour may come from.</p>
<p>Maybe I can describe main features.</p>
<p>It is a convolutional neural network with</p>
<ul>
<li>1 conv-layer (7 filters 11x11, stride and paddings are all equal to 5, no maxpools nor batchnorms) and</li>
<li>2 fully-connected layers (with the second layer being an output layer). I apply batchnorm to the first FC-layer.</li>
</ul>
<p>I use ReLU activation functions everywhere.</p>
<p>The loss function is defined by <code>torch.nn.MSELoss()</code>, i.e. mean-squared-error loss function.</p>
<p>I use Adam optimization method.</p>
<p>Please, let me know if there is anything else that needs to be mentioned.</p>
<p>I tested my code on the MNIST hand-written digits dataset, and it worked well (provided that loss is cross-entropy type loss and few other corrections to a classification problem).</p>

Capo_Mestre: <p>I just saw your answer. Yes! That is probably it. I will try to use <code>sigmoid</code> function at the end.</p>

patrickwilliams3: <p>A sigmoid function would not be any better since that would bound your answer to [0,1]. It seems that your output does not have a specific range so it may be better to use no activation function at the end in that case.</p>

Capo_Mestre: <p>I will try that, thanks!</p>