akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mask R-CNN native PyTorch Implementation

username: mbordi
cooked: <p>Hello,</p>
<p>I was currently working with Mask R-CNN and was  reading a wonderful <a href="https://medium.com/@fractaldle/mask-r-cnn-unmasked-c029aa2f1296" rel="nofollow noopener">article</a> about it.</p>
<p>In my project we already know which parts of the image we want to proceed with box and mask heads of the networks. Using <code>roi_heads</code> I can get the predictions for mask and and bounding boxes but have some problems with processing it and converting to the original image size.</p>
<p>Following the article above the mask logits postprocessing look like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://miro.medium.com/max/2000/1*SQCtcZTTaHj7Ao8-8IY0rQ.jpeg" title="" rel="nofollow noopener"><img src="https://miro.medium.com/max/2000/1*SQCtcZTTaHj7Ao8-8IY0rQ.jpeg" alt="" width="690" height="66"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">1934×187</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p>And my question is where I can find these steps in the native PyTorch implementation. I tried to look at the torchvision/detection/roi_align. py and transforms, but still do not understand how it works.</p>
<p>I would appreciate any help.<br>
Thank you!</p>
Issue with utils.save_image

username: 4158ndfkvBHJ1
cooked: <p>I want to use SPADE to do image synthesis. My input is a bunch of .tif files, which I need to convert to jpeg. I wrote the following script :</p>
<p>for img in img_files:<br>
tempstrg = img.split("_")<br>
lab = [fl for fl in label_files if (tempstrg[1] in fl and tempstrg[2] in fl)]<br>
image = Image.open(path_image + img)<br>
label = Image.open(path_label + lab[0])<br>
image = transforms.ToTensor()(image)<br>
label = transforms.ToTensor()(label)<br>
print(“uniques in image :”, len(np.unique(np.array(image))))<br>
print(“uniques in label :”, len(np.unique(np.array(label))))<br>
utils.save_image(image, img_jpg + str(index) + “.jpg”)<br>
utils.save_image(label, label_jpg + str(index) + “.jpg”)<br>
index = index + 1</p>
<p>expectedly, the label map has a very low number of unique values (6 in my case), so all is well. But when I do</p>
<p>for f in files:<br>
label = Image.open(path + f)<br>
label = transforms.ToTensor()(label)<br>
print(label.size())<br>
label = np.array(label)<br>
print(len(np.unique(label)))</p>
<p>the number of label doesn’t stay the same, and SPADE training crashes because of it. Is there some sort of normalization / interpolation done somewhere that I’m not aware of ?</p>

username: ptrblck
cooked: <p><code>ToTensor</code> will normalize the input image, such that its output will have values in the range <code>[0, 1]</code>.<br>
While this is expected for input image tensors, this might be problematic, if you are dealing with mask images, which contain class indices.<br>
Instead of <code>ToTensor</code> you should probably create the tensor via <code>torch.from_numpy</code>, which should keep the values. Also, I’m not sure, how <code>save_image</code> will act on the mask, so you should double check, that it doesn’t change the values either.</p>

username: 4158ndfkvBHJ1
cooked: <p>I ended up doing something very simple, converting to a numpy array and saving with PIL. But thanks for the heads up in any case !</p>
Methods to extract features from CNN

username: gktejus
cooked: <p>I’m dealing with an encoder-decoder network where the decoder network is followed by a fully-connected layer and a linear layer (output). However, before flattening the features from the decoder, I’ve seen people do either of two things,i.e either use a single <code>nn.AdaptiveAvgPool2d()</code> with an output size of say <code>(7,7)</code>, flatten it and pass it onto the fully-connected layer or use <code>nn.AdaptiveAvgPool2d()</code> and <code>nn.AdaptiveMaxPool2d()</code> each with an output size of <code>(1,1)</code>, concatenate them and flatten them and pass it onto the fully-connected layers.I’m not sure what are the benefits/disadvantages of both the methods are and why they are done so?</p>
How to use the sum output of the two models to update the two models

username: Qu_Yukun
cooked: <p>I try to train a two-stram network. I have two models, and I hope I can use the sum output of the two models to update the two models.</p>
<p>Now I have two models, get their loss respectively: they are loss1 and loss2.<br>
And then I add the two loss values: loss = loss1 + loss2.<br>
The question is that：<br>
loss.backward() will update wich models’ gradient?  The last one or both of them?</p>
<p>The code is as follows:</p>
<blockquote>
<p>spat_out = spat_model(spat_data)<br>
temp_out = temp_model(temp_data)</p>
<p>spat_loss = spat_criterion(spat_out, labels)<br>
temp_loss = temp_criterion(temp_out, labels)</p>
<p>loss = spat_loss + temp_loss<br>
loss.backward()</p>
<p>spat_optimizer.step()<br>
temp_optimizer.step()</p>
</blockquote>
<p>Am I using the correct method?</p>

username: ptrblck
cooked: <p>The <code>loss.backward()</code> call will let Autograd calculate the gradients for both models, as the final <code>loss</code> is a sum of both partial losses.<br>
You can also verify it by printing the gradients of some parameters of these models after the backward call.</p>
Should class-weights be used to compensate for imbalance?

username: FaultyBagnose
cooked: <p>Standard image classification design with CNN: The cross-entropy loss function has a weight option, which can apparently be used to compensate for class imbalance. This essentially scales up the learning rate for images of the smaller classes, right? How would this be better than including multiple copies of the images? This should on average have the same effect on the gradient without introducing “shocks” in the way of occasional long jumps in the parameter space?</p>
<p>Of course, you can use non-integer class weights, which does not quite fit the multiple copies approach, but has up-weighting other benefits compared to using multiple copies?</p>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="FaultyBagnose" data-post="1" data-topic="83808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/96bed5/48.png" class="avatar"> FaultyBagnose:</div>
<blockquote>
<p>This essentially scales up the learning rate for images of the smaller classes, right?<br>
[…]<br>
This should on average have the same effect on the gradient without introducing “shocks” in the way of occasional long jumps in the parameter space?</p>
</blockquote>
</aside>
<p>The loss will be relatively scaled for each class and the corresponding weight.<br>
I.e. in the default setup (calculating the mean of the batch loss), you would normalize the final loss with the used weights, so that the overall gradient doesn’t explode in case you sample a batch full of samples with a high weight.</p>
<p>From my experience, I got better results from oversampling the minority class, but I wouldn’t generalize it and others might prefer a weighted criterion.</p>
Random split per class in ImageFolder

username: Luca_Pamparana
cooked: <p>I am using torchvision <code>ImageFolder</code> to load my training data where each directory contain samples for a particular class instance.</p>
<p>I want to split the data into training and validation set. How can I do it so that the split is done per class i.e. if the split is 90-10 for 90% training data and 10% validation, it should randomly choose 90% of the data for training from each of the folders.</p>
<p>Is there an easy way to do this from within pytorch with the <code>ImageFolder</code> dataset?</p>

username: ptrblck
cooked: <p>You could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.train_test_split</code></a> with the <code>stratify</code> option to create the desired sample indices.<br>
Once you have the indices, create the training and validation datasets by wrapping the whole dataset into <code>Subset</code>s and passing the corresponding indices to them.</p>
BatchNorm cause "CUDA out of memory"

username: JTiC
cooked: <p>Hi. My code works well with small batch size on single GPU. However, training samples with larger batch size on multiple GPUs, an error will raise when operating BN layers, e.g., “CUDA out of memory. Tried to allocate 418.00 MiB (GPU 0; 11.91 GiB total capacity; 11.13 GiB already allocated; 194.56 MiB free; 11.18 GiB reserved in total by PyTorch)”.</p>
<p>Besides, for some reasons, it is unsuitable to use smaller batch size and other normalization methods (e.g., LayerNorm).</p>
<p>Is there any advice to fix this bug? Thank you in advance.</p>

username: surya00060
cooked: <p>Hi</p>
<p>Can you give more details about how you are training on multiple GPUs.</p>
<p>For training on MultiGPUs, One way is to use <code>DataParallel()</code> where batches of input data are split across GPUs and after each step of computation gradient accumulation happens on a single GPU. You can refer to this tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">link</a>.</p>
Graph_executor behavior change in 1.5

username: seanprime7
cooked: <p>Hi all,</p>
<p>It seems that <code>.graph_for</code> doesn’t seem to run optimizations anymore with PyTorch 1.5.</p>
<p>For example, from the following simple code, <code>graph_for</code> dumps different graphs between PyTorch 1.4 and PyTorch 1.5.</p>
<pre><code class="lang-auto">$ cat small.py
import torch

def f(x,y): return x + y * 3

print(torch.jit.script(f).graph_for(torch.rand(2, 2, device='cuda'), torch.rand(2, 2, device='cuda')))
</code></pre>
<p><strong>PyTorch 1.4</strong>:</p>
<pre><code class="lang-auto">$ python small.py
graph(%x.1 : Float(*, *),
      %y.1 : Float(*, *)):
  %6 : Float(*, *) = prim::FusionGroup_0(%x.1, %y.1)
  return (%6)
with prim::FusionGroup_0 = graph(%0 : Float(*, *),
      %4 : Float(*, *)):
  %2 : int = prim::Constant[value=1]()
  %5 : int = prim::Constant[value=3]() # small.py:3:27
  %6 : Float(*, *) = aten::mul(%4, %5) # small.py:3:23
  %3 : Float(*, *) = aten::add(%0, %6, %2) # small.py:3:19
  return (%3)
</code></pre>
<p><strong>PyTorch 1.5</strong>:</p>
<pre><code class="lang-auto">$ python small.py
graph(%x.1 : Tensor,
      %y.1 : Tensor):
  %3 : int = prim::Constant[value=3]() # small.py:3:27
  %2 : int = prim::Constant[value=1]()
  %4 : Tensor = aten::mul(%y.1, %3) # small.py:3:23
  %5 : Tensor = aten::add(%x.1, %4, %2) # small.py:3:19
  return (%5)
</code></pre>
<p>Is this expected?  With PyTorch 1.5, I don’t see any fusion group.</p>

username: seanprime7
cooked: <p>Found the description of the change in the release note: <a href="https://github.com/pytorch/pytorch/releases/tag/v1.5.0" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases/tag/v1.5.0</a>.</p>
<p>But even with <code>torch._C._jit_set_profiling_mode(True)</code>, I don’t see any fusion group in the output.</p>
<pre><code class="lang-auto">$ cat small.py
import torch
torch._C._jit_set_profiling_mode(True)

def f(x,y): return x + y * 3

print(torch.jit.script(f).graph_for(torch.rand(2, 2, device='cuda'), torch.rand(2, 2, device='cuda')))

$ python small.py
graph(%x.1 : Tensor,
      %y.1 : Tensor):
  %2 : int = prim::Constant[value=3]() # small.py:4:27
  %3 : int = prim::Constant[value=1]()
  %4 : Tensor = prim::profile(%y.1)
  %5 : Tensor = aten::mul(%4, %2) # small.py:4:23
  %6 : Tensor = prim::profile(%x.1)
  %7 : Tensor = prim::profile(%5)
  %8 : Tensor = aten::add(%6, %7, %3) # small.py:4:19
  %9 : Tensor = prim::profile(%8)
   = prim::profile()
  return (%9)
</code></pre>

username: ptrblck
cooked: <p>Could you try to run the code in the latest nightly build?<br>
If you can still reproduce this issue, feel free to create an issue <a href="https://github.com/pytorch/pytorch/issues">here</a> including the reproducible code snippets.</p>
Unable to import torch

username: michellemay
cooked: <p>I have installed pytorch and torchvision via annaconda, I can see torchvision 0.6.0 using “conda list” command and I can see torch 1.5.0 under “pip list”.<br>
I can find these 2 under C:…\Anaconda3\Lib\site-packages<br>
But when I tried to import torch, the following error occurs.</p>
<p>Traceback (most recent call last):<br>
File “.\inference_combined_all.py”, line 2, in <br>
import torch<br>
File “C:\Users\qizha\Anaconda3\lib\site-packages\torch_<em>init</em>_.py”, line 81, in <br>
ctypes.CDLL(dll)<br>
File “C:\Users\qizha\Anaconda3\lib\ctypes_<em>init</em>_.py”, line 364, in <strong>init</strong><br>
self._handle = _dlopen(self._name, mode)<br>
OSError: [WinError 126] The specified module could not be found</p>
<p>Does anyone have any thoughts on this? Thank you!</p>

username: albanD
cooked: <p>cc <a class="mention" href="/u/peterjc123">@peterjc123</a> do you have an idea where this could be coming from?</p>

username: ptrblck
cooked: <p>This error seems to be raised, if a PyTorch version with CUDA support was installed on a CPU-only machine, as described <a href="https://discuss.pytorch.org/t/cannot-import-torch-on-jupyter-notebook/79334/3">here</a> by <a class="mention" href="/u/peterjc123">@peterjc123</a>.</p>
How to keep only top k percent values

username: Absurd
cooked: <p>Supports that x is a tensor with shape = [batch_size, channel_number, height, width], some may be positive value and others may be 0. And I only want to keep the value of top 10% positive values of each channel and make others 0. But the problem is that the number of pixels of positive value in each channel is different, so I wonder how to solve it efficiently?</p>

username: ptrblck
cooked: <p>It shouldn’t matter, if some values are positive and others zero, since the result tensor would also contain zeros. So even if the top 10% includes zeros (if not enough positive values were found), you wouldn’t change the output, would you?</p>
<p>If that’s the case, this code should work:</p>
<pre><code class="lang-python"># Setup
x = torch.randn(2, 3, 10, 10)
orig_shape = x.size()

# Reshape and calculate positions of top 10%
x = x.view(x.size(0), x.size(1), -1)
nb_pixels = x.size(2)
ret = torch.topk(x, k=int(0.1*nb_pixels), dim=2)
ret.indices.shape

# Scatter to zero'd tensor
res = torch.zeros_like(x)
res.scatter_(2, ret.indices, ret.values)
res = res.view(*orig_shape)
</code></pre>

username: Absurd
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="83706">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>matter</p>
</blockquote>
</aside>
<p>Thanks for your answer. Yes, it seems that the number of positive pixel actually doesn’t matter! Your answer helps me a lot, thanks again!</p>
shuffle=True makes the weird result

username: Yu_Syuan_Sean_Lin
cooked: <p>I am trying to make a prediction on stock data with a simple LSTM model.<br>
I spilt first 80% data as train set, and last 20% data as test set.</p>
<p>The data shape is like ( batch_size , seq_len , feature_dims)</p>
<p>here is my code for data loader</p>
<pre><code class="lang-auto">train_loader=Data.DataLoader(dataset=trainDataset,batch_size=BATCH_SIZE,shuffle=True)
</code></pre>
<p>when I set shuffle = false, everything work fine. (Left pics)</p>
<p>However, when I change shuffle to False, the prediction value become weird.(Right pics),the output value will stuck in a very small range.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e" title="shufflefalse"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e_2_690x222.png" alt="shufflefalse" data-base62-sha1="cRxjduvdEodDvxas4cJ7U4pCBLE" width="690" height="222" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e_2_690x222.png, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a275d42ae96ce1b38a9b52e1922bfa13e0f107e.png 2x" data-dominant-color="F6F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">shufflefalse</span><span class="informations">884×285 48.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone give me any suggestion? Thanks!</p>

username: ptrblck
cooked: <p>If shuffling interferes negatively with your training, it seems that your training routine or model depends on the sequential input of the data.<br>
E.g. if you are using an RNN-like model, you might pass the batches sequentially and reusing the hidden states. In that case, shuffling the data might destroy the temporal correspondence.<br>
You could try to shuffle “segments” of the data, but it depends of course on your actual use case.</p>
Impact on Performance when model is trained at float32 and inference is done at float16 (half precision)?

username: Nisan_Aryal
cooked: <p>I am trying to train the model in float32 and inference in float16. I think my code is working fine. I wanted to know if anyone else has done this to reduce the size of the model and is there any change in accuracy while decreasing the precision? I know that this helps to increase the speed a little but I am worried about the performance.</p>

username: ptrblck
cooked: <p>To be save, you could use native mixed-precision via <a href="https://pytorch.org/docs/master/amp.html"><code>torch.cuda.amp</code></a>, which will make sure that only “safe” operations are performed in FP16.<br>
Calling <code>half()</code> on your model directly might work in a lot of use cases, but might also yield overflows etc., so you would have to verify your model manually.</p>
<p>If you want to manually use FP16, you could check e.g. the validation accuracy before deploying the model as a quick sanity check.</p>
Question about nn.LSTMCell

username: Wjk666
cooked: <p>Recently, I deccided to  realize a dialogue system. But focus on the formula as followings, I’m confused how to realize it by pytorch.</p>
<p>At time t, the encoding hidden state of the first layer l1<br>
and second layer l2 are defined as:</p>
<p>$h_{t}^{l_1} = LSTM(x_t,h_{t-1}^{l_{1}})$<br>
$h_{t}^{l_2} = LSTM(&lt;pad&gt;,h_{t}^{l_1},h_{t-1}^{l_{2}})​$<br>
where $x_t$ is the input word embedding. The special symbol<code>&lt;pad&gt;</code> denotes padding with zero. $h_{0}^{l_1} $ and $h_{0}^{l_2} $  are initialized with zero vectors.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e928e53e59b0e5d1ed81755a3a2a42e3de65227a.png" data-download-href="https://discuss.pytorch.org/uploads/default/e928e53e59b0e5d1ed81755a3a2a42e3de65227a" title="RG~PZC2({)46DDZRK_2XC"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e928e53e59b0e5d1ed81755a3a2a42e3de65227a_2_329x500.png" alt="RG~PZC2({)46DDZRK_2XC" data-base62-sha1="xgCRRB6aLJrcUOvO3VppwfFg9zI" width="329" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e928e53e59b0e5d1ed81755a3a2a42e3de65227a_2_329x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e928e53e59b0e5d1ed81755a3a2a42e3de65227a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e928e53e59b0e5d1ed81755a3a2a42e3de65227a.png 2x" data-dominant-color="E8EAE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RG~PZC2({)46DDZRK_2XC</span><span class="informations">487×738 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so how to deal with it when LSTM need multi inputs? I think it needs <code>LSTMCell</code> because of the differences between layer 1 and layer 2. But LSTMCell’s input are <code>input,h,c</code>. <strong>How to realize it as the representation of formula (2) and (3)</strong> Or shoud I concatenate $y_{t-1}$ and $s_{t-1}^{l_2}$ as <code>input</code>?</p>
PWCNet not working

username: soumilkanwal80
cooked: <p>I am trying to setup PWCNet for optical flow extraction using PyTorch, from this link: <a href="https://github.com/NVlabs/PWC-Net/tree/master/PyTorch" rel="nofollow noopener">https://github.com/NVlabs/PWC-Net/tree/master/PyTorch</a>.</p>
<p>After following the steps in ReadMe it says<br>
ImportError(“torch.utils.ffi is deprecated. Please use cpp extensions instead”)<br>
On this link <a href="https://github.com/pytorch/pytorch/issues/15645" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15645</a> they say so there is no easy solution to this? it looks like it requires a full revamp of the code that used to work for pytorch &lt;= 0.4.1<br>
But the code is written in PyTorch 0.2</p>
<p>Neither I am able to install cudatoolkit 8.0 using  conda as mentioned in the ReadMe.</p>
How to keep the accuacy when convert a pytorch model to TensorRT in FP16 mode

username: lcxywfe
cooked: <p>I want to use TensorRT FP16 mode to accelerate my pytorch model, but there is a loss of precision. My question is how can I use the mixed precision training of pytorch, to avoid the loss of accuracy when converting to a TensorRT FP16 model. I have tried the  <code>torch.cuda.amp.autocast</code>  to training the model, but it still lost some precision.</p>

username: ptrblck
cooked: <p>Do you see a loss in accuracy comparing plain PyTorch code to mixed-precision training using <code>torch.cuda.amp</code>?<br>
How are you exporting the model to TensorRT?</p>

username: lcxywfe
cooked: <p>The mixed-precision training is OK, the loss is introduced by the conversion to the TensorRT fp16 engine.<br>
PyTorch -&gt; ONNX -&gt; TensorRT</p>

username: ptrblck
cooked: <p>How large is the accuracy drop after the conversion to TensorRT?</p>
Accessing accumulated quantized result

username: samgd
cooked: <p>Hi,</p>
<p>The PyTorch quantized operations are great but they return the result after it has been quantized back to 8-bit integer. Is there a simple way of accessing the accumulated result at int32? Are there other libraries, such as Caffe or a different quantization backend, that have this API?</p>

username: supriyar
cooked: <p>Re-quantization (conversion from int32 to int8) is performed in the quantized operator libraries - FBGEMM and QNNPACK.<br>
Currently there is no easy way to access this value from pytorch API.</p>
Which combination of criterion and activation function to get 1 probability prediction

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>Currently I have a model to predict a binary target using <code>NLLLoss()</code> with the final layer going through a <code>log_softmax()</code> which gives me probability for target being 0 and probability for target being 1.  I would like to output 1 final probability in order to use the <code>Captum</code> package to calculate feature importance.  What is the correct combination of loss function and final activation function to give me one probability prediction instead of two?</p>

username: chetan_patil
cooked: <p>Hi, I think <code>nn.Sigmoid()</code> as the last activation layer, and <code>nn.BCELosswithLogits()</code> would help here.</p>

username: Jordan_Howell
cooked: <p>thank you.  The <code>nn.sigmoid()</code> worked but the <code>nn.BCELosswithLogits()</code> gave two outputs.  I’m testing sigmoid with just <code>BCELoss</code> now.</p>

username: chetan_patil
cooked: <p>Hi, actually I was wrong, you can either use <code>nn Sigmoid()</code> and <code>BCELoss()</code> , or you can use one standalone function <code>nn.BCELosswithLogits()</code> which combines both the Sigmoid as well as the Log-likelihood formula.<br>
Apparently, nn.BCELoss() and nn.BCEwithLogitsLoss() should yield same number of outputs.</p>
Modifying a parameter in forward pass after getting Nan in the backward pass

username: Abdelpakey
cooked: <p>Hi,</p>
<p>I hope I can explain my scenario well so I can get the correct answer.<br>
I have a customized loss function say:</p>
<p>loss= function (op1, op2, op3,…, op3), where op is an operation.</p>
<p>After a couple of epochs, I get Nan in the gradient, but I know which operation, say op3, causes this Nan (SVDBackward operation in my case) and I know how to fix it mathematically.</p>
<p>Aside note:<br>
1-  if you’re wondering why  I’m not simply using  IF-Else statement since I know the problem and how to solve it. The answer is, the problem happens in the backward pass and cant be noticed from only the forward pass.</p>
<p>2- torch.nn.utils.clip_grad_norm_ will not work in my case.</p>
<p>My question is: is there a way to modify the op3 after the backward pass, i.e (loss.backward()?</p>
<p>Something like this:</p>
<pre><code>loss.backward()
doing the magic code to modify op3 that will be executed once in this current forward 
pass..</code></pre>

username: Abdelpakey
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you advise on this?</p>

username: ptrblck
cooked: <p>I don’t know, how the function to modify the forward should work, but you could register a hook on the parameters, which create NaNs, check for invalid values during the <code>backward</code> pass, and set a flag which can be checked to modify your model.<br>
After the faulty backward pass, you should zero out the gradients, check for the flag, and modify your model.</p>
<p>Would this workflow generally work?</p>
Deep learning question

username: VladislavPrh
cooked: <p>Hello everyone, can you give some tips for my task?</p>
<p>I have context and response where context some dialog and response just response to this dialog so my model need to classify: this response human-generated or machine-generated.<br>
I have tried seq2seq, RNN, and CNN(text classification) but the results were not so good as I wanted.</p>
<p>I know that this question does not relate to the Pytorch, but may be you have some ideas? <img src="//discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=5" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Thank you!</p>

username: timbmg
cooked: <p>In principal a RNN should do the job here… What did you try, and why was it not satifactory? What data do you use?</p>

username: VladislavPrh
cooked: <p>RNN works fine for me it gives ~ 80% accuracy on the validation set but there is some way to get higher accuracy(Up to 90%). This data from Hackathon and the size is 7 millions examples</p>

username: sarace
cooked: <p>Hi,I have the same problem as you,could you explain how you do it? Thanks a lot!!</p>
How to save MNIST as jpg files

username: J_Na
cooked: <p>Hi, all</p>
<p>How to save MNIST as .jpg format ?<br>
Is it possible with<br>
<code>from torchvision.utils import save_image</code>?</p>
<p>(I use default dataloader from pytorch.)</p>

username: ptrblck
cooked: <p>Since the <code>torchvision.datasets.MNIST</code> dataset returns PIL Images, if you don’t specify a transformation, the easiest way would be to directly save them:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(root=PATH)

for idx, (img, _) in enumerate(dataset):
    img.save('{:05d}.jpg'.format(idx))
</code></pre>

username: saba
cooked: <p>Sorry ptrblck,</p>
<p>How I can same this images. I use this command it save it but it is nothing in it just white.</p>
<pre><code class="lang-auto">real_batch = next(iter(trainloader))
fig=plt.figure()
plt.figure(figsize=(8,8))
plt.axis("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))
fig.savefig(os.path.join(root_dirDurringTraining13+str(1)+str(2))+'Images.jpg',dpi=fig.dpi)
</code></pre>
<p>would you please guide me how to save it ?</p>

username: ptrblck
cooked: <p>Is <code>plt.imshow</code> showing the expected figure and is only <code>savefig</code> failing or do you also get a while output in <code>imshow</code>?<br>
I would recommend to check the min and max values of your image array, as <code>matplotlib</code> could try to normalize the images thus potentially removing information.</p>

username: saba
cooked: <p>it is solved I should used plt=plt.figure<br>
many thanks</p>
CUDA memory explosion when using torch.cat or torch.stack

username: DrHB
cooked: <p>Whenever with big <code>tensors</code> I use <code>torch.stack</code> or <code>torch.cat</code> I start to get  <code>cuda</code> Runtime Error. However writing custom <code>jit</code> script which does similar operation doesn’t result in memory issue.</p>
<p>Below is simple example:</p>
<pre><code class="lang-auto">import torch
from torch.jit import script
x =torch.rand([1000, 1000, 1000]).cuda()
y =torch.rand([1000, 1000, 1000]).cuda()
z =torch.rand([1000, 1000, 1000]).cuda()

torch.stack((x, y, z))
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-8-515e22a6cd7a&gt; in &lt;module&gt;
----&gt; 1 torch.stack((x, y, z))

RuntimeError: CUDA out of memory. Tried to allocate 11.18 GiB (GPU 0; 11.93 GiB total capacity; 11.18 GiB already allocated; 354.81 MiB free; 11.18 GiB reserved in total by PyTorch)
</code></pre>
<p>This some reshaping which I need for my project written in jit runs without error.</p>
<pre><code class="lang-auto">@script
def get_grids(z_os, y_os, x_os, fs):
    empty = []
    for i in range(z_os.shape[0]//fs):
        ten_ = torch.stack((z_os[i*fs:(i+1)*fs, i*fs:(i+1)*fs, i*fs:(i+1)*fs], 
                            y_os[i*fs:(i+1)*fs, i*fs:(i+1)*fs, i*fs:(i+1)*fs], 
                            x_os[i*fs:(i+1)*fs, i*fs:(i+1)*fs, i*fs:(i+1)*fs]),3)
        
        empty.append(ten_)
    return torch.stack(empty)
</code></pre>
<pre><code class="lang-auto">res = get_grids(x, y, z, torch.tensor(10))
res.shape
torch.Size([100, 10, 10, 10, 3])
</code></pre>
<p>Usually I work with much bigger <code>tensors</code> . What is the explanation for such behavior? And is there better way to achieve what I want without using <code>jit</code>?</p>
How to create a validation dataset for siamese network model

username: Luca_Pamparana
cooked: <p>I am trying to implement a siamese network model using pytorch and pytorch lightning and I have a question about how to split such a dataset into training and validation dataset.</p>
<p>So, for a siamese network we create a triplet of input data point where we have an anchor image, a positive image (belonging to the same class as the anchor image) and a negative image (belonging to a different class). So, I am trying to use the pytorch dataset functionality as follows:</p>
<p>So, the dataset creation at the moment is as follows:</p>
<pre><code class="lang-auto">class SiameseTriplet(Dataset):
    def __init__(self, torchvision_dataset, transform=None):
        # Store things to class variables
        ....

    def __getitem__(self, index):
        img0_tuple = random.choice(self.dataset.imgs)
       # randomly sample to get a negative sample
       ...
       anchor_class_name = img0_tuple[0].split('/')[-2]
       all_files_in_class = glob.glob(self.dataset.root + anchor_class_name + '/*')

      # randomly sample now to get the positive image
      positive_image = random.choice(all_files_in_class)

     # Do some image transformations
     ....
     return anchor, positive, negative
</code></pre>
<p>Now what I basically want to do create a validation dataset. I am not sure what the best strategy is for that. Say I want to do a 95-5 split between training and validation. I thought the best way to do so would be split the dataset so that each class has the 95-5 split. Is that something reasonable to do? So say I have three classes and my data is organized as follows:</p>
<p>|— root<br>
|-------- cat<br>
|-------- dog<br>
| ------- horse</p>
<p>In that case, I would go the 95 -5 split on each of these classes. Is that something reasonable to do for Siamese networks?</p>
<p>Now in my training code, I do something like:</p>
<pre><code class="lang-auto">import torchvision.datasets as dset
train_folder_dataset = dset.ImageFolder(root=data_path)

train_dataset = SiameseTriplet(image_folder_dataset=train_folder_dataset,
                                                 transform=transform)
</code></pre>
<p>Now, it seems to me that the ideal place to split the dataset would be after the <code>train_folder_dataset = dset.ImageFolder(root=data_path)</code> line. However, I cannot figure out how to do so with the strategy I discussed earlier where each of the class has the specified split.</p>
<p>I know there is a <code>torch.utils.data.Subset</code> class for doing a split on ImageFolder but how do I tell it to apply the split <em>per class</em> as well?</p>
How to load labels a small dataset in a single batch Pytorch

username: Muhammad_Ahmed
cooked: <p>At the moment, I just want labels of a dataset, Whenever i try to load the labels in following way,<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9920bc7ca84c4a7fb7ec8c489bcc9b30cc91d5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9920bc7ca84c4a7fb7ec8c489bcc9b30cc91d5e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9920bc7ca84c4a7fb7ec8c489bcc9b30cc91d5e.png" alt="image" data-base62-sha1="oc5ARQUJOyJE2n4LZxG9J5tm1l4" width="690" height="75" data-dominant-color="282828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">712×78 3.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I get error PID was killed. Any solution for this ?</p>

username: Muhammad_Ahmed
cooked: <p>The length of trainloader is 1.<br>
I try to iterate through it once and get all the labels of dataset.<br>
I get error PID “Any number 160/240” was killed.</p>
<p>This clearly shouldn’t exceed memory limit since its mere labels.</p>

username: Nikronic
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="Muhammad_Ahmed" data-post="2" data-topic="83762">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammad_ahmed/48/22672_2.png" class="avatar"> Muhammad_Ahmed:</div>
<blockquote>
<p>The length of trainloader is 1.</p>
</blockquote>
</aside>
<p>The size of <code>trainloader</code> has to be 1 as you defined <code>batch_size=len(dataset)</code>.<br>
Your code is completely correct. Are you using custom datasets? Obviously, the problem is not from the code you have mentioned.</p>
<p>Here is small snippet that demonstrates your approach:</p>
<pre><code class="lang-auto">x = torch.randint(0, 2, size=(10, 3, 2, 2)).float()
y = torch.randint(0,2, size=(10, 1))
dataset = data.TensorDataset(x, y)
trainloader = data.DataLoader(dataset, batch_size=len(dataset), shuffle=True, )

temp=iter(trainloader)
labels = next(temp)[1]
labels.shape  # torch.Size([10, 1])
</code></pre>
<p>Bests</p>
Softmax Across 3D tensor

username: Jack_Rolph
cooked: <p>I have a tensor:</p>
<p><code>A = torch.randn(B,C,X,Y,Z)</code></p>
<p>I would like to perform a softmax activation over the channels <code>C</code>.</p>
<p>What I hope to achieve is that the sum of every non-zero element over channels <code>C</code> is equal to one.</p>
<p>I have the softmax function, which operates over some dimension. In my case, I would imagine that I use <code>dim=1</code>, if I wanted it over the channels.</p>
<pre><code class="lang-auto">sm = torch.nn.Softmax(dim=1)
</code></pre>
<p>Will this achieve the intended result?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Yes, it will.<br>
For instance, if you have <code>[1, 3, 2, 2, 2]</code> tensor, then validating softmax value at position <code>[0, 0, 0, 0, 0]</code>, you need following calculation:</p>
<pre><code class="lang-auto">torch.exp(x[0, 0, 0, 0, 0])/ (torch.exp(x[0, 0, 0, 0, 0])+torch.exp(x[0, 1, 0, 0, 0])+torch.exp(x[0, 2, 0, 0, 0]))
</code></pre>
<p>Bests</p>
How do GAN's differ?

username: German_1
cooked: <p>I am aware variations of GAN’s exist; i.e DCGAN, DiscoGAN, CicyleGAN; Source: <a href="https://github.com/eriklindernoren/PyTorch-GAN;" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN;</a> My question is, what are the primary constructs that differentiate GANs?<br>
I would imagine that having a specified network architecture acting as the Discriminator creates a variety of a GAN network, but I’m unsure and need clarification. Please let me know if this thinking is correct or not, and in addition please elaborate on what other factors lead to a variant GAN network.<br>
Thank you very much!</p>

username: ptrblck
cooked: <p>Besides changes in the architecture, different GAN variants use different loss functions, connections, inputs etc.<br>
<a href="https://arxiv.org/abs/1710.07035">This older paper</a> gives a brief overview. You can also find some blog posts about different GANs, but I’m not sure if there is a good “complete” overview.<br>
If I’m not mistaken, at one point a lot of new GAN architectures were published daily on arxiv, so it was quite hard to follow which variant might be promising.</p>

username: German_1
cooked: <p>That was helpful! Thanks a lot</p>
Extending PyTorch Example

username: maralm
cooked: <p>I looked at the custom backward example for extending pytorch. My question is that how we can access grad_input, grad_weight, grad_bias or any other returned variable from the backward function  after calling loss.backward() in this example?</p>
<pre><code class="lang-auto"># Inherit from Function
class LinearFunction(Function):

    # Note that both forward and backward are @staticmethods
    @staticmethod
    # bias is an optional argument
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    # This function has only a single output, so it gets only one gradient
    @staticmethod
    def backward(ctx, grad_output):
        # This is a pattern that is very convenient - at the top of backward
        # unpack saved_tensors and initialize all gradients w.r.t. inputs to
        # None. Thanks to the fact that additional trailing Nones are
        # ignored, the return statement is simple even when the function has
        # optional inputs.
        input, weight, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None

        # These needs_input_grad checks are optional and there only to
        # improve efficiency. If you want to make your code simpler, you can
        # skip them. Returning gradients for inputs that don't require it is
        # not an error.
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0)

        return grad_input, grad_weight, grad_bias
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Could you explain why you want to access these?<br>
Also these are regular python functions so you can simply store them in any structure in the parent scope.</p>

username: maralm
cooked: <p>In my actual example, I would like to copy the gradients to cpu (after computing them on gpu) and then access the cpu copies of gradients after loss.backward().</p>

username: albanD
cooked: <p>In that case, you can store them in the parent scope to access them later.<br>
Or use a hook on the Tensors directly as <code>t.register_hook()</code> to have a function that is called once the gradient for that Tensor is computed.</p>
Resnet152 size mismatch error

username: Mughees
cooked: <p>hi, I am trying to use resnet152 and stuck in the error in FC layer. I am unable to understand it right now.<br>
error is<br>
size mismatch, m1: [1 x 32768], m2: [2048 x 1000] at c:\programdata\miniconda3\conda-bld\pytorch_1533094653504\work\aten\src\th\generic/THTensorMath.cpp:2070<br>
Pytorch version: 0.4.1<br>
Torchvision: ‘0.2.1’</p>
<p>To reproduce:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
import torch.nn as nn
import numpy as np
model = models.resnet152()#pretrained=True)
tst = torch.rand(1,3,299,299)

output = model(tst)
![image|690x289](upload://6Qpxosqr1pkWw6R4QeaW3JanCnG.png)</code></pre>

username: chetan_patil
cooked: <p>Since you are hitting an error in FC layer, the problem is in the input size, could you change the input sizes to (1,3,224,224) or (1,3,229,229) ?</p>

username: Mughees
cooked: <p>Thanks it worked but I thought pytorch updated its model to accept any input size, which i have used as well in previous examples.</p>
Slicing Tensor in Pytorch

username: Jack_Rolph
cooked: <p>Suppose I have a tensor:</p>
<pre><code class="lang-auto">a = torch.randn(B,N,V)
</code></pre>
<p>I want to get the third column of the tensor <code>A</code> along axis <code>V</code>, in the format <code>(B,N,3,1)</code>.</p>
<p>I could do this by:</p>
<pre><code class="lang-auto">a_slice = input[:,:,3,None]
</code></pre>
<p>Particularly, I worry that this element of my code may not be differentiable. Is this the case?</p>
<p>If so, is there a way of doing this with Torch functions so that I don’t run into issues?</p>

username: albanD
cooked: <p>Hi,</p>
<p>All the indexing ops are differentiable. So no need to worry about it !</p>
Why we use Categorical

username: slavavs
cooked: <p>Please explain why we use Categorical? I can’t figure it out. What do we get at the categorical output?</p>
<pre><code class="lang-auto">def predict(state):
    # Select an action (0 or 1) by running policy model
    # and choosing based on the probabilities in state
    state = torch.from_numpy(state).type(torch.FloatTensor)
    action_probs = policy(state)
    distribution = Categorical(action_probs)
    action = distribution.sample2()
</code></pre>

username: Tim2
cooked: <pre><code class="lang-auto">distribution = Categorical(action_probs)
    action = distribution.sample2()
</code></pre>
<p>As I have understod it Categorical returns the propabillety of every possible action. Then one possible action gets sampled from this distribution (You can’t just do argmax(action_probs) because in some sence then there would be no exploration also the intension is to increase the propabillety of good actions and decrease the propabillety of bad actions that is why whe have to sample from the distripution instead of chosing the highest value)</p>
<p>I am fairly new to reinforcement learning and python so pleas correct me if I am wrong.</p>

username: kengz
cooked: <p>Why we use Categorical (or any other distribution) is a direct result of the policy formulation, where action is defined to be sampled from a distribution, <code>a ~ π(s)</code>, with probability of action = <code>π(a|s)</code>. It’s an implementation of the algorithm. You may change the distribution (e.g. to Gaussian for continuous actions), but you may not change it to an argmax because that is fundamentally a different algorithm/sampling mechanism (argmax is not a distribution).</p>
<p>Policy-based methods in RL like ActorCritic are different from Q-learning that uses a greedy-selection of action based on it’s Q-estimation (hence the argmax in Q-learning).</p>
Is this the right way to compute gradients of two losses from two different NN's?

username: Murtaza_Basu
cooked: <p>I have a NN defined in pytorch and I have created two instances of that net as <code>self.actor_critic_r1</code> and <code>self.actor_critic_r2</code>. I calculate the losses of each net i.e. <code>loss1</code> and <code>loss2</code> and I sum it up and calculate the grads in the following way,</p>
<pre><code class="lang-auto">#approach1
    loss_r1 = value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef
    loss_r2 = value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef
    self.optimizer_r1.zero_grad()
    self.optimizer_r2.zero_grad()
    loss = loss_r1 + loss_r2
    loss.backward()
    self.optimizer_r1.step()
    self.optimizer_r2.step()
    clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
    clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)
</code></pre>
<p>Alternatively, should I update the loss individually like this,</p>
<pre><code class="lang-auto">#approach2
self.optimizer_r1.zero_grad()
(value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef).backward()
self.optimizer_r1.step()
clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
self.optimizer_r2.zero_grad()
(value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef).backward()
self.optimizer_r2.step()
clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)
</code></pre>
<p>I am not sure if this the right approach to update a network with multiple loss please provide your suggestion.</p>

username: ptrblck
cooked: <p>I don’t know, how the losses were calculated, but both approaches should yield the same result, if no parameters are shared between the optimizers.<br>
To verify it, you could seed the code, run a single iteration using both approaches, and compare the gradients and final parameters in both runs.</p>
<p>Also, it seems you are clipping the gradients too late, as the optimizer already performed the <code>step</code> method.</p>

username: Murtaza_Basu
cooked: <p>Thank you for the clarification!</p>
Log probability of RelaxedBernoulli

username: Yashas_Annadani
cooked: <p>Here is a small snippet of code which initializes a relaxed bernoulli with temperature 0.1 and relaxed bernoulli parameter 0.8:</p>
<pre><code class="lang-auto">import torch
dist =  torch.distributions.relaxed_bernoulli.RelaxedBernoulli(torch.tensor(0.1), probs = torch.tensor([0.8]))
samp = dist.rsample()
samp_prob = torch.exp(dist.log_prob(samp))
print(samp_prob)
</code></pre>
<p>This most of the times gives probability much greater than 1. What am I missing here?</p>