akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[Resolved] Mini-batch training on PyTorch and Input&Param Size

username: 111137
cooked: <p>Hi,</p>
<p>I am <strong>new</strong> to PyTorch.</p>
<p>I made my own custom model and now run with <em>online training</em> (<strong>batch size = 1</strong>). As a next step, I want extend the model to take mini-batch. Then I want to study about how to support mini-batch on PyTorch.</p>
<ul>
<li>Does it need to extend parameter size holding for all indentical one propagation?</li>
</ul>
<p>I think that it is natural to have amount of mini-batch size for input and its label. But regarding parameters, I think the model should not have the size because of use the model and parameter for identical inference after its deploying.</p>
<p>Is my thinking a correct?</p>

username: ptrblck
cooked: <p>Yes, the size of the parameters is independent of the batch size.<br>
Your code is most likely already using “batched” inputs with a batch size of 1, so increasing the batch size should work out of the box.</p>

username: 111137
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> -san,</p>
<p>Again, thank you for your comment, before running, I did replace constant “1” in loading input with hyper-parameter being set to large number. And now it works (^ - ^).</p>
<p>My model did not work on TensorFlow but now it works fine with PyTorch.</p>
Minimal char RNN in PyTorch

username: Intel_Novel
cooked: <p>Anyone having PyTorch version of this minimal RNN written by Karpathy back in 2015.<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/karpathy/d4dee566867f8291f086" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/karpathy/d4dee566867f8291f086" target="_blank" rel="nofollow noopener">https://gist.github.com/karpathy/d4dee566867f8291f086</a></h4>
<h5>min-char-rnn.py</h5>
<pre><code class="Python">"""
Minimal character-level Vanilla RNN model. Written by Andrej Karpathy (@karpathy)
BSD License
"""
import numpy as np

# data I/O
data = open('input.txt', 'r').read() # should be simple plain text file
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)</code></pre>
This file has been truncated. <a href="https://gist.github.com/karpathy/d4dee566867f8291f086" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>This is part of his <a href="https://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noopener">The Unreasonable Effectiveness of Recurrent Neural Networks</a></p>
<p>I have hard time learning RNNs but no progress.</p>

username: starman
cooked: <p>A bit late but here it is <a href="https://github.com/nikhilbarhate99/Char-RNN-PyTorch" rel="nofollow noopener">char-rnn-pytorch</a></p>
How to clear memory after training

username: nicememesonnn
cooked: <p>I have a 1060 6GB and after training the model about 4 gigabytes still remain occupied. I either need to do a pkill -u [username] and login again before I retrain the model. Is there any way that it clears itself or is there something I’m doing wrong ?</p>

username: ptrblck
cooked: <p>Are you seeing this zombie process after your script has finished?<br>
Is your script crashing at some point?</p>

username: nicememesonnn
cooked: <p>No, the process does not crash. But I do see zombie processes.</p>

username: ptrblck
cooked: <p>Do you have a (small) reproducible code snippet so that we could have a look?</p>
Pytorch dataloader

username: mvadrev
cooked: <p>Hello,</p>
<p>I am using librosa to convert some greeting phases as audio files to an STFT spectrogram. I the used the ImageFolder command to load and covert the images to tensors using transforms.ToTensor() as transforms</p>
<p>[code:]</p>
<p>TRAIN_DATA_PATH = ‘…\librosa\images’<br>
train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)<br>
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)</p>
<p>[/code:]</p>
<p>There are a few things I want to do but did not find clear cut explanations on them.</p>
<p>I want to visualize each of the images from the data loader. Can someone explain with some code about how to do this? I tried looking in this forum but found a few ways. Can someone give me one standard way with an example?</p>

username: mvadrev
cooked: <p>I tried this from the udemy pytorch course but got an error</p>
<p>dataiter = iter(train_data_loader)<br>
images, labels = dataiter.next()</p>
<p>for idx in np.arange(2):<br>
ax = fig.add_subplot(2,20/2,idx+1, xticks=[],yticks=[])<br>
ax.imshow(np.squeeze(images[idx]))</p>
<p>Error: TypeError: Invalid shape (3, 256, 256) for image data</p>
Help me understand transform normalize

username: mayur_newase
cooked: <p>I am trying to normalize image(mean 0 std 1) before giving it to neural network.<br>
This is my code.</p>
<pre><code class="lang-auto">def stats(image):
    #expects image in [H,W,C] format
    print("channel 1 min {} max {} mean {} std {}".format(image[:,:,0].min(), image[:,:,0].max(), image[:,:,0].mean(), image[:,:,0].std()))
    print("channel 2 min {} max {} mean {} std {}".format(image[:,:,1].min(), image[:,:,1].max(), image[:,:,1].mean(), image[:,:,1].std()))
    print("channel 3 min {} max {} mean {} std {}".format(image[:,:,2].min(), image[:,:,2].max(), image[:,:,2].mean(), image[:,:,2].std()))

image = cv2.imread(train.loc[0, "path"]); print("original shape", image.shape)
print("original image stats")
stats(image)
image_tensor = transforms.ToTensor()(image); print("\nimage tensor shape ", image_tensor.shape)  #scales in [0,1] but also changes dimension[c, H, W]

#now lets normalize it per channel and get mean 0 and std 1
#for which I hope it will use following formula
"""
image_n[:,:,0] = (image_n[:,:,0] - image_n[:,:,0].mean()) / image_n[:,:,0].std()
image_n[:,:,1] = (image_n[:,:,1] - image_n[:,:,1].mean()) / image_n[:,:,1].std()
image_n[:,:,2] = (image_n[:,:,2] - image_n[:,:,2].mean()) / image_n[:,:,2].std()
"""

image_n = transforms.Normalize(mean = [image[:,:,0].mean(), image[:,:,1].mean(), image[:,:,2].mean()],
                               std = [image[:,:,0].std(), image[:,:,1].std(), image[:,:,2].std()])(image_tensor)

print("normalized image shape", image_n.shape) #lets change to [H,W,C] and check stats
image_n = np.transpose(image_n.detach().numpy(), [1,2,0]); print("changing dims", image_n.shape)
stats(image_n)

</code></pre>
<p>I get following output</p>
<pre><code class="lang-auto">original shape (2136, 3216, 3)
original image stats
channel 1 min 0 max 63 mean 16.429443081875267 std 9.372180776313941
channel 2 min 0 max 202 mean 43.16988136913746 std 26.907299737201175
channel 3 min 0 max 255 mean 80.74282933413457 std 49.58311051825643

image tensor shape  torch.Size([3, 2136, 3216])


normalized image shape torch.Size([3, 2136, 3216])
changing dims (2136, 3216, 3)
channel 1 min -1.753001093864441 max -1.726640224456787 mean -1.7461274862289429 std 0.003921598196029663
channel 2 min -1.6043928861618042 max -1.5749527215957642 mean -1.5981013774871826 std 0.0039215791039168835
channel 3 min -1.6284340620040894 max -1.608265995979309 mean -1.622046947479248 std 0.003921573981642723
</code></pre>
<p>Here image is not normalized.<br>
What am I doing wrong ?</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/mayur_newase">@mayur_newase</a></p>
<p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor" rel="nofollow noopener">ToTensor</a> already “normalizes” the [0,255] uint input image into [0.0, 1.0] floats. Hence, the <code>mean</code> and <code>std</code> args expected by <code>transforms.Normalize</code> should be also scaled down to the  [0.0, 1.0] range.</p>
How to check if a number is in a tensor or not?

username: zcajiayin
cooked: <p>How to check if the number <code>1</code> is in <code>torch.tensor([1, 2, 3])</code>?</p>

username: LeviViana
cooked: <pre><code class="lang-python">import torch
a = torch.arange(3)
b = a + 3
print((a == 1).nonzero().numel() &gt; 0) # 1 is in a -&gt; true
print((b == 1).nonzero().numel() &gt; 0) # 1 is in b -&gt; false
</code></pre>

username: zcajiayin
cooked: <p>Thank you for your answer, I understand.</p>
What is the algorithm used for torch.nn.AdaptiveAvgPool2d?

username: Haimin_Hunter_Zhang
cooked: <p>I have read the documentation of torch.nn.AdaptiveAvgPool2d, and I understand how to use this function now. But I don’t get how this function works. Can help be provided to explain the algorithm of AdaptiveAvgPool2d?<br>
Much appreciated.</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/tom">@tom</a> explains it beautifully in <a href="https://discuss.pytorch.org/t/what-is-adaptiveavgpool2d/26897/2?u=ptrblck">this post</a>.</p>

username: crcrpar
cooked: <p>Could you tell me a paper which proposed adaptive poolings?</p>

username: ptrblck
cooked: <p>I’m not sure, if adaptive pooling was proposed in a scientific paper or if frameworks just implemented it to support variable sized inputs.</p>
<p>This is just speculation, but I think I came across adaptive pooling layers for the first time in the <a href="https://arxiv.org/pdf/1312.4400.pdf">Network in Network</a> paper. They use a global adaptive pooling and skip the linear layers at the end, argumenting that this might prevent overfitting to some degree.<br>
PyTorch’s adaptive pooling layers are more flexible as you can set the output size yourself, instead of pooling globally over the whole spatial size.</p>

username: crcrpar
cooked: <p>Thanks!</p>
<p>I couldn’t find any paper before and<br>
I didn’t want to miss a paper if exists.</p>

username: n0obcoder
cooked: <p>the idea of adaptive pooling is very similar to the novel idea proposed in <a href="https://arxiv.org/abs/1406.4729" rel="nofollow noopener">SPP Net</a> ( Spatial Pyramid Pooling )</p>
Pytorch Multiprocessing Queue error

username: drapado
cooked: <p>Hi, I’m building an application where I receive images from a socket connection, then I process them and return back the results in another socket connection. All three steps are separated in a multiprocessing independent process. Furthermore, since the dataflow will be constant they operate parallel in while loops. In order to allow communication between the processes I use queues. I pass the queues as arg to the processes, and in some processes I use queue.put() while in others queue.get() (normal producer-consumer behaviour).</p>
<p>I have an strange error, lets say the process that put the tensors from the socket into the queue is faster and puts 15 batches of images in the queue before the process with the pytorch model analyzes nothing. You would expect these 15 objects (cpu tensors) of the queue being different, however they are all the same which corresponds to the last objects. It’s like if the queue is filled in all their posititons with the last object it gets.</p>
<p>If I send the images from the socket very slowly (with break points for instance) then this does not happen and everything is computed properly.</p>
<p>Is this a know bug? Is there a limit size in terms of memory for the queues which I fill with my cpu tensors?</p>
Get features of pretrained model when forwarding

username: Toan_Bui
cooked: <p>I use resnet50 and change fc layer to classifying, like this:<br>
resnet50.fc = nn.Sequential(<br>
nn.Linear(fc_inputs, 256),<br>
nn.ReLU(),<br>
nn.Dropout(0.4),<br>
nn.Linear(256, num_classes),<br>
nn.LogSoftmax(dim=1)<br>
)<br>
I want to get both outputs and the features of nn.Linear(fc_inputs, 256) when forwarding, it looks like:<br>
features, outputs = resnet50(data)<br>
Thanks for all your supporting. (Sorry for my bad English)</p>

username: alex.veuthey
cooked: <p>I think it would be easier to <strong>not</strong> use the <code>Sequential</code> API in this case, but rather define each layer in the <code>.__init__()</code> method of your ResNet network, and call them in order in the <code>.forward()</code> method, this way you can have intermediate variables for <code>features</code> and <code>outputs</code>.</p>

username: Toan_Bui
cooked: <p>Thanks for your suggestion, it work for me.</p>
On which size tensor can you use "bicubic" upsampling? Should bias be used on upsample?

username: zacheberhart
cooked: <p>I’m playing around with various upsampling techniques and tried using <code>bicubic</code> but am getting the following error:</p>
<p><code>NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 4D) for the modes: nearest | linear | bilinear | trilinear (got bicubic)</code></p>
<p>When can/should <code>bicubic</code> be used?</p>
<p>And on a slightly unrelated note: is there a best practice for using bias on the upsampling convolution? I’ve seen a lot of models that don’t use bias but I’m not entirely sure why…</p>
<p>Thank you!</p>
<p>Code below:</p>
<pre><code class="lang-auto">def upconv(ni, nf, stride = 1):
    return nn.Conv2d(ni, nf, kernel_size = 3, stride = stride,
                     padding = 1, bias = True)

def upblock(ni, nf, mode, scale_factor = 2):
    return nn.Sequential(
        nn.Upsample(scale_factor = scale_factor, mode = mode),
        upconv(ni, nf),
        nn.BatchNorm2d(nf),
        nn.ReLU(inplace = True)
    )

self.decoder = nn.Sequential(
            
    upblock(1024, NDF * 8, mode = self.uptype),
    upblock(NDF * 8, NDF * 4, mode = self.uptype),
    upblock(NDF * 4, NDF * 2, mode = self.uptype),
            
    nn.Upsample(scale_factor = 2, mode = self.uptype),
    upconv(NDF * 2, NC),
    nn.Sigmoid()

)
</code></pre>

username: nareshr8
cooked: <p>I am also facing similar issue. Got any solution?</p>
Pytorch pretrained models and caffe pretrained models

username: zemen_tourist
cooked: <p>Why people did not use pytorch pretrained models (vgg, googlelenet) which are easy to use instead of building caffe library and use its pretrained models</p>

username: ptrblck
cooked: <p>Pretrained <code>torchvision</code> models are used in a lot of scripts.<br>
Where did you see Caffe models in PyTorch?</p>

username: zemen_tourist
cooked: <p>thank for answering my question. I see people using it in kaggle and some github repo which use caffe as feature extractor</p>
How to train layerwise in Pytorch?

username: Shisho_Sama
cooked: <p>Hello everyone, hope you are having a great day.<br>
How can I have layer wise training in Pytorch? I mean, suppose I have a network that trains like normal<br>
but parts of the network also gets optimized independently ? its something like this</p>
<pre><code class="lang-auto">|________|
|module1 |  
|________|
|module2 |
|________|
|module3 |
|________|
</code></pre>
<p>It goes like this I feed the model like normal :</p>
<pre><code class="lang-auto">for imgs, labels in dataloader:
    imgs = imgs.to(device)
    labels = labels.to(device)
    preds = model(imgs) 
    loss = criterion(preds, labels)
    ....
</code></pre>
<p>but what is not shown here is that each module can have its own criterion,<br>
each module receives a batch of images for example, runs the normal thing it does<br>
and it also trains until the loss reaches to a specific quantity then gets out of the loop for example and<br>
sends its outputs to the next layer, this goes on until the network predictions are made available which<br>
then the loss is calculated and the network weights is updated.</p>
<p>Thank you all very much in advance</p>