akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Control the training batch

username: AleTL
cooked: <p>Hello,</p>
<p>I’m working with an autoencoder and I want to visualize the output and the original image. I already did it but my problem is that everytime the images are plotted, are different images, so I cannot make a visual comparison of the improvement.</p>
<p>My code is next:</p>
<pre><code>    # training and so on
for batch_features, _ in train_loader:
    batch_features = batch_features.view(-1, 784)
    
    optimizer.zero_grad()
    outputs = model(batch_features)
    train_loss = criterion(outputs, batch_features)
    train_loss.backward()

    optimizer.step() #update the weights (net.parameters)
    loss += train_loss.item()

loss = loss / len(train_loader)
print("epoch : {}/{}, loss = {:.6f}".format(epoch + 1, epochs, loss))

#VISUALIZATION
reconstructed = outputs.view(-1,1,28,28)
original = batch_features.view(-1,1,28,28)
img = T.ToPILImage()(reconstructed[0]) # plot the first element of the last batch
img2 = T.ToPILImage()(original[0])


plt.subplot(121)
imgplot = plt.imshow(img)
plt.subplot(122)
imgplot2 = plt.imshow(img2)
plt.suptitle("Reconstructed vs Original")
plt.show()
</code></pre>
<p>Every time I plot the images, the [0] image is different, why? I think I don’t exactly understand the concept of batch. Because as far as I understand, the [0] I plot should always be the [0] image of the last plot, but it is not.<br>
How could I control the number of the batch that is inputing the NN? How could I plot everytime the same picture? For instance, the image [23] of the batch 30, or the image [540] of the whole training set?</p>
<p>Thank you so much!</p>

username: ptrblck
cooked: <p>I guess you might be shuffling the data in your training <code>DataLoader</code>, which would thus yield random samples in all batches.<br>
If you want to use the same image, you could directly get it by indexing the dataset via: <code>data, target = train_loader.dataset[index]</code>.</p>
DeepHist - Differentiable Histogram for Image-to-Image Translation - Pytorch implementation?

username: simonjaq
cooked: <p>Hello all</p>
<p>I stumbled over this paper:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.6/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/2005.03995" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://arxiv.org/abs/2005.03995" target="_blank" rel="noopener nofollow ugc">DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image...</a></h3>

<p>We present the DeepHist - a novel Deep Learning framework for augmenting a
network by histogram layers and demonstrate its strength by addressing
image-to-image translation problems. Specifically, given an input image and a
reference color...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Looks extremely interesting. I’m working on an image-to-image translation model and have been unsuccessfully trying to implement a histogram loss.</p>
<p>Now unfortunately I’m too stupid to translate the math in this paper into Pytorch code.</p>
<p>Maybe someone more intelligent here in the forum can?</p>

username: Ibrahim_Salihu_Yusuf
cooked: <p>Hi Simon! I have a simpler version of a differentiable that can be of help. We can collaborate on your work if you don’t mind</p>

username: simonjaq
cooked: <p>Yes. That would be very interesting. Would you share here? Or PM me.</p>
Is torch.nn.functional.avg_pool1d deterministic?

username: ooyamatakehisa
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/nn.functional.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/nn.functional.html" target="_blank" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>There is no note in the description of avg_pool1d on this page that says "may choose a non-deterministic algorithm, so this function is deterministic?</p>
<p>The function conv1d, which is similar to avg_pool1d, has its Note, so please tell me what avg_pool1d is doing.</p>
<p>Thank you!!</p>

username: ptrblck
cooked: <p>As described in the note for <code>F.conv1d</code> cudnn might non-deterministically pick a kernel or the kernel itself might yield non-deterministic results, which is not the case for the pooling layer.</p>
Tensor.backward() low-level computation

username: dav_n
cooked: <p>Hi everyone,<br>
I’m trying to understand what .backward() does when called, in terms of types of involved tensors (during computation). I am running the following code:</p>
<p>—Before training code:<br>
net.to(‘cuda’)<br>
net.half()      # 16 bit net</p>
<p>—Training code (first part):<br>
outputs = net(inputs)      # 16 bit<br>
loss = criterion(outputs, targets)      <span class="hashtag">#16</span> bit<br>
scaledloss = scalefactor * loss.float()      # 32 bit<br>
net.zero_grad()<br>
scaledloss.backward()      <span class="hashtag">#16</span> bit grads inside net (parameter.grad)</p>
<p>May I conclude that, if my network has fp16 parameters and its computed gradients are fp16, then backpropagation is computed in fp16?<br>
Furthermore, inspecting C++ code, I found out there is a library of functions with their own backward methods (<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp</a>): are these the ones called during .backward() computation?</p>

username: ptrblck
cooked: <p>Yes, the <code>dtype</code> would be recorded by Autograd and the backward computation would be executed in the same precision as its forward part, if you don’t change it.<br>
This would also mean that your FP16 gradient updates might easily underflow, which is why we recommend to use the mixed-precision training utility via <code>torch.cuda.amp</code>.</p>
Older PyTorch version on Azure Functions

username: Jishma
cooked: <p>Hi, I’m new to Pytorch and Azure functions. The project I’m currently working on requires an older version of Pytorch( version 1.0.1). The deployed function results in the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e266b79a6cf1e27fc1369e44b8ecd0b88150d7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e266b79a6cf1e27fc1369e44b8ecd0b88150d7f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e266b79a6cf1e27fc1369e44b8ecd0b88150d7f.png" alt="image" data-base62-sha1="4iIDzh3pXWDOkAqe2mnQCcSEZXx" width="690" height="192" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1059×296 26.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t get the above error when I use a higher version. Does this mean Azure functions does not support older Pytorch versions?</p>

username: ptrblck
cooked: <p>Based on the error it seems you are right and would need to stick to newer PyTorch releases.<br>
Generally I would also recommend to use the latest PyTorch version to get bug fixes and new utilities.</p>
Converting a VGG model into sequential, but getting different outputs

username: sunfishcc
cooked: <p>Background:<br>
I’m working on an adversarial detector method which requires to access the outputs from each hidden layer (Local Intrinsic Dimensionality Detector - a.k.a.: LID).<br>
I loaded a pretrained VGG16 from <code>torchvision.models</code>.</p>
<p>To access the output from each hidden layer, I put it into a sequential model:</p>
<pre><code class="lang-auto">vgg16 = models.vgg16(pretrained=True)

vgg16_seq = nn.Sequential(*(
    list(list(vgg16.children())[0]) + 
    [nn.AdaptiveAvgPool2d((7, 7)), nn.Flatten()] + 
    list(list(vgg16.children())[2])))
</code></pre>
<p>I looked into the <em>torchvision</em> <a href="https://github.com/pytorch/vision/blob/d3d393672b877f80fedd2d11de6b84fb19599c2e/torchvision/models/vgg.py" rel="noopener nofollow ugc">VGG</a> implementation, it uses the <code>[feature..., AvgPool, flatten, classifier...]</code> structure.<br>
Since <code>AdaptiveAvgPool2d</code> layer and <code>Flatten</code> layer have <strong>no parameters</strong>, I assume this should work, but I have different outputs.</p>
<pre><code class="lang-auto">output1 = vgg16(X_small)
print(output1.size())
output2 = vgg16_seq(X_small)
print(output2.size())
torch.equal(output1, output2)
</code></pre>
<p>They are in the same dimension but different outputs.</p>
<blockquote>
<p>torch.Size([32, 1000])<br>
torch.Size([32, 1000])<br>
False</p>
</blockquote>
<p>I tested the outputs right after the <code>AdaptiveAvgPool2d </code> layer, the outputs are equal:</p>
<pre><code class="lang-auto">output1 = nn.Sequential(*list(vgg16.children())[:2])(X_small)
print(output1.size())
output2 = nn.Sequential(*list(vgg16_seq)[:32])(X_small)
print(output2.size())
torch.equal(output1, output2)
</code></pre>
<blockquote>
<p>torch.Size([32, 512, 7, 7])<br>
torch.Size([32, 512, 7, 7])<br>
True</p>
</blockquote>
<p>Can someone point out what went wrong?<br>
Thank you</p>

username: sunfishcc
cooked: <p>By adding <code>.eval()</code>, both models get the same result.</p>
<pre><code class="lang-auto">vgg16.eval()
output1 = vgg16(X_small)
print(output1.size())

vgg16_seq.eval()
output2 = vgg16_seq(X_small)
print(output2.size())
torch.equal(output1, output2)
</code></pre>
<p>But I’m not sure why.</p>

username: ptrblck
cooked: <p>VGG16 uses dropout layers, which will be disabled when <code>model.eval()</code> is used and which explains the difference during the <code>model.train()</code> comparison.</p>
Is it possible to rename Event Log scalar

username: teixeirajpf
cooked: <p>Hi,<br>
I’ve been fairly busy running and updating several pieces of code as I learn pytorch.<br>
After a bunch of experiments that took a fair amount of time I noticed that some of my logged scalars on tensorboard had different names/paths. I’d like to set them under the same name/path, offline, instead of re-running all experiments just to get new logs with the issue fixed.</p>
<p>Is there any possibility of doing this in tensorboard or tensorboardX ? I can’t seem to find much information online about this, nor the documentation seems to have any functions for renaming stuff inside already finished logs.</p>
<p>Thanks in advance.</p>

username: teixeirajpf
cooked: <p>Well after a while I finaly found something that proved useful:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/60079644/how-do-you-edit-an-existing-tensorboard-training-loss-summary" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/12846524/alexp" target="_blank" rel="noopener nofollow ugc">
    <img alt="AlexP" src="https://lh4.googleusercontent.com/-d5DtBySzA9A/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3reJ4DvpOj5mTfAKjjKxbyD-nmUkOw/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/60079644/how-do-you-edit-an-existing-tensorboard-training-loss-summary" target="_blank" rel="noopener nofollow ugc">How do you edit an existing Tensorboard Training Loss summary?</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, machine-learning, edit</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/12846524/alexp" target="_blank" rel="noopener nofollow ugc">
    AlexP
  </a>
  on <a href="https://stackoverflow.com/questions/60079644/how-do-you-edit-an-existing-tensorboard-training-loss-summary" target="_blank" rel="noopener nofollow ugc">04:01PM - 05 Feb 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The accepted answer managed to work for my case. Although I  have to keep calling the main function and go about making a new output event file per scalar to change. I could probably change it to work for multiple pairs of oldnames/newnames but as it is works well enough and is made to replace multiple old names to a single new name.</p>
How to shuffle sentence based on ids?

username: 111429
cooked: <p>lets’ say I have a tensor with content:<br>
[0,55,49,38,2,0,96,28,2,0,73,2]<br>
0 is begin of sentence token<br>
2 is end of sentence token<br>
I want to do random sentence permutation as like<br>
[0,73,2,0,55,49,38,2,0,96,28,2]<br>
or<br>
[0,73,2,0,96,28,2,0,55,49,38,2]<br>
how can I do it?<br>
Thanks.</p>

username: ptrblck
cooked: <p>This code might work:</p>
<pre><code class="lang-python">x = torch.tensor([0,55,49,38,2,0,96,28,2,0,73,2])
idx = torch.cat((torch.tensor([0]), torch.randperm(len(x)-2)+1, torch.tensor([len(x)-1])))
print(idx)
out = x[idx]
print(out)
</code></pre>
Help Explaining Latency of Multi-threaded Implementation

username: its_sandy
cooked: <p>Hello all <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"> .<br>
I have written the below C++ extension (CPU) to fill nodes into fixed-size buckets in a FIFO manner. Each node goes into a single bucket in each of the L tables. I parallelize over L to avoid collisions.</p>
<p>Dimensions Used:- num_nodes=524288, num_buckets=4096, bucket_size=128, num_threads=48<br>
(I use a system with about 1000G RAM, supporting up to 96 parallel threads.)</p>
<p>When using values of L &lt; num_threads, I expect latency to be relatively constant w.r.t L. However, I find that latency varies significantly when I change L. What could be the reason for this? Does it have to do with caching? Thanks in advance.</p>
<p>L = 12 =&gt; 5.3450 ms<br>
L = 24 =&gt; 12.8900 ms<br>
L = 36 =&gt; 22.2119 ms<br>
L = 48 =&gt; 29.6163 ms</p>
<pre><code class="lang-auto">void fill_buckets_FIFO(
        const torch::Tensor&amp; indices,
        torch::Tensor&amp; buckets,
        torch::Tensor&amp; bucket_counts) {
    
    int32_t num_nodes = indices.size(0);
    int32_t L = buckets.size(0);
    // int32_t num_buckets = buckets.size(1);
    int32_t bucket_size = buckets.size(2);

    auto buckets_0 = buckets.accessor&lt;int32_t, 3&gt;(); // L x num_buckets x bucket_size
    auto bucket_counts_0 = bucket_counts.accessor&lt;int32_t, 2&gt;(); // L x num_buckets
    auto indices_0 = indices.accessor&lt;int32_t, 2&gt;(); // num_nodes x L

    at::parallel_for(0, L, 0, [&amp;](int32_t start, int32_t end) {
        for (int32_t l = start; l &lt; end; l++) {
            auto buckets_1 = buckets_0[l];
            auto bucket_counts_1 = bucket_counts_0[l];

            for(int32_t i = 0; i &lt; num_nodes; i++) {
                int32_t bucket_index = indices_0[i][l];
                int32_t &amp;bucket_count = bucket_counts_1[bucket_index];
                buckets_1[bucket_index][bucket_count % bucket_size] = i;
                bucket_count++;
            }
        }
    });
}
</code></pre>
Problem with torch.optim.lr_scheduler.StepLR

username: gn_torch
cooked: <p>Hi, I am trying to use the StepLR scheduler</p>
<p>I have defined it here<br>
‘optimizer = optim.SGD(model.parameters(), lr=0.03, momentum=0.9, weight_decay=0.0005, nesterov=True)’</p>
<p>‘scheduler1 = optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.55)’</p>
<p>and calling it to step in my training loop<br>
‘scheduler1.step()’</p>
<p>which results in the error “Attribute error: tuple object has no attribute step”<br>
Can anyone help with this?</p>

username: ptrblck
cooked: <p>Could you check, if you have a trailing comma at the end of the <code>scheduler1 = ...</code> definition?<br>
If not, could you post an executable code snippet to reproduce this issue?</p>
What does a 2d tensor target mean when my target label is either 1 or 0?

username: grizzlycoder
cooked: <p>In my model, I take two tracks (track1, track2) and I have an interaction label (1 or 0). In the training epoch, when I print my target I get something like torch.tensor([1,0]) or torch.tensor([0,0]) etc.</p>
<p>How does it convert my single number into a 2d vector of different values? I’d imagine label 1 becomes [1,1] and label 0 becomes [0,0]. I don’t see where it would get [0,1] or [1,0] from.</p>

username: ptrblck
cooked: <p>I’m not sure how to understand the question as you are defining the model architecture and are responsible for the interpretation of the model output. Based on your description I guess you are using two output neurons, apply a <code>sigmoid</code> on them, and are dealing with a multi-label classification use case?<br>
If that’s the case, your model would try to output a high value (close to 1), if the class is active, and a 0 otherwise.</p>
Dataloader return wrong batch shape

username: trungdang97
cooked: <p>I’m having trouble while fine-tuning pretrained monolingual Roberta model.</p>
<p>This is my custom dataset:</p>
<pre><code class="lang-auto">class VnParaDataset(Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __len__(self):
        return len(self.encodings)

    def __getitem__(self, index):
        item = {key: torch.tensor(val[index]) for key, val in self.encodings.items()}
        return item
</code></pre>
<p>‘input_ids’,‘token_type_ids’,‘attention_mask’ of encodings has shape of [5440,193].</p>
<p>My Dataloader is:</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
</code></pre>
<p>However, when I loop through batches, my batch have shape of [3,193] and len(train_loader)=1:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    count = 0
    for batch in train_loader:
        print('Epoch {0} - Batch {1}'.format(epoch, count))
        optimizer.zero_grad()
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        outputs = phobert(input_ids, attention_mask=attention_mask, token_type_ids=None)
        loss = outputs[0]
        loss.mean().backward()
        optimizer.step()
        count += 1
</code></pre>
<p>Shouldn’t the shape of one batch be [32,193] and there are 5440/32=170 batches in train_loader?</p>

username: ptrblck
cooked: <p>The length of a <code>dict</code> would return the number of keys, so you might want to use:</p>
<pre><code class="lang-python">    def __len__(self):
        return len(self.encodings['input_ids'])
</code></pre>
<p>instead.</p>
Delete row at index

username: Praveen_Tirupattur
cooked: <p>I have a 4-dimensional tensor and I want to select all the elements from that tensor except for one row along a dimension. How can I do that?<br>
Here is the code snippet I have</p>
<p>features = Variable(torch.rand(2, 128, 19, 128))<br>
selected = features[:, :, ~2, :] # I want to select all rows except the 2nd row along 3rd dimension.</p>

username: nicofish
cooked: <p>You could use a double for loop and iterate through the tensor removing the second rows. Im sure there are better ways to do this  but seeing how your tensor isn’t that big it should take that long.</p>
Is there a Mask-Keypoint R-CNN available?

username: Flicic_Suo
cooked: <p>In torchvision we got keypoint R-CNN to perform object detection as well as keypoint detection; and we got mask R-CNN to perform object detection as well as instance segmentation. Say, if for an detected object, I want to perform both keypoint detection and instance segmentation, is there a “mask-keypoint R-CNN” for this?</p>

username: Flicic_Suo
cooked: <p>Wrote one myself <a href="https://github.com/feiyangsuo/MaskKeypointRCNN-pytorch" rel="noopener nofollow ugc">here</a>. Yet may not be very stable.</p>
Can't find momentum buffer for Linear layer

username: J3ssi
cooked: <p>Hi,<br>
I have a problem with my code. The error first emerged after upgrading to 1.7.0</p>
<p>I used</p>
<p>for name, param in model.named_parameters():<br>
mom_param = optimizer.state[param][‘momentum_buffer’]</p>
<p>before to get the momentum buffer. As Iread this is no longer working in 1.7.0</p>
<p>For all except one layer (Linear) I can use:</p>
<p>k = 0<br>
for name, param in model.named_parameters():<br>
mom_param = optimizer.state_dict()[k][param][‘momentum_buffer’]</p>
<p>What can I do to get the momentum buffer of my Linear layer?</p>
<p>I seachred hot to make the code inside one of this fancy code snips. But I haven’t found out how to do it.</p>
<p>Thanks for helping.<br>
Jessi</p>

username: ptrblck
cooked: <p>The first code snippet seems still to work in <code>1.8.0.dev20201116</code>:</p>
<pre><code class="lang-python">model = models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)
model(torch.randn(1, 3, 224, 224)).mean().backward()
optimizer.step()

for name, param in model.named_parameters():
    mom_param = optimizer.state[param]['momentum_buffer']
    print(mom_param)
</code></pre>
<p>Could you share a code snippet, where this would be failing?</p>
"AssertionError: Torch not compiled with CUDA enabled"

username: ihsan
cooked: <p>I get an error “AssertionError: Torch not compiled with CUDA enabled”. I’m using GTX950 gpu. CUDA was available 1 week ago, but not anymore. And I can’t solve this problem.  I could not find the solution to the problem in any title related to this subject. Please help. Thank you in advance</p>

username: ptrblck
cooked: <p>Check if you’ve updated any drivers, if the GPU is still recognized by <code>nvidia-smi</code>, or if you’ve accidentally installed the CPU version of PyTorch.</p>

username: ihsan
cooked: <p>Hi, Since I am using Win10, I did nvcc - version control and the result is as follows. Also the pytorch version isn’t the cpu version.</p>
<p>nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2020 NVIDIA Corporation<br>
Built on Wed_Jul_22_19:09:35_Pacific_Daylight_Time_2020<br>
Cuda compilation tools, release 11.0, V11.0.221<br>
Build cuda_11.0_bu.relgpu_drvr445TC445_37.28845127_0</p>

username: ptrblck
cooked: <p>Are you able to compile and/or run any other CUDA applications?</p>

username: ihsan
cooked: <p>Sorry, I might have misunderstood. Saying “CUDA application” are you talking about a .cu file or any nvcc command usage? By the way i can access GPU and use it, using “numba” library.</p>
Pytorch Forward Hook

username: Juz_Z
cooked: <p>I am trying to extract activation vectors from some spotlighted layers using hook.<br>
Referring to the usage from the forum, I applied this hook function to my resnet50 model (trained with dataparallel, multi-GPUs).</p>
<p>While I am sending batched test images to the network and expecting to get batched feature vectors, I am surprised that the size of the activations is not the same as my batch size. I use 64 images and the output logits form the network is also 64. But the activation tensor is only 32.</p>
<p>I don’t know what made this failure. Please help me out.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe5621b05a6a4fa0c8321ce936100140467079b.png" data-download-href="https://discuss.pytorch.org/uploads/default/9fe5621b05a6a4fa0c8321ce936100140467079b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fe5621b05a6a4fa0c8321ce936100140467079b_2_430x500.png" alt="image" data-base62-sha1="mOvmSNruwV4aSs5GQ691m13Z1mr" width="430" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fe5621b05a6a4fa0c8321ce936100140467079b_2_430x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe5621b05a6a4fa0c8321ce936100140467079b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe5621b05a6a4fa0c8321ce936100140467079b.png 2x" data-dominant-color="EFEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">617×716 68.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Diego
cooked: <p>You are using <code>nn.DataParallel</code> so the batch is split evenly across all GPUs so the initial 64 batch size is split between both gpus into 32 and 32. I think that this might be the problem in your case.</p>

username: Juz_Z
cooked: <p>Exactly. After observing the outputs, I realized that I used two gpus in parallel. Thank you so much for quick reply.</p>
Multilevel inheritance for Mask R-CNN-Calling the base class (GeneralizedRCNN)

username: sigma_x
cooked: <p>Pytorch implementation of Mask R-CNN is multilevel inheritance:</p>
<pre><code class="lang-auto">class GeneralizedRCNN(nn.Module):
     def __init__(self, sm inputs):
           self.sm_inputs=sm_inputs
     def forward(self, x):
           sm_outputs=self.sm_inputs(x)
           return sm_outputs

class FasterRCNN(GeneralizedRCNN):
     def __init__(self, sm_cool_inputs):
           super(FasterRCNN, self).__init__(sm_inputs=sm_cool_inputs)

class MaskRCNN(FasterRCNN):
     def __init__(self, sm_awsm_inputs):
           super(MaskRCNN, self).__init__(sm_cool_inputs=sm_awsm_inputs)
           self.sm_module.sm_more = sm_awsm_inputs.sm_awsm_input
</code></pre>
<p>Finally, you write a script where you call</p>
<pre><code class="lang-auto">mymodel=MaskRCNN(**awsm_kwargs)
mymodel(sm_awsm_img)
</code></pre>
<p>So how does one actually get to run the <code>forward</code> method in the base class? I think it is some ‘magic method’ like <code>__getitem__</code> for dataset interfaces. Is this correct?</p>
Model parallelism for branching model

username: jtencer
cooked: <p>I have a branching model in which several submodules may be executed independently in parallel.  Below is the simplest example I could come up with:</p>
<pre><code class="lang-auto">class myModule(nn.Module):
    def __init__(self, size_x, size_y=5):
        super(myModule, self).__init__()

        self.f0 = nn.Linear(size_x, size_y)
        self.f1 = nn.Linear(size_x, size_y)

        self.g  = nn.Linear(2*size_y, size_y)

        self.h0 = nn.Linear(2*size_y, size_x)
        self.h1 = nn.Linear(2*size_y, size_x)

    def forward(self, x):
        y0 = self.f0(x[:,0,:])
        y1 = self.f1(x[:,1,:])
        y = torch.cat((y0,y1),dim=1)

        yg = self.g(y)

        out = torch.zeros(x.shape)
        out[:,0,:] = self.h0(torch.cat((y0,yg),dim=1))
        out[:,1,:] = self.h1(torch.cat((y1,yg),dim=1))
        return x
</code></pre>
<p>The functions f0 and f1 act independently on separate channels of the input.  Function g combines the outputs of f0 and f1 into a single vector fg which is then used by the h functions (along with the output of the corresponding f function) to attempt to recreate the input.  I’ve just used linear layers as submodules in the example, but in my application, these are more complicated, are large in their own right, and there are more than 2 of them.</p>
<p>My goal is to execute f0 and f1 in parallel as well as h0 and h1 in parallel.  I’m OK with blocking while g executes.  I’m particularly interested in how to make sure that the backward pass executes in parallel.</p>

username: pritamdamania87
cooked: <p>You can use <a href="https://pytorch.org/docs/stable/generated/torch.jit.fork.html" rel="noopener nofollow ugc">torch.jit.fork</a> for this, however note that your forward pass needs to run in TorchScript for this to actually work asynchronously.</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="jtencer" data-post="1" data-topic="105064">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jtencer/48/31917_2.png" class="avatar"> jtencer:</div>
<blockquote>
<p>I’m particularly interested in how to make sure that the backward pass executes in parallel.</p>
</blockquote>
</aside>
<p>If the two branches are on different GPU devices, then the backward pass would run then in parallel on the two different GPUs. If everything is on CPU, currently the backward pass only has single threaded execution and everything would run on a single thread on the CPU.</p>

username: jtencer
cooked: <p>Thanks!  This was very helpful.</p>
Image classification using GAN

username: khadidja
cooked: <p>Hi everyone, I want to make a classification of images using PyTorch, knowing that the database contains 3 regions (ROI A, ROI B, ROI C) of the same images, I proposed to apply the gan of multiple generators (generator for each region). You can help me with the programming part, please.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="khadidja" data-post="1" data-topic="105239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/khadidja/48/31978_2.png" class="avatar"> khadidja:</div>
<blockquote>
<p>You can help me with the programming part, please.</p>
</blockquote>
</aside>
<p>I’m sure we can help you when you explain what you’ve already implemented and where you are stuck. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: khadidja
cooked: <p>thank you <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
my problem is in the programming part (I am a beginner in PyTorch), I make a classification of images knowing that I have two classes. I tried to make the classification with multi generator GAN.</p>
Distributed datasets on multi-machines

username: sakh251
cooked: <p>Hey,</p>
<p>I have big dataset, and it is stored on different machines. I am trying to run DistributedDataParallel to train models on it. So each machine has its own data to train. In each machine I have 3 GPUs, one process for each. I am trying to use DistributedSampler to distribute batches between GPUs in each machine. So I just set world_size to number of GPUs in each machine and also set rank in each process.<br>
For 2 machines, my real world size is 6 and rank from, 0-6. But for DistributedSampler, I set 3 and rank from 0-3.<br>
I am not sure if I am doing the correct way.<br>
Anyway by doing this I got error on this assertion:</p>
<pre><code class="lang-auto">assert len(indices) == self.num_samples
</code></pre>
<p>Thank you</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="sakh251" data-post="1" data-topic="105113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/bbce88/48.png" class="avatar"> sakh251:</div>
<blockquote>
<p>But for DistributedSampler, I set 3 and rank from 0-3.</p>
</blockquote>
</aside>
<p>I think this might be the issue, for <code>num_replicas=3</code>, the ranks should be from 0-2 and not 0-3. I was able to reproduce the error you’re seeing if I do something like this (which is invalid):</p>
<pre><code class="lang-auto">sampler = DistributedSampler(dataset, num_replicas=3, rank=3)
</code></pre>

username: sakh251
cooked: <p>It was correct, I converted global rank to local rank incorrectly. I just send number replica and local rank and it works.</p>
<p>Thank you</p>
Exporting the operator max_unpool2d to ONNX opset version 11 is not supported.  Please let me know if this issue has got fixed ? Is there any work around solution available for this issue.

username: tlk2abhishek
cooked: <p>Exporting the operator max_unpool2d to ONNX opset version 11 is not supported.</p>
<p>Please let me know if this issue has got fixed ?<br>
Is there any work around solution available for this issue.</p>
Is pytorch mobile successor of caffe2go

username: alcaster
cooked: <p>I just read paper(<a href="https://arxiv.org/pdf/2008.12298v2.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2008.12298v2.pdf</a>) in which facebook talks about running their model on devices. It is very recent from september this year but I see that they are still using internal tool “caffe2go” which was never open sourced.<br>
The only talk about doing this and about this framework in this blogpost <a href="https://engineering.fb.com/2016/11/08/android/delivering-real-time-ai-in-the-palm-of-your-hand/" rel="noopener nofollow ugc">https://engineering.fb.com/2016/11/08/android/delivering-real-time-ai-in-the-palm-of-your-hand/</a> . As we have now pytorch mobile I assumed caffe2go is deprecated but facebook is still developing new products with it.</p>
<ol>
<li>Is pytorch mobile not stable enough to use on production and that’s why facebook still uses caffe2go ?</li>
<li>Is pytorch mobile borrows concepts or shares some implementation with caffe2go ?</li>
</ol>
Time-series lookup in loss function

username: superunification
cooked: <p>So I’ve got an optimization problem where the goal is to predict a time series envelope; it’s the estimated fundamental pitch of a signal, at a series of equally spaced time steps.</p>
<p>Now the trouble starts. My loss function is as follows - roughly speaking it iterates through the predicted periods from the pitch envelope, testing how well it minimizes the difference successive audio segments of period-length (if the successive segments cancel each other out, the period must be correct) – all differences are accumulated to give an overall error value of the fitness of that pitch envelope to the audio signal.</p>
<pre><code class="lang-auto">def get_fold_alignment_loss(period_envelope: torch.Tensor, audio: torch.Tensor) \
        -&gt; torch.Tensor:
    cume_offset = torch.zeros(1, device=period_envelope.device)
    error = torch.zeros(1, device=period_envelope.device)

    while cume_offset &lt; audio.size(0):
        prog_fraction = cume_offset / audio.size(0)
        current_idx = int(prog_fraction * period_envelope.size(0))

        prev_val = period_envelope[current_idx]
        next_val = period_envelope[current_idx + 1]
        prev_prog = current_idx / period_envelope.size(0)
        next_prog = (current_idx + 1) / period_envelope.size(0)
        prev_period = torch.pow(2, prev_val)
        next_period = torch.pow(2, next_val)
        prog_to_next = (prog_fraction - prev_prog) / (next_prog - prev_prog)
        period = prog_to_next * next_period + (1 - prog_to_next) * prev_period

        prev_pos = max(0, round((cume_offset - period).item()))
        pos = round(cume_offset.item())
        next_pos = min(audio.size(0), round((cume_offset + period).item()))

        if next_pos - pos == 0:
            break
        cume_offset += period
        min_size = min(pos - prev_pos, next_pos - pos)
        error += torch.linalg.norm(audio[pos - min_size:pos] - audio[pos + min_size])
    return error</code></pre>
<p>Now this isn’t differentiable as I have needed to truncate continuous values to do look-ups into the learned time-series, and to calculate error quantities requires indexing into ranges of the signal.</p>
<p>In principle, the time series (audio and pitch envelope) could be represented by some high-degree polynomial or sinc basis functions… so that they could be ‘indexed’ by continuous values, but I have to wonder if there is a better way. Can I estimate the gradients with small random perturbations of the pitch envelope?</p>
<p>Would appreciate any help pointing me in the right direction.</p>

username: superunification
cooked: <p>I suppose just a hint as to whether this is appropriate; it’s my first try diving into the autograd stuff:</p>
<p>If providing my own gradients, I need to calculate d_loss / d_params, correct? So for each element in my learnable time series, what is the ratio of loss delta versus the adjustment delta.</p>
<p>What amount of smoothness is required to be smooth enough for Adam/SGD to function? This would, I believe, directly map to how many iterations of gradient sampling are required.</p>
Cuda initalization time on a multi-GPU machine for inference

username: Martin_Castin
cooked: <p>I am trying to parallelize the initialization of cuda on multiple GPUs. Indeed, I am trying to make my Pytorch model’s initialization faster and I have noticed that this initialization takes about 3s per GPU on a P6000.<br>
A simple code like the following one</p>
<pre><code class="lang-auto">pool = mp.get_context('spawn').Pool(torch.cuda.device_count())
pool.map(torch.Tensor([0]).cuda, range(torch.cuda.device_count()))
</code></pre>
<p>does not seem to show any performance improvement compared to</p>
<pre><code class="lang-auto">for gpu in range(torch.cuda.device_count()):
    torch.Tensor([0]).cuda(device=gpu)
</code></pre>
<p>It looks like these operations cannot be parallelized. Am I doing something wrong? Is there anyway to make this faster?</p>
DQN Pytorch not working

username: Chris_Freiburg
cooked: <p>Hello I am new here<br>
my name is Chris I study CS.<br>
I am implementing the DQN algorithm in Pytorch<br>
and I made an simple example.<br>
It has 2 States and 4 actions.</p>
<p>For state 1 the action 1 gets a reward of plus 5 and state 2 the action 3 gets -5<br>
after training for a wile the q function for bothstate are almost equal.</p>
<p>I think for the state 1 the q value for action1 should be close to 5</p>
<p>and for state 2 the q value action 3 should be close to -5</p>
<p>Does anyone now what could I do wrong</p>
<p>Thank you for your help</p>

username: Pdieter
cooked: <p>Not only the rewards but also the transition probabilities determine your q-values. So please check your transition probabilities.</p>
<p>If you only have 2 states and 4 actions you can also compute your q-function numerically with e.g. value iteration. You can then compare the numerically computed results with the ones of your Pytorch model.</p>
Log_softmax function returns "nan" in raspbian OS running torch wheels, but works fine in anaconda (windows OS)

username: Lahiru_Welagedara
cooked: <p>Hello Guys</p>
<p>I have been working on a pytorch model for one of my testing purposes. So far I trained the model using anaconda in windows and the model is working fine. I wanted to train the same model in raspberry pi 4 (aarch64). So I did the following,</p>
<ol>
<li>Deployed RaspbianLite OS</li>
<li>Used torch wheels to install torch in Raspberry PI as it has ARM architecture ( got the wheels for torch(v1.7) and torchvision(v.0.8.1) from - <a href="https://mathinf.eu/pytorch/arm64/" rel="noopener nofollow ugc">https://mathinf.eu/pytorch/arm64/</a> )</li>
</ol>
<p>Torch was imported successfully and I ran the model in raspberry pi without errors.<br>
But the loss got a value of nan. So I debugged and found out that there was no issue with the data or data transformation (no nan inputs). But found out that F.log_softmax(r_out2, dim=1) returns a nan value from the beginning of first batch of data itself.</p>
<p>Anaconda environment versions -<br>
numpy.<strong>version</strong> - 1.18.5<br>
torch.<strong>version</strong>     - 1.7.0<br>
torchvision.<strong>version</strong> - 0.8.1</p>
<p>Raspberry Pi versions -<br>
torch - v1.7<br>
torchvision - v.0.8.1<br>
numpy - v1.18.5 (tried upgrading to latest version as well but no luck)</p>
<p>I have no clue why the same model and same data returns nan, when it runs perfectly well in the anaconda environment. The only issue is with the log_softmax  as of yet.</p>
<p>Does it have something to do with the torch wheels I installed ?<br>
Or does it have to do something with the processing power on raspberry ?</p>
<p>I am very glad if someone can help me with this issue</p>
<p>Thanks !</p>

username: Lahiru_Welagedara
cooked: <p>Solution Found -<br>
The issue was with the torch version 1.7 in <a href="https://mathinf.eu/pytorch/arm64/" rel="noopener nofollow ugc">https://mathinf.eu/pytorch/arm64/</a>.<br>
The model worked fine for torch version 1.6 available in <a href="https://mathinf.eu/pytorch/arm64/" rel="noopener nofollow ugc">https://mathinf.eu/pytorch/arm64/</a>.</p>
Making a custom dataset like MNIST

username: qq_kim
cooked: <p>Hello! I’m struggling with making custom datasets like MNIST.<br>
The dimension of MNIST datasets is like torch.size [6000,2,32,32],<br>
and 2 means that it includes the image as well as the label.</p>
<p>I have images with label on its image file name but I don’t know how to include image file name(label)<br>
into one tensor, my tensor dimension is   [6000,1,32,32], and it means the tensor has only image data not<br>
the label.</p>
<p>how to make label within image tensor?</p>
Net.load_state_dict does not work

username: Jingnan_Jia
cooked: <p>I saved a trained net with an unused layer instance in self function:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)
    self.layer2 = nn.Conv2d(32, 64, 3)
    self.layer_not_used = nn.Conv2d(64, 2, 3)  # not used in forward
  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    return out
</code></pre>
<p>After several days I want to load the net from disk. But in my new code I have already removed the unused layers in self function. The net is:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)
    self.layer2 = nn.Conv2d(32, 64, 3)
  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    return out
</code></pre>
<p>And I found that I could not load the trained net successfully.</p>
<p>How to load my trained net successfully when the trained net has some unused layers?</p>

username: dav_n
cooked: <p>A way to fix your issue might be hidden here: <a href="https://pytorch.org/tutorials/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.html</a></p>
<p>Maybe it’s enough to do something like:<br>
net = MyNet()<br>
net.load_state_dict(torch.load(PATH), strict=False)</p>
Appending augmented data to original training set

username: karthikramesh55
cooked: <p>Hello,</p>
<p>I am new to PyTorch and currently looking to perform data augmentation on the FashionMNIST data, through flip and crop transformations. After that, would like to add these transformed data tensors to the training set to get an expanded dataset. The transforms I am using are as follows:</p>
<p>aug_transform = transforms.Compose([transforms.RandomVerticalFlip(p = 0.5),<br>
transforms.RandomResizedCrop((28, 28),(0.9, 1.0)),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5,), (0.5,)),<br>
])</p>
<p>Are there any ways that you would suggest adding the augmented tensors to the training set? Also, is it possible to add the transformation on the existing training data on the fly  (dynamically) while training the model?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="karthikramesh55" data-post="1" data-topic="105213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/7ba0ec/48.png" class="avatar"> karthikramesh55:</div>
<blockquote>
<p>Are there any ways that you would suggest adding the augmented tensors to the training set?</p>
</blockquote>
</aside>
<p>You could create two separate datasets, the standard one and the other one with transformations, and combine them into a <code>ConcatDataset</code>.</p>
<aside class="quote no-group" data-username="karthikramesh55" data-post="1" data-topic="105213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/7ba0ec/48.png" class="avatar"> karthikramesh55:</div>
<blockquote>
<p>Also, is it possible to add the transformation on the existing training data on the fly (dynamically) while training the model?</p>
</blockquote>
</aside>
<p>The transformations will be applied on-the-fly by default on each sample in the <code>__getitem__</code> method of the dataset.</p>

username: karthikramesh55
cooked: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>. It works like a charm.</p>
Concat weights from two pre-trained models

username: simongeek
cooked: <p>Hi.</p>
<p>I would like to concat a high-level feature from two own pre-trained U-Net models.</p>
<p>Let:</p>
<pre><code class="lang-auto">model1 = torch.load("UNetmodel1.pt")
model2 = torch.load("UNetmodel2.pt")
</code></pre>
<p>Model1 is pre-trained on heart ultrasound dataset and model2 is pre-trained on abdomen ultrasound dataset. Now, I would like to extract high-level features from two pre-trained segmentation network U-Net. Then, I would like to classify these features by AlexNet or other state-of-the-art classification networks. On the output, I need two classes: the heart and abdomen.<br>
How to do it?</p>

username: ptrblck
cooked: <p>Based on your description of the use case I don’t think you would like to concatenate the parameters of the models, but are looking for something like a <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">model ensemble</a>.<br>
In the linked code snippet you could replace the two submodules with your pretrained UNets and add <code>Alexnet</code> as the <code>classifier</code>.</p>

username: simongeek
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you very much. I made it yesterday, based on the above example.<br>
And a little question: If I will make a classification task, based on 2-3 pre-trained segmentation network, should I train again a whole model from scratch?</p>

username: ptrblck
cooked: <p>I think you should at least compare retraining the submodules and see, if it would improve your use case.</p>
Adapting cnn to image regression

username: rui_zhou
cooked: <p>Hello, is there any guide for adapting cnn to regression?<br>
I have images and csv labels, there are demos in keras, can I do it in pytorch, the following are the adaption codes in keras, how should I do the same work in pytorch ?</p>
<pre><code class="lang-auto">from keras.applications.xception import Xception
from keras.models import Model

model = Xception(weights='imagenet', include_top=True, input_shape=(299,299, 3))

x = model.get_layer(index=len(model.layers)-2).output

print(x)
x = Dense(1)(x)

model = Model(inputs=model.input, outputs=x)
model.summary()
</code></pre>
<pre><code class="lang-auto">opt = RMSprop(lr=0.0001)
model.compile(loss='mean_squared_error', optimizer=opt, metrics=['mae'])
</code></pre>

username: ptrblck
cooked: <p>You could replace the last linear layer (often called <code>model.classifier</code>) with a new <code>nn.Linear</code> layer with a single output neuron and use e.g. <code>nn.MSELoss</code> as the criterion.</p>