akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CPU/GPU usage rate sometimes drops to 0 in 32G V100

username: Philokey
cooked: <p>I encountered a strange problem when using 32G V100, and it’s not happened in titan xp or 16G V100. When running the follow code, the useage of CPU sometimes will drops to 0. In the mean time, the time cost of aten::addmm will increase to more than 100ms. However, it’s less than 1s under normal circumstances. You can get trace.json generated by profiler <a href="https://drive.google.com/file/d/1m7Lh2dIpHUsb8WNLVHwOgLpY32ZKwo1H/view?usp=sharing" rel="noopener nofollow ugc">here</a></p>
<p>I am using pytorch 1.7.1+cu101, and no other programs are running on this machine when I test the follow code. I wonder why this phenomenon occurs.</p>
<p>Thank you very much.</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn as nn
import torch.autograd.profiler as profiler

torch.set_num_threads(1)
mlp = nn.Linear(1000, 1000)
a = torch.rand([10, 1000])

print(time.strftime("%Y-%m-%d %H:%M:%S"), 'start')
t0 = time.perf_counter()
with profiler.profile(record_shapes=True) as prof:
    while True:
        t1 = time.perf_counter()
        out = mlp(a)
        t2 = time.perf_counter()
        if t2 - t1 &gt; 0.05:
            print(time.strftime("%Y-%m-%d %H:%M:%S"), t2 - t1)
        if t2 - t0 &gt; 120:
            break
print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))
prof.export_chrome_trace("trace.json")
print(time.strftime("%Y-%m-%d %H:%M:%S"), 'finish')
</code></pre>
Why model is not learning to output 0. Is there any mistake i am doing?

username: sandeep1
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d403f3b68de641e10e2b90d30ce03f549a515ecd.png" data-download-href="https://discuss.pytorch.org/uploads/default/d403f3b68de641e10e2b90d30ce03f549a515ecd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d403f3b68de641e10e2b90d30ce03f549a515ecd_2_671x499.png" alt="image" data-base62-sha1="ufzGPGyXgtzn6UUD0kr5zB2CGrb" width="671" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d403f3b68de641e10e2b90d30ce03f549a515ecd_2_671x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d403f3b68de641e10e2b90d30ce03f549a515ecd_2_1006x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/4/d403f3b68de641e10e2b90d30ce03f549a515ecd.png 2x" data-dominant-color="ECEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1252×932 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your training loop misses the <code>optimizer.step()</code> call so the parameters of your model won’t be updated.<br>
I would recommend to take a look at the <a href="https://pytorch.org/tutorials/">tutorials</a> and play around with some of these models and training loops first.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: sandeep1
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>  Is the code corrected now?</p>
<pre><code class="lang-auto">y=np.array([0,1,0,1,0,1,0,1,0,1])
inputs=torch.from_numpy(train)
target= torch.from_numpy(y)
train_ds=TensorDataset(inputs,target)
batch_size=5
loss_fun=F.mse_loss
train_dl=DataLoader(train_ds,shuffle=True)
model=nn.Linear(1,1)
opt=torch.optim.SGD(model.parameters(),lr=10e-5)
loss_fun=F.mse_loss
def fit(num_epochs,model,loss_fn,opt):
  for epoch in range(num_epochs):
    for xb,yb in train_dl:
      pred=model(xb.float())
      #print(xb)
      loss=loss_fn(pred,yb.float())
      loss.backward()
      opt.step()
      opt.zero_grad()

      # if (epoch+1) % 10 == 0:
      #       print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

fit(10000,model,loss_fun,opt)

model(torch.from_numpy(np.array([11.0])).float())```</code></pre>

username: ptrblck
cooked: <p>It looks alright for a toy example, yes.</p>
Swapping BatchNorm for LayerNorm in ResNet

username: jacobbuckman
cooked: <p>Question about the interface to ResNet in torchvision.</p>
<p>I’m trying to create a ResNet with LayerNorm (or GroupNorm) instead of BatchNorm. There’s a parameter called norm_layer that seems like it should do this: <code>resnet18(num_classes=output_dim, norm_layer=nn.LayerNorm)</code></p>
<p>But this throws an error,<br>
<code>RuntimeError('Given normalized_shape=[64], expected input with shape [*, 64], but got input of size[128, 64, 14, 14]')</code><br>
about the shapes being wrong. Is this deprecated? Or am I using the interface wrong? What is the recommended way to do this?</p>

username: googlebot
cooked: <p>Layernorm was not designed for images, thus it works with “last dimension” tensors usual elsewhere. Technically, it will work with two permute() calls, how well it substitutes batchnorm is another matter…</p>

username: jacobbuckman
cooked: <p>Okay, let me ask a different way then: what are the currently-implemented valid arguments to norm_layer? I’ll take anything besides BatchNorm. (Any of the myriad approaches that normalizes locally, without being reliant on the batch.)</p>

username: googlebot
cooked: <p>You can write a wrapper around nn.LayerNorm that permutes dimensions before&amp;after. Or try <a href="https://github.com/Cerebras/online-normalization/blob/master/online-norm/pytorch/online_norm_pytorch/online_norm_2d.py" rel="noopener nofollow ugc">online norm</a>. IDK how well these will work with resnet untuned.</p>

username: jacobbuckman
cooked: <p>I see, thanks.</p>
<p>Related q, although maybe this deserves a new thread…what is this about?</p>
<pre><code class="lang-auto">if groups != 1 or base_width != 64:
  raise ValueError('BasicBlock only supports groups=1 and base_width=64')
</code></pre>
<p>Can I not adjust the width of the resnet? Is there a non-basic block I should be using?</p>
<p>I don’t care about width, per-say: I really just want a single parameter that I can adjust which will change the “scale” of the model, making it larger or smaller. What’s the best way to do this?</p>

username: googlebot
cooked: <p>Perhaps you need to generate a network with Bottleneck blocks, not sure.</p>
<p>PS also check GroupNorm, that’s like parallel LayerNorm blocks, and it works with N,C,* tensors.</p>
How to speed up image segmentation results transfer from GPU to CPU

username: constant.roussel
cooked: <p>I am running an image segmentation network in C++ for in production application. The output of my network is a 1x1x640x400 tensor. My forward is taking 3 ms on GPU, but the problem is that the transfer from GPU to CPU is 22 ms long.</p>
<p>Is that because of the size of my output tensor ? I saw I need to synchronize CUDA stream, but</p>
<pre><code class="lang-auto">at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
<p>is not recognized in libtorch 1.7.1 cuda 10.2.</p>
<p>Is there a way to speed up the transfert ?</p>

username: ptrblck
cooked: <p>The transfer time is defined by e.g. the connection of the GPU to the host.<br>
For your particular issue, make sure you are timing the operations correctly. Since CUDA operations are executed asynchronously, you would have to synchronize the code before starting and stopping the timer.<br>
If you don’t synchronize, you might be profiling the kernel launch (not the forward pass), while the transfer to the host would automatically synchronize the code and accumulate the forward pass + the transfer time.</p>
Training multiple models on multiple GPUs hangs

username: MehMeh
cooked: <p>Hi,</p>
<p>When I run multiple train sessions on multiple GPUs (one model per GPU), I am getting repeatable problems on one GPU (GPU 3). Note that this GPU is the only one configured for video output as well.</p>
<p><strong>Symptoms:</strong><br>
a.) If I run the first training on the affected GPU 3, the training hangs as soon as I start two or more training sessions on other GPUs. (GPU 3 is not available afterward, reboot required.)<br>
b.) If GPU 3 is the last to run the training, the entire system freezes (hard reboot required).</p>
<p>If I train only on the affected GPU 3, it runs without any problems.</p>
<p><strong>Environment:</strong><br>
PyTorch version: 1.7.1<br>
CUDA used to build PyTorch: 11.0<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Enterprise</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: 11.0<br>
GPU models and configuration: 4 x RTX3090<br>
Nvidia driver version: 461.40<br>
cuDNN version: 8.0.4</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.2<br>
[pip3] torch==1.7.1<br>
[pip3] torchvision==0.8.2<br>
[conda] blas                      1.0                         mkl    conda-forge<br>
[conda] cudatoolkit               11.0.221             h74a9793_0    anaconda<br>
[conda] mkl                       2020.2                      256    anaconda<br>
[conda] mkl-service               2.3.0            py38hb782905_0<br>
[conda] mkl_fft                   1.2.0            py38h45dec08_0<br>
[conda] mkl_random                1.1.1            py38h47e9c7a_0    anaconda<br>
[conda] numpy                     1.19.2           py38hadc3359_0<br>
[conda] numpy-base                1.19.2           py38ha3acd2a_0<br>
[conda] pytorch                   1.7.1           py3.8_cuda110_cudnn8_0    pytorch<br>
[conda] torchvision               0.8.2                py38_cu110    pytorch</p>
<p>CPU: AMD Ryzen Threadripper 3970X</p>
<p>I have already tried the solutions suggested here: <a href="https://forum.level1techs.com/t/repeatable-system-freezes-under-gpu-load-with-threadripper-ubuntu-18-04/138237" class="inline-onebox" rel="noopener nofollow ugc">Repeatable system freezes under GPU load with Threadripper &amp; Ubuntu 18.04 - GPU - Level1Techs Forums</a></p>
<p>Current workaround: I figured out that if I use smaller batch sizes and set num_workers of the training DataLoaders not to 0 there seem to be no problems.<br>
However, I want to find the cause of the problem.</p>
<p>Edit: a simple example in Tensorflow runs without any problems on all GPUs.</p>
BatchNorm with track_running_stats=True versus False leads to major performance difference

username: asu89
cooked: <p>I have a model that I trained for colorization (Imagenet), containing BatchNorm layers with the parameter <strong>track_running_stats=False</strong>. The results were very good, but I can’t convert that model to an <strong>ONNX</strong> one because it’s not supported at the moment (I get an error and from what I can tell, it’s not possible at the moment).<br>
So I loaded the model weights to a model with BatchNorm (track_running_stats=True) using strict=False and resumed training, but found that the performance of the model significantly decreased even after many epochs.</p>
<p>Q1. Can someone please explain why making this change impacts the performance so much? I know it has to do with <strong>the mean and variance at the batch level</strong>. When track_running_stats=False it does not track these global values, correct?</p>
<p>Q2. Also, is there a way to convert a PyTorch model to ONNX even when using BatchNorm with track_running_stats=False?</p>

username: ptrblck
cooked: <ol>
<li>I would assume the training results would not see any impact, since the batch statistics would be used to normalize the activations in both cases (<code>track_running_stats=True/False</code>).<br>
The validation results might be different, since the running stats might not represent the batch stats well (e.g. if the data comes from different distributions etc.).</li>
</ol>

username: asu89
cooked: <p>Thank you for the reply, <a class="mention" href="/u/ptrblck">@ptrblck</a> !<br>
Indeed, this happens on test images, which can be quite different from the training dataset.</p>
<p>I will try to include some augmentation on the train data, so that it resembles the test data better.</p>
TypeError: __init__() missing 1 required positional argument: 'num_features'

username: Samiha
cooked: <p>Here is the error details:</p>
<p>Traceback (most recent call last):<br>
File “test_toy_v2_model.py”, line 25, in <br>
model = get_unet_model(norm_layer=torch.nn.BatchNorm2d, with_proposals=True)<br>
File “…/adaptis/model/toy/models.py”, line 13, in get_unet_model<br>
norm_layer=norm_layer),<br>
File “…/adaptis/model/toy/unet.py”, line 12, in <strong>init</strong><br>
self.image_bn = norm_layer()<br>
TypeError: <strong>init</strong>() missing 1 required positional argument: ‘num_features’</p>
<p>Related Codes:<br>
1)<br>
with self.name_scope():<br>
self.image_bn = norm_layer(in_channels = 3)<br>
prev_ch = 0</p>
<ol start="2">
<li>
</li></ol>
<p>def get_unet_model(norm_layer, channel_width=32, max_width=512, with_proposals=False, rescale_output=(0.2, -1.7)):<br>
return AdaptIS(<br>
feature_extractor=UNet(num_blocks=4, first_channels=channel_width, max_width=max_width,<br>
norm_layer=norm_layer),<br>
adaptis_head=ToyAdaptISHead(<br>
SimpleConvController(3, channel_width, norm_layer=norm_layer),<br>
ch=channel_width, norm_radius=42, with_coord_features=True,<br>
norm_layer=norm_layer,<br>
rescale_output=rescale_output<br>
),<br>
segmentation_head=ConvHead(2, channels=32, num_layers=3, norm_layer=norm_layer),<br>
proposal_head=ConvHead(1, channels=32, num_layers=2, norm_layer=norm_layer),<br>
with_proposals=with_proposals<br>
)</p>
<ol start="3">
<li>
</li></ol>
<p>model = get_unet_model(norm_layer=torch.nn.BatchNorm2d, with_proposals=True)</p>
<p>Could anyone please why this error is occurring and how to solve it? Thanks.</p>

username: ptrblck
cooked: <p>The error is raised in:</p>
<pre><code class="lang-python">File “…/adaptis/model/toy/unet.py”, line 12, in __init__
self.image_bn = norm_layer()
</code></pre>
<p>where <code>norm_layer</code> most likely expects the <code>num_features</code> argument, while nothing is passed.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to detect large gradient fluctuations?

username: PhysicsIsFun
cooked: <p>Hello community!</p>
<p>I suspect that during my training procedure, a small number of parameters in my model undergo drastic changes from batch to batch in certain training epochs (exploding gradients). In other words it does not happen all the time. Other than simply printing out the gradients at those epochs/batches (which might be unfeasible for large models), is there a simple way to do this? Like, getting a warning if at one point a gradient becomes “larger than usual”?</p>
<p>Best,<br>
PIF</p>

username: googlebot
cooked: <p>If you’re using something like adam, gradient variance is already tracked, so I’d consider using that information for tracking. For crude fixing, nn.utils has some clipping functions. And IIRC tensorboard can plot gradient densities over epochs (for tracked params/tensors).</p>

username: PhysicsIsFun
cooked: <p>I am using SGD. Is the variance tracked there as well?</p>
<p>And isn’t Tensorboard part of Tensorflow? Little confused.</p>

username: googlebot
cooked: <p>No, it is done by adaptive optimizers. Actually, switching to one may solve your problem by itself, as they smooth steps strongly. RMSProp may be the least intrusive one.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="3" data-topic="112253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>And isn’t Tensorboard part of Tensorflow?</p>
</blockquote>
</aside>
<p>Yes, but pytorch supports writing its trace files. I re-checked my older code that produced gradient plots, and it is a bit non-trivial - I used TensorboardLogger and events from ignite library, and Tensor.register_hook callbacks to record gradients for exporting.</p>
Gated Recurrent Unit Biases General Question

username: J_Johnson
cooked: <p>Why does Pytorch’s implementation of Gated Recurrent Units use two sets of biases for the update, reset and new gates? It seems redundant since these biases are summed before activation. Were there any performance gains for this approach, as opposed to just one set of biases?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.GRU.html</a></p>

username: googlebot
cooked: <p>pytorch just mirrors cudnn I think; what usually happens is Wx+b non-recurrent computation is merged (W = [Wr Wz Wn]) and done first, so <em>theoretical</em> uses of two biases are imaginable, but generally it beats me too why cudnn has this…</p>

username: J_Johnson
cooked: <p>I can’t imagine how two separate biases for z and r would increase efficiency or performance in any way. The gradients will be identical. Perhaps the new gate might* have some nominal additional information from two separate biases. as one is added prior to the Hadamard product. But I wonder if it was tested first for any performance gains. Or if, as you mentioned, the underlying architecture in cudnn just made it a better choice to leave in. That’s good to know. But then I wonder how Tensorflow got around that with their v3 implementation.</p>

username: googlebot
cooked: <p>performance impact here is negligible - same as replacing mm with addmm, because blas gemm routine fuses this bias summation.</p>
Memory usage for UNET

username: einrone
cooked: <p>Hey!</p>
<p>Before building model, i was trying to use mateuszbuda brainsegmentation unet to see how it perfom on my multiple scelerosis dataset. I  have rtx 3070 with 8gb ram, and I am getting following error/warning: RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 7.77 GiB total capacity; 6.12 GiB already allocated; 35.75 MiB free; 6.22 GiB reserved in total by PyTorch)</p>
<p>Does anyone have an advice for me?</p>
<p>The link of the model is down below:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/mateuszbuda/brain-segmentation-pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/5720638?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/mateuszbuda/brain-segmentation-pytorch" target="_blank" rel="noopener nofollow ugc">mateuszbuda/brain-segmentation-pytorch</a></h3>


  <p><span class="label1">U-Net implementation in PyTorch for FLAIR abnormality segmentation in brain MRI</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>You would have to reduce the memory usage of the script e.g. by lowering the batch size, using <code>torch.utils.checkpoint</code> to trade compute for memory, or by using a smaller model or input data.</p>

username: einrone
cooked: <p>Is there a way to see how much memory the model using without any input data?</p>

username: ptrblck
cooked: <p>That’s not easily doable, since you would need to infer the output shapes of the intermediate activations for all operations. In case you are able to do so, you would also have to stick to all native kernels, since e.g. cudnn uses different workspace sizes for different algorithms, which might be selected if <code>cudnn.benchmark</code> is active.</p>

username: einrone
cooked: <p>Because a friend of mine managed to fit the unet and train it on GTX1070, but i get this following error:</p>
<p>Runetimeerror : cuda out of memory. Tried to allocate 12mib (GPU 0; 7.77 GB total capacity; 6.12 GB already allocated; 35mb free; 6.22 GB is resvered in total by pytorch)</p>
<p>So kinda confused. This happens when it goes inside the training phase.</p>
High memory consumption + long training time on a new version

username: 111439
cooked: <p>Good day,<br>
I am trying to reproduce some results of <a href="https://github.com/jeya-maria-jose/KiU-Net-pytorch/blob/master/arch/ae.py#L306" rel="noopener nofollow ugc">this model</a> and the memory consumption is crazy (40gb for 256*256, batch size =3), and the computation time is also very slow, although the number of parameters is very low (~300k). My suggestion that maybe on newer version some changes were done so the code from the older version doesn’t work properly on a new pytorch.<br>
Unfortunately I can’t check the code on an older version because I have CUDA 11 and as far as I understand only pytorch 1,7 is compatible with it.<br>
I am wondering is it possible that on a new version the code runs much slower and consumes a lot of memory? If so, what is the possible reason for that and how it can be fixed?<br>
I also do not exclude the fact the this model has complex connections or wasn’t implemented correctly and simply doesn’t work.</p>

username: ptrblck
cooked: <p>The PyTorch conda binaries and pip wheels ship with their own CUDA runtime, so you could install different versions using virtual environments on the same system. Your local CUDA toolkit would be used, if you build PyTorch from source or compile a custom CUDA extension.<br>
Since you suspect that framework changes are responsible for the increased memory usage, this would allow you to compare different release versions.</p>

username: 111439
cooked: <p>thank you for the reply. I tried to install 1.4 version via<br>
<code>conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch </code><br>
The installation was successful, cuda_is_available returns True, but the computation takes forever even for the models for that it worked really fast on 1.7 version. My GPU model is Nvidia A100. nvidia-smi says that 500mb are busy, before it was about 2-3GB for the prediction. Could it be possible that older pytorch versions aren’t compatible with this gpu?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="111439" data-post="4" data-topic="112101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111439/48/29874_2.png" class="avatar"> 111439:</div>
<blockquote>
<p>Could it be possible that older pytorch versions aren’t compatible with this gpu?</p>
</blockquote>
</aside>
<p>They are compatible, but the CUDA JIT will compile all kernels for your architecture, since PyTorch <code>1.4</code> didn’t ship with <code>sm_80</code>, which is needed for the A100 you are using.</p>

username: 111439
cooked: <p>Thank you for your help. I asked the developer and they confirmed that the model requires a lot of memory and time for training, so I do not think that there is an issue with the performance because of another pytorch version</p>
Machine A running on GCP (VM) and machine B running locally (laptop)

username: bobross
cooked: <p>Hi there,</p>
<p>So I was playing around with <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial and took the code from <a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py" rel="noopener nofollow ugc">here</a>, and got it working fine locally (single machine without the GCP).</p>
<p>So my idea was to actually have rank=0 running in the VM on GCP and have rank=1 run on my laptop. This means that both of these workers are on completely different networks.</p>
<p>I am unable to get this setup working, it seems to be hanging when I run the worker with rank=1.</p>
<p>This is my setup and the changes I made in as much detail as possible.</p>
<ul>
<li>For the tutorial code, the only change I made was changing port 29500 to port 5000.</li>
<li>The /etc/hosts file got a new entry on my local machine (laptop). Specifically the IP of the NIC, for me, it was <strong>wlp2s0</strong> and let’s assume the address was <strong>11.22.33.44</strong>. So the /etc/hosts file would have the new entry <strong>11.22.33.44 mycomputer</strong>.</li>
<li>The /etc/hosts file got a new entry in my VM on GCP, but this is set by default. The NIC for the VM seems to be <strong>ens4</strong>? And let’s assume the IP address is <strong>44.33.22.11</strong> and let’s also assume the IP address of the VM is <strong>33.33.33.33</strong>. So the new entry of the /etc/hosts file would be <strong>44.33.22.11 gcp-vm</strong>
</li>
<li>I also made sure the ports of the VM are open and listening, so I updated the firewall settings and to verify this I simply created a flask server and queried the IP, in my case it would be <strong>http:// 33.33.33.33:5000</strong>
</li>
</ul>
<p>I’m not surprises it’s hanging I think the way how I’m running the parameter server isn’t the correct way, but I’m unsure. What are the correct changes I should make to get this working properly?</p>

username: Yanli_Zhao
cooked: <p>Do you want to have a way to let the two machines ping each other and make sure the network connection between them is working?</p>

username: bobross
cooked: <p>Hi there <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a>! Thanks for the reply. Yes, so basically it seems the two machines are not communicating with each other. I want to figure out how to get these two machines (one running on GCP and the other being my laptop (in a different network)) to be able to communicate and train the MNIST model using the RPC Framework. Or is this something that the Pytorch RPC is unable to do currently in this manner?</p>

username: bobross
cooked: <p>Did that answer your question <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a>? Or did I miss something?  I can give any extra information that is needed.</p>

username: Yanli_Zhao
cooked: <p>I think the laptop and VM should physically be in the same network, otherwise the two nodes will not get hand shake and get connected via RPC framework.</p>
RL expediting loss computation

username: crnyu
cooked: <p>I have an RL setup with both a value and a policy function. They each need to be updated in turn. Is it possible to do this update in one go without having to do something like this (<a href="https://github.com/openai/spinningup/blob/master/spinup/algos/pytorch/sac/sac.py#L230" class="inline-onebox" rel="noopener nofollow ugc">spinningup/sac.py at master · openai/spinningup · GitHub</a>) where the updates are computed for each with the other having requires_grad=True?</p>
What do you use for loading standard text datasets?

username: Bjorn_Lindqvist
cooked: <p>For images there is torchvision that works pretty well. I tried torchtext but I couldn’t get it to work and the documentation is very sparse. Is there any other library for loading text that is recommended?</p>

username: sipan17
cooked: <p>You can use datasets provided by huggingface (<a href="https://github.com/huggingface/datasets" class="inline-onebox" rel="noopener nofollow ugc">GitHub - huggingface/datasets: 🤗 The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools</a>)</p>
Scipy into PyTorch

username: kjans123
cooked: <p>I am attempting to integrate this scipy function into PyTorch:<br>
<a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.UnivariateSpline.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.UnivariateSpline.html</a></p>
<p>I’ve been looking at this guide:<br>
<a href="https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html</a></p>
<p>This is the code I have written so far:</p>
<blockquote>
<p>from scipy.interpolate import UnivariateSpline<br>
import torch<br>
from torch.autograd import Function<br>
class SPLFunction(Function):</p>
</blockquote>
<pre><code>@staticmethod
def forward(ctx, input, s):
    np_input = input.detach().numpy()
    s = s.detach()
    
    x, y = np_input[0], np_input[1]

    spl = UnivariateSpline(x, y,s=s)
    result = spl(y)
    ctx.save_for_backward(input, s)

    return torch.as_tensor(result, dtype=input[0].dtype)

@staticmethod
def backward(ctx, grad_output):
    input, s = ctx.saved_tensors
    np_input = input.detach().numpy()
    s = s.detach()
    
    x, y = np_input[0], np_input[1]
    spl = UnivariateSpline(x, y,s=s)
    df = spl.derivative(x)
    df = torch.from_numpy(df*grad_output)
    
    return(df)
</code></pre>
<p>However, this does not seem to be working. I am getting a variety of errors…if anyone has experience integrating scipy functions in to PyTorch, any tips in the right direction would be hugely appreciated!</p>
Generate New Images using data augmentation and save those images

username: thesekyi
cooked: <p>I have a set of 3 images. I want to use random data augmentation to generate 3000 new images from these 3 images. I haven’t found much by googling. I guess I am googling the wrong keywords. Can anyone please give me a useful direction to look?</p>

username: Dwight_Foster
cooked: <p>You could use the torchvision transformations to transform an image and then use the torchvision utils to save a tensor as an image <a href="https://pytorch.org/vision/stable/utils.html" rel="noopener nofollow ugc">here</a>.</p>
Concatenating tensors error: Expected object of scalar type Float but got scalar type Double

username: SU801T
cooked: <p>Hi,</p>
<p>I want to concatenate a tensor with a number that has multiple decimal places (0.990989 etc.) basically a float.</p>
<p>I extract features from a pretrained network like this:</p>
<pre><code class="lang-auto">class DenseNetConv(torch.nn.Module): 
    def __init__(self):
        super(DenseNetConv,self).__init__()
        original_model = models.densenet161(pretrained=True)
        self.features = torch.nn.Sequential(*list(original_model.children())[:-1])
        for param in self.parameters():
            param.requires_grad = False

    def forward(self, x):
        x = self.features(x)
        x = F.avg_pool2d(x, kernel_size=7).view(x.size(0), -1)
        return x

</code></pre>
<p>where I then define the fully connected layer like:</p>
<pre><code class="lang-auto">classifier = nn.Linear(2208 + 1, args.num_classes)
</code></pre>
<p>where “2208” is the number of features from the above class and the “1” is the additional score I want to concatenate:</p>
<pre><code class="lang-auto">
score = score.to(device)
x10 = densenet(inputs10x)
x_all = torch.cat([x10, score], dim=1)
</code></pre>
<p>However, when I concatenate I get an error:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Float but got scalar type Double for sequence element 1 in sequence argument at position #1 'tensors'

</code></pre>
<p>Is the problem with the different dimensions of the score tuple and features tuple?</p>

username: Dwight_Foster
cooked: <p>you can convert x10 to a double by doing</p>
<pre><code class="lang-auto">x10.double()
</code></pre>
RuntimeError when trying to backprop through slicing operations

username: plaviajvar
cooked: <p>Hello everyone,</p>
<p>I’m trying to backprop through a dynamic programming algorithm. The algorithm is definitely backpropable, but the forward pass involves operating on tensor slices, which is not a standard scenario for autograd. The arising RuntimeErorrs don’t inspire much condifence. I have isolated a small example highlighting the issue.</p>
<pre data-code-wrap="python"><code class="lang-python"># Allocate random 3 x 3 x 2 energy volume of values from [0, 1]
height, width = 3, 3
ndisp = width-1
torch.manual_seed(0)
energy_init = torch.rand(height, width, ndisp, requires_grad=True)

# Clone to avoid "RuntimeError: leaf variable has been moved into the graph interior"
energy = energy_init.clone()
torch.autograd.set_detect_anomaly(True)

#######################
# Copy operation here
#######################

# Backprop and print gradients
energy_sum = energy.sum()
energy_sum.backward()
print(energy_init.grad)
</code></pre>
<p>For the following 4 operations, the code executes without errors and computes the correct gradients:</p>
<pre data-code-wrap="python"><code class="lang-python">energy[:, 0] = energy[:, 1] 
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">energy[:, 0] = energy[:, 0] + energy[:, 1]
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">energy[:, 0] = torch.minimum(energy[:, 1].clone(), torch.tensor([2]))
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">two = torch.ones_like(energy[:, 1]) * 2
mask = (energy[:, 1] &lt; two)
energy[:, 0] = mask * energy[:, 1] + (~mask) * two
</code></pre>
<p>This one, however, generates a <strong>RuntimeError</strong>:</p>
<pre data-code-wrap="python"><code class="lang-python">energy[:, 0] = torch.minimum(energy[:, 1], torch.tensor([2]))
</code></pre>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 2]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead.</p>
</blockquote>
<p>It seems that only <strong>torch.minimum</strong> causes issues, and other binary operations like addition are fine. I’m not sure why autograd considers the computation be “in-place”.</p>
<p>My main concern is that the correctness is somehow compromised. But also, since pointwise minimum operations are performed lots of times, it would be preferable to avoid cloning the slice every time.</p>
<p>Is the code correct and is there a less hacky way to resolve the issue?</p>
<p>Thank you in advance.</p>
CUDA Device Assert Trigger when moving tensor from CPU to GPU

username: serenayj
cooked: <p>Hi,</p>
<p>I tried to move a tensor from cpu to gpu by doing:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.tensor(1)
&gt;&gt;&gt; t.cuda()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>All cpu tensors can’t be moved to GPU. My pytorch cuda is 10.2, pytorch version 1.5.1. Nvidia-SMI info as follows:</p>
<pre><code class="lang-auto">NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0
</code></pre>
<p>Could someone help me out?</p>

username: ptrblck
cooked: <p>This error points to an installation (or unlikely a hardware) issue.<br>
Did you restart your workstation after installing or upgrading some drivers?<br>
If so, make sure you are able to run some CUDA samples.</p>

username: serenayj
cooked: <p>Thanks for your reply. Reinstalling pytorch and cudatoolkit solved everything.</p>
How to calculate the square of a tensor?

username: luodahei
cooked: <p>just like numpy.square in numpy.<br>
Thank you very much</p>

username: Sunshine352
cooked: <p>Assume a tensor A, the the square of A equals A * A, that is B = A * A</p>

username: justusschock
cooked: <p>You should also be able to do <code>A**2</code> or <code>A.pow(2)</code></p>

username: Adam_Conkey
cooked: <p>There is also a <a href="https://pytorch.org/docs/stable/generated/torch.square.html" rel="noopener nofollow ugc"><code>torch.square</code></a> function now (<a href="https://github.com/pytorch/pytorch/pull/30719" rel="noopener nofollow ugc">PR</a>) that you can call like <code>A.square()</code>.</p>
Why any numerical operations on nn.Parameter return nn.Tensor?

username: Naruto-Sasuke
cooked: <pre><code class="lang-python">import torch
import torch.nn as nn
learning_alpha = torch.ones(1)
learning_beta = torch.zeros(1)
learning_alpha = nn.Parameter(learning_alpha)
learning_beta = nn.Parameter(learning_beta)

c = learning_alpha + learning_beta
print(type(c))

</code></pre>
<p><code>&lt;class 'torch.Tensor'&gt;</code></p>

username: albanD
cooked: <p>Hi,</p>
<p><code>nn.Parameter</code> are just a Tensor subclass for <code>nn.Module</code> to know what is a parameter there or not. But there is no need for them to remain Parameters and they can be used just as any regular Tensor.</p>
Questions about dropout

username: morphism
cooked: <p>Hello, everyone</p>
<p>I have questions about Dropout.</p>
<ol>
<li>
<p>Should I use different dropout mask when forwarding batch data?<br>
That is, should I generate different dropout mask for each mini-batch data at the same layer?</p>
</li>
<li>
<p>Should the dropout mask when doing backpropagation be the same as the dropout mask when doing forward processing?<br>
That is, should I save the dropout masks when forwarding and apply the same mask to the corresponding layer when doing backpropagation?</p>
</li>
<li>
<p>Should I not use dropout to the output layer?<br>
Should I not use dropout to the input?</p>
</li>
</ol>
<p>Thank you in advance and have a nice weekend</p>

username: nivesh_gadipudi
cooked: <ol>
<li>A dropout mask is to be used can be applied to the layers by multiplying a tensor of the same size elementwise, this comes in your forward() method so as to apply for the layers irrespective of the input</li>
<li>When you create a dropout mask and apply it to forward propagations the neurons will be inactive or ‘0’. so, while calculating the gradients it will not propagate back through the dropout mask</li>
<li>Dropout can’t be used on output layers</li>
</ol>

username: morphism
cooked: <p>Hello,<br>
Thank you for your answer.<br>
And this question is assuming that I am not using PyTorch nor TensorFlow.<br>
That is, I am implementing dropout algorithm by myself for studying.</p>

username: nivesh_gadipudi
cooked: <p>Yes, I understood when you were referring to dropout mask instead of dropout layer</p>

username: morphism
cooked: <p>Thank you…!<br>
…</p>
Questions about dropout (2)

username: morphism
cooked: <p>Hello,<br>
I am studying dropout algorithm<br>
I am implementing dropout using Python or Julia.</p>
<ol>
<li>
</li></ol>
<p>If I have two hidden layers and dropout ratios are 0.5, 0.3 respectively,<br>
Then how much ratio should I multiply to the output when evaluating?</p>
<p>Also, should (may) I multiply some ratio (such as 1/(1-p) for some appropriate p) to the output when training?</p>
<ol start="2">
<li>When using Pytorch, when I use dropout, I have never multiplied some ratio to the output when evaluating? Does PyTorch automatically multiply some ratio to the output when evaluating?</li>
</ol>
<p>Thank you in advance</p>

username: ptrblck
cooked: <p>PyTorch scales the activation during training with the inverse as seen <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/aten/src/ATen/native/Dropout.cpp#L62">here</a>.<br>
This inverse scaling is mentioned in the original dropout paper (if I’m not mistaken) and allows you to just disable the dropout layer during evaluation.</p>

username: morphism
cooked: <p>Thank you for your answer.<br>
It helped me a lot<br>
Have a nice day!</p>
Network based transfer learning

username: Sahil_malik
cooked: <p>Could you please elaborate the concept of network-based transfer learning. Is it the same concept that we use for image classification through pret-trained models? Can we call it network-based transfer learning?</p>
<p>It’s explained here: <a href="https://link.springer.com/chapter/10.1007/978-3-030-01424-7_27" rel="noopener nofollow ugc">https://link.springer.com/chapter/10.1007/978-3-030-01424-7_27</a>, but I am confused. I need clarification on this concept. It would be a great help.</p>
<p>Thank you!</p>
Get 2d indices for max and min

username: Ameen_Ali
cooked: <p>Hi<br>
lets say I have a tensor of shape [batchsize , 7 , 7]<br>
I wish to get for each sample in the batch the maximum two values with 2d indices + the minimum two values with 2d indices</p>
<p>Example : lets say we have a tensor of shape [2 , 3  , 3]<br>
<br></p>
<pre><code>tensor = [[[1,2,3] , [4,5,6] , [6,7,8]] , 
         [[-9,-8,-7],[-6,-5,-4],[-3,-2,-1]]]
</code></pre>
<p>the maximum is : [(2,2) , (2,1)] , [(2,2) , (2,1)] for the two tensors, <br>
the minimum is : [(0 , 0) , (0,1)] , [(0,0) , (0,1)] for the two tensors as well.</p>
<p>I have tried something around :</p>
<pre><code class="lang-auto">torch.nonzero ((a == a.max (dim = 1, keepdim = True)[0]))
</code></pre>
<p>but it works over rows/columns not on the whole tensor<br>
thanks</p>

username: Eta_C
cooked: <p>maximum</p>
<pre><code class="lang-python">index = torch.topk(tensor.reshape(2, -1), 2, dim=1)[1][..., None]
a, b = index // 3， index % 3
res = torch.cat([a, b], dim=2)
print(res)
</code></pre>
<p>for minimum, use <code>tensor =  -tensor</code></p>
Pytorch-Geometric graph classification where graphs all have the same shape

username: gil_bc
cooked: <p>Hi all,<br>
as the headline suggests, I want to train a model for graph classification using Pytorch-geometric.<br>
As a matter of fact, I actually have many samples based on the same graph structure, differing in their node features (one node-feature vector per sample).</p>
<p>I tried to create a data object for each sample (referring to it as a different presentation of the same graph), but that is very computationally expensive, because there are relatively many such samples.</p>
<p>Therefore, I’m looking for a way to do graph-level classification, with a batching system that will refer to all graphs as having the same nodes, and the same number of node features.</p>
<p>Is that possible?</p>
<p>Thanks!</p>
How to solve `ValueError: Expected input batch_size (16) to match target batch_size (32).`?

username: xsquared
cooked: <p>well here is the network but Im getting the error <code>ValueError: Expected input batch_size (16) to match target batch_size (32).</code> I think its because I have the wrong linear channels but i dont know how to come up with the right number. so if you could help me i will appreciate it.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    """A representation of a convolutional neural network comprised of VGG blocks."""
    def __init__(self, n_channels):
        super(Net, self).__init__()
        # VGG block 1
        self.conv1 = nn.Conv2d(n_channels, 64, (3,3))
        self.act1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d((2,2), stride=(2,2))
        # VGG block 2
        self.conv2 = nn.Conv2d(64, 64, (3,3))
        self.act2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d((2,2), stride=(2,2))
        # VGG block 3
        self.conv3 = nn.Conv2d(64, 128, (3,3))
        self.act3 = nn.ReLU()
        self.pool3 = nn.MaxPool2d((2,2), stride=(2,2))
        # Fully connected layer
        self.f1 = nn.Linear(8192, 1000)
        self.act4 = nn.ReLU()
        # Output layer
        self.f2 = nn.Linear(1000, 10)
        self.act5 = nn.Softmax(dim=1)

    def forward(self, X):
        """This function forward propagates the input."""
        # VGG block 1
        X = self.conv1(X)
        X = self.act1(X)
        X = self.pool1(X)
        # VGG block 2
        X = self.conv2(X)
        X = self.act2(X)
        X = self.pool2(X)
        # VGG block 3
        X = self.conv3(X)
        X = self.act3(X)
        X = self.pool3(X)
        print(X.shape)
        # Flatten
        X = X.view(-1, 8192)
        # Fully connected layer
        X = self.f1(X)
        X = self.act4(X)
        # Output layer
        X = self.f2(X)
        X = self.act5(X)

        return X
</code></pre>

username: nitaifingerhut
cooked: <p>What’s the input shape?<br>
it’s hard to answer without it<br>
but if I would guess, the view operation reduce the batch dim by 2<br>
also attach the rest of your code</p>
Performance issues with pytorch conda package

username: constantinpape
cooked: <p>I have benchmarked the performance of pytorch for a network using 3d convolutions with different gpus on our cluster for two different pytorch builds:</p>
<ul>
<li>the conda package from the pytorch channel (<code>py3.8_cuda11.0.221_cudnn8.0.5_0</code>)</li>
<li>pytorch build on our cluster with easybuild (I will provide a link to the recipe and build options later)</li>
</ul>
<p>The performance of the easybuild version is, for some configurations, significantly better than the performance of the conda package; especially for Volta and Ampere cards with half precision.<br>
For details on the benchmarks, see <a href="https://github.com/constantinpape/3d-unet-benchmarks#results" class="inline-onebox" rel="noopener nofollow ugc">GitHub - constantinpape/3d-unet-benchmarks</a>.</p>
<p>Is it possible that the conda package is built without the correct options to fully leverage the features of the newer architectures like tensorcores?</p>

username: ptrblck
cooked: <p>Which CUDA and cudnn version are you using in your source build?<br>
Besides different versions you might also be hitting <a href="https://github.com/pytorch/pytorch/issues/50153">this issue</a>, which we are working on.</p>

username: constantinpape
cooked: <p>The source build uses CUDA 11.1.1 and CUDNN 8.0.4<br>
The issue you have linked looks relevant, I will have a closer look.</p>
How to reverse a multi-hot encoding

username: kristen
cooked: <p>I have my data encoded as multi-hot vectors for a multi-label classification task (4-classes in this example):</p>
<pre><code class="lang-auto">multihot_batch = torch.tensor([[0,1,0,1], [0,0,0,1], [0,0,1,1]])
</code></pre>
<p>How can I undo this encoding and have each entry be a list of the classes present, like this:</p>
<pre><code class="lang-auto">tensor([[1, 3],
        [3],
        [2, 3]])
</code></pre>
<p><code>torch.argmax</code> only returns the index of the first max:</p>
<pre><code class="lang-auto">
torch.argmax(multihot_batch, dim=1, keepdim=True)
tensor([[1],
        [3],
        [2]])
</code></pre>
<p>This approach gives them all, but it’s not formatted how I want:</p>
<pre><code class="lang-auto">(multihot_batch == torch.tensor(1)).nonzero()
tensor([[0, 1],
        [0, 3],
        [1, 3],
        [2, 2],
        [2, 3]])
</code></pre>
<p>I’d appreciate any suggestions.</p>

username: Naruto-Sasuke
cooked: <p>Hi, it is not easy, after hard thinking for one hour, I work out this problem:</p>
<pre><code class="lang-python">import torch

multihot_batch = torch.tensor([[0,1,0,1], [0,0,0,1], [0,0,1,1]])
#multihot_batch = torch.tensor([[0,1,0,1], [0,0,0,1], [0,0,1,1], [0,0,0,1]])
vnon = (multihot_batch == torch.tensor(1)).nonzero(as_tuple=False)
v0 = vnon[:,0]
v1 = vnon[:,1]

# 0-based index -&gt; 1-based index
split_ind = ((torch.roll(v0, -1, 0) - v0) == 1).nonzero(as_tuple=False)[:,0] + 1
# the first index is excatly the split size of the first split
# the other splits(apart from the last one) can be obtained by this
split_size = torch.cat([split_ind[0].view(1),(torch.roll(split_ind, -1, 0) - split_ind)[:-1]])
# add the final split size
final_size = torch.tensor([torch.numel(v1) - torch.sum(split_size)])
split_size = torch.cat([split_size, final_size])

print(torch.split(v1, split_size.tolist()))

</code></pre>