akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't save/load model with state_dict

username: usami
cooked: <p>The following code snippet works:</p>
<pre><code class="lang-python">trained_network = ...
torch.save(trained_network, 'final-model.pt')
new_network = torch.load('final-model.pt', map_location='cpu')

evaluation(new_network)  # new_network performs good
</code></pre>
<p>But using state_dict to save/load does not work:</p>
<pre><code class="lang-python">trained_network = ...
torch.save(trained_network.state_dict(), 'final-model.pt')
new_network = init_model(...)
new_network.load_state_dict(torch.load('final-model.pt'))

evaluation(new_network)  # new_network performs bad, just like a model without training
</code></pre>
<p>I would like to use latter approach as it is the recommended way. How can I fix it ?</p>

username: usami
cooked: <p>I figure out this problem now. I first compare the parameters name and values using the following code:</p>
<pre><code class="lang-python">trained_network = Net() # network with trained parameters
my_network = Net() # network with default initialization

my_network.load_state_dict(trained_network.state_dict())

for ((k1, v1), (k2, v2)) in zip(my_network.state_dict().items(), trained_network.state_dict().items()):
    assert k1 == k2, "Parameter name not match"
    if not torch.equal(v1, v2):
        print("Parameter value not match", k1)
</code></pre>
<p>And then I found the parameter values are different in following module:</p>
<pre><code class="lang-python">class ActNorm(nn.Module):

    def __init__(self, channel: int):
        super(ActNorm, self).__init__()

        self.logs = nn.Parameter(torch.zeros((1, channel, 1)))
        self.bias = nn.Parameter(torch.zeros((1, channel, 1)))
        self.eps = 1e-6
        self.is_inited = False

    def forward(self, x: Tensor):
        if not self.is_inited:
            self.__initialize(x)

        z = x * torch.exp(self.logs) + self.bias
        logdet = x.shape[2] * torch.sum(self.logs)
        return z, logdet

    def __initialize(self, x: Tensor):
        with torch.no_grad():
            bias = -torch.mean(x.detach(), dim=[0, 2], keepdim=True)
            logs = -torch.log(torch.std(x.detach(), dim=[0, 2], keepdim=True) + self.eps)
            self.bias.data.copy_(bias.data)
            self.logs.data.copy_(logs.data)
            self.is_inited = True
</code></pre>
<p>When I use <code>torch.load(...)</code>, the above <code>is_inited</code> is set to <code>True</code>, while <code>is_inited</code> is set to <code>False</code> when using <code>my_network.load_state_dict(...)</code>. As a result, the loaded parameters are always overwrited whenever I use <code>my_network.load_state_dict()</code>, which cause to bad performance.</p>
Time issues with F.unfold

username: Micha_Christ
cooked: <p>Hi!<br>
I’m currently struggling with some computation time issues.<br>
I have an input and a filter like this:</p>
<pre><code class="lang-auto">z = torch.from_numpy(np.random.randint(1, 10, (bs, channels, z_w, z_h))).float()
f = torch.from_numpy(np.random.randint(1, 5, (n_fm, channels, filter_w, filter_h))).float()

</code></pre>
<p>I now want to extract patches from that input. First I did this manually:</p>
<pre><code class="lang-auto">def getPatches(z,  f, p=0, s=1):
    # Retrieve dimensions
    n_c, z_w, z_h = z.shape

    # Retrieve dimensions
    n_f, n_c, f_w, f_h = f.shape

    # Compute the dimensions of the CONV output volume
    o_h = int((z_h + 2 * p - f_h) / s) + 1
    o_w = int((z_w + 2 * p - f_w) / s) + 1
    p = torch.zeros((o_w*o_w, n_c, f_w, f_h))

    for currW in range(o_w):
        for currH in range(o_h):
            # Get the image patch
            x1 = currW * s
            x2 = currW * s + f_w
            y1 = currH * s
            y2 = currH * s + f_h

            # get patches
            p[currW*o_h + currH] = z[:, x1:x2, y1:y2]

    return p
</code></pre>
<p>Later in my code I’m using different <code>torch.einsum</code> calls. Each of them get some quantities that were calculate with the patches as input. Imagine s.th. like <code>torch.einsum("kcwhp,kp-&gt;pcwh", A, B)</code>, where A is based on <code>p</code> (would be to much to share the whole code).</p>
<p>The above <code>einsum</code> calls needs <code>0.048s</code> to compute the result, based on the manually created patches.</p>
<p>Now I’ve notices that there is already such a utility in the <code>torch.nn.functional</code> package <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
<code>p_fold = F.unfold(z, kernel_size=(filter_w, filter_h), stride=(1,1)).transpose(1, 2).reshape(-1, channels,  filter_w, filter_h)</code>.</p>
<p>I’ve checked it that both methods are returning the same patches. But now  <code>torch.einsum("kcwhp,kp-&gt;pcwh", A, B)</code> with the quantity <code>A</code> computed based on <code>p_fold</code> needs <code>1.03s</code>. This is a big difference and I can’t explain it, as the inputs <code>A</code> and <code>B</code> and the result of the <code>einsum</code> call are the same.</p>
<p>Are you aware of any time issues with <code>F.unfold</code> or  is there something general about Pytorch and <code>torch.einsum</code> I might not know?</p>
<p>Thanks for helping!</p>

username: ptrblck
cooked: <p>Could you check, if the <code>einsum</code> time would be reduced in the second case, if you manually add a <code>.contiguous()</code> call on the output of <code>unfold</code>? The output of <code>unfold</code> might not be contiguous and I’m not sure, if <code>einsum</code> would call it internally, which could explain the difference in performance.</p>
Making a new type of tensor [1,2,2] -> [[0, 1, 1] [1, 1, 0] [1,1, 0]]

username: Uk_Jo
cooked: <p>I have 3 classes and want to make a new type of tensor with a label tensor.</p>
<p>Label tensor = [1,2,2]</p>
<p>New tensor =<br>
[[0, 1, 1]<br>
[1, 1, 0]<br>
[1,1, 0]]</p>
<p>How can I make this easily?<br>
Thanks in advance!!</p>

username: ptrblck
cooked: <p>Could you explain how the labels would be converted to the new tensor, as I might miss the obvious pattern?</p>
Can't load onnx model converted from pythorch model with opencv

username: neouyghur
cooked: <p>Hi, I got the following error when I tried to read it with opencv (<code>ort_session = cv2.dnn.readNetFromONNX("test.onnx") </code>):</p>
<p><code>File "open_cv_test.py", line 12, in &lt;module&gt;     ort_session = cv2.dnn.readNetFromONNX("test.onnx") cv2.error: OpenCV(4.4.0) /tmp/pip-req-build-njn2fp78/opencv/modules/dnn/include/opencv2/dnn/shape_utils.hpp:222: error: (-215:Assertion failed) clamped.start &lt; clamped.end in function 'clamp'</code></p>
<p>However, I can read it with python onnx module. How can I fix this problem? Thanks.</p>

username: neouyghur
cooked: <p>I fixed this problem by upgrading my opencv version.</p>
How to make a GPU dedicated to data processing

username: Sangyeon_Kim
cooked: <p>Hi, I’m working on building a model via weakly-supervised learning.</p>
<p>In order to achieve data generation in a weakly-supervised manner during training process, I’d like to make a GPU dedicated to data processing and pass them to other GPUs, while other GPUs learn using the processed data in parallel.</p>
<p>Is it possible? and could you give me some example or snippet?<br>
I’m thinking of it is important for data synchronization among a data-dedicated GPU and other GPUs.</p>
<p>And I’m also curious it’s possible to select GPU’s device in data loader.</p>

username: gcramer23
cooked: <p>Hello <a href="https://discuss.pytorch.org/u/Sangyeon_Kim">Sangyeon_Kim</a>,</p>
<p>This might help you with your implementation.</p>
<aside class="quote no-group" data-username="Sangyeon_Kim" data-post="1" data-topic="119946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sangyeon_kim/48/33791_2.png" class="avatar"> Sangyeon_Kim:</div>
<blockquote>
<p>I’d like to make a GPU dedicated to data processing</p>
</blockquote>
</aside>
<p>You can use multiple processes and assign a process to be the data processer. The data processer would use a GPU device.</p>
<aside class="quote no-group" data-username="Sangyeon_Kim" data-post="1" data-topic="119946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sangyeon_kim/48/33791_2.png" class="avatar"> Sangyeon_Kim:</div>
<blockquote>
<p>pass them to other GPUs</p>
</blockquote>
</aside>
<p>You can communicate the processed data over RPC.</p>
<aside class="quote no-group" data-username="Sangyeon_Kim" data-post="1" data-topic="119946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sangyeon_kim/48/33791_2.png" class="avatar"> Sangyeon_Kim:</div>
<blockquote>
<p>while other GPUs learn using the processed data in parallel.</p>
</blockquote>
</aside>
<p>Each trainer will be a process. GPU devices will be assigned to the process. Each trainer will create a DDP instance.</p>
<p>ddp doc <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a><br>
distributed rpc doc <a href="https://pytorch.org/docs/stable/rpc.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch 2.1 documentation</a></p>

username: Sangyeon_Kim
cooked: <p>Thanks <a href="/u/gcramer23">gcramer23</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’ll give it a shot.</p>
Custom RNNs: what is the best strategy to implement them?

username: gnufabio
cooked: <p>Hi all,</p>
<p>I’d like to implement a custom RNN architecture and I’m trying to understand which is the best strategy, from the computational point of view, to do it.</p>
<p>After some search in this (fantastic) forum, I realized there are several ways to proceed:</p>
<ol>
<li>Use a for loop inside a forward to compute the dynamic evolution of the network</li>
<li>Manually unroll the RNN (the length of the input is fixed), which would be similar to what done in TF v1 by the function <code>dynamic_rnn</code>
</li>
<li>Follow <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="noopener nofollow ugc">this PyTorch post</a> which however looks a bit more complicated, at least for me, as I am a beginner  with PyTorch.</li>
</ol>
<p>Thank you for your precious advices!</p>
Getting multi-class segmentation mask from multi-channel model prediction

username: banikr
cooked: <p>I have 9 channel(class) 3D prediction of size: <code>b,C,48,64,64</code>. (<code>b</code> for batch size, and prediction is 0~1)</p>
<p>In original segmentation mask the class labels are: <code>labels = [0, 1, 2, 3, 4, 8, 10, 11, 56]</code><br>
Are there convenient ways to reconstruct the final output with shape: <code>b,48,64,64</code> with all the label values assigned, something opposite to one-hot encoding.</p>
<p>Moreover, I got the following code:</p>
<pre><code class="lang-auto">pred = model(img)
pred = pred.data.max(1)[1].cpu().numpy()[:, :, :].astype('uint8')
</code></pre>
<p>Does it help somehow?</p>
Error when compiling pytorch v1.3.0 from source

username: PKUFlyingPig
cooked: <p>I tried to build pytorch v1.3.0 from source with CUDA10.1. But when I run <code>python setup.py install</code>, an error occurred:</p>
<pre><code class="lang-auto">[3121/3304] Linking CXX executable bin/utility_ops_test
FAILED: bin/utility_ops_test 
: &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-mis$
ing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-fu$
ction -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-$
rror=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEF$
NITION -O3 -rdynamic -Wl,-rpath -Wl,/usr/local/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/utility_ops_test.dir/operators/utility_ops_test.cc.o $
o bin/utility_ops_test  -Wl,-rpath,/home/CORP.PKUSC.ORG/flyingpig/anaconda3/envs/pipeswitch/lib:/opt/intel/lib/intel64:/home/CORP.PKUSC.ORG/flyingpig/pyto$
ch/build/lib:/usr/local/cuda-10.1/lib64:  lib/libgtest_main.a  -Wl,--no-as-needed,/home/CORP.PKUSC.ORG/flyingpig/pytorch/build/lib/libtorch.so -Wl,--as-ne$
ded  lib/libprotobuf.a  /home/CORP.PKUSC.ORG/flyingpig/anaconda3/envs/pipeswitch/lib/libmkl_intel_lp64.so  /home/CORP.PKUSC.ORG/flyingpig/anaconda3/envs/p$
peswitch/lib/libmkl_gnu_thread.so  /home/CORP.PKUSC.ORG/flyingpig/anaconda3/envs/pipeswitch/lib/libmkl_core.so  -fopenmp  -lpthread  /usr/lib64/libm.so  /$
sr/lib64/libdl.so  /opt/intel/lib/intel64/libirc.so  lib/libmkldnn.a  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda-10.1/lib64/libnvToolsExt.so  /usr$
local/cuda-10.1/lib64/libcudart.so  /usr/local/cuda-10.1/lib64/libcufft.so  /usr/local/cuda-10.1/lib64/libcurand.so  /usr/local/cuda-10.1/lib64/libcudnn.so
  /usr/lib64/libcublas.so  lib/libgtest.a  -lpthread &amp;&amp; :
/home/CORP.PKUSC.ORG/flyingpig/pytorch/build/lib/libtorch.so：对‘cudaSetupArgument’未定义的引用
/home/CORP.PKUSC.ORG/flyingpig/pytorch/build/lib/libtorch.so：对‘cudaConfigureCall’未定义的引用
/home/CORP.PKUSC.ORG/flyingpig/pytorch/build/lib/libtorch.so：对‘cudaLaunch’未定义的引用
collect2: 错误：ld 返回 1
</code></pre>
<p>I tried to run <code>python setup.py clean</code> then rerun <code>python setup.py install</code>, but the error still occurred.</p>
Im2col im2col_backward ops lowering not supported in xla. Any temporary workaround to replace `nn.unfold` without trigger the unsupported ops?

username: Cloud_Huang
cooked: <p>I am trying to speed up my pytorch code by using pytorch_lightning, which supports TPU training. However, the training runs very slowly. Each iter takes 20secs in TPU, whereas only takes half a sec in GPU.</p>
<p>Upon profiling, the issue is below:</p>
<pre><code class="lang-auto">pt-xla-profiler: TransferFromServerTime too frequent: 449 counts during 6 steps
pt-xla-profiler: Op(s) not lowered: aten::im2col, aten::im2col_backward,  Please open a GitHub issue with the above op lowering requests.
</code></pre>
<p>XLA does not support <code>aten::im2col, aten::im2col_backward</code></p>
<p>Upon investigation of the pytorch code, it is <code>nn.unfold</code> that is triggering <code>im2col</code> and <code>im2col_backward</code></p>
<p>I can’t afford to rent multi-GPU instances and really need to use TPU to speed up things.</p>
<p>Wihout XLA supporting <code>im2col</code> and <code>im2col_backward</code> any time soon, is there a way to replace what <code>nn.unfold</code> is doing but without triggering the 2 ops ?</p>

username: Mazhar_Shaikh
cooked: <p>Hi Cloud_Huang, Could you provide a sample of the code that is using unfold and fold?</p>

username: Cloud_Huang
cooked: <p>Here you go</p>
<pre><code class="lang-auto">
class PixelEmbed(nn.Module):
    """ Image to Pixel Embedding
    """

    def __init__(self, img_size=224, patch_size=16, in_chans=3, in_dim=48, stride=4):
        super().__init__()
        num_patches = (img_size // patch_size) ** 2
        self.img_size = img_size
        self.num_patches = num_patches
        self.in_dim = in_dim
        new_patch_size = math.ceil(patch_size / stride)
        self.new_patch_size = new_patch_size

        self.proj = nn.Conv2d(in_chans, self.in_dim, kernel_size=7, padding=3, stride=stride)
        self.unfold = nn.Unfold(kernel_size=new_patch_size, stride=new_patch_size)

    def forward(self, x, pixel_pos):
        B, C, H, W = x.shape
        assert H == self.img_size and W == self.img_size, \
            f"Input image size ({H}*{W}) doesn't match model ({self.img_size}*{self.img_size})."
        x = self.proj(x)
        x = self.unfold(x)
        x = x.transpose(1, 2).reshape(B * self.num_patches, self.in_dim, self.new_patch_size, self.new_patch_size)
        x = x + pixel_pos
        x = x.reshape(B * self.num_patches, self.in_dim, -1).transpose(1, 2)
        return x


</code></pre>
Torch.reciprocal(x) vs 1.0/x

username: vadimkantorov
cooked: <p>Is there any difference between the two? Does 1.0/x call torch.reciprocal internally?</p>

username: Mazhar_Shaikh
cooked: <p>From this <a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py#L393" rel="nofollow noopener">line</a> in the code, it can be seen that 1.0/x will call torch.reciprocal internally.</p>

username: pascal_notsawo
cooked: <p>This : <a href="https://github.com/pytorch/pytorch/blob/master/torch/_tensor.py#L543" class="inline-onebox" rel="noopener nofollow ugc">pytorch/_tensor.py at master · pytorch/pytorch · GitHub</a>? Do you mean</p>

username: Mazhar_Shaikh
cooked: <p>Hi pascal_notsawo, Yes, that link is correct.</p>
How to resize image to bounding box?

username: Switchback
cooked: <p>How can i resize image to bounding box?<br>
I mean dynamically set <code>(xmin, ymin, xmax, ymax)</code> values to all images, for future model training. I think <code>transforms.Resize()</code> can help me, but <code>Resize()</code> only takes two arguments and its not accurate to bounding box</p>

username: Switchback
cooked: <p>seems like there are no way to crop it with pytorch, i’ll use Pillow lib.</p>

username: eqy
cooked: <p>Note that cropping can be expressed as a slicing index operation (e.g., given an image in CHW format you can just do <code>crop = image[:, ymin:ymax, xmin:xmax]</code> to crop and then call resize if necessary).</p>
What's the difference between BUILD.bzl and cmake?

username: j_sun
cooked: <p>Hi,<br>
i want to know about the difference between ‘BUILD.bzl’ file and cmake.<br>
What is ‘BUILD.bzl’ file for and what is PyTorch/cmake for?<br>
I think both of them are necessary to build a pytorch, but if it’s wrong, please let me know.<br>
Have a nice day.<br>
Thank you.</p>

username: eqy
cooked: <p>If I understand correctly the <strong>BUILD.bazel</strong> file is for building with well, <a href="https://bazel.build/" rel="noopener nofollow ugc">bazel</a>. cmake builds seem to be more common in my experience (what most users use when building natively from souce) and do not depend on the BUILD.bazel file.</p>

username: j_sun
cooked: <p>Why do cmake need if the PyTorch building with Bazel?<br>
I want to know what Bazel builds and what cmake builds in PyTorch.</p>

username: eqy
cooked: <p>As far as I can tell they are not really for building different things, but rather for different build systems depending on which you use. And most users seem to use CMake. You can verify that you don’t need BUILD.bazel when building with cmake because you can just delete the file and it will build just fine. Both build the underlying implementation of PyTorch (kernels, implementation of the tensor representation, etc.)</p>
Catching tensor shape errors using the type checker

username: mrahtz
cooked: <p>Hey all!</p>
<p>Tl;dr: come to our talk <strong>Catching Tensor Shape Errors Using the Type Checker</strong> at the PyCon 2021 Typing Summit next week! <a href="https://us.pycon.org/2021/summits/typing/" rel="noopener nofollow ugc">https://us.pycon.org/2021/summits/typing/</a></p>
<p>Longer version: over the past year a small group of us have been thinking about how to use Python’s type system to make shape information more visible, and to catch shape errors automatically using type checkers. In this talk we’ll discuss our tentative solution, talking about a) how it works, b) some examples of what it looks like in practice, and c) what its limitations are.</p>
<p>Hope to see you there!<br>
Matthew</p>

username: JuanFMontesinos
cooked: <p>It’s only for 3.9 right?</p>
Weight of layer as a result of dot operation

username: jorgeorozco
cooked: <p>Hi, I’m trying to implement a network, in which the weights of the layers should be calculated as a result of a tensor operation. This is the code I have:</p>
<pre><code class="lang-auto">class NOWANet(nn.Module):
    def __init__(self, V, Wstacked):
        super(NOWANet, self).__init__()

        # first conv layer
        # define layer
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)

        # define tensors that will be used to calculate weights and biases
        self.conv1_W_V = nn.Parameter(V[0], requires_grad=True)
        self.conv1_B_V = nn.Parameter(V[1], requires_grad=True)
        self.conv1_W_stack = nn.Parameter(Wstacked[0], requires_grad=False)
        self.conv1_B_stack = nn.Parameter(Wstacked[1], requires_grad=False)
        
        # set layer weights and bias
        self.conv1.weight = nn.Parameter(torch.tensordot(self.conv1_W_V, 
                                                         self.conv1_W_stack, 
                                                         dims=1, out=None),
                                         requires_grad=True)
        self.conv1.bias = nn.Parameter(torch.tensordot(self.conv1_B_V, 
                                                       self.conv1_B_stack, 
                                                       dims=1, out=None),
                                       requires_grad=True)
</code></pre>
<p>The idea is to perform a tensordot between the V and W_stack tensors, which then should be used as weights and bias (one tensordot each). The catch is that I only want to optimize V so that W_stack remains the intact. The way it is currently written initializes the weights correctly, but backprop optimizes over the actual weights of the layer, there is no change to the V vector.</p>
<p>Any ideas or suggestions on how to do it?</p>

username: JuanFMontesinos
cooked: <p>Use the functionals instead of the convolutional module. Functionals takes weights as inputs.</p>
Softmax_backward C implementation data type switching/casting

username: Mohamed_Ahmad
cooked: <p>Hey everyone, I’ve been trying to write up my own implementations in Numpy for some of the ops handled  by Pytorch, I was writing my an implementation for softmax_backwards and it just ended up looking like this</p>
<pre><code>    dx = output * grad
    s = dx.sum(axis=dimension, keepdims=True)
    temp = grad - s
    out = output * temp
</code></pre>
<p>Here output is just the output of Softmax itself and grad is the incoming gradient, and dimension is just whatever was supplied as the dim argument to softmax initially<br>
. I believe this implementation lines up with what eventually happens in <code>pytorch/aten/src/ATen/native/SoftMax.cpp</code></p>
<p>However I couldn’t match my output with the Pytorch generated one. Eventually I noticed that in the torch way of doing it in Softmax.cpp, defined in host_softmax_backward()I noticed that the output it returns ends up having the same data type as the grad input (which in this case lets say a float). However in the calculations happening in the function itself some of the math end us as doubles and when you assign it to the output It just gets casted down to a float and returned. In that sense if I change my attempted implementation of softmax_backwards to look like this</p>
<pre><code>    dx = output * grad
    s = dx.sum(axis=dimension, keepdims=True, dtype=np.float64)
    temp = grad - s
    out = output * temp
    out = out.astype(grad.dtype)
</code></pre>
<p>Then the numbers match up perfectly, so I guess I was wondering is this whole input_float—&gt;intimidated_calculation_double—&gt;output_float switching and then casting data types intentional behaviour?</p>
<p>Thanks!</p>
How do I delete the computation graph after each trainloader batch?

username: ProGamerGov
cooked: <p>So, I’m trying to make sure that the computation graph is deleted after processing each batch, but none of the stuff I’ve tried seems to work:</p>
<pre><code class="lang-auto">model.eval()

for inputs, _ in loader:
      outputs = model(inputs)
      # Do stuff with outputs that requires grad
      x = some_func(outputs).detach()

      # Somehow delete computation graph

      del outputs
      model.zero_grad() # this doesn't seem to fix the memory problem.
</code></pre>
<p>Do I have to include <code>backward()</code> even though I don’t need to use it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Deleting all the Tensors that reference the graph is enough to free it.<br>
In your case, the del outputs should do the trick. How do you know the computational graph is kept around?</p>
<p>Note that zeroing the gradients does not remove the .grad field, it just zeros them. So the .grad attributes will still consume some memory. You can give <code>set_to_none=True</code> to the zero grad function to actually free these TEnsors.</p>

username: Brando_Miranda
cooked: <p>does deleting all Tensors that reference the graph enough to free it from my GPU too or does your suggestion only free the computers memory?</p>
<p>I am seeing different errors. The most common with a batch size that is “normal” e.g. 16 or more I get this error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 3; 11.93 GiB total capacity; 11.34 GiB already allocated; 64.00 KiB free; 11.36 GiB reserved in total by PyTorch)
</code></pre>
<p>I believe this is caused by my giant trees.</p>
<p>At other times my error is simply freezing when my model is computing the forward pass.</p>

username: albanD
cooked: <p>It frees all memory.<br>
Note that there is a caching allocating on the cuda side so you need to use <code>torch.cuda.memory_allocated()</code> to know how much memory is actually used by Tensors.</p>
RuntimeError: `initialize_buckets` must NOT be called during autograd execution

username: xylon
cooked: <p>I wanna use ddp to train my model, while I encounter this issue:</p>
<blockquote>
<p>RuntimeError: <code>initialize_buckets</code> must NOT be called during autograd execution.</p>
</blockquote>
<p>Could anyone give me help ?</p>
<p>This error will appear at the end of the <strong>second epoch</strong>, and the first epoch is good.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62a840a09f2d0bc38e766e85a361f97b11390d6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/62a840a09f2d0bc38e766e85a361f97b11390d6f" title="Snipaste_2021-05-04_16-24-00"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62a840a09f2d0bc38e766e85a361f97b11390d6f.png" alt="Snipaste_2021-05-04_16-24-00" data-base62-sha1="e4LgOWz3Jc7VMyVt0bl5KDmA9c3" width="690" height="468" data-dominant-color="09323C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2021-05-04_16-24-00</span><span class="informations">1129×766 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrzzd
cooked: <p>Could you please share a self-contained reproducible source file for thie problem?</p>

username: rvarm1
cooked: <p>A repro would be helpful, and in addition <a href="https://github.com/pytorch/pytorch/issues/46175" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel cannot handle multi-device runs with non-evenly divisible batch sizes · Issue #46175 · pytorch/pytorch · GitHub</a> might also provide valuable context.</p>
<p>Are you using single GPU per process or multiple GPUs per process? The latter mode is deprecated and no longer maintained by PyTorch and we suggest moving to single GPU per process which is the more performant and supported way going forward.</p>
Where does the torch/csrc/autograd/generated/VariableType.h file move to?

username: xie
cooked: <p>I want to define my own operator for the pytorch to onnx, and the <a href="https://pytorch.org/docs/master/onnx.html" rel="nofollow noopener">docs</a> shows that we can find the declaration of the function in torch/csrc/autograd/generated/VariableType.h for an ATen operator,but I can’t find the file in the master version, and where does the file move to?</p>

username: xie
cooked: <p>OK，this problem is solved. I have downloaded the pytorch of C++ installed vesion which the file VariableType.h is in</p>

username: JP-Lisn
cooked: <p><a class="mention" href="/u/xie">@xie</a>, also running into this problem, but don’t quite understand what you did to resolve the issue. If you have a sec could you elaborate?</p>
<p>Would be very grateful, thanks in advance.</p>
How to find the unused parameters in network

username: kealennieh
cooked: <p>When using torch.nn.parallel.DistributedDataParallel to train network, I’ve got " please add <strong>find_unused_parameters=True</strong> into DistributedDataParallel " error. After adding this flag into DistributedDataParallel,  I can train the network normally.</p>
<p>I know the reason why the error occurs is that the network has some unused parameters. <strong>I wonder whether we have some tools to find what they are.</strong></p>

username: e0357894
cooked: <p>Hi,</p>
<p>I have the same issues and I need to find those unused parameters also. Please let me know if you have got any solutions.</p>
<p>Thanks</p>

username: Xin_Qiu
cooked: <p>hi, nihao.<br>
i have the same issues too. i trained the model a few step and torch.save the model to compare the parameters, and find the one not be updated(unused).<br>
for example:</p>
<pre><code class="lang-auto">import torch
sd1 = torch.load("./work_dir/step_1.pth")["state_dict"]
sd4 = torch.load("./work_dir/step_5.pth")["state_dict"]
for k in sd1:
    v1 = sd1[k]
    v4 = sd4[k]
    if (v1==v4).all():
        print(k)
</code></pre>

username: e0357894
cooked: <p>Hi~,</p>
<p>I moved all the trainable parameters to the forward pass and then the problem got solved. Hope this help you.</p>
<p>Regards,</p>

username: faizan
cooked: <p>An easy way to find unused params is train your model on a single node without the DDP wrapper. after <code>loss.backward()</code> and before <code>optimizer.step()</code> call add the below lines</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    if param.grad is None:
        print(name)
</code></pre>
<p>This will print any param which did not get used in loss calculation, their grad will be None.</p>

username: Philipflyg
cooked: <p>Very good method!!!  thank you very much!!!</p>
TorchScript object detection model is slower

username: blackberry
cooked: <p>I have compiled an SSD-based object detection model in PyTorch with <code>torch.jit.script(model)</code>. I benchmarked the scripted and the original models on Tesla K80 GPU (AWS p2 instance). Looks like the scripted model is slower than the original model.</p>
<p>Averaged over 100 images:<br>
Original model: 0.1787 seconds per image<br>
Scripted model: 0.1928 seconds per image</p>
<p>I also benchmarked a ResNet50 model, got similar slow-down.<br>
Original ResNet50: 0.0281<br>
Scripted ResNet50: 0.0303</p>
<p>I was expecting some speed-up, and disappointed by the slow-down.<br>
Is this normal, or could I have missed something?</p>

username: ptrblck
cooked: <p>Could you share the code you’ve used to profile the code?<br>
Note that CUDA operations are executed asynchronously, so you would have to synchronize the timer via <code>torch.cuda.synchronize()</code> before starting and stopping the timers.</p>

username: blackberry
cooked: <p>I am aware of the synchronization. I measure by averaging over 100 forward passes (no difference with/without synchronization). If I use <code>with torch.jit.optimized_execution(False):</code> I get similar time as the original model.</p>
<pre><code class="lang-auto">model = torchvision.models.resnet50(pretrained=False).cuda()
model.eval()
model = torch.jit.script(model)

# Disable all gradient computations 
torch.set_grad_enabled(False)

# Load &amp; transform the image

# warm-up
embedding = model(image)

t1 = time.time()
N = 100
for i in range(N):
   embedding = model(image)
t2 = time.time()
t = t2 - t1
print('Time: {:.4f}, avg: {:.4f}'.format(t, t / N))
</code></pre>
<p>I wonder if TorchScript is sensitive to GPU architecture (faster on the recent GPUs, but slower on old GPUs?). I have not done the same comparison on the new GPUs yet (in any case, I am interested in the old GPUs as they will be used in prod).</p>

username: 111480
cooked: <p>I have the same experience now. Did you find a reason?</p>

username: blackberry
cooked: <p>Not yet. I am guessing it might be related to two things: (1) Model’s network architecture, (2) Type of GPU. I will update here if/when I find a definitive answer.</p>
Nan values with log_prob()

username: nolwen_brosson
cooked: <p>Hello,</p>
<p>I’m having a lot of NaN values with the log_prob function.</p>
<p>See the code:</p>
<pre><code class="lang-auto">def forward(self, observation):
        # Convert tuple into tensor
        observation_as_list = []
        observation_as_list.append(observation[0])
        observation_as_list.append(observation[1])
        observation_as_list = np.asarray(observation_as_list)
        observation_as_list = observation_as_list.reshape(1,2)
        observation = observation_as_list
        
        state = torch.from_numpy(observation).float()
        state = F.tanh(self.affine(state))
        
        state_value = self.value_layer(state)
        weight_prelu = torch.tensor([[5.0]])
        action_parameters = (self.action_layer(state)) # A voir quelle activation function mettre
        action_distribution = Normal(action_parameters[0][0], action_parameters[0][1])
        
        action = action_distribution.sample() # Torch.tensor; action
        self.logprobs.append(action_distribution.log_prob(action+ 1e-6)) # HERE ###
        self.state_values.append(state_value)
        #print(action.item())
        return action.item() # Float element
</code></pre>
<p>The whole code is here: <a href="https://github.com/nbrosson/Actor-critic-MountainCar/blob/master/agent.py" rel="nofollow noopener">https://github.com/nbrosson/Actor-critic-MountainCar/blob/master/agent.py</a></p>
<p>Any ideas how I’m getting this problem?</p>
<p>Thanks !!</p>

username: Pratik_Kulkar
cooked: <p>Hello, I am also facing the same problem ! have you found the solution to it ?</p>
Indexing and retrieving tensor rows from a multidimensional tensor

username: asabater
cooked: <p>I have a source multidimensional tensor of shape <code>(a,b,c,c,d)</code> which stores vectors/data of size <code>d</code>, and another tensor of shape <code>(a,b,e,2)</code> that stores <code>e</code> indices of size 2. 2-dimensional values correspond to the indices 2-3 of the data tensor (both dimensions of size <code>c</code>). Note that both tensors share the same <code>a,b</code> dimension sizes.</p>
<p>What I want to do is to use these indices to retrieve <code>e</code> rows of size <code>d</code> from the first tensor. So that, the output tensor should have size <code>(a,b,e,d)</code>, i.e. <code>e</code> vectors of size <code>d</code> along the <code>a,b</code> dimensions.</p>
<pre><code class="lang-auto">a, b, c, d = 3,5,7,9
e = 11
data = torch.rand(a,b,c,c,d)
inds = torch.randint(0,c, size=(a,b,e,2))
    
res = data[:, :, inds[:,:,:,0], inds[:,:,:,1],:]
print(' - Obtained shape:', res.shape) 
print(' - Desired shape:', (a,b,e,d))

# - Obtained shape: (3, 5, 3, 5, 11, 9)
# - Desired shape: (3, 5, 11, 9)
</code></pre>