akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to quickly generate a random permutation moving each element with a distance less than K?

username: 111414
cooked: <p>I know the following PyTorch API can perform a global random shuffle for 1D array [0, … , n-1]:</p>
<pre><code class="lang-python">torch.randperm(n)
</code></pre>
<p>but I’m confused on how to quickly generate a random permutation, such that each element of the shuffled array satisfying:</p>
<pre><code class="lang-python">K = 10  # should be positive
shuffled_array = rand_perm_func(n, K)  # a mysterious function
for i in range(n):
    print(abs(shuffled_array[i] - i) &lt; K)  # should be True for each i
</code></pre>
<p>which means that each element is moved with a distance less than K. Does there exists a fast implementation for 1D arrays?</p>

username: googlebot
cooked: <p>hm, perhaps:<br>
r = arange(n).expand(num_arrays,-1)<br>
scores = r + randint_like(r, high=k)<br>
r2 = argsort(scores, dim=-1)</p>

username: 111414
cooked: <p>It seems that this problem is very difficult… I just read this webpage: <a href="https://stackoverflow.com/questions/30747690/controlling-distance-of-shuffling" class="inline-onebox" rel="noopener nofollow ugc">python - Controlling distance of shuffling - Stack Overflow</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: 111414
cooked: <p>The following code works:</p>
<pre><code class="lang-python">import torch

# randomly produces a 1-D permutation index array,
# such that each element of the shuffled array has
# a distance less than K from its original location
def rand_perm(n, K):
    o = torch.arange(0, n)
    if K &lt;= 1:
        return o
    while True:
        p = torch.randperm(n)
        d = abs(p - o) &lt; K
        if bool(d.all()):
            return p

if __name__ == '__main__':
    for i in range(10):
        print(rand_perm(10, 2))
</code></pre>
<p>but it seems that when n is large, and K is small, the generation will take a very long time. Does there exists a more efficient implementation?</p>

username: 111414
cooked: <p>The following code is inspired by <a class="mention" href="/u/googlebot">@googlebot</a>:</p>
<pre><code class="lang-python">def rand_perm(n, K):
    o = torch.arange(0, n)
    d = torch.randint_like(o, low=-K, high=K + 1)
    return torch.argsort(o + d)
</code></pre>
<p>but it does not actually meet my demands. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: 111414
cooked: <p>I am going to use the following code first and see if it works: <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>
<pre><code class="lang-python">def rand_perm(n, K):
    o = torch.arange(0, n).float()
    d = (torch.rand_like(o) - 0.5) * K
    return torch.argsort(o + d)
</code></pre>
How is default gradient parameter is passed to backward fuction?

username: AbishekBashyal
cooked: <p>In many blogpost and discussion section it is said that by default <code>F.backward()</code> is <strong>equivalent</strong> as <code>F.backward(gradient=torch.Tensor([1.]))</code>.<br> But looking at the implementation of <strong>tensor.py</strong> and <strong>autograd.backward()</strong> implementation the default value for external gradient is <code>none</code> i.e</p>
<pre><code class="lang-python">#tensor.py
def backward(self, gradient=None, retain_graph=None, create_graph=False):
    ...
</code></pre>
<pre><code class="lang-python">#autograd.backward
def backward(
    tensors: _TensorOrTensors,
    grad_tensors: Optional[_TensorOrTensors] = None,
    ....
):
    ...
</code></pre>
<p>So, when is the <code>gradient</code> set to default <code>torch.Tensor([1.])</code> ??<br>
Also, what will be difference telling that:<br> <br> 1) <code>F.backward()</code> is <strong>equivalent</strong> as <code>F.backward(gradient=torch.Tensor([1.]))</code><br>
2) <code>F.backward()</code> is <strong>equal</strong> as <code>F.backward(gradient=torch.Tensor([1.]))</code></p>

username: tom
cooked: <p>These blogposts should really use <code>torch.ones_like(loss)</code> (but only for scalar-valued loss) even when ignoring the memory layout to keep things simple. <code>torch.autograd.backward</code> will call an auxiliary function<br>
<a href="https://github.com/pytorch/pytorch/blob/7caa464631c1fcc9f9a4290cf0fe7913c36d1796/torch/autograd/__init__.py#L51" rel="noopener nofollow ugc"><code>_make_grads</code></a> that creates those if they are not passed in.<br>
Having a <code>None</code> formal default argument and then somehow computing the “effective default” is a common pattern in Python much more generally applicable than just PyTorch. Sometimes - like here - it is because the default isn’t fixed (depends on the device, memory layout) or because the default is mutable and so you want to instantiate a new object every time the function is called (blog posts on “mutable default arguments” seem to be very common, too).</p>
<p>I’m guessing “equivalent” means “gives you the same result as” and it is a bit weaker than equal which is more “does literally the same thing”.</p>
<p>Best regards</p>
<p>Thomas</p>

username: AbishekBashyal
cooked: <p>thanks, Thomas :D, Its clear now!</p>
ResnetBlocks/Skip Connections only in deepest layers

username: Carsten_Ditzel
cooked: <p>Recently I see a lot of convolutional encoder/decoder architectures which only after downsampling within several layers, appernd resnet blocks. In other words, instead of using skip connection from the beginning like in traditional ResNet architectures those are only used when the spatial extend of the input has already been shrunk by a factor of 4 or 8. Surprisingly the result quality seems to be on par if not succeeding the skip-connections all the way approach. Is there some intuition behind this or can someone shed some light on this for me?</p>
Convolution without summation?

username: Shiny_Sheepy
cooked: <p>Hi there,</p>
<p>Is there any way to perform a convolution but without summation? Concretely, assume we have a Bx3x5x5 input and a 3x3x3 kernel, when we do a conv2d operation, we get a Bx1x4x4 output. For <strong>each conv-ed kernel</strong>, the result (Bx3x3x3) is summed both over H,W dimension and C dimension and produces a Bx1x1x1 scalar, and performing this operation over all locations gives the final Bx1x4x4 output. Is there any way to get the intermediate Bx3x3x3 results for <strong>each conv-ed kernel</strong>? i.e. without performing the summation over H,W, and C dimension, and thus the final result has a size of Bx3x3x3x4x4.</p>
<p>I know there is a <code>groups</code> option in conv2d, and by setting <code>groups=channel_in</code>, we essentially prevent the summation over the C dimension, but how can we prevent the summation over the H and W dimension?</p>
<p>Thanks</p>

username: tom
cooked: <p>(Edit: I wrote Unbind first, but it needs to be torch.nn.Unfold, thank you for pointing this out, I changed it here).</p>
<p>torch.nn.Unfold gets you the patches that you could feed into matrix multiplication to get the convolution result (after reshaping).<br>
If you then use pointwise multiplication with broadcasting, you should roughly get what you want (albeit in a different shape).</p>
<p>The torch.nn.Unfold documentation has an example of how to mimic convolution with torch.nn.Unfold and matmul.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Shiny_Sheepy
cooked: <p>Hi Thomas,</p>
<p>Thanks for your reply. I took a look at the torch.nn.unbind <a href="https://pytorch.org/docs/stable/generated/torch.unbind.html" rel="noopener nofollow ugc">documentation</a> but didn’t see the example you mentioned. Could you paste a link to it?<br>
Thanks.</p>
<p>Regards</p>
<p>Charlie</p>

username: tom
cooked: <p>Oh, I’m sorry, I meant <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html" rel="noopener nofollow ugc"><code>torch.nn.Unfold</code></a>!</p>
Running PyTorch model on Metal

username: HashedViking
cooked: <p>Trying to run FOM model on iOS but got following error</p>
<pre><code class="lang-auto">2021-03-22 19:43:00.122800+0300 Test[4525:1169258] Metal GPU Frame Capture Enabled
2021-03-22 19:43:00.123248+0300 Test[4525:1169258] Metal API Validation Enabled
2021-03-22 19:43:00.242356+0300 Test[4525:1169477] fopen failed for data file: errno = 2 (No such file or directory)
2021-03-22 19:43:00.242415+0300 Test[4525:1169477] Errors found! Invalidating cache...
2021-03-22 19:43:00.269292+0300 Test[4525:1169477] fopen failed for data file: errno = 2 (No such file or directory)
2021-03-22 19:43:00.269342+0300 Test[4525:1169477] Errors found! Invalidating cache...
sample at time CMTime(value: 0, timescale: 1000, flags: __C.CMTimeFlags(rawValue: 1), epoch: 0)
2021-03-22 19:43:02.301383+0300 Test[4525:1169258] The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/modules/fom/___torch_mangle_305.py", line 53, in forward
    input0: Tensor,
    input1: Tensor) -&gt; Tensor:
    out = torch.constant_pad_nd(input, [6, 6, 6, 6], 0)
          ~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    _0 = getattr(self, "prepack_folding._jit_pass_packed_weight_0")
    _1 = ops.metal_prepack.conv2d_run(out, _0)

Traceback of TorchScript, original code (most recent call last):
/python3.8/site-packages/torch/nn/functional.py(3997): _pad
development/first-order-model/first-order-model-my/modules/util.py(241): forward
/python3.8/site-packages/torch/nn/modules/module.py(860): _slow_forward
/python3.8/site-packages/torch/nn/modules/module.py(887): _call_impl
development/first-order-model/first-order-model-my/modules/keypoint_detector.py(51): forward
/python3.8/site-packages/torch/nn/modules/module.py(860): _slow_forward
/python3.8/site-packages/torch/nn/modules/module.py(887): _call_impl
development/first-order-model/first-order-model-my/modules/fom.py(69): forward
/python3.8/site-packages/torch/nn/modules/module.py(860): _slow_forward
/python3.8/site-packages/torch/nn/modules/module.py(887): _call_impl
/python3.8/site-packages/torch/jit/_trace.py(934): trace_module
/python3.8/site-packages/torch/jit/_trace.py(733): trace
development/first-order-model/first-order-model-my/convert.py(75): make_animation
development/first-order-model/first-order-model-my/convert.py(112): &lt;module&gt;
RuntimeError: 'memory_format' argument is incompatible with Metal tensor
</code></pre>
<p>On CPU backend model works fine. How can I fix or bypass this issue? <a class="mention" href="/u/xta0">@xta0</a></p>
Add memory to chatbot

username: Rey
cooked: <p>I’m new to programming, so forgive me if this is a stupid question.</p>
<p>Currently I’m working on this tutorial: <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Chatbot Tutorial — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>But I want the chatbot remembering what I say to it, so it can give me recommendations based on my previous input.<br>
I’d really appreciate your answer.</p>
Hessian of output image pixels w.r.t. input image

username: Jeffrey_Lai
cooked: <p>Hello,</p>
<p>I have inputs of size (Bx2xHxW) and outputs of size (Bx1xHxW). In my problem, the channel dimension represents spatial coordinates e.g. (x, y) or (x, y, z) etc. I’m trying to create a laplacian ‘matrices’ of shape (Bx1xHxW) where for each image, each pixel represents the laplacian at that point in the original image. This means that for every pixel (i, j), I basically need the trace of the hessian of the output[:, :, i, j] w.r.t. input[:, :, i, j].</p>
<p>For now i’m trying to get this to work for a single image (B=1) and the input ‘image’ is a 2x100x100 tensor where the first channel represents the x-coordinate and the second channel represents the y-coordinate. The model for the toy example is X^2 + Y^2 + XY. Thus, the laplacian ‘matrix’ for this example should be a (1x100x100) tensor with the value 4 everywhere. However, im having trouble producing the expected result. Is there a nice way to do this that is not too expensive?</p>
<p>Here’s what I’ve tried so far:</p>
<pre><code class="lang-auto">size = 32
x = torch.linspace(-1, 1, size)
y = torch.linspace(-1, 1, size)
X, Y = torch.meshgrid(x, y)

grid = torch.stack([X, Y], axis=-1).permute(2, 0, 1)

class func2d(nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        return x[0]**2 + x[1]**2 + x[0]*x[1]
    
grid.requires_grad = True
model = func2d()
out = model(grid)

d1 = torch.autograd.grad(
    outputs=[out],
    inputs=[grid],
    grad_outputs=torch.ones_like(out),
    create_graph=True,
    only_inputs=True
)[0]
d2 = torch.autograd.grad(
    outputs=[d1],
    inputs=[grid],
    grad_outputs=torch.ones_like(d1),
    only_inputs=True
)[0]
print(d2)
</code></pre>
Difference between MULBackward, requires_grad and Sum_backward

username: Sumi
cooked: <p>I am trying to find contrastive loss by sending an instance of input using:<br>
loss = criterion(*input_loss)<br>
criterion is contrastive loss.</p>
<p>The following code updates loss:<br>
loss_value = 0.5 * (target.float() * distances + (1 + -1 * target).float() * F.relu(self.margin - (distances + self.eps).sqrt()).pow(2))</p>
<p>Loss_value here is of type (tensor MULBackward) and distance is (tensor Sum_backward)</p>
<p>But when I try to detach the input and find the loss using numpy, the <strong>loss isn’t getting updated</strong>:</p>
<p>out1=output1<br>
out2=output2<br>
out1=out1.detach().numpy()<br>
out2=out2.detach().numpy()<br>
dist=np.sum(np.power(out2 - out1, 2),axis=1)<br>
loss_value = 0.5 * (target.float() * dist +(1 + -1 * target).float() * np.maximum(0,np.power(self.margin - np.sqrt(dist + self.eps),2))<br>
loss_value.requires_grad_()<br>
dist=torch.from_numpy(dist)<br>
dist.requires_grad_()<br>
out1.requires_grad_()<br>
out2.requires_grad_()</p>
<p>I tried to do dist.requires_grad_().sum()  but it doesn’t help. Can someone please help me understand the issue.</p>
Torch.no_grad() has no effect

username: Gwon
cooked: <p>Hi,<br>
I want to extract all features from first convolution Layer in ResNet Model.<br>
So, i construct the codes like below</p>
<pre><code class="lang-auto"># Model file
class ResNet(nn.Module):
    def __init__().__init__()
         .....
        # First Convolution Layer
        self.conv1 = nn.Conv2d(...)
        ....
        # Fully-Connected Layer
        self.fc = nn.Linear(...)

    def forward(self, data):
        .....
    def extract_features(self, data):
        # Extract First Convolution features
        return self.conv1(data)
</code></pre>
<pre><code class="lang-auto"># Implement code
model = ResNet()

with torch.no_grad():
    for batch_idx, (inputs, targets) in enumerate(train_loader):
         outputs = model.extract_features(inputs) # &lt;- It causes memory leakage
        
</code></pre>
<p>Running this code, i encountered an error</p>
<pre><code class="lang-auto">CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.91 GiB total capacity; 11.33 GiB already allocated; 2.88 MiB free; 11.34 GiB reserved in total by PyTorch)
</code></pre>
<p>I know that <code> torch.no_grad()</code>  means do not using gradient options when inference time<br>
But in my code, GPU memories are increase every iteration.</p>
<p>Anybody can help?</p>

username: sio277
cooked: <p>Yes, in the context of <code>torch.no_grad()</code>, no backwards graph is created so the GPU memory allocation must be smaller than without it. Can you provide more code lines saving <code>outputs</code> tensor? Or, check if any other pre-calculated tensors on GPU were freed properly.</p>

username: Gwon
cooked: <p>Thank you for reply.</p>
<p>The more code lines are like below</p>
<pre><code class="lang-auto">features = []
with torch.no_grad():
    for batch_idx, (inputs, targets) in enumerate(train_loader):
         outputs = model.extract_features(inputs) # &lt;- It causes memory leakage
         features.append(outputs)
    # Concat all features
    npy_data = torch.cat((features[:]))
    npy_data = np.array(npy_data.cpu())
    np.save('data_npy/data.npy', npy_data)
</code></pre>
<p>When i extract last convolution layer’s features, GPU memory is not raising<br>
I think, The first convolution layer’s dimension is [-1, 64, 32, 32], last convolution layer’s dimension is [-1, 512, 4, 4].</p>
<p>[-1, 64, 32, 32] is more bigger than [-1, 512, 4, 4]. So I think it is the reason that the problem to occur</p>

username: sio277
cooked: <p>Try to move <code>outputs</code> tensor to RAM before you <code>append</code> it, such as,</p>
<pre><code class="lang-auto">features = []
with torch.no_grad():
    for batch_idx, (inputs, targets) in enumerate(train_loader):
         outputs = model.extract_features(inputs)
         features.append(outputs.cpu())
</code></pre>

username: Gwon
cooked: <p>It works well!</p>
<p>Thank you,</p>
When to use add_module function?

username: Marvin_Teichmann
cooked: <p>The module class contains a function <a href="http://pytorch.org/docs/master/nn.html#torch.nn.Module" rel="nofollow noopener">add_module</a> to initialize submodules. I am a bit confused what the purpose of this function is.</p>
<p>Most people initialize submodules as simple member variables of the supermodule (like in example 1). Is there any case where it is advantageous to initialize supmodules using the <a href="http://pytorch.org/docs/master/nn.html#torch.nn.Module" rel="nofollow noopener">add_module</a> (as in example 2)? What is the recommended way of initializing submodules?</p>
<p>Is there any difference between example 1 and example 2?</p>
<p><em>Example 1: Initialize submodules as Member Variabels</em></p>
<pre><code>class Net(nn.Module):
    def __init__(self):
	super(Net, self).__init__()
	self.conv1   = nn.Conv2d(3, 16, 5, padding=2)
	self.pool    = nn.MaxPool2d(2, 2)
	self.dropout = nn.Dropout2d(p=0.5)
	self.conv2   = nn.Conv2d(16, 16, 5, padding=2)
	self.conv3   = nn.Conv2d(16, 400, 11, padding=5)
	self.conv4   = nn.Conv2d(400, 200, 1)
	self.conv5   = nn.Conv2d(200, 1, 1)
</code></pre>
<p><em>Example 2: Initialize submodules using <code>add_module</code></em></p>
<pre><code>class Net(nn.Module):
    def __init__(self):
    super(Net, self).__init__()
    self.add_module("conv1", Conv2d(3, 16, 5, padding=2))
    self.add_module("pool", MaxPool2d(2, 2))
    self.add_module("dropout", Dropout2d(p=0.5))
    self.add_module("conv2", Conv2d(16, 16, 5, padding=2))
    self.add_module("conv3", Conv2d(16, 400, 11, padding=5))
    self.add_module("conv4", Conv2d(400, 200, 1))
    self.add_module("conv5", Conv2d(200, 1, 1))</code></pre>

username: richard
cooked: <p>I think example 1 and example 2 do the same thing. The <a href="https://github.com/pytorch/pytorch/blob/157f949cefb9b15d01160d2becb16c6572023536/torch/nn/modules/module.py#L368" rel="nofollow noopener">setattr method</a> of <code>nn.Module</code> doesn’t call add_module directly but it does something similar.</p>
<p>In general, you won’t need to call <code>add_module</code>. One potential use case is the following:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
	super(Net, self).__init__()
        modules = [...]  # some list of modules
        for module in modules:
            self.add_module(...)
</code></pre>

username: itisianlee
cooked: <p>it just looks simple ~</p>

username: Jay_Super
cooked: <p>Well, I think example 2 provides more simplicity when you need to add the same module using for loop.<br>
Let me know if there are other use cases!</p>
<pre><code class="lang-python">for l in range(3):
     self.add_module(f'networkA_{l}'}
     self.add_module(f'networkB_{l}'}
</code></pre>
Reproducibility typo

username: digivoro
cooked: <p>To whom it may concern:</p>
<p>There’s a typo in the <a href="https://pytorch.org/docs/stable/notes/randomness.html#python" rel="noopener nofollow ugc">Reproducibility</a> page</p>
<p>It says “inport”<br>
It should say “import”</p>
<p>(I didn’t know where else to put this)</p>
<p>Greetings! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Deterministic PRNG across CPU + CUDA?

username: turian
cooked: <p>What is a <em>fast</em> deterministic PRNG for torch that gives near identical results on CUDA + CPU?</p>
<p>Unfortunately, torchcsprng is a bottleneck in our code, and we are looking for other deterministic, non-crypto-secure pseudo-random number generation. However, even CUDA and CPU give different results:</p>
<pre><code class="lang-auto">import torch
torch.use_deterministic_algorithms(True)
torch.cuda.manual_seed(0)
torch.rand(3,3, device="cuda")

tensor([[0.3990, 0.5167, 0.0249],
        [0.9401, 0.9459, 0.7967],
        [0.4150, 0.8203, 0.2290]], device='cuda:0')
</code></pre>
<pre><code class="lang-auto">import torch
torch.use_deterministic_algorithms(True)
torch.manual_seed(0)
torch.rand(3,3)

tensor([[0.4963, 0.7682, 0.0885],
        [0.1320, 0.3074, 0.6341],
        [0.4901, 0.8964, 0.4556]])
</code></pre>
<p>Related to: <a href="https://discuss.pytorch.org/t/reproducibility-over-different-machines/63047" class="inline-onebox">Reproducibility over Different Machines</a></p>
Impliment a Cost Matrix in Pytorch

username: Yelnat
cooked: <p>Hey guys,</p>
<p>I’m kinda new to pytorch/ML in general and haven’t tried writing my own loss function before, I’m looking at <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387/9" class="inline-onebox">Custom loss functions - #9 by ptrblck</a> and it doesn’t seem too difficult. My specific question is to do with implementing a cost matrix which is what I think <a href="https://discuss.pytorch.org/t/how-to-make-cost-function-out-of-confusion-matrix/38138" class="inline-onebox">How to make cost function out of confusion matrix?</a> is trying to do but it seems confusing.</p>
<p>Specifically I’m training a EfficientNet on the ISIC 2019 dataset and wanted to ask if there is any specific standard way for doing this in pytorch?</p>
<p>For those of you who don’t know a confusion matrix is something looks like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1423f0662593fee4253b5f7157c65280b8d88af.png" alt="image" data-base62-sha1="n0yNSHkTQGRW4pLVE6xaJcfb6dp" width="283" height="75"></p>
<p>Using this matrix I would multiply the loss by a factor of 10 if my model predicted ‘not faulty’ when it was in fact ‘faulty’, and multiply the loss by only a factor of 1 if it predicted something that was ‘not faulty’ as ‘faulty’ (and multiply by 0 if it got it right). My cost matrix won’t look like this exactly, as mine is on skin lesions and ISIC has 8 different unbalanced classes of skin cancer, and some classifications are more dangerous than others, which is why it’s important that my model learns which incorrect predictions are more costly.</p>
<p>Any help would be appreciated</p>

username: M_J_A
cooked: <p>Hi, did you find a way to solve this? I have a similar problem. Thanks!</p>

username: Yelnat10
cooked: <p>Hey, sorry bout that, Forgot I made this post (and then subsequently my password). The way I’m implementing my cost matrix is not by building it into the loss function but instead by using the method defined in Bishops book “Pattern Recognition And Machine Learning”</p>
<p>Bishop gives us this formula to implement a cost matrix:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52b54b104e0f47fce0b5baca0b015a18699f2a54.png" alt="image" data-base62-sha1="bNFzqRpIJYnfTikuPE1IlNBWhH6" width="502" height="198"></p>
<p>This implementation effectively balances a softmax output of a neural network with the cost associated with misclassification.</p>
<p>e.g. lets say I have some probability distribution (0.35, 0.65) that is the softmaxed output from my network, so in this case we’re predicting 35% faulty and 65% not faulty. However, using Bishops sum I would get the Expected cost values of ((0.35 * 0 + 0.35 * 1), (10 * 0.65 + 0.65 * 0)) = (0.35, 6.5).</p>
<p>From these two expected cost values, 0.35 and 6.5, we choose the one that gives us the lowest expected cost, in this cause we predict faulty.</p>
<p>My explanation is likely rather poor, I believe Bishop talks about this in chapter 1.52 if you want to check that out.</p>
Can I run two subnetworks of one model parallelly?

username: tlsshh
cooked: <p>I have a network with four subnetworks: A_model, B_model, C_model, D_model</p>
<p>During forwarding,</p>
<pre><code class="lang-auto">  out_A = A_model(input) 
  out_B = B_model(out_A)
  out_C = C_model(out_A)
  out_D = D_model(out_B, out_C)
</code></pre>
<p>B_model and C_model both take out_A as input, but they run independently.</p>
<p>To my understanding, PyTorch will run these lines serially, right? If B_model and C_model can run in parallel (e.g., multi-threading), will it save a lot of time?</p>

username: Enamex
cooked: <p>Very, very, <em><strong>very</strong></em> interested in this! Anyone?<br>
I’m hoping there would be a way to parallelize this somehow on <em>one</em> GPU (which should generalize to multi-GPU and multi-node). But lack the details to judge if that’s possible.</p>

username: googlebot
cooked: <p>Briefly, you need either multiple torch.cuda.stream contexts, or jit.fork (in jit compiled code) to also separate cpu threads (that enqueue cuda operations). Unfortunately, speedup on one gpu may be limited, if gpu utilization in affected code is already high, or code fragments are small.</p>
[dataparallel] Trained on one GPU but Inference used on multiple GPUs

username: jS5t3r
cooked: <p>I have trained ResNet50 on one GPU… Yes, I was reading other topics using DataParallel, but I have not found any solution for this kind of training…<br>
Later, I use the model in an other way and I want to use multiple GPUs, but I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "attack.py", line 57, in &lt;module&gt;
    model.load_state_dict(torch.load(settings.CHECKPOINT_PATH))
  File "/home/user/.conda/envs/bm/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
        Missing key(s) in state_dict: "module.conv1.0.weight", "module.conv1.1.weight", "module.conv1.1.bias", "module.conv1.1.running_mean", "module.conv1.1.running_var", "module.conv2_x.0.residual_function.0.weight", "module.conv2_x.0.residual_function.1.weight", "module.conv2_x.0.residual_function.1.bias",
</code></pre>

username: sio277
cooked: <p>I guess you had saved your DataParallel-wrapped model, and tried to load it before DP wrapping (keys were mismatched while loading and I guess it’s due to the ‘module’ attribute, where your original model is placed after DP construction). Try to save your model after unwrapping DP, such as,</p>
<pre><code class="lang-auto">state = getattr(model, 'module').state_dict()
torch.save(state, file)
</code></pre>

username: jS5t3r
cooked: <p>Sorry, for that I confused you. I trained the model on a single GPU without DP wrapping.<br>
Now, I need the DP wrapping for executing the trained model on multiple GPUs.</p>

username: sio277
cooked: <p>Then, try to load your model before DP construction.</p>
Seq2seq with lstm help

username: annamae
cooked: <p>has anyone implemented this tutorial with an lstm?<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a><br>
I have tried to implement it with an lstm but i am getting an error.  Below is my code.</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(Encoder, self).__init__()
        self.hidden_size = hidden_size

        self.word_embed = nn.Embedding(input_size, hidden_size)
        self.pos_embed = nn.Embedding(input_size, hidden_size)
        self.lstm = nn.LSTM(hidden_size, hidden_size)

    def forward(self, words, hidden):
        word_embed = self.word_embed(words).view(1, 1, -1)
        output = torch.cat([word_embed, pos_embed], 2)
        output, hidden = self.lstm(output, hidden)
        return output, hidden

    def initHidden(self):
        return (torch.zeros(1, 1, self.hidden_size, device=device), torch.zeros(1, 1, self.hidden_size, device=device))

class AttnDecoder(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoder, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.lstm = nn.LSTM(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        attn_weights = F.softmax(
            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
        attn_applied = torch.bmm(attn_weights.unsqueeze(0),
                                 encoder_outputs.unsqueeze(0))

        output = torch.cat((embedded[0], attn_applied[0]), 1)
        output = self.attn_combine(output).unsqueeze(0)

        output = F.relu(output)
        output, hidden = self.lstm(output, hidden)

        output = F.log_softmax(self.out(output[0]), dim=1)
        return output, hidden, attn_weights
</code></pre>
<p>I am getting the error</p>
<pre><code class="lang-auto">self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
RuntimeError: Tensors must have same number of dimensions: got 2 and 3
</code></pre>
<p>Thanks!</p>
'forward_pre_hook' on module causes CUDA OOM errors

username: sio277
cooked: <p>Hi.<br>
I tried to add zero padding before particular convolution layers, but when I tried the below code, CUDA OOM occurs after ~40 epochs (without the pre hook, no OOM). I used <code>forward_pre_hook</code> as follows:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
from torch.nn.utils import spectral_norm


def conv_layer(inplanes, planes, kernel_size, stride, padding, bias, do_spectral_norm):
    conv = nn.Conv2d(in_channels=inplanes, out_channels=planes, kernel_size=kernel_size, stride=stride,
                     padding=padding, bias=bias)

    if conv.kernel_size == (3, 3) and conv.stride == (2, 2) and conv.padding == (1, 1):
        conv.padding = (0, 0)

        def forward_pre_hook(module, input):
            print('forward_pre_hook has been fired.')
            modified_input = F.pad(input[0], pad=(0, 1, 0, 1))
            return modified_input

        conv.register_forward_pre_hook(forward_pre_hook)

    if do_spectral_norm:
        return spectral_norm(conv)
    else:
        return conv
</code></pre>
<p>This <code>conv_layer</code> function is called several times in the main module.</p>
<p>Question1.<br>
After I turned on the <code>forward_pre_hook</code>, increment of allocated GPU memory was observed (18116MB to 21408MB). I think this is a natural behavior as the extra backwards graph corresponding to the added padding node would be created. However, as the training proceeds, it increased to 24146MB and 24154MB - here is I do not understand. How can I stop the memory leak? (Note that, I set cudnn.deterministic as True and cudnn.benchmark as False, for exact observation of the memory.)</p>
<p>Question2.<br>
Or, <code>spectral_norm</code> can be the problem? As far as I know, the spectral_norm implementation uses <code>forward_pre_hook</code> as well and duplication of those hooks can be the source of problem? (FYI, I don’t fully know about the details inside  <code>torch.nn.utils.spectral_norm</code>).</p>
<p>Any suggestions are welcome. Thank you.</p>
<p>Torch version: 1.7.1<br>
Python version: 3.7.4</p>
Re-training autoencoder while ensuring new models are different to last

username: Peter_Featherstone
cooked: <p>I have a pair of models, A1 and B1 that I train on some data on date 1. How can i train a new pair A2 and B2 on date 2, such that the combinations (A1,B2) and (A2,B1) are guaranteed to fail?<br>
Currently if i have some loss L(A,B) which i minimize, i update it as follows:<br>
L → 2*L(A2,B2) - L(A1,B2) - L(A2,B1)<br>
So when training A2 and B2, i load A1 and B1, freeze their weights and i ensure that their parameters are not added to the optimizer. So i have 4 models loaded in total, but only 2 are training.<br>
That seems to work during training, but not during evaluation. Like training accuracy is fine but evaluation accuracy goes down the toilet. Is there a known way to do this?</p>

username: Peter_Featherstone
cooked: <p>For example, if you are training a new auto-encoder, given the same input, how can you ensure the code, i.e. the output of the encoder, is maximally different to the output of the first trained auto-encoder.</p>

username: Peter_Featherstone
cooked: <p>Ok, the whole thing about training accuracy being “good” and evaluation accuracy being “bad” is to do with labels. It’s like training a binary classifier where the first half of a batch is label 0 and the second half of the batch is label 1. You will find that training accuracy goes up, but evaluation accuracy is almost always close to 0. It’s the same kind of behavior here. You need to randomize your batch and therefore your labels. By using the loss L → 2*L(A2,B2) - L(A1,B2) - L(A2,B1), I’m passing all “good” labels in one batch and all “bad” labels in another. So the workaround is to have another loss function where you can minimize the following L = L1(randperm(cat(A1,A2)), B2) + L2(A2,randperm(cat(B1,B2)). Something like that. You need to randomize the batch when concatenating A1 and A2, or B1 and B2. You need to appropriately define L1 and L2 for your problem.</p>
<p>So having done all of that, you can train (A2,B2) to be maximally different to (A1,B1). And likewise you can train (A3,B3) to be maximally different to (A2,B2). But if you only ever load 4 models, 2 of which are frozen and the other 2 are training, there is no guarantee that (A3,B3) is maximally different to (A1,B1). Your models could be theoretically oscillating between two local minima. And there is a limit to how many models you can load at once. In theory you could load all of (A1,B1), (A2,B2) and (A3,B3) and make (A3,B3) different to the last two pairs, but not everyone has 24GB of GPU memory.</p>
<p>So, …, it would be nice if there was an easy and computationally friendly way to guarantee different models every time you trained them. What is it?</p>
When using LSTM, how can the output of the previous LSTMCell become the input of the next LSTMCell?

username: Sjk_Jae
cooked: <pre><code class="lang-auto">max_length = 20

# Sample from a category and starting letter
def sample(category, start_letter='A'):
    with torch.no_grad():  # no need to track history in sampling
        category_tensor = categoryTensor(category)
        input = inputTensor(start_letter)
        hidden = rnn.initHidden()

        output_name = start_letter

        for i in range(max_length):
            output, hidden = rnn(category_tensor, input[0], hidden)
            topv, topi = output.topk(1)
            topi = topi[0][0]
            if topi == n_letters - 1:
                break
            else:
                letter = all_letters[topi]
                output_name += letter
            input = inputTensor(letter)

        return output_name

# Get multiple samples from one category and multiple starting letters
def samples(category, start_letters='ABC'):
    for start_letter in start_letters:
        print(sample(category, start_letter))

samples('Russian', 'RUS')

samples('German', 'GER')

samples('Spanish', 'SPA')

samples('Chinese', 'CHI')
</code></pre>
<p>Like this example, it can control all parts of the recursive network. However, when using LSTM directly, you cannot control the relationship between input and output before and after LSTM like this example. How can I directly use LSTM to realize an example that the name can be output by a letter?</p>
One Network for multiple predictions

username: moreshud
cooked: <p>Hi,</p>
<p>I am currently experimenting with an approach of using a single Unet network for 2 predictions for image segmentation task. The model output is <code>[32, 7, 256, 256]</code> to be matched with pathology and lunglobes grondtruth i.e <code>[32, [0,1], 256, 256] --&gt; [32, 2, 256, 256]</code> and <code>[32, [2,3,4,5,6], 256, 256] --&gt; [32, 5, 256, 256]</code>. However, I am stuck on efficiently backpropagating individual losses i,e loss1.backward() and loss2.backward() and getting the evaluation metric after each epoch.</p>
<p>Your suggestions are well appreciated</p>

username: Dwight_Foster
cooked: <p>One possibility is you could add the two losses together. This would save you computation and time so you do not have to perform two backwards passes. And then for the evaluation metric you can just print the two losses individually.</p>

username: moreshud
cooked: <p>I appreciate your input <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a>.</p>
<p>This is currently the approach I am adopting (as discussed in some threads) but I could not figure the understanding of using the computed parameters (weights and bias) for individual predictions in the next epoch.</p>

username: Dwight_Foster
cooked: <p>I am confused what you mean. Does something change in the next epoch? The next epoch the same thing should happen. Your weights and biases will be updated from the last step and should work in the next one. Is there something I am missing?</p>

username: moreshud
cooked: <p>Based on my understanding, after your forward pass (for the 1st iteration), the differentiable loss makes updated weights and bias on backward pass that will be used on in the 2nd iteration and the process follows till number of iterations that makes an epoch.</p>
<p>In this case - a single network for 2 predictions with different weighted loss functions, how will the network keep associating parameters during iterations for an epoch for each predictions that enables it master what has been learnt in previous iterations (or epochs)?</p>
I am trying to do brain tumor segmentation. I need to load my data. How to make customized dataset of a brain images ( .nii)?

username: Naveed_Ilyas
cooked: <p>I am trying to build the customized dataset for brain image. I have two folders</p>
<ol>
<li>HGG</li>
<li>LGG<br>
In each folder we have 5 MRI images including Flair, t1, t1c, t2 and a labeled image.<br>
How to create custom dataset. Your help in this situation is highly appreciable. Thank you</li>
</ol>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> might be hellful.<br>
In case you get stuck, could you add more information about your data and how each sample would be loaded?</p>

username: FreddyJ
cooked: <p>You can create your own dataset class by subclassing a dataset like so (minimal example):</p>
<pre><code class="lang-auto">class Lazy_Dataset(torch.utils.data.Dataset):

    def __init__(self, path):
        self.steps = [function that figures out the size of your dataset located in the path]
        self.path = path

    def __len__(self):
        return self.steps
    
    def __getitem__(self, idx):
        array = [function that loads your nifti file as a numpy array]
        data = torch.Tensor(array)
        label = [function that figures out your label]
        return data, label, idx
</code></pre>
<p>You can later create an instance of that dataset (and a dataloader) by calling:</p>
<pre><code class="lang-auto">mydataset = Lazy_Dataset(mypath)
mydataloader = torch.utils.data.DataLoader(mydataset, batch_size=24, shuffle=True)
</code></pre>

username: Naveed_Ilyas
cooked: <p>Thank you for your kind response. Here What i did so far. Can you please highlight the mistake.<br>
Here is my code and folder organization.<br>
(upload://pr9pSRiQKjIGy9X03dr2BAlfY4F.png) ![Presentation4|690x388]</p>
<p>patient_dir = “D:\Brats2020\”<br>
l_patnt =os.listdir(patient_dir)<br>
<span class="hashtag">#print</span>(l_patnt)</p>
<p>class Brats2022(Dataset):<br>
def <strong>init</strong>(self,patient_dir):<br>
self.patient_dir = patient_dir</p>
<pre><code>def __len__(self):
    return len(self.patient_dir)

def __getitem__(self, index):
    #return self.patient_dir[index]
    patient_dir = self.patient_dir[index]
    volumes = []
    patient_id = os.path.split(patient_dir)[-1]
    volume_path = os.path.join(patient_dir,patient_id + "_" + "seg"+".nii.gz")
    volume_seg = nib.load(volume_path)
    volu_im = volume_seg.get_fdata()
    np_arr = np.array(volu_im)
    data_transform = transforms.ToTensor()
    return data_transform(np_arr)</code></pre>
Loss function of multiple inputs and one target

username: Arii
cooked: <p>Hello,</p>
<p>I am now dealing with the combined images of two mnist digits. And the only label I have is the mod 10 of the sum of two digits. I built the network as the following:</p>
<pre><code class="lang-auto">class module(nn.Module):
    def __init__(self, input_dim, hidden_dim1, hidden_dim2):
        super(module, self).__init__()
        self.layer = nn.Sequential(
            nn.Linear(input_dim, hidden_dim1),
            nn.ReLU(),
            nn.Linear(hidden_dim1, hidden_dim2),
            nn.ReLU(),
            nn.Linear(hidden_dim2, 10),
        )

    def forward(self, input1, input2):
        output = torch.cat((input1, input2), dim=0) 
        output = output.view(output.size(0), -1)       
        output = self.layer(output)            

        output = torch.split(output, input1.size(0))
        return output[0], output[1]
</code></pre>
<p>There are now two separated outputs from the given image. Then I need to make a calculation using these two outputs to match one target value(=mod10(image1_label, image2_label)).</p>
<p>Here is where I am stuck for a couple of days. I have no idea how to build the loss function using these two inputs and one target, and still have a doubt it is really a classification problem, so it’s the right decision to use the CrossEntropyLoss or not. So, I would appreciate it if you can give me any hints to solve it out so that I can move on.</p>
<pre><code class="lang-auto">model = Module(input_dim=28*28, hidden_dim1=2048, hidden_dim2=1024).to(device)
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)

def train_model(model, dataloader, optimizer, loss_fn):
    losses = []
    predictions = 0
    for images1, images2, labels, _,_ in dataloader:
        images1 = images1.to(device)
        images2 = images2.to(device)
        labels = labels.to(device)

        output1, output2 = model(images1, images2)
        optimizer.zero_grad()

        ??loss = loss_fn(????????, labels)
        loss.backward()
        optimizer.step()
            
        losses.append(loss.item())
        predicted = (output1.argmax(1)+output2.argmax(2)) % 10
        predictions += accuracy(labels, predicted)
    
    train_loss  = sum(losses) / len(losses)
    train_accuracy = 100 * predictions / len(dataloader.dataset)
    return train_loss, train_accuracy
</code></pre>
<p>Best,</p>
Why element-wise multiplication is applied when matrix multiplication should be used?

username: oat
cooked: <p>[Context]<br>
Book: <a href="https://www.amazon.sg/Deep-Learning-Scratch-Building-Principles/dp/1492041416" rel="noopener nofollow ugc">Deep Learning from Scratch</a></p>
<p><a href="https://github.com/SethHWeidman/DLFS_code/blob/master/01_foundations/Code.ipynb" rel="noopener nofollow ugc">Jupyter Notebook on GitHub</a>, Code block 55</p>
<p>[Question]<br>
Why <strong>element-wise multiplication</strong> is applied to calculate <strong>dLdN = dLdS*dSdN</strong>, rather than <strong>matrix multiplication</strong> via either <code>np.dot()</code> or <code>np.matmul()</code>?</p>
<p>I assume this is to make the dimensionality of the rest derivatives correct as shown below in the comment following each derivative. However, I still don’t understand why the calculation of dLdN is different from that of dLdX… Or, am I missing something important here?</p>
<pre><code class="lang-auto">def matrix_function_backward_sum_1(X: ndarray,
                                   W: ndarray,
                                   sigma: Array_Function) -&gt; ndarray:
    '''
    Compute derivative of matrix function with a sum with respect to the
    first matrix input
    '''
    assert X.shape[1] == W.shape[0] # X: (m x n), W: (n x p)

    # matrix multiplication
    N = np.dot(X, W) # N: (m x p)

    # feeding the output of the matrix multiplication through sigma
    S = sigma(N) # S: (m x p)

    # sum all the elements
    L = np.sum(S) # L: a scalar 

    # note: I'll refer to the derivatives by their quantities here,
    # unlike the math where we referred to their function names

    # dLdS - just 1s
    dLdS = np.ones_like(S) # (m x p)

    # dSdN
    dSdN = deriv(sigma, N) # (m x p)
    
    # dLdN 
    dLdN = dLdS * dSdN # (m x p)  element-wise multiplication?

    # dNdX
    dNdX = np.transpose(W, (1, 0)) # (p x n)

    # dLdX
    dLdX = np.dot(dSdN, dNdX) # (m x p) x (p x n) = (m x n)

    return dLdX
</code></pre>
Approximate convolutional layer

username: Cadmium_29
cooked: <p>Hello all,<br>
I would like to build a new convolutional layer, where the addition and multiplication in this layer are all approximated(I could use accurate addition and multiplication to model that). I look at some code related to customized Conv layer but it seems that they rely on the con2d primitive which use accurate addition and multiplication. Is there any way I can implement this? Thanks!</p>

username: tom
cooked: <p>You would have to share a bit more about what you’re trying to do to be sure, but one bridge to more bespoke convolution-like operations can be the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Unfold" rel="nofollow noopener"><code>torch.nn.Unfold</code></a> module or the corresponding <code>torch.nn.functional.unfold</code>. What it does is reducing the convolution to Unfold + batch matrix multiplication with the kernel reshaped to a matrix form - the linked documentation has a demonstration.<br>
This allows to replace the linear transformation in the convolution with other ops.<br>
Note, though, that this is quite memory (and compute) intensive – each pixel is duplicated to all filters it appears in, but it might work for prototyping.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Cadmium_29
cooked: <p>Hi,<br>
Thanks for your reply. What I am trying to prototype is replacing the adder/multiplier in the processor unit in CPU/GPU with approximated adders/multipliers which may have inaccurate results. The behaviors of these proposed adders/multipliers could be emulated using logic operations. So basically, I need to replace the "<em>" and “+” in conv2d primitive with the function I wrote as follows:<br>
y = a</em>b+c;<br>
-&gt; y = approximate_add(approximate_mul(a,b),c);<br>
I look at the unfold function, it seems that it will still need to call the primitive which will use the accurate adder/multiplier if I understand it correctly. Does this mean I may need to modify the source code for conv2d?<br>
Thanks,<br>
Di</p>

username: Burak_Toker
cooked: <p>Hi, Can you find any solution about approximate adder</p>