akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Overlapping Windows over Batched Audio

username: tobiaaa
cooked: <p>I have a Batched Audio Signal with the Shape [N, C, L] with e.g. N=4, C=4, L=2048, which I want to apply a <strong>Sliding Window with Overlap</strong> to, and later I want to reconstruct the original shape by using <strong>overlap and add</strong>.</p>
<h4>
<a name="example-1" class="anchor" href="#example-1"></a>Example:</h4>
<p>Input Shape → [4, 4, 2048]<br>
After Sliding Window with Window Size 256 and Overlap 128 → [4, 4, 15, 256]<br>
Overlap and Add → [4, 4, 2048]</p>
<p>Using <code>Tensor.unfold()</code> works well for applying the sliding window but I haven’t found a way to overlap and add later on.<br>
Also I played around with <code>nn.Fold</code> and <code>nn.Unfold</code> but if I understand correctly those transformation mess with the Channel Dimension which I don’t want.</p>

username: ptrblck
cooked: <p>Could you explain what “mess with the channel dimension” would mean in the usage of <code>nn.Fold/Unfold</code>?<br>
Did you see any errors using these modules?</p>

username: tobiaaa
cooked: <p>Thank you for your reply! When using <code>nn.Fold</code>, the Channel Dimension is divided by the product of  <code>kernel_size</code>. Since I only have 4 Channels, I can use only very small Kernels so I don’t know how I can achieve for example a window size of 256 with 128 overlap.<br>
I want the Channels to stay the same and only “cut” along the Signal Length.</p>
<p>This snippet does exactly what I want but I haven’t found a way to achieve the same behavior with <code>nn.Fold/Unfold</code>.</p>
<pre><code class="lang-auto">x = torch.arange(0., 16384).view(2, 4, 2048)
# shape: (2, 4, 2048)
x_framed = x.unfold(-1, 256, 128)
# shape: (2, 4, 15, 256)
</code></pre>

username: ptrblck
cooked: <p><code>nn.Unfold</code> would return the same output if you reshape it accordingly:</p>
<pre><code class="lang-python">x = torch.arange(0., 16384).view(2, 4, 2048)
# shape: (2, 4, 2048)
x_framed = x.unfold(-1, 256, 128)
# shape: (2, 4, 15, 256)


# module approach
unfold = nn.Unfold(kernel_size=(256, 1), stride=(128, 1))
out_ref = unfold(x.unsqueeze(3))
out_ref = out_ref.view(out_ref.size(0), 4, 256, -1).permute(0, 1, 3, 2).contiguous()

print((x_framed - out_ref).abs().max())
# &gt; tensor(0.)
</code></pre>
Implement custom layer: local binary patern

username: bernhard
cooked: <p>(I changed the code so one can make it run on their own machine easily…)</p>
<p>Hi together,</p>
<p>I want to implement Local Binary Pattern on pytorch tensors. At the moment my code looks like this:</p>
<blockquote>
<p>import numpy as np<br>
import torch<br>
import torch.nn as nn<br>
import torchvision<br>
from PIL import Image<br>
import torchvision.transforms as transforms<br>
from torchvision.transforms import ToTensor</p>
<p>class LBP(nn.Module):</p>
<pre><code>def __init__(self):
    super(LBP, self).__init__()
    self.multiply = torch.tensor([[1,2,4],[128, 0, 8], [64, 32, 16]]).to("cuda")
    self.padding = transforms.Pad(1)

def get_value_of_crop(self, crop):
    crop_center = crop[1,1]
    comparison = crop &lt;= crop_center
    result_matrix =  comparison * self.multiply
    result = torch.sum(result_matrix)
    return result

def forward(self, grey_image):
    lbp_tensor = torch.zeros_like(grey_image)
    padded_image = self.padding(grey_image)
    for i in np.arange(0, padded_image.shape[1] - 2):
        for j in np.arange(0, padded_image.shape[1] - 2):
            print(f"{i}, {j}")
            lbp_tensor[0, i, j] = lbp_machine.get_value_of_crop(padded_image[0, i:i + 3, j:j + 3])

    return lbp_tensor
</code></pre>
<p>lbp_machine = LBP()</p>
<p>data = torchvision.datasets.CIFAR10(“.”, download= True)<br>
testimage = data.<strong>getitem</strong>(0)[0].convert(“L”)<br>
tensortransform = ToTensor()<br>
testimage = tensortransform(testimage).to(“cuda”)<br>
lbp_machine(testimage)</p>
</blockquote>
<p>It is not really performant as there are two for-loops. Any ideas about how to avoid these two for-loops and make the layer more performant?</p>
<p>Thanks a lot for any feedback!</p>
<p>Bernhard</p>
<p>P.s.: I know I could use packages like scikit-image, but they dont work on pytorch-tensors and by transforming my tensors to numpy-arrays I lose information like “grad-fn” that I will need later…</p>
How to combine multiple features into 1 image

username: Laudwika_Ambardi
cooked: <p>Hi, this is my first time posting. i’m starting with GANs and I’m wondering how to combine multiple features that i generated, where when I generate them i can generate each region then have the generator combine them.<br>
i want to generate each region of the face, where the end result when i combine will be like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f7857c15981d5c560d16c7a2d2cb3759e1549dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f7857c15981d5c560d16c7a2d2cb3759e1549dc" title="0"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f7857c15981d5c560d16c7a2d2cb3759e1549dc.png" alt="0" data-base62-sha1="ktc6UGEnRpaROH0gwJWJVmyKITG" width="500" height="500" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0</span><span class="informations">512×512 2.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Currently i’m using a DCGAN capable of generating the whole image from noise.</p>
InstanceNorm behavior changed?

username: Allerhand
cooked: <p>I recently updated a network from PT from 1.2 tp 1.10.<br>
After the update I started getting the error “ValueError: Expected more than 1 value per channel when training…”<br>
I understand completely. At the end of the network, I have a tensor of size [batch, 1000, 1, 1] that enters an InstanceNorm layer.<br>
However, it never used to cause a problem with PT 1.2 or 1.1.</p>
<p>I tried removing this instance norm, and the network gives significantly worse results.<br>
For completeness, I also had to change F.sigmoid to torch.sigmoid in some places.</p>
<p>I tired training multiple times using PT 1.2 and 1.10 and I get a consistent large difference. Also tried looking at the layer’s source code, but it doesn’t show any relevant difference within the Python part.</p>
<p>Can someone help recreate the old behavior? Thanks!</p>
Loading Mask_RCNN script model fails with no error

username: KRG
cooked: <p>Hi all,</p>
<p>I converted and saved  “maskrcnn_resnet50_fpn” model to script model. But when trying to load this model in c++, it fails without any error.</p>
<p>torch : 1.10.2+cu113<br>
torchvision : 0.11.3+cu113<br>
LibTorch: libtorch-win-shared-with-deps-1.10.2+cu113<br>
OS: Windows10</p>
<p>Python  Code -</p>
<pre><code class="lang-auto">kwargs = {"trainable_backbone_layers": None}
checkpoint_name = "checkpoint.pth"
model = torchvision.models.detection.__dict__["maskrcnn_resnet50_fpn"](pretrained=False, num_classes=5, **kwargs)
model.load_state_dict(torch.load(checkpoint_name)['model'])
model.eval()

script_model = torch.jit.script(model)
torch.jit.save(script_model, 'maskrcnn_ts.pt')
</code></pre>
<p>c++ code -</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include&lt;torch/script.h&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
        return -1;
    }
    std::string model_pt_name = argv[1];

    std::cout &lt;&lt; "Loading Model " &lt;&lt; model_pt_name &lt;&lt; "\n";

    torch::jit::script::Module module = torch::jit::load(model_pt_name);
   /* try {*/
        // Deserialize the ScriptModule from a file using torch::jit::load().
        
   std::cout &lt;&lt; "Loaded Model\n";
   /* }
    catch (const c10::Error&amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }*/

    std::cout &lt;&lt; "ok\n";
}

</code></pre>
<p>C++ project is build in Release mode.</p>
<p>cmd : LibTorchScriptProject.exe maskrcnn_ts.pt</p>
<p>This fails in the step " torch::jit::script::Module module = torch::jit::load(model_pt_name);" without any error message. Can you please advise what is the issue here.</p>

username: KRG
cooked: <p>I was able to load the model by including “<span class="hashtag">#include</span> &lt;torchvision/vision.h&gt;”</p>
Federated learning and differential privacy with pytorch

username: jarvico
cooked: <p>Hi,</p>
<p>I would like to do some experiments on federated learning and differential privacy using pytorch.</p>
<p>I saw the below blog post on <a href="http://openmined.org" rel="noopener nofollow ugc">openmined.org</a><br>
<a href="https://blog.openmined.org/pysyft-opacus-federated-learning-with-differential-privacy/" class="inline-onebox" rel="noopener nofollow ugc">Opacus + PySyft - A short demo</a></p>
<p>Then, I wanted to give it a try on Google Colab.</p>
<p>But unfortunately, the code on this blog post is not working. I think there are some compatibility issues with the current versions of syft, opacus, torch etc.</p>
<p>I would be glad if someone can share a working copy of the similar example?</p>
<p>error was at this line:</p>
<p>—&gt; hook = sy.TorchHook(th)<br>
AttributeError: module ‘syft’ has no attribute ‘TorchHook’</p>

username: jarvico
cooked: <p>The question is still open from my side. I need a working code (pytorch) example of a federated learning demo utilizing differential privacy.</p>
<p>Any help on this will be highly appreciated.</p>

username: ptrblck
cooked: <p>You seem to hit <a href="https://github.com/OpenMined/PySyft/issues/4820#issuecomment-729089956">this issue</a> which seems to be a deprecated approach:</p>
<blockquote>
<p>it looks like you are trying to use the old 0.2.x method of <code>TorchHook</code> in 0.3.0 which is no longer needed.<br>
This raises a very good issue which is we need to add a 0.2.x upgrade explanation to our README.md so thanks, I will get this in the pipeline asap.</p>
</blockquote>
<p>Unfortunately the linked README is broken, so unsure what the replacement would be.</p>
_pickle.UnpicklingError: invalid load key, '='

username: 1206089395k
cooked: <p>checkpoint = torch.load(model_path)<br>
File “/home/lab/.conda/envs/lll/lib/python3.6/site-packages/torch/serialization.py”, line 386, in load<br>
return _load(f, map_location, pickle_module, **pickle_load_args)<br>
File “/home/lab/.conda/envs/lll/lib/python3.6/site-packages/torch/serialization.py”, line 563, in _load<br>
magic_number = pickle_module.load(f, **pickle_load_args)<br>
_pickle.UnpicklingError: invalid load key, ‘=’.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/1206089395k">@1206089395k</a> Do you have different versions of pickle while saving and loading the model? That might be one of the reasons</p>

username: 1206089395k
cooked: <p>No,  the version is the same.</p>
Is there a way to make the following code run faster or to avoid the loop?

username: tyesh
cooked: <pre><code class="lang-auto">#input
x = torch.rand((32, 512))
#10 nodes in a graph
nodes = torch.rand(10, 512)
#edges between the nodes
edges = torch.zeros([10,10])
#each input has an assigned node
correct_nodes = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1])

for i, xi in enumerate(x):
    #get the correct node for the input
    correct_node = correct_nodes[i]
    #for each node in the graph
    for node in range(nodes.shape[0]):
        #edge operations between the correct node and other nodes in the graph
        if edges[correct_node, node] &lt; e_min:
            edges[correct_node, node] = 0
        if edges[correct_node, node] &gt; 0:
            edges[correct_node, node] = edges[correct_node, node] * epsilon
</code></pre>

username: Waffles
cooked: <p>I might have an idea, but I’m making a few assumptions…<br>
This also might just be useful if in your scenario your edges matrix is larger than the given example.<br>
If it is significantly larger I would suggest looking at sparse representations, and maybe computing this outside of python itself.</p>
<h1><a name="h-1" class="anchor" href="#h-1"></a></h1>
<p>Observations, it seems you aren’t using xi when looping over x and you are only using i to go over correct nodes, so this can be written like this.</p>
<pre><code class="lang-auto">for i in correct_nodes:
    for j in range(nodes.shape[0]):
        if edges[i, j] &lt; e_min:
            edges[i, j] = 0
        if edges[i, j] &gt; 0:
            edges[i, j] = edges[i, j] * epsilon
</code></pre>
<p>Other observation, correct_nodes elements is contained within the (i) 0th dimension of edges.<br>
If all the conditions are the following, then we are simply passing through each elements of a row multiple times for every node in “correct_nodes”, in other words if “correct_nodes” contains duplicates we are going over (i,j) multiple times.<br>
Lastly, I’m going on a limb here and assume epsilon is used to reduce the edge element value.</p>
<pre><code class="lang-auto">cond1&gt; if (e &lt; e_min) : e -&gt; 0
(revisit e): if e := 0 we will never reset it.
cond2&gt;if e &gt; 0:  e -&gt; e * epsilon 
(revisit e):can trigger cond1.
    
Therefore run:
&gt; e:= e * epsilon^(encounters)
&gt; and then check condition 1
</code></pre>
<p><strong>So maybe something like this would be what you are looking for if correct_nodes is a long list of nodes ?</strong></p>
<pre><code class="lang-auto">from collections import Counter
visited_node_encounters = torch.ones(10)
visits_counter = Counter(visits)
for i in range(10):
    if i in visits_counter:
        visited_node_encounters[i] = visits_counter[i]
encounters = torch.ones(10,10) * visited_node_encounters
epsilon = epsilon * torch.ones(10,10)

cond2 = epsilon**encounters

edges = cond1 = (1 - ((cond2 &lt; epsilon) * 1)) * cond2
</code></pre>
<p>*Note here I’m ignoring the order of these conditions, if they do matter apply edges = edges * epsilon^(encounters-1), followed by edges = cond1 * epsilon</p>
[Resolved] Missing gradients do to silly typo

username: Waffles
cooked: <p>I have two tensors of 3 features that are linked to their own rank values.</p>
<p>I’m attempting to predict &amp; learn in a pairwise fashion if I have a lower or higher rank match, however, it seems my problem is I cannot compute my gradients?<br>
(The model here is simplified for debugging purposes.)<br>
I was hoping someone could give me some pointers.<br>
<em>Example:</em><br>
<em>f1 = [1,3,5]. f2 = [1,4,5], y1 = 1, y2 = 5</em><br>
<em>If f1 scores lower than f2 and the rank y1 is lower than y2 then we have a positive match otherwise we don’t.</em><br>
Pytorch version: 1.7.3</p>
<pre><code class="lang-auto">w = torch.randn(3, 1, requires_grad=True)
softmax = torch.nn.Softmax(dim=0)

def model(x, w):
    z = torch.zeros(1, dtype=torch.float, requires_grad=True)
    z  = z + x @ w
    return z

def custom_loss(p, t):
    #Cross Entropy
    total = torch.zeros(1, dtype=torch.float, requires_grad=True)
    left_comp = (p) * torch.log(t)
    right_comp = (1-p) * torch.log(1-t)
    total = total + -1 * (left + right)
    return total

current_sample_ys = [int(5 * random.random()) for i in range(2)]
current_sample_xs = [[random.random() for i in range(3)] for i in range(2)]

labels = softmax(torch.tensor(current_sample_ys, dtype=torch.float))
features = [ torch.tensor(list(x), dtype=torch.float) for x in current_sample_xs]

m1 = model(features[0], w)
m2 = model(features[1], w)

outputs = softmax(torch.cat((m1, m2), dim=0))
# Turning -1:1 range to 0:1
pred_outputs = (torch.sub(*outputs) + 1) / 2
target = (torch.sub(*labels) + 1) /2

r = custom_loss(target, pred_outputs)

r.backward()

print(w.grad) #None
</code></pre>

username: KFrank
cooked: <p>Hi Phil!</p>
<aside class="quote no-group" data-username="Waffles" data-post="1" data-topic="144358" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/waffles/48/46767_2.png" class="avatar"> Waffles:</div>
<blockquote>
<pre><code class="lang-auto">def custom_loss(p, t):
    #Cross Entropy
    total = torch.zeros(1, dtype=torch.float, requires_grad=True)
    left_comp = (p) * torch.log(t)
    right_comp = (1-p) * torch.log(1-t)
    total = total + -1 * (left + right)
    return total
</code></pre>
</blockquote>
</aside>
<p>As written, I believe than neither <code>left</code> nor <code>right</code> is defined in your function<br>
<code>custom_loss()</code> (nor elsewhere in the code you posted), so I don’t think<br>
that your code will run at all.</p>
<p>If you fix that (in a sensible way), gradients should backpropagate through<br>
<code>custom_loss()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Waffles
cooked: <p>This is embarrassing. I’ve been going crazy over this. Thank you Frank!<br>
Any recommendations for a jupyter notebook linter or something of sorts that would have helped me catch this?</p>
Why doesn't UNet allow any backbone?

username: gonzrubio
cooked: <p>I am currently going over various architectures for object detection and came across the <a href="https://arxiv.org/abs/1612.03144" rel="noopener nofollow ugc">Feature Pyramid Network (FPN)</a> which at first glance looks incredibly similar to the <a href="https://arxiv.org/abs/1505.04597" rel="noopener nofollow ugc">UNet architecture</a> which made me wonder if it had any major differences other than implementation details to reduce memory footprint or inference/training time.</p>
<p>After doing some digging around I came across <a href="https://forums.fast.ai/t/differences-between-u-net-and-feature-pyramid-networks-fpn/33166/3" rel="noopener nofollow ugc">this answer</a> which says:</p>
<blockquote>
<p>The Feature Pyramid Network (FPN) looks a lot like the [U-net] (<a href="https://vitalab.github.io/deep-learning/2017/02/27/unet.html" rel="noopener nofollow ugc">https://vitalab.github.io/deep-learning/2017/02/27/unet.html </a>). The main difference is that there is multiple prediction layers: one for each upsampling layer. Like the U-Net, the FPN has laterals connection between the bottom-up pyramid (left) and the top-down pyramid (right). But, where U-net only copy the features and append them, FPN apply a 1x1 convolution layer before adding them. This allows the bottom-up pyramid called “backbone” to be pretty much whatever you want.</p>
</blockquote>
<p>Why is it that the UNet doesn’t allow us to chose any “backbone” as the encoding portion of the architecture?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I can’t speak for the author of the linked question, but would guess</p>
<blockquote>
<p>But, where U-net only copy the features and append them, FPN apply a 1x1 convolution layer before adding them. This allows the bottom-up pyramid called “backbone” to be pretty much whatever you want.</p>
</blockquote>
<p>means that the FPN implementation could be more flexible as the skip connections are processes (and you might be thus able to change the number of channels, spatial size etc.) while the (original) UNet implementation might have just concatenated the skip activations and would thus be more shape-dependent.<br>
In any case, that’s just my interpretation of this answer so you might want to follow up with the authod.</p>
Official Docker Image

username: abelbrown
cooked: <p>Is there an official docker image for PyTorch on DockerHub?</p>

username: smth
cooked: <p>no, but we do provide an official Dockerfile: <a href="https://github.com/pytorch/pytorch#docker-image">https://github.com/pytorch/pytorch#docker-image</a></p>

username: gpakosz
cooked: <p>Hello <a class="mention" href="/u/smth">@smth</a>, this is really confusing.</p>
<p>Who’s behind <a href="https://hub.docker.com/r/pytorch/pytorch/" rel="nofollow noopener">https://hub.docker.com/r/pytorch/pytorch/</a> then if not PyTorch developers?<br>
And if things changed recently, what makes you guys decide to push new images to that registry?</p>

username: smth
cooked: <p><a class="mention" href="/u/gpakosz">@gpakosz</a> we have control of that registry, but we haven’t committed to actually doing release pushes onto it.</p>

username: gpakosz
cooked: <p>Thanks for your reply <a class="mention" href="/u/smth">@smth</a>. I’m crossing fingers you’ll push <code>1.0</code> there, and eventually decide to push on a regular basis as it makes things easy for us imho🤞</p>

username: barkerje
cooked: <p>What is the relationship between the official docker file and the pytorch image on docker hub?  Is the hub entry created using the official docker file?</p>

username: Carsten_Ditzel
cooked: <p>I have been meaning to ask this as well.</p>
<p>for now I am using a tensorflow image and building upon this to equip it with pytorch abilities. But there should be a more natural way</p>

username: jxtps
cooked: <p>I’m a bit confused about the intended way to consume recent pytorch versions. As of this writing:</p>
<ul>
<li>On pytorch dot org I can download v1.10.2.</li>
<li>On <a href="https://hub.docker.com/r/pytorch/pytorch/tags" class="inline-onebox" rel="noopener nofollow ugc">Docker Hub</a> the “latest” tag was pushed 4 months ago and contains 1.10.0.</li>
<li>
<a href="https://github.com/pytorch/pytorch/issues/70881" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/70881</a> makes it sound like docker images are built for more or less every single commit. But where are those images? Are they externally accessible?</li>
</ul>
<p>For a host of reasons we use the published docker images, but really need to access 1.10.2. How can we achieve that? Thanks!</p>

username: ptrblck
cooked: <p><span class="hashtag">#70881</span> most likely targets the CI docker containers used to run unit tests.<br>
You could use the provided <code>Dockerfile</code> and rebuild PyTorch <code>1.10.2</code> inside it manually if you need this version asap.</p>
Splitting tensor into sub-tensors in overlapping fashion

username: nicozhou
cooked: <p>Is there any way to Splitting tensor into sub-tensors in overlapping fashion? My input is video frames B x T x C x H x W. I want to split it become sub-tensors for example, frame1~5, frame 2~6, frame 3~7,… . Is there any built-in pytorch function to do that?</p>

username: ptrblck
cooked: <p>Yes, you can use <code>tensor.unfold</code>:</p>
<pre><code class="lang-python">x = torch.arange(10)
y = x.unfold(dimension=0, size=5, step=1)
print(y)
&gt; tensor([[0, 1, 2, 3, 4],
          [1, 2, 3, 4, 5],
          [2, 3, 4, 5, 6],
          [3, 4, 5, 6, 7],
          [4, 5, 6, 7, 8],
          [5, 6, 7, 8, 9]])
</code></pre>
Pytorch's weight_decay and batch normalization layer

username: samin_hamidi
cooked: <p>I was wondering if the parameters of batch_norm layers are considered when computing the L2_norm of weight decay in Pytorch’s implementation?</p>

username: ptrblck
cooked: <p>The <code>weight_decay</code> argument will be applied to the current parameter group. I.e. if you are passing the batchnorm parameters to this group (or re just using a single group and are passing all parameters) weight decay will also be applied on them.</p>
Speeding up 7D sliding windows summation

username: sebastienwood
cooked: <p>Hi !<br>
I’ve been having troubles with switching an implementation from numpy to Pytorch on a 7 dimensional tensor. The numpy code looks as follow:</p>
<pre><code class="lang-auto">@nb.njit
def numba_7d_conv_cpu(gamma, new_gamma, ratio_sq: float, kernel_size: int):
  for b in range(new_gamma.shape[0]):
    for c in range(new_gamma.shape[1]):
      for i in range(new_gamma.shape[2]):
        for j in range(new_gamma.shape[3]):
          for cp in range(new_gamma.shape[1]):
            for k in range(new_gamma.shape[2]):
                for l in range(new_gamma.shape[3]):
                  new_gamma[b,c,i,j,cp,k,l] = ratio_sq * np.sum(gamma[b,c,kernel_size*i:kernel_size*(i+1),kernel_size*j:kernel_size*(j+1),cp,kernel_size*k:kernel_size*(k+1),kernel_size*l:kernel_size*(l+1)])
</code></pre>
<p>Notice I make use of Numba.jit in order to achieve a competitive performance. First dimension is the batch size, then it’s repeating twice the number of channels, image width, image height (it is a covariance matrix on the original input).<br>
I’m interested in porting it to Pytorch in order to prevent some back and forth with the GPU/scale the batch size while keeping runtime low.</p>
<p>Using the multiple for loops with pure Pytorch code and Torch.jit do not yield improved runtimes (if I understood right loops aren’t very good with JIT at the moment ?). Should I consider writing CUDA kernels for this task ?</p>

username: sebastienwood
cooked: <p>I managed to it by reshaping → convolution → reshaping → convolution → reshaping. I’d be happy to help if anyone encounter the same issue.</p>
Pytorch equivalent of TensorFlow IntegerLookup?

username: toliveira
cooked: <p>Is there an equivalent of TensorFlow <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/IntegerLookup" rel="noopener nofollow ugc">IntegerLookup</a> layer in PyTorch? Does this equivalent support out-of-vocabulary (OOV) tokens?</p>
Simple skip connection code not working in pytorch

username: Emmanuel357
cooked: <p>My simple code is below.</p>
<blockquote>
<p>class AutoEncoder(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(AutoEncoder, self).<strong>init</strong>()</p>
<pre><code>    edim1 = n_imgs

    self.input_layer = nn.Sequential(

        nn.LayerNorm(simg_size),
        nn.Linear(simg_size, edim1, bias = False),
        nn.LeakyReLU(0.2, inplace=True)
    )

    self.h_layer = nn.Sequential(
            nn.LayerNorm(edim1),
            nn.Linear(edim1, edim1, bias = False),
            nn.LeakyReLU(0.2, inplace=True)
        )

    self.output_layer = nn.Sequential(
        nn.LayerNorm(edim1),
        nn.Linear(edim1, img_size, bias = False),
        nn.Tanh()
    )

def decoder(self, img):
    z1 = self.input_layer(img)
    z2 = self.h_layer(z1)
    z2 += z1  #THE LINE PRODUCING THE ERROR
    out = self.output_layer(z2)
    return out
</code></pre>
</blockquote>
<p>THE ERROR</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 64]], which is output 0 of LeakyReluBackward1, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/emmanuel357">@Emmanuel357</a><br>
Replicating the error in a simpler way</p>
<pre><code class="lang-auto">a=torch.autograd.Variable(torch.Tensor([1]), requires_grad=True)
b=torch.autograd.Variable(torch.Tensor([2]), requires_grad=True)
print([a, b])
</code></pre>
<pre><code class="lang-auto">[tensor([1.], requires_grad=True), tensor([2.], requires_grad=True)]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_4808/4177809845.py in &lt;module&gt;
      2 b=torch.autograd.Variable(torch.Tensor([2]), requires_grad=True)
      3 print([a, b])
----&gt; 4 a+=b
      5 print(a)

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

</code></pre>
<p><strong>SOLUTION</strong></p>
<pre><code class="lang-auto">a=a+b
print(a)
</code></pre>
<pre><code class="lang-auto">tensor([3.], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
How to index a tensor by a tensor

username: zyc
cooked: <p>I have a value tensor value.shape=[b, l, c], and an index tesnor index.shape=[b, n], s.t. n&lt;=l, max(n) &lt;= len(l)-1<br>
How to retrieve value tensor by index tensor?</p>
<p>The following code seems to work perfectly but involve a for loop.</p>
<pre><code class="lang-python">torch.stack([value[i, index[i] ]for i in range(len(value))])
</code></pre>

username: zyc
cooked: <pre><code class="lang-python">value[torch.arange(len(index)), index.T].transpose(0,1)
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/zyc">@zyc</a> you can use this</p>
<pre><code class="lang-auto">enhanced_index = torch.Tensor([[i, y] for i,x in enumerate(index) for y in x]).type(torch.long)
values[enhanced_index[:,0], enhanced_index[:,1]]
</code></pre>
Multiple training codes are stopped without error

username: oasjd7
cooked: <pre><code class="lang-auto">cuda 11.4 
pytorch 1.10.2
</code></pre>
<p>It works normally when running up to two training codes.<br>
However, when I add one more, the added code fail to run 1 epoch and stops without error.</p>
<p>The code still hold GPU memory but not running.</p>
<p>I tried to apply <code>num_workers</code> from 0 to 4 but nothing changed.<br>
How can I fix it?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85321cf563c250aca93772b72f09e327829e8a20.png" alt="스크린샷 2022-02-16 13.05.52" data-base62-sha1="j0iRXWuV2d0iMTqzJmtPl3xGgYU" width="652" height="108"></p>
<ul>
<li>When the codes are stopped, CPU are not working.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88822b2cf528de3328cb083ba39264043bc311e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/88822b2cf528de3328cb083ba39264043bc311e2" title="스크린샷 2022-02-16 14.35.58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88822b2cf528de3328cb083ba39264043bc311e2_2_690x252.png" alt="스크린샷 2022-02-16 14.35.58" data-base62-sha1="jtBOU4grDkJJNI5bj8Ew4PxySmm" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88822b2cf528de3328cb083ba39264043bc311e2_2_690x252.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88822b2cf528de3328cb083ba39264043bc311e2_2_1035x378.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88822b2cf528de3328cb083ba39264043bc311e2_2_1380x504.png 2x" data-dominant-color="2C2E2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2022-02-16 14.35.58</span><span class="informations">2706×990 222 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
Interactive between Matlab and Pytorch

username: Kevinzhu
cooked: <p>Hey, I’m running pytorch in Matlab. In specific, I import Pytorch lib into the matlab and so I can call functions from PyTorch. I would like to know how to convert matlab variables such as double, complex double to the pytorch tensor? In Python I can simply do A = torch.tensor(B) that B is a variable assigned before. However, I cannot do that in Matlab.</p>

username: ptrblck
cooked: <p>I cannot find PyTorch in MATLAB’s supported <a href="https://www.mathworks.com/products/deep-learning.html#frm">frameworks interop</a> so I guess you are exporting to ONNX first?<br>
In any case, I would expect the MATLAB docs should mention how to use 3rd party frameworks but again I couldn’t find any supported use case so far.</p>

username: Kevinzhu
cooked: <p>Emm…I’m not sure what ONNX is…what I did is config environment in Conda first, and then start Matlab from Conda, then I could use following code to import torch in matlab</p>
<pre><code class="lang-auto">torch = py.importlib.import_module("torch");
</code></pre>
<p>This allows me to call PyTorch function such as</p>
<pre><code class="lang-auto">p2 = py.numpy.array(b);
b_Ten = py.torch.from_numpy(p2);
</code></pre>
<p>so I can convert matlab variable ‘b’ to a tensor ‘b_Ten’. Actually I just figured out I can do it in this way.However, it is still unclear to me that how to convert tensor back to matlab variable. I tried</p>
<pre><code class="lang-auto">b_Ten.numpy(); 
</code></pre>
<p>but it makes error…</p>

username: ptrblck
cooked: <p>Oh, interesting. My MATLAB usage dates a few years back and was mostly targeting signal processing so I wasn’t aware of a Python API and thus don’t know what its current support is, sorry.</p>

username: Kevinzhu
cooked: <p>Yea, Im using MATLAB for signal processing as well, but I need Pytorch to compute a gradient, which I cannot get the explicit expression easily…</p>
PyTorch Profiler kineto_event.name() shows "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 5: invalid start byte"

username: seanM29
cooked: <p>hi, I use PyTorch Profiler on V100 (pytorch:1.8.0-cuda11.1-cudnn8),<br>
According to the instruction on <a href="https://pytorch.org/docs/stable/profiler.html" class="inline-onebox" rel="noopener nofollow ugc">torch.profiler — PyTorch 1.9.0 documentation</a>, I update my code. But I got this error:</p>
<p>"Traceback (most recent call last):<br>
File “/opt/conda/lib/python3.8/site-packages/torch/autograd/profiler.py”, line 1141, in parse_kineto_results<br>
filter_name(kineto_event.name())<br>
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x89 in position 5: invalid start byte<br>
"</p>

username: Vidit_Goel
cooked: <p>Hi <a class="mention" href="/u/seanm29">@seanM29</a> , I am also facing the same issue. Did you find the solution?</p>

username: shenh10
cooked: <p>Same issue here. Any update?</p>
Passing image as a condition to Discriminator to make it conditional

username: KURUVILLA_ABRAHAM
cooked: <p>Hi team,</p>
<p>How can we pass a image to a Discriminator block to condition the same .</p>
<p>Below is my image</p>
<p>Main image -  torch.randn(1,3,256,256)<br>
Conditonal image - torch.randn(1,3,256,256)<br>
Target_colors = torch.zeros(1,1,3,dtype=torch.float)<br>
Target_colors[:,:,0] = 255</p>
<p><span class="hashtag">#Below</span> is the way i am trying -</p>
<p>self.label_condition_color = nn.Sequential(nn.Embedding(300,3),nn.Linear(3,256*256),<br>
nn.ReLU(True))</p>
<p>self.embed_sketch = nn.Embedding(1,self.image_size*self.image_size)</p>
<h1>
<a name="i-need-to-concat-both-to-main-image-above-1" class="anchor" href="#i-need-to-concat-both-to-main-image-above-1"></a>i need to concat both to main image above</h1>
<p>concat = torch.cat((x, label_output), dim=1)</p>
<h1>
<a name="actual-code-i-am-using-2" class="anchor" href="#actual-code-i-am-using-2"></a>Actual Code i am using -</h1>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self,image_size):<br>
super(Discriminator, self).<strong>init</strong>()<br>
self.image_size = image_size</p>
<pre><code>    self.main = nn.Sequential(
        nn.Conv2d(6, 64, 4, stride=2, padding=1),
        nn.LeakyReLU(0.2, inplace=True),

        nn.Conv2d(64, 128, 4, stride=2, padding=1),
        nn.InstanceNorm2d(128),
        nn.LeakyReLU(0.2, inplace=True),

        nn.Conv2d(128, 256, 4, stride=2, padding=1),
        nn.InstanceNorm2d(256),
        nn.LeakyReLU(0.2, inplace=True),

        nn.Conv2d(256, 512, 4, padding=1),
        nn.InstanceNorm2d(512),
        nn.LeakyReLU(0.2, inplace=True),

        nn.Conv2d(512, 1, 4, padding=1),
        nn.Sigmoid()
    )
    self.label_condition_color = nn.Sequential(nn.Embedding(300,3),nn.Linear(3,256*256),
                        nn.ReLU(True))
    # self.label_condition_disc = nn.Sequential(nn.Linear(3,3*256*256),
    #                     nn.ReLU(True))
    self.embed_sketch = nn.Embedding(1,self.image_size*self.image_size)

def forward(self,inputs):
    x,sketch,label = inputs
    label_output = self.label_condition_color(label)
    label_output = label_output.view(-1,3,self.image_size,self.image_size)
    print(sketch.shape)
    # sketch_embed = self.embed_sketch(sketch).view(sketch.shape[0],1,self.image_size,self.image_size)
    # print("skecth embed shape is:",sketch_embed.shape)
    concat = torch.cat((x, label_output), dim=1)
    x = self.main(concat)
    x = F.avg_pool2d(x, x.size()[2:])
    
    x = torch.flatten(x, 1)
    return x.</code></pre>
How to apply element-wise map function with tensor?

username: ming_zhang
cooked: <p>I have a dict H，which key is（i，j） and value is a tensor of 4 elements.<br>
And a one-dim tensor e, such as e =[1,2,3]<br>
I want to construct a 3-dim tensor T:<br>
T = torch.tensor(H[(e[i] ,e[j])]   for i in range(3) for j in range(3)])<br>
that is to say T_ij = H[(e[i],e[j])].<br>
T will be a tensor with size (3,3,4)<br>
is there any possible way to parallel this process?<br>
Thanks very much!!</p>

username: ptrblck
cooked: <p>I don’t quite understand what <code>H</code> is since you are mentioning a <code>dict</code>, but are trying to call it with two indices. Are you using a nested <code>list</code> instead or a tensor?</p>

username: ming_zhang
cooked: <p>Thanks your reply!<br>
H is a python dictionary，which key is a tuple.<br>
Like H[(0,0)] = [1,2,3,4]<br>
H[(0,1)] = [1,2,3,4]</p>
Inplace-operation in F.binary_cross_entropy() or squeeze() during gradient computation

username: yty
cooked: <p>I am running into the error:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4608, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>And I’ve narrow it down to one single line of the code:</p>
<pre><code class="lang-auto">g_adversarial_loss = F.binary_cross_entropy(d_fake_output.squeeze(), valid)
</code></pre>
<p>it seems that there is some inplace operation in the code above, but I can’t find it. How can I solve the Error?</p>
<p>Thank you in advance!</p>

username: ptrblck
cooked: <p>I guess a previous operation might have manipulated a tensor inplace and raises the error in the backward pass. Since you are working with a GAN, have a look at <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this post</a> and check if you could be hitting the same issue.</p>
How to efficiently apply transforms?

username: oasjd7
cooked: <p>I want to make two images which are applied with different strong augmentations.<br>
I gave up configuring two datasets to which each augmentation was applied. Because the dataset is too large and the CPU failed to process it.</p>
<p>So, I tried compose only one dataset and make two versions of augmentations in each mini-batch. Below code is works well, but I think it’s very inefficient.<br>
(I failed to convert 4D data (BxCxHxW) to PIL Image directly, so I use <code>for-loop</code>.<br>
Please tell me a more efficient way to implement this.</p>
<pre><code class="lang-auto">&gt;&gt; In Dataset:
transforms.Compose([
                    transforms.Resize([256, 256])
                    transforms.ToTensor(),
                ]),

...
for step, data in enumerate(data_loader):
        import torchvision.transforms as transforms
        strong_trans = transforms.Compose([
            transforms.ToPILImage(),
            transforms.RandomCrop(size=224,
                                  padding=int(224 * 0.125),
                                  padding_mode='reflect'),
            transforms.RandAugment(),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

        img_strong_trans = []
        for x in data:
            img_strong_trans.append(strong_trans(x))

        img_strong_trans = torch.stack(img_strong_trans)
</code></pre>

username: ptrblck
cooked: <p>You could apply both transformations inside the <code>Dataset.__getitem__</code> and return both transformed samples to avoid the additional transformation with the for loop in the <code>DataLoader</code> loop.</p>
How to copy network parameters in libtorch c++ API

username: pszabo
cooked: <p>Hi,<br>
My question is how to copy the values of trainable parameters from one network to another<br>
using the libtorch c++ API.<br>
More precisely:<br>
I have a custom Network class derived from torch::nn::Module and two instances of this class named n1 and n2. I want to copy the trainable parameters from n2 to n1.<br>
In pytorch this can be achieved by n1.load_state_dict(n2.state_dict()), but the network class has no such methods in the c++ API. (Seems to be related to the clone merhod, but I could not figure out how to use it)</p>

username: kolkir
cooked: <p>Hi, as I understand now you should implement load_state_dict function for c++ manually, it will looks like:</p>
<pre><code class="lang-auto"> torch::autograd::GradMode::set_enabled(false);  // make parameters copying possible
 auto new_params = ReadStateDictFromFile(params_path); // implement this
 auto params = model-&gt;named_parameters(true /*recurse*/);
 auto buffers = model-&gt;named_buffers(true /*recurse*/);
 for (auto&amp; val : new_params) {
     auto name = val.key();
     auto* t = params.find(name);
     if (t != nullptr) {
          t-&gt;copy_(val.value());
      } else {
          t = buffers.find(name);
          if (t != nullptr) {
            t-&gt;copy_(val.value());
          }
      }
}
torch::autograd::GradMode::set_enabled(true);
</code></pre>

username: pszabo
cooked: <p>Thanks a lot Kirill, I managed to implement a similar solution, but thanks for posting me, this code is neater than mine.</p>

username: afshin67
cooked: <p>Are you gonna add something like <code>load_state_dict</code> in the next version?</p>

username: kolkir
cooked: <p>I’m not a PyTorch developer, but you can look at my implementation <a href="https://github.com/Kolkir/mlcpp/blob/master/mask_rcnn_pytorch/stateloader.cpp" rel="nofollow noopener">here</a></p>

username: afshin67
cooked: <p>Thanks, I also implemented a function for myself, though it would be nicer if libtorch had a built-in function.</p>

username: tom
cooked: <p>This is an old thread, but linked recently:</p>
<p>These days, there also is a save / load functionality in C++, but is based on the JIT file format. So you can save a traced model (but use the model, not the function) und load weights back into your PyTorch C++ API-built model.</p>
<p>Best regards</p>
<p>Thomas</p>

username: reza1
cooked: <p>I have a model in the format hdf5 which is taught using keras pyhton. Now I want to load this model in C ++ and use it. I do not know if libtorch loads the model. How should this be done?</p>
Convert 3D CNN model to 2D CNN

username: Siddu_javali
cooked: <p>How can I convert a 3D CNN model trained for Nifti images to 2d so that I can inference on 2D nifti slices?</p>

username: ptrblck
cooked: <p>You could either:</p>
<ul>
<li>create a “depth” dimension in your inputs and <code>repeat</code> the data <code>D</code> times to match the expected input to the original 3D model</li>
<li>try to reduce the parameters from the 3D model to 2D ones</li>
</ul>
<p>For the 2nd approach: I don’t know which approach would work at all, but e.g. you could reuse the kernels from the <code>nn.Conv3d</code> layers, apply a <code>mean</code> (or <code>sum</code> or slice) in the depth dimension of the kernel and load it into a newly created <code>nn.Conv2d</code> layer.<br>
Depending on the actual model architecture this model “surgery” might not be trivial so the 1st approach might be easier.</p>
DataLoader that uses workers to load samples rather than batches

username: AntreasAntoniou
cooked: <p>Hello everyone,</p>
<p>So here’s what I’ve been running into.</p>
<p>Imagine you are trying to scale up your batch size by some factor N, and you can only increase the number of your CPUs but not so much your RAM.</p>
<p>The default behaviour of DataLoader is to spin up num_workers processes, each of which samples a full batch before returning it. The issue that this causes is that in order for me to benefit from the increased availability of CPUs to scale my batch size up efficiently, I need to basically convert a situation where:</p>
<p>batch_size=B, num_workers=W</p>
<p>to</p>
<p>batch_size=NxB, num_workers=NxW.</p>
<p>This will ensure that I am able to properly utilize the extra CPUs I can get access to. However, in doing so I am also effectively increasing the memory that I need to use from B x W to N x B x N x W, which is problematic since I have access to more CPUs, but not additional RAM. And in most cases, such as on cloud compute, the factor of CPU to RAM increase is usually equal, therefore, an increase of N^2 complexity for memory could be very problematic.</p>
<p>I was wondering if there is a way to change the default behaviour of the DataLoader from a situation where instead of each worker loading a full batch, it instead has a pool of workers sampling single samples each which are then integrated into a batch once they reach a certain number. That way I could do something like what I need above, without needing all that extra memory, and potentially saving memory as well.</p>
<p>Please let me know your thoughts.</p>

username: ptrblck
cooked: <p>This feature request was mentioned a few times and e.g. I could quickly find <a href="https://github.com/pytorch/pytorch/issues/24985">this here</a>, which also includes some references.<br>
I don’t think this behavior is implmented yet, but might be tracked in the RFCs for the new data loading pipeline, so feel free to add your use case to e.g. the <a href="https://github.com/pytorch/pytorch/issues/41292">tracking issue</a>.</p>
Help,my BucketIterator doesn't work

username: 1376213926
cooked: <p>I creat a BucketIterator from my train_set.<br>
But when I begin my training ,the BucketIterator object doesn’t  work :<br>
“for batch in train_iterator:”---------my code<br>
it said that “object of type ‘numpy.int64’ has no len()”</p>
<p>detailed code as follows:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/pyf1234/sentimental-classification-by-lstm-pytorch">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/pyf1234/sentimental-classification-by-lstm-pytorch" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://storage.googleapis.com/kaggle-avatars/thumbnails/default-thumb.png" class="thumbnail onebox-avatar" width="160" height="160">

<h3><a href="https://www.kaggle.com/pyf1234/sentimental-classification-by-lstm-pytorch" target="_blank" rel="noopener nofollow ugc">sentimental classification by LSTM(pytorch)</a></h3>

  <p>Explore and run machine learning code with Kaggle Notebooks | Using data from multiple data sources</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: AbdulsalamBande
cooked: <p>I recommend you debug the code step by step. You also have to know that  data processing using torchtext has been heavily discussed here <a href="https://github.com/pytorch/text/issues/664" rel="noopener nofollow ugc">torchtext issue</a>. You can follow the migration tutorial from torctext data processing to something better <a href="https://colab.research.google.com/github/pytorch/text/blob/master/examples/legacy_tutorial/migration_tutorial.ipynb" rel="noopener nofollow ugc">Migrate torchtext from the legacy API to the new API</a></p>