akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Create a binary tensor with n% ones

mhadnanali: <p>I want to create a binary tensor with 0 and 1s only, but the number of 1s should be n%.</p>
<p>I am using this code but it takes 7+ seconds and 19926MB memory.</p>
<p>Is there any faster way to do that?</p>
<pre><code class="lang-auto">import torch
import time
start_time = time.time()
def create_random_binary_tensor(shape, percentage, device):
    size = shape[0] * shape[1]
    num_ones = int(size * (percentage / 100.0))
    
    # Create a tensor filled with zeros
    tensor = torch.zeros(size, dtype=torch.float32, device=device)
    
    # Set random indices to ones until the desired number of ones is reached
    indices = torch.randperm(size, device=device)[:num_ones]
    tensor[indices] = 1
    
    # Reshape the tensor to the desired shape
    tensor = tensor.view(shape)
    
    return tensor

# Set the shape and percentage of ones
shape = (19000, 19000)
percentage = 0.5

# Check if CUDA is available and use it if possible
device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")

# Create the random binary tensor
tensor = create_random_binary_tensor(shape, percentage, device)

# Print the tensor
execution_time = time.time() - start_time
print("Execution Time: {:.4f} seconds".format(execution_time))

</code></pre>
Modified PPO Example: loss_value.backward(retain_graph=True)?

kgetzand: <p>Hi all,</p>
<p>I’ve modified the PPO tutorial to use a custom environment. However, in the training step, when I call “loss_value.backward()” it eventually throws “RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)…”</p>
<p>If I naively change it to “loss_value.backward(retain_graph=True)” it works, but I’m not sure why the error was thrown in the original case. Any information on why this is occurring, and what the implications are for using “loss_value.backward(retain_graph=True)” would be appreciated.</p>
<p>I stepped through the output of my custom env and it seems like the obs, reward, etc. retained their graphs, as far as I can tell.</p>
<p>Thanks in advance!</p>

samuel_beaussant: <p>Hi,</p>
<p>Please post your code and/or provide a link to the mentionned tutorial to help solve your issue.</p>
Installing PyTorch under Python 3.8 - Question about networkx version

bonen: <p>I’m trying to install PyTorch on Ubuntu 20.04 with pip under Python 3.8.</p>
<p>According to the official, PyTorch can work under Python 3.8, however, PyTorch requires <code>networkx</code> package and the latest version of it requires Python 3.9+.</p>
<p>So, before installing PyTorch, I have to install the older version of <code>networkx</code> manually by <code>pip install networkx==3.1</code>.<br>
My question here is, is it OK to use this older <code>networkx</code> package with the latest PyTorch?</p>

ptrblck: <p>Yes, it should be OK since the CI uses a docker container with <a href="https://github.com/pytorch/pytorch/blob/fcc36de9d6c3d13e0995b17637e19ad10741ca87/.ci/docker/requirements-ci.txt#L94"><code>networkx==2.8.8</code></a> so even older.</p>

bonen: <p>I understand. Thank you!</p>

mcamurri: <p>So why isn’t the Pytorch installer already selecting the correct version of <code>networkx</code>?</p>
Analyzing PyTorch Profiling with Tensorboard Intergration

a2jinhee: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6f980c9b4219c455cb92386bed43b026496d1f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6f980c9b4219c455cb92386bed43b026496d1f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6f980c9b4219c455cb92386bed43b026496d1f3_2_690x362.png" alt="image" data-base62-sha1="zeQ0Y06LAh6HqafSUBITuWZV0Wv" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6f980c9b4219c455cb92386bed43b026496d1f3_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6f980c9b4219c455cb92386bed43b026496d1f3_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6f980c9b4219c455cb92386bed43b026496d1f3_2_1380x724.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2190×1152 272 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is my profiling output in tensorboard.<br>
I understand that ‘Device Self Duration’ only counts for the parent class, and ‘Device Total Duration’ counts for the parent and child class combined. However, the output names in ‘Device Total Duration’ confuse me, because there seem to be 8 convolution operations similar yet different to each other. I am guessing they are in a hierarchical relationship with each other, but I’m not sure because I’m new to profiling.</p>
<p>I tried to check the hierarchical relationship by setting with_modules=True, but it does not show anything in the tensorboard visualization. Is there any way to analyze/visualize the hierarchy between the module calls so I can see exactly which operation is the bottleneck?</p>
<p>The top picture is sorted by the ‘Device Total Duration Time’.</p>
Single GPU Windows System

Jay4: <p>I’m sorry to ask such a basic question, but I can’t find the answer and don’t know where else to ask.</p>
<p>If I have a windows 11 machine, with a single GPU, and an Intel CPU <strong>without integrated graphics</strong>, can that GPU be used for PyTorch development?  Or will PyTorch and the output display be fighting over the card?</p>
Return queries, keys and values from nn.multiheadAttention

chrathans: <p>I am using pytorch function for <code>multiheadAttention</code> to build a vision transformer in a similar fashion with this <a href="https://www.akshaymakes.com/blogs/vision-transformer" rel="noopener nofollow ugc">blog</a>. The function looks as follows:</p>
<p><code>self.attn = nn.MultiheadAttention(embed_dim, num_heads,dropout=dropout)</code></p>
<p>Then during the forward pass I am calling it to transform my input embeddings: <code>x = x + self.attn(inp_x, inp_x, inp_x)[0]</code></p>
<p>What I want is to return also the keys, queries and values to visualize the weights and their gradients. Can I do that with <code>nn.MultiheadAttention</code>? Or is there any other implementation that could help with that?</p>
Fine-tuning Denoising Diffusion Probabilistic Model

Vojtech_Kadlec: <p>Hi, I have pretrained Denoising Diffusion Probabilistic Model (DDPM) for image generation and I want to fine-tune it for image segmentation. I am using the official implementation from <a href="https://github.com/lucidrains/denoising-diffusion-pytorch/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lucidrains/denoising-diffusion-pytorch: Implementation of Denoising Diffusion Probabilistic Model in Pytorch</a>.</p>
<p>Does anyone have experience with extracting weights from the U-Net model? The last layer of this implemented U-Unet has only one value as the output but I want output as the tensor with shape (batch, dim, img_shape, img_shape).</p>
<p>Thank you for the answers.</p>
Understanding per layer learning rates with scheduler

funnym0nk3y: <p>Hey everybody,</p>
<p>I’m trying to use a LR-scheduler for transfer learning. My backbone (an efficientnet_v2) should also be tuned, but with lower learning rates than the classifier. AFAIK this could be done like this for at least a fixed learning rate:</p>
<pre><code class="lang-auto">LR = 1e-3

params = [
          {'params': model.conv1.parameters(), 'lr': LR / 10},
          {'params': model.bn1.parameters(), 'lr': LR / 10},
          {'params': model.layer1.parameters(), 'lr': LR / 8},
          {'params': model.layer2.parameters(), 'lr': LR / 6},
          {'params': model.layer3.parameters(), 'lr': LR / 4},
          {'params': model.layer4.parameters(), 'lr': LR / 2},
          {'params': model.fc.parameters()}
         ]


optimizer = optim.Adam(params, lr = FOUND_LR)
</code></pre>
<p>I now want to use a OneCycleLR, how do I get the scheduler to respect the defined LR fractions? How are scheduler and optimizer interacting with each other? Is the scheduler updating one (to me yet unknown) variable or does the scheduler update all the ‘lr’ fields of the parameter groups?</p>

ptrblck: <p>The learning rate schedulers will iterate all <code>.param_groups</code> of the <code>optimizer</code> as seen <a href="https://github.com/pytorch/pytorch/blob/3cc103132205820fc0c571e3e68dd5e9b5b85727/torch/optim/lr_scheduler.py#L45-L164">here</a>.</p>

fnands: <p>Just to go back to the original topic, I can confirm what <a class="mention" href="/u/funnym0nk3y">@funnym0nk3y</a> sees:</p>
<p>when using <code>OneCycleLR</code> it doesn’t respect the difference in layer learning rates (while e.g. <code>CosineAnnealingLR</code> does).</p>
<p>Example:</p>
<pre><code class="lang-auto">optimizer = torch.optim.AdamW([
        {'params': self.backbone.decoder.parameters()},
        {'params': self.backbone.encoder.parameters(), 'lr': self.initial_lr*0.1}
    ], 
    lr=self.initial_lr, 
    weight_decay=self.weight_decay)
</code></pre>
<p>When then using:</p>
<pre><code class="lang-auto">scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
                optimizer, T_max=self.trainer.max_epochs, eta_min=1e-5
            )
</code></pre>
<p>This works as expected, i.e. the LR for the encoder parameters stays a factor of 10 (disregarding <code>eta_min</code> funniness) lower than the decoder LR.</p>
<p>Using <code>OneCycleLR</code> however the decoder and encoder LR is the same at all times, so the behavior differs from <code>CosineAnnealingLR</code>, and from what I would naively expect.</p>
<p>I’m still trying to parse through the <code>OneCycleLR</code> code, but doesn’t seem like it ever refers back to current parameter LR when changing LRs.</p>
Error in data loading

Christian: <p>I run code on someone’s data. Her data is in npz format.<br>
But when I run the code dataloader gives error. I thought it’s data problem. But I can load data by numpy. Could some help me to sort out the problem? Thank you so much.<br>
Here’s the error:<br>
Traceback (most recent call last):<br>
File “train.py”, line 66, in <br>
for i, data in enumerate(trainloader):<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 630, in <strong>next</strong><br>
data = self._next_data()<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1345, in _next_data<br>
return self._process_data(data)<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1371, in _process_data<br>
data.reraise()<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/_utils.py”, line 694, in reraise<br>
raise exception<br>
zlib.error: Caught error in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py”, line 308, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 51, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/export/home/y/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 51, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/export/home/y/NTS-Net/core/dataset.py”, line 129, in <strong>getitem</strong><br>
X, Y = self.data[‘x’][index], self.data[‘y’][index]<br>
File “/export/home/y/.local/lib/python3.8/site-packages/numpy/lib/npyio.py”, line 253, in <strong>getitem</strong><br>
return format.read_array(bytes,<br>
File “/export/home/y/.local/lib/python3.8/site-packages/numpy/lib/format.py”, line 823, in read_array<br>
data = _read_bytes(fp, read_size, “array data”)<br>
File “/export/home/y/.local/lib/python3.8/site-packages/numpy/lib/format.py”, line 958, in _read_bytes<br>
r = fp.read(size - len(data))<br>
File “/usr/lib/python3.8/zipfile.py”, line 940, in read<br>
data = self._read1(n)<br>
File “/usr/lib/python3.8/zipfile.py”, line 1016, in _read1<br>
data = self._decompressor.decompress(data, n)<br>
zlib.error: Error -3 while decompressing data: invalid stored block lengths</p>
Multilinear maps in torch?

Ono: <p>Hi all, first post in the forum! Hoping the community can point me in the right direction.</p>
<p>I’m trying to implement multi-linear tensor maps in torch (more below). I have some implementations that work well, but before I go further down the rabbit hole I thought I’d ask if this is already in the torch library. My cursory searches haven’t yielded anything, but I’m fairly new to this so may not be looking in the right places.</p>
<p>The basic idea is I want to implement a multi-linear tensor map between n-dimensional tensors that obviates the need to flatten and unflatten tensors. So, for example, instead of taking a 2d tensor x^{ij}, flattening it, hitting it with a linear map, then unflattening it, I want to simply implement the multi-linear tensor L^{ij}_{kl} that acts as a multilinear map as follows:</p>
<p>x^{ij} = L^{ij}_{kl} x^{kl}</p>
<p>This can easily be generalized to higher order by increasing the contracted indices. nn.Bilinear is a special case, but it is a highly restricted special case of the above.</p>
<p>For context, the reason the reason I’m interested in this is that these multilinear maps more readily preserve spatial distance information between neighboring pixels, which tend to get lost in the flattening/unflattening steps. For example, for a multi-linear map from a MxM to an NxN tensor, you can easily implement a normalized distance tensor d^{ij}_{kl} as follows:</p>
<p>d^{ij}_{kl} = \sqrt{(i/N-k/M)^2 + (j/N-l/M)^2},</p>
<p>which tells you the normalized euclidean distance between pixel <em>kl</em> in the input tensor and pixel <em>ij</em> in the output tensor. Using this you can do some cool things like introduce a distance based attenuation function, or a non-locality penalty in the error function.</p>
<p>As I said I have implementations of these, but might as well use canned tools if they exists. Does this look familiar? Is it already in the library or in add-ons?</p>

KFrank: <p>Hi Andy!</p>
<aside class="quote no-group" data-username="Ono" data-post="1" data-topic="197615" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/b19c9b/48.png" class="avatar"> Ono:</div>
<blockquote>
<p>I want to simply implement the multi-linear tensor L^{ij}_{kl} that acts as a multilinear map as follows:</p>
<p>x^{ij} = L^{ij}_{kl} x^{kl}</p>
<p>This can easily be generalized to higher order by increasing the contracted indices.</p>
</blockquote>
</aside>
<p>Check out <a href="https://pytorch.org/docs/2.2/generated/torch.einsum.html" rel="noopener nofollow ugc">torch.einsum()</a> for performing general tensor contractions.  It might<br>
be just what you’re looking for.</p>
<p>Best.</p>
<p>K. Frank</p>

Ono: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<p>I implemented it using torch.tensordot, but I’ll look into einsum. Looks like it might be a cleaner implementation.</p>
Can save optimized_model created with torch_tensorrt.compile

Steven_Griset: <p>I’m using torch.tensorrt.compile to inference models, the runtime is working great.  However I would like to save the compiled model and use next time without having to go through compile again.</p>
<p>model = models.resnet50(pretrained=True).eval()<br>
At the end of the compile I get this information<br>
INFO optimized model type<br>
&lt;class ‘torch._dynamo.eval_frame.OptimizedModule’&gt;<br>
How do I save this?  The IR I’m is ir="torch_complie(only one that works) not dynamo.</p>

ptrblck: <p>I don’t know if the experimental <a href="https://pytorch.org/tutorials/intermediate/torch_export_tutorial.html"><code>torch.export</code></a> util. supports TorchTRT, but <a class="mention" href="/u/narendasan">@narendasan</a> might know.</p>

Steven_Griset: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for help I tried the export approach but with torch_tensorrt.dynamo.export (torch.export does not support TorchTRT) but it I got the following exception with that approach:<br>
[ERROR] Preprocessing failed: Detected that you are using FX to torch.jit.trace a dynamo-optimized function. This is not supported at the moment.</p>

narendasan: <p>In general you cannot serialize torch.compile outputs. torch.export is the right tool for this. The error you saw may be from a previous version of torch-tensorrt. Please take a look at this section of the docs <a href="https://pytorch.org/TensorRT/user_guide/saving_models.html" class="inline-onebox" rel="noopener nofollow ugc">Saving models compiled with Torch-TensorRT — Torch-TensorRT v2.3.0.dev0+85971ff documentation</a></p>
Trouble following instance segmentation finetuning tutorial

trueenglishuser: <p>I was following <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial from pytorch</p>
<p>unfortunatly due to being a new user i can only put one image here. So i cant provide clean examples of images from the dataset in this post, only the model output</p>
<p>I’m working on a problem: there are metal parts (in this case metal nuts) scattered in a box. Some of them are unobscured and lie ~flat (within angle threshold), while some lie on the edge or are obscured by other parts. I need to detect the first class of parts.</p>
<p><strong>After finetuning resnet50 im getting results like this:</strong><br>
Model output<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/5359c3f8b770a8177d8fbb531434bbf71287a369.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5359c3f8b770a8177d8fbb531434bbf71287a369" title="output"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/5359c3f8b770a8177d8fbb531434bbf71287a369.jpeg" alt="output" data-base62-sha1="bTlX0CkF9VFppld1VpA4knygcel" width="506" height="500" data-dominant-color="633953"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">output</span><span class="informations">672×664 74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In this particular dataset i have 100 images, i did try with a different one that has 1000 images (but the quality wasnt very realistic so i scraped that)</p>
<p><strong>I mainly have these questions:</strong></p>
<ol>
<li>what loss function is used in this tutorial? from my digging in source code it seems to be calculated somewhere very deep within the model class and theres no way to set loss function that i want</li>
</ol>
<p>To that end, whatever is being outputed as loss looks like this over 100 epochs:<br>
Starts at 4.65, drops to 4.50 over first 15 epochs and then oscillates there till the end.</p>
<ol start="2">
<li>
<p>Why are boudning boxes such a mess? the masks in the output are not too bad, but the bounding boxes are all over the place</p>
</li>
<li>
<p>Am i even using the right model? in the tutorial examples the masks didnt necessarily outline every person in the picture, only the fully visible ones so it seemed like what im looking for, but the results are still pretty bad</p>
</li>
</ol>
<p>any advice from your experience working on similar problems is also appritiated</p>
PyTorch 2.1.1 build error

FuncJ: <p>Hello everyone, I encountered a build problem when compiling PyTorch, as follows:</p>
<p>/home/xx/project/pytorch-v2.1.1/caffe2/perfkernels/common_avx512.cc:18:2: error: <span class="hashtag-raw">#error</span> ( "You found a build system error: <strong>AVX512F</strong>, <strong>AVX512DQ</strong>, <strong>AVX512VL</strong> " "is defined (via e.g. -mavx512f, -mavx512dq, and -mavx512vl) " “but CAFFE2_PERF_WITH_AVX512 is not defined.”);</p>
<pre><code class="lang-auto">command:
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}

export BLIS_HOME=/xx/xx/blis

export PATH=$BLIS_HOME/include/blis:$PATH LD_LIBRARY_PATH=$BLIS_HOME/lib:$LD_LIBRARY_PATH

export BLAS=BLIS

USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 USE_FLASH_ATTENTION=0 BUILD_TEST=1 BUILD_CAFFE2=1 BUILD_CAFFE2_OPS=1 USE_OPENMP=1 python setup.py develop 2&gt;&amp;1 | tee m.txt
</code></pre>

ptrblck: <p>Caffe2 is deprecated and I don’t think its build is tested so disable it again of just remove <code>BUILD_CAFFE2=1</code> to use the default (which won’t build with it).</p>
Dynamically change a models forward function during runtime

Stephen_Condon: <p>I am attempting to search for good branches in a model by dynamically adding a branch in the forward function by editing the forward function ast, unparsing the updated ast and writing it back to the file where the model is stored, and making a new object based on the updated model, with the old weights reloaded. But I can’t get the model to use the new forward function unless the process is completely shutdown and manually restarted. Is it even possible to edit a model in such a manner, or to reload its forward function at runtime?</p>
<p>An example workflow of how I’m attempting it is:</p>
<ol>
<li>Have a model ModelA in models.py</li>
<li>In training.py, train the model</li>
<li>In generation.py, take a reference to the model, get the ast of the model, and edit the forward function</li>
<li>Unparse the edited ast and rewrite to models.py</li>
<li>Save the weights of the model</li>
<li>Reload models.py with importlb.reload</li>
<li>Create a new model object</li>
<li>Load the saved weights to the new object</li>
<li>Train again to train the branch layers</li>
</ol>
<p>After using importlib.reload, the model still has the old forward function in memory and does not use the updated one unless I kill the script and restart it, but I need to reload during runtime without killing the script.</p>

JuanFMontesinos: <p>Well I think you can do that without writting the file<br>
if you call “eval” over a text string you should be able to create a working function.<br>
then you just need to assign self.forward=new_fn and that’s all<br>
You can ofc dump the function somewhere but that’s more logging that a requirement.</p>
<p>Another options is ofc given your new files a different name so you skip all the import issues.</p>

Stephen_Condon: <p>Sorry for getting back so late, I was actually just being stupid. I was still recreating the object with the old non reloaded module. I would have rathered not to write the new model to a file, but I need to show how the model is being affected for what I’m doing.</p>
Is there a dilated k-nearest neighbour solution available fast execution?

Adex: <p>I am implementing the dilated k Neares Neighbour algorithm. The algorithm unfortunately has nested loops. The presence of loops severely hampers the execution speed.</p>
<pre><code>import torch
dilation=3
nbd_size=5
knn_key=torch.randint([0,30,(64,12,198,100)])


dilated_keys=torch.zeros([knn_key.shape[0],knn_key.shape[1],knn_key.shape[2],nbd_size])

for i in range(knn_key.shape[0]):
    for j in range(knn_key.shape[1]):
        for k in range(knn_key.shape[2]):
            list_indices=[]
            while (len(list_indices))&lt;nbd_size:
                for l in range(knn_key.shape[3]):
                    if knn_key[i][j][k][l]%dilation==k%dilation:
                        list_indices.append(knn_key[i][j][k][l])
                        if (len(list_indices))&gt;=nbd_size:
                            break
            list_indices_tensor=torch.tensor(list_indices)
            dilated_keys[i][j][k]=list_indices_tensor
</code></pre>
<p>The variable <code>knn_key</code> stores the <code>100</code> nearest neighbours among the 1000 data points originally available. The <code>dilated_keys</code> stores the <code>nbd_size=5</code> selected indices of the neighbours that are used after applying dialation filter. Any help to use broadcasting solution to remove the three nested loops will be highly helpful.</p>

Adex: <p>Looks like the inner loop can be reduced as follows,</p>
<pre><code class="lang-auto">import torch

dilation = 3
nbd_size = 5
knn_key = torch.randint(0, 30, (64, 12, 198, 100))

dilated_keys = torch.zeros((knn_key.shape[0], knn_key.shape[1], knn_key.shape[2], nbd_size), dtype=torch.int64)

for i in range(knn_key.shape[0]):
    for j in range(knn_key.shape[1]):
        for k in range(knn_key.shape[2]):
            key = knn_key[i, j, k]
            indices = torch.nonzero(key % dilation == k % dilation).squeeze()
            selected_indices = indices[:nbd_size]
            dilated_keys[i, j, k] = key[selected_indice
</code></pre>
<p>However three nested loops are still there. Any help will be highly helpful. <a href="https://stackoverflow.com/questions/78015382/is-there-a-dilated-k-nearest-neighbour-solution-available-fast-execution/78019217#78019217" rel="noopener nofollow ugc">[1]</a></p>
Prediction task

Ryz: <p>I got this error when I am running my main code, I need your help"<br>
Traceback (most recent call last):<br>
File “C:\Users\solo\Desktop\EXPERIMENT\ML-master\main.py”, line 188, in <br>
w, loss = local.train(net=copy.deepcopy(net_glob).to(args.device))<br>
File “C:\Users\solo\Desktop\EXPERIMENT\ML-master\models\Update.py”, line 59, in train<br>
loss = self.loss_func(log_probs, labels)<br>
File “C:\Users\solo\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\solo\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\loss.py”, line 1164, in forward<br>
return F.cross_entropy(input, target, weight=self.weight,<br>
File “C:\Users\solo\AppData\Roaming\Python\Python310\site-packages\torch\nn\functional.py”, line 3014, in cross_entropy<br>
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)<br>
RuntimeError: “nll_loss_forward_reduce_cuda_kernel_2d_index” not implemented for ‘Float’</p>
Converting float16 tensor to numpy causes rounding

gholste: <p>I am running mixed-precision training, converting my model’s outputs (float16) to numpy, and storing those outputs for later evaluation. I just noticed that all numpy arrays are rounded to 2 decimal points! Why would this happen?! This is very easily reproducible:</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([64.6250, 61.7812, 61.8750, 64.7500, 64.5000], dtype=torch.float16)
print(x)  # tensor([64.6250, 61.7812, 61.8750, 64.7500, 64.5000], dtype=torch.float16)
print(x.numpy())  # array([64.6 , 61.78, 61.88, 64.75, 64.5 ], dtype=float16)
</code></pre>
<p>The solution I’ve found is simply to run <code>x.float().numpy()</code>… but why is this happening in the first place? And, no, this is not a matter of printing preferences; I can set the precision as high as I want with np.set_printoptions and it is still rounded to 2 decimal places.</p>
<p>Here is my PyTorch version info from <code>conda list</code>:</p>
<pre><code class="lang-auto">pytorch                   2.1.0           py3.10_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h7e8668a_5    pytorch
</code></pre>
<p>Linux version info:</p>
<pre><code class="lang-auto">NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
</code></pre>

ptrblck: <p>Increase the <code>precision</code> in <code>numpy</code>’s <code>printoptions</code> and it’ll show the same values:</p>
<pre data-code-wrap="python"><code class="lang-python">np.set_printoptions(precision=4, floatmode="fixed")
print(x.numpy())
# [64.6250 61.7812 61.8750 64.7500 64.5000]
</code></pre>

gholste: <aside class="quote no-group" data-username="gholste" data-post="1" data-topic="197690">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/53a042/48.png" class="avatar"> gholste:</div>
<blockquote>
<pre><code class="lang-auto">x = torch.tensor([64.6250, 61.7812, 61.8750, 64.7500, 64.5000], dtype=torch.float16)
</code></pre>
</blockquote>
</aside>
<p>Wow, okay thanks! I tried this, but did not specify the proper <code>floatmode</code>. I guess I don’t understand the default numpy print option behavior here: 64.625 is rounded to 64.6 but 61.875 is rounded to 61.88?</p>
<p>This was causing issues for me because I was storing these numpy arrays in pandas data frame cells (now I see why this is not recommended…), causing them to be represented as string objects, meaning I was actually using the 2-decimal-place-rounded values for evaluation.</p>
Can the cumulative layer normalization (cLN) be added?

Max_Liu: <p>The cumulative layer normalization proposed by the Conv-TasNet paper (<a href="https://arxiv.org/abs/1809.07454" class="inline-onebox" rel="noopener nofollow ugc">[1809.07454] Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation</a>) has been validated by multiple papers to have excellent effects, but I am unable to implement this Normalization Layer myself. Could you help with its implementation?</p>
Scaled_dot_product_attention implementations

zhong2: <p>Is PyTorch’s memory efficient attention implementation of scaled_dot_product_attention<br>
the same as xFormer’s memory_efficient_attention (which uses Flash-Decoding according to <a href="https://pytorch.org/blog/flash-decoding/" class="inline-onebox" rel="noopener nofollow ugc">Flash-Decoding for long-context inference | PyTorch</a>)? I’m interested in testing Flash-Decoding and PyTorch’s scaled_dot_product_attention page (<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.scaled_dot_product_attention — PyTorch 2.2 documentation</a>) links to xFormer’s GitHub.</p>
How to understand this schematic?

YPOC: <p>Hello,</p>
<p>I’m a pytorch beginner and also new to CNNs. I’m trying to apply a model from a paper to my problem, however I get very poor results (R² = 0.1 for regression). I think I don’t understand the schematic drawing of the CNN used in the paper. Could you please help me understand it?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f6a33f762756369596f0de1b1ad84bc1491e5d1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3f6a33f762756369596f0de1b1ad84bc1491e5d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f6a33f762756369596f0de1b1ad84bc1491e5d1_2_690x348.jpeg" alt="image" data-base62-sha1="92ZCpmNQcupTScyhELuvzIOrJcJ" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f6a33f762756369596f0de1b1ad84bc1491e5d1_2_690x348.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f6a33f762756369596f0de1b1ad84bc1491e5d1.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f6a33f762756369596f0de1b1ad84bc1491e5d1.jpeg 2x" data-dominant-color="EEEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">782×395 69.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the schematic of what the authors call convp-20(1x2x2). As input they are using an image with several channels, that represent the principal components. Is my implementation correct?</p>
<pre><code class="lang-auto">step1 = nn.Sequential(                                                 # in_dim = (n, n_components, 15, 15)
    nn.Conv2d(n_components, n_components*20, 2, groups=n_components),  # dim = (n, n_components*20, 14, 14)
    nn.BatchNorm2d(n_components*20),
    nn.ReLU(),
    nn.MaxPool2d(2),                                                   # dim = (n, n_components, 7, 7)
)
</code></pre>
<p>The next step in the paper is called convp-20(20x2x2). This comes immediately after convp-20(1x2x2) and is repeated 3 times. Is my Implementation correct? I’m very unsure about this step.</p>
<pre><code class="lang-auto">step2 = nn.Sequential(
    nn.Conv2d(n_components*20, n_components, 2, groups=n_components),  # dim = (n, n_components, 6, 6)
    nn.BatchNorm2d(n_components),
    nn.ReLU(),
    nn.MaxPool2d(2),                                                   # dim = (n, n_components, 3, 3)
)
</code></pre>
<p>As final layer they use this. I think my code should be correct?</p>
<pre><code class="lang-auto">step_final = nn.Sequential(
    nn.Flatten(),
    nn.Linear(20*3*3, 1)  # I want to use regression instead of classification
)
</code></pre>
<p>If you want to read the paper, you can <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7146316/" rel="noopener nofollow ugc">find it here</a> and look under the section “4. 2D-CNN with Principal Components”</p>
In image segmentation, how to deal with the inconsistency between pred and true shape when calculating loss

mst72: <p>For example，I’m calculating dice loss right now.<br>
My pred shape is [8,2,512,512], representing batch, channel, H, W<br>
My true label shape is [8,512,512], representing batch,  H, W<br>
Calculating loss requires the same shape, how do I handle it?</p>

mst72: <p>my true label is a mask</p>

ptrblck: <p>If you are dealing with a multi-class segmentation, the shapes will work for <code>nn.CrossEntropyLoss</code>.</p>
Captum LLMAttribution generates AttributeError: 'NoneType' object has no attribute 'get'

Yemmy1000: <p>Hi. I have used Captum for LLaMA-2 successfully before. But now I wanted to use it again and it generates the error below. I also copy and pasted the example from <a href="https://captum.ai/tutorials/Llama2_LLM_Attribution" rel="noopener nofollow ugc">Understanding Llama2 with Captum LLM Attribution</a> same problem.</p>
<p>What is the problem. Somebody help!</p>
<pre><code class="lang-auto">
Traceback (most recent call last):

  Cell In[6], line 9
    attr_res = llm_attr.attribute(inp, target=target)

  File ~/Documents/Research/January 2024/captum/captum/attr/_core/llm_attr.py:375 in attribute
    cur_attr = self.attr_method.attribute(

  File ~/Documents/Research/January 2024/captum/captum/log/__init__.py:42 in wrapper
    return func(*args, **kwargs)

  File ~/Documents/Research/January 2024/captum/captum/attr/_core/feature_ablation.py:289 in attribute
    initial_eval = self._strict_run_forward(

  File ~/Documents/Research/January 2024/captum/captum/attr/_core/feature_ablation.py:599 in _strict_run_forward
    forward_output = _run_forward(*args, **kwargs)

  File ~/Documents/Research/January 2024/captum/captum/_utils/common.py:531 in _run_forward
    output = forward_func(

  File ~/Documents/Research/January 2024/captum/captum/attr/_core/llm_attr.py:249 in _forward_func
    model_kwargs = self.model._update_model_kwargs_for_generation(

  File ~/anaconda3/envs/llmft/lib/python3.11/site-packages/transformers/generation/utils.py:681 in _update_model_kwargs_for_generation
    model_kwargs["cache_position"] = model_inputs.get("cache_position", None)

AttributeError: 'NoneType' object has no attribute 'get'
</code></pre>
Custom backward step for convolutions

giommariapilo: <p>Hello, I’m trying the way the inputs of the backward pass for different layers and the way the output gradients are saved are saved. In short I managed to write a backward computation for a linear layer that computes the gradient only for a subset of the weights, but I can’t find any example of the backward pass implementation for a convolutional layer. Alternatively, is there a way to call the convolutional backward function inside my backward?</p>

albanD: <p>Hey!</p>
<p>I think <a href="https://github.com/pytorch/pytorch/blob/d3839b624b5f6451a13bd9b5ecbbce4c2a9b1db6/test/test_flop_counter.py#L137-L161" class="inline-onebox">pytorch/test/test_flop_counter.py at d3839b624b5f6451a13bd9b5ecbbce4c2a9b1db6 · pytorch/pytorch · GitHub</a> contains an implementation of convolution backward as a set of convolution ops.<br>
It doesn’t cover all the arguments of a real convolution but is a good base for what you want I think</p>

giommariapilo: <p>Yes that was a good way to start but in the end I found out about torch.nn.grad.conv2_input and torch.nn.grad.conv2_weight and that worked even better! Thank you for your answer though</p>

albanD: <p>Hey!</p>
<p>the torch.nn.grad functions are not necessarily accurate so I wouldn’t rely on them for anything serious (that’s why I didn’t suggest them here).<br>
But if they work for you, that’s good.</p>

giommariapilo: <p>Hi! Sorry but can you clarify what you mean by not being accurate? I thought they replicated the backward behavior of a conv layer?</p>

albanD: <p>They are not used much and so not very carefully reviewed. We had quite a few issues about it before: <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aissue+nn.grad+is%3Aclosed" class="inline-onebox">Issues · pytorch/pytorch · GitHub</a></p>

giommariapilo: <p>I see, thank you, I’ll evaluate accordingly whether they suit my use… I think provisionally they’ll stay before I move to something more reliable</p>
Conv on CUDA docs

vitaliivovk: <p>Hi!<br>
I want to understand how convolution inference works on CUDA. I found ATen sources, but it’s a bit hard to understand just from code. If you could share some docs it would help me.<br>
Especially interesting how parallelization is implemented.<br>
Thanks!</p>

vitaliivovk: <p>I still need answer to this, thanks</p>

ptrblck: <p>It seems you’ve already found the code, which should show a native <code>im2col</code> followed by a <code>matmul</code> approach and other paths which dispatch to accelerating libraries, such as <code>cuDNN</code>. Which part exactly are you struggling with?</p>
How to deal SQL query in tabular dataset?

iamexperimentingnow: <p>Hi, I have a dataset which contains numerical columns, categorical columns and SQL Query as a separate column. Can someone recommend me how to deal with it?</p>
<p>I would like to get suggestion on how to deal with SQL Query column? what sort of feature engineering should I apply for SQL Query column?</p>
<p>Is there any specific model to deal with this?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p><a class="hashtag-cooked" href="/c/nlp/8" data-type="category" data-slug="nlp" data-id="8"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>nlp</span></a> <a class="hashtag-cooked" href="/c/data/37" data-type="category" data-slug="data" data-id="37"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>data</span></a> <a class="hashtag-cooked" href="/c/projects/33" data-type="category" data-slug="projects" data-id="33"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>projects</span></a> <a class="hashtag-cooked" href="/c/uncategorized/1" data-type="category" data-slug="uncategorized" data-id="1"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>Uncategorized</span></a></p>

JamesTrick: <p>This really depends on what you’re trying to achieve. If you’re doing a pure numerical prediction for example, the SQL query may not add any value to your model.</p>
<p>On the other hand, if you’re looking at generating SQL through, starting off with a standard NLP approach of tokenising, then developing a language model would be a good start.</p>

iamexperimentingnow: <p><a class="mention" href="/u/jamestrick">@JamesTrick</a>  really?<br>
I wanted to predict numerical value, and I feel SQL query will add weightage to the model. so, hence why I thought to check with community, is there any better feature engineering method to deal with SQL query.</p>
How to train a parameter initialized outside of nn.Module

thanrl: <p>I have a somewhat global parameter that I’d like to train.</p>
<pre><code class="lang-auto">global_param = torch.tensor(...).requires_grad_()
</code></pre>
<p>This <code>global_param</code> is used to initialize parts of a network</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, global_param):
        super().__init__()
        self.param1 = global_param*2
        self.param2 = global_param/2
        # other nn.Parameters
    def forward(self, x):
        # the forward pass uses self.param1, self.param2 and other nn.Parameters
</code></pre>
<p>If I write a normal training loop, I get the “Trying to backward through the graph a second time” error.</p>
<pre><code class="lang-auto">net = Model(global_param)
optimizer = torch.optim.Adam(list(net.parameters())+[global_param])
for sample in data:
    pred = net(sample)
    loss = loss_fn(pred, target)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Understandably, because part of the computation graph <code>self.param1 = global_param*2; self.param2 = global_param/2</code> is created before the forward pass. This error is gone if I do <code>loss.backward(retain_graph=True)</code>, but it results in this new error</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor []] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>I am having a hard time debugging this in-place error, probably related to <a href="https://discuss.pytorch.org/t/problem-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/131107/4">retaining the computation graph</a>. So Is there a better way to write this training loop without setting <code>retain_graph=True</code>? Thank you!</p>
Does Pytorch support Linux with Apple Silicon?

Billy_Mazotti: <p>I would like to be able to use mps in my Linux VM (my setup is Mac M1 + Ubuntu 22.04 via VMWare Fusion), however it seems like there are two major barriers in my way/questions that I have:</p>
<ol>
<li>Does there exist a Linux + arm64/aarch64 with M1 Pytorch build?</li>
</ol>
<p>I have not been able to find such a build. From what I’ve seen, most people who are looking for a Linux + arm64/aarch64 build are typically using NVIDIA GPUs.</p>
<ol start="2">
<li>Is VMware’s 3D acceleration actually providing “expected” GPU passthrough?</li>
</ol>
<p>By expected, I mean that I expect the OpenGL values for Renderer to be the same regardless of whether I’m on host (macOS) or guest (Ubuntu). Following Antonin Carette’s tutorial <a href="https://carette.xyz/posts/opengl_and_cpp_on_m1_mac/" rel="noopener nofollow ugc">OpenGL, in C++, on m1 mac</a>, I get the following print out for my GPU renderer specs:</p>
<pre><code class="lang-auto">Renderer: Apple M1 Max
OpenGL version supported: 4.1 Metal - 86
</code></pre>
<p>however when I follow GetIntoGameDev’s tutorial <a href="https://www.youtube.com/watch?v=LxEFn-cGdE0" rel="noopener nofollow ugc">Setting up an OpenGL Project in Ubuntu</a> (and add in the print out lines from the m1 mac tutorial), I get the following print out:</p>
<pre><code class="lang-auto">Renderer: SVGA3D; build: RELEASE;  LLVM;
OpenGL version supported: 4.3 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
</code></pre>
<p>I have no idea if the change in renderer names makes a difference, however I thought it was worth mentioning as another possible issue.</p>
Model.eval() in predicting step yields SyntaxError

sebastianwjc: <p>I adopted a 2-layer neural network in the tutorial:</p>
<pre><code>model = nn.Sequential(
    nn.Linear(D_in, H),
    nn.Tanh(),
    nn.Linear(H, D_out),
)
</code></pre>
<p>Then I trained the model, save and load the model in the way as they mentioned <a href="https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch" rel="nofollow noopener">here</a> (second answer, case <span class="hashtag">#1</span>). But when the output yields a SyntaxError:</p>
<pre><code>  File "eval01.py", line 63
    model.eval() 
        ^
SyntaxError: invalid syntax
</code></pre>
<p>I don’t have any Batchnorm or Dropout layer in my model. Is this the reason of this error?</p>

ptrblck: <p>Could you post the code for loading the state dict?<br>
Your error is most likely due to a typo, e.g. a missing parenthesis.</p>

sebastianwjc: <p>Thanks for the hint! It’s indeed a missing parenthesis at the very beginning of the code and the mismatch tracks down to these line. I’m really really sorry <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=5" title=":sob:" class="emoji" alt=":sob:"></p>

Akash_Sardar: <p>Gosh. Same thing happened to me too <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>