akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to dynamically change the dimensions of a layer in the forward pass without losing the weights?

username: sals13
cooked: <p>Hi all,</p>
<p>I am trying to calculate the input of a linear layer based on the output of the previous convolutional layer. This calculation is dependent on the input size; for each input the output of a conv layer will be different. Below is my network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self,k=16):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=(3,3)) 
        self.act1 = nn.ReLU()
        self.conv2 = nn.Conv2d(16, 16, kernel_size=(3,3))
        self.act2 = nn.ReLU()
        self.conv3 = nn.Conv2d(16, 16, kernel_size=(3,3))
        self.act3 = nn.ReLU()
        self.flat = nn.Flatten()
        self.fc = nn.Linear(k, k//2) # k is to be calculated with each step.
        self.act4 = nn.ReLU()
        self.fc2 = nn.Linear(k//2 ,1)
    def forward(self,x):
        if x.size(-1) &lt;= 4 or x.size(0) &lt;= 4:
            x = self.act1(self.conv1(x))
        elif x.size(-1) &gt;= 32 and x.size(0) &gt;= 32:
            x = self.act1(self.conv1(x))
            x = self.act2(self.conv2(x))
            x = self.act3(self.conv3(x))
        else:
            x = self.act1(self.conv1(x))
            x = self.act2(self.conv2(x))
            
        x = self.flat(x).T
        # here the dimension should be calculated 
        # and here the new layer dimensions should be updated
        x = self.act4(self.fc(x))
        x = self.fc2(x)
        return x
</code></pre>
<p>PS: k is the number of output neurons in the last convolutional layer<br>
and my input is in the form [128,1,n,m] where n and m &lt;=16 and are powers of 2 &gt; 2, i.e., n,m = 4,8,16. Any idea on how to change the input size with each pass without losing the weights? If I simply define a new layer (i.e., self.fc = nn.Linear(k, k//2) after calculating k) the gradient won’t converge.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>The weight matrix of a linear layer has a predefined shape and changing it by initializing a new module won’t work since you would lose the already trained parameters. You could try to define a custom weight <code>nn.Parameter</code> and either slice or stack it with a new parameter. However this approach would also train only a subset of the parameter of the input is smaller than the largest expected shape.<br>
The common approach is to use adaptive pooling layers after the last conv layer which allows you to define the output shape and thus also the <code>in_features</code> of the first linear layer.</p>

username: sals13
cooked: <p>Thanks for the reply.</p>
<p>If I use pooling, don’t I need to define the output size manually? If so, this is not what I want. I want to extract the output size from the last conv layer and use it in the fc layer. This is the description of the network that I am trying to realise “The convolutional layers are followed by two fully connected layers, with k/2 and 1 output neurons, respectively, where k is the number of output neurons in the last convolutional layer.”</p>
Segmentation fault (core dumped) with torch.compile

username: Qian_Li
cooked: <p><strong>Describe the Bug</strong><br>
when I run this code, error with “Segmentation fault (core dumped)” appeared. Does someone know how to resolve it?</p>
<pre><code class="lang-auto">import torch

batch_n = 100
input_data = 10000
hidden_layer = 100
output_data = 10

class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.lr1 = torch.nn.Linear(input_data, hidden_layer, bias=False)
        self.relu = torch.nn.ReLU()
        self.lr2 = torch.nn.Linear(hidden_layer, output_data, bias=False)
    def forward(self, x):
        x = self.lr1(x)
        x = self.relu(x)
        x = self.lr2(x)
        return x

device = torch.device("cuda:0")
input = torch.randn(batch_n, input_data).to(device)
input.requires_grad = True
label = torch.randn(batch_n, output_data).to(device)

model = MyModel().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

compiled_model = torch.compile(model)
compiled_model.train()
optimizer.zero_grad()
out = compiled_model(input)
loss = loss_fn(out, label)
loss.backward()
optimizer.step()
</code></pre>
<p><strong>Environment</strong></p>
<p>Python version: 3.8.13<br>
torchversion: 1.14.0<br>
CUDA version: 11.7</p>

username: marksaroufim
cooked: <p>Looks like your’e getting the error even without <code>torch.compile()</code> - I’ve seen that erro show up if something is off with my CUDA installation. Worth trying out a fresh environment</p>

username: James_Liu
cooked: <p>I have the same issue when compiling the model. Without compiling, there is no segmentation fault. I run on RTX 2080 Ti.</p>
<p>NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6</p>

username: Wheest
cooked: <p>I also had this issue, which seemed to occur on both CPU and GPU.</p>
<p>However, the issue went away on CPU when running <code>CUDA_VISIBLE_DEVICES=""</code> first.  This allowed a CPU inference using <code>torch.compile</code> to run.</p>
<p>I assume that something is up with my CUDA config.  My PyTorch version is <code>2.0.1+cu117</code>, but my CUDA version is <code>12.1</code>.  However <a href="https://discuss.pytorch.org/t/install-pytorch-with-cuda-12-1/174294/16">in principle this should not be an issue</a>.  Will try a fresh Docker image.</p>
TorchConfig.cmake and MS Visual debug / release projects at once

username: Sebastien_Maraux
cooked: <p>Hello,</p>
<p>From what I have found so far, it seems not really easy, or even doable at all, to build a standard visual studio project with two configurations Release and Debug, whereas it is a commmon practice for windows projects.</p>
<p>As the FindPackage(Torch) relies on the TorchConfig.cmake and caffe2Target.cmake, the cleanest modifications I see that could allow a standard behavior compliant with cmake workflow, would be to use set_property on each dependency lib target, with IMPORTED_LOCATION_RELEASE<br>
and IMPORTED_LOCATION_DEBUG. This is not a trivial work, and it should be done again for each version if it was done on customer side.</p>
<p>Would it be possible to do the required work for the current version ? It may also mean that the debug version lib for windows may include the release libs to insure a consistant directory tree ?</p>
<p>What do you think about that ?</p>
<p>Thank you for reading</p>
Convert a tensorflow model to PyTorch

username: yavatan
cooked: <p>Hi,<br>
I need to convert a model with PyTorch codes.<br>
CLASSES = 5<br>
input_size = 224<br>
channels = 3<br>
Here is the tensorflow code:</p>
<p>def build_feature_extractor(inputs):</p>
<pre><code>x = tf.keras.layers.Conv2D(16, kernel_size=3, activation='relu', input_shape=(input_size, input_size, channels))(inputs)
x = tf.keras.layers.AveragePooling2D(2,2)(x)

x = tf.keras.layers.Conv2D(32, kernel_size=3, activation = 'relu')(x)
x = tf.keras.layers.AveragePooling2D(2,2)(x)

x = tf.keras.layers.Conv2D(64, kernel_size=3, activation = 'relu')(x)
x = tf.keras.layers.AveragePooling2D(2,2)(x)

return x
</code></pre>
<p>Now I need to convert it to PyTorch.<br>
Conv2d in TensorFlow receive a parameter =&gt; input_shape=( *, *, *)<br>
Is it necessary to define input_shape in PyTorch? If the answer is yes, then how can I define it?</p>
<p>Thank you</p>

username: eqy
cooked: <p>You do not need to specify the input shape to define a Conv2D layer in PyTorch, only the number of input channels, as that is all that is needed to determine the shape of the weights.</p>
<p>See the Conv2D docs here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Conv2d — PyTorch 2.0 documentation</a> which contains information about which arguments are required to define the layer.</p>

username: yavatan
cooked: <p>That means Conv2d in PyTorch is flexible and I don’t need to specify the image size and width.<br>
Thank you eqy.</p>
Uploading functions and classes on gpu in pytorch

username: Vaibhav_Chahar
cooked: <p>Hi to all,<br>
Is it possible to upload any external function or class present in a external module on GPU in pytorch?<br>
I tried to do it by<br>
I_Val=pf.linear_index(m,n).to(dev)<br>
but it says ::<br>
AttributeError: ‘list’ object has no attribute ‘to’<br>
Please help!!</p>

username: ptrblck
cooked: <p>No, you cannot move arbitrary objects to the GPU as the underlying library/frameworks needs to support GPUs. Your current code snippet fails since you are trying to move a plain <code>list</code> to the GPU, which Python doesn’t understand as it does not support GPUs natively.<br>
Instead, create a PyTorch tensor and move this object to the device.</p>

username: Vaibhav_Chahar
cooked: <p>Hi,<br>
def linear_index(m,n):<br>
Mu=[]<br>
for i in range(m):<br>
for j in range(m):<br>
for k in range(m):<br>
for l in range(m):<br>
for p in range(n):<br>
x=m<em>m</em>m<em>n</em>i+m<em>m</em>n<em>j+m</em>n<em>k+n</em>l+p<br>
Mu.append(x)<br>
I_Val_1=[]<br>
for i in Mu:<br>
a=(i//(m<em>m</em>m<em>n))<br>
b=(i%(m</em>m<em>m</em>n))//(m<em>m</em>n)<br>
c=((i%(m<em>m</em>m<em>n))%(m</em>m<em>n))//(m</em>n)<br>
d=(((i%(m<em>m</em>m<em>n))%(m</em>m<em>n))%(m</em>n))//n<br>
e=(((i%(m<em>m</em>m<em>n))%(m</em>m<em>n))%(m</em>n))%n<br>
I_Val_1.append([a,b,c,d,e])<br>
return torch.as_tensor(I_Val_1)<br>
This is the function i am talking about. in short it is giving a nested lists as [[1,1,1,1,1][2,1,2,0,1]…]. It is a tensor, if i send it .to(dev). Will it be on GPU?</p>

username: ptrblck
cooked: <p>Your code is quite unreadable as it’s not properly formatted, but I assume you want to call <code>to(device)</code> on the returned tensor. If so, then yes the result will be moved the the GPU.</p>

username: Vaibhav_Chahar
cooked: <p>Hi,<br>
I have a question, while running the code on GPU, my code is finished running but the memory allocated to the GPU is still remain. I want to remove this memory. I tried del tensors, model etc. but it is still there.<br>
I used command nvtop to see the usage and stats of GPU, after finishing it shows this, 894MiB GPU memory is still there. How can I completely remove it.<br>
gauge_neural.py  is the name of my python script.</p>
<pre><code>PID    USER DEV    TYPE  GPU        GPU MEM    CPU  HOST MEM Command                                                                                                                                                                   
</code></pre>
<p>1265975 vchahar   0 Compute   0%    894MiB   8%     0%   2429MiB python3 gauge_neural.py</p>

username: ptrblck
cooked: <p>The ~900MB could be used by the CUDA context and will be freed once you close your application.</p>
[Q] YoloV3/V4: Where to position a sequence-model to model the spatio-temoporal signal in video data?

username: weight_theta
cooked: <p>Dear Community,</p>
<p>I am currently working on turning a YoloV3/V4 Network into a video object detector. That is, we want to add simple sequence model like an RNN or transformer to the architecture to model. We are currently not sure where to place the sequence model within the architecture that looks roughly as followed:</p>
<ol>
<li>EfficentNet Backbone</li>
<li>Neck consisting of Spatial Pyramid Pooling and a Path Aggregation Network</li>
<li>A prediction head consisting of a linear convolution</li>
</ol>
<p>In particular, we feel that it is a little tricky, since YoloV3/V4 makes prediction on the feature maps using a linear convolution. That is unlike a sequence model it does not make predictions on a flattened input.</p>
<p>There are a number of approaches which we think can work, but they come with their own problems:</p>
<ol>
<li>
<p>Repalce the prediction head with an RNN:  flatten the output of the preceedig layer, give it as input to the RNN and predict Nclasses x (class_prob, x, y, height width), however the output can quickly grow with 1000 classes there are 1000 * 5 = 5000 predictions. Can RNNs handle these type of predictions? Loss adjustment may be neccessary, but doable.</p>
</li>
<li>
<p>Add an RNN right after the backbone: flatten the output of the backbone, give it as input to the RNN. Here the problem arises on how we can adjust the output shape of the RNN for the following layers deeper down the architecture i.e., the Neck ? Furthermore, the Network then also only aggregates the spatio-temporal signal up to the backbone and not further, which may not be ideal.</p>
</li>
</ol>
<p>If there is a more suitable approach or place to put the RNN within the architecture please let us know.</p>
<p>all the best,<br>
weight_theta</p>
"Expected to have finished reduction" error when dropping layers with DDP

username: divinho
cooked: <p>I want to do layerdrop. Unfortunately when I do so I get an error:<br>
<code>Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. [...]</code></p>
<p>I thought this could be caused by the fact that I was doing DDP and different workers were dropping different layers leading to problems when trying to sync gradients. However I set the seed so the different workers should be dropping the same layer at the same time and I still get the error.</p>
<p>Why is this happening?</p>

username: wanchaol
cooked: <p>Thanks for posting <a class="mention" href="/u/divinho">@divinho</a>. From your description it seems that your model is dropping a particular layer and the loss calculation does not use that layer at all. If you want to train in such way with DDP, can you try use <code>find_unused_parameters=True</code> when initializing DDP?</p>
<p>Currently, <code>find_unused_parameters=True</code> must be passed into <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc"><code>torch.nn.parallel.DistributedDataParallel()</code></a> initialization if there are parameters that may be unused in the forward pass.</p>

username: divinho
cooked: <p>Thank you for the suggestion I will try that!</p>

username: divinho
cooked: <p>That solved it thank you!</p>

username: Shashikant_Verma
cooked: <p>Hi, adding find unused parameters = True throws the following error<br>
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the <code>forward</code> function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple <code>checkpoint</code> functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.<br>
Parameter at index 52 has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.</p>
What's the equivalent way for C++ to change dtype in place?

username: weicu
cooked: <p>This is the python code:</p>
<pre><code>  x = torch.ones([4], dtype=torch.float64)
  y = x.view(torch.int64)
</code></pre>
<p>However, in C++, it doesn’t work:</p>
<pre><code>  x = torch::ones({4}, ...)
  y = x.view(torch::kInt64); // Failed
</code></pre>
<p>Which function should I use to do the python-equivalent thing in C++?</p>
Cannot assign different embed_dim and kdim in nn.MultiheadAttention

username: kallyaleksiev
cooked: <p>Hi everyone!</p>
<p>I am trying to build a multi-head attention model that has different values for d_k, d_v, and d_model (where d_k, d_v, and d_model are as in <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1706.03762.pdf</a>)</p>
<p>I do</p>
<p><code>attention = nn.MultiheadAttention(embed_dim=200, num_heads=20, kdim=300, vdim=300, batch_first=True)</code></p>
<p>and then with</p>
<p><code>input = torch.randn((10, 4, 300)) output, output_weights = attention(input, input, input)</code></p>
<p>I get an error</p>
<p><code>assert embed_dim == embed_dim_to_check, \ AssertionError: was expecting embedding dimension of 200, but got 300</code></p>
<p>I realise that mathematically we can just have embed_dim (AKA dim_model) equal 300 and then just add a linear projection so that we have output multiplied by a matrix W_O’ of dimensions 300 x 200 which will be mathematically equivalent to what we need but I don’t understand why is the above assertion needed here?</p>
<p>This is somewhat similar to this <a href="https://github.com/pytorch/pytorch/issues/27623" class="inline-onebox" rel="noopener nofollow ugc">Not possible to use different key/value dimensionalities in nn.MultiheadAttention · Issue #27623 · pytorch/pytorch · GitHub</a>,  however we need d_model and d_key to be different, not d_v and d_k.</p>
<p>What am I missing? Thanks in advance</p>

username: LukasHedegaard
cooked: <p>I’ll second this - the assertions around embed_dim are too restrictive. The embed_dim seems to be a “catch-all” parameter, although the multi-head attention fundamentally supports <strong>four</strong> different embeddings dimensions to coexist:</p>
<ol>
<li>The input dimension of the query - doesn’t need to be divisible by num_heads,</li>
<li>The embed_dim for query and key - this should be divisible by num_heads,</li>
<li>The embed_dim for value as well the input dimension of the out_projection - this should be divisible by num_heads,</li>
<li>The output dimension of out_proj - doesn’t need to be divisible by num_heads.</li>
</ol>
Help with out of bounds error while plotting

username: dhsusf
cooked: <p>I’ve been running into this problems for days with the Pytorch Forcasting package.</p>
<p>Running this code:</p>
<p>for idx in range(10):  # plot 10 examples<br>
best_tft.plot_prediction(x, raw_predictions, idx=idx, add_loss_to_title=True);</p>
<p>I get this message:</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_forecasting/models/temporal_fusion_transformer/<strong>init</strong>.py in plot_prediction(self, x, out, idx, plot_attention, add_loss_to_title, show_future_observed, ax, **kwargs)<br>
671             show_future_observed=show_future_observed,<br>
672             ax=ax,<br>
 → 673             **kwargs,<br>
674         )<br>
675</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_forecasting/models/base_model.py in plot_prediction(self, x, out, idx, add_loss_to_title, show_future_observed, ax, quantiles_kwargs, prediction_kwargs)<br>
759         ):<br>
760<br>
 → 761             y_all = torch.cat([encoder_target[idx], decoder_target[idx]])<br>
762             max_encoder_length = x[“encoder_lengths”].max()<br>
763             y = torch.cat(</p>
<p>IndexError: index 1 is out of bounds for dimension 0 with size 1</p>
<p>I’m very new at this. If there is more information or code that is needed to trouble shoot, then please let me know. Thanks in advance.</p>

username: ptrblck
cooked: <p>Based on the error message the indexing operation is failing in once of these tensors:</p>
<pre><code class="lang-python">encoder_target[idx]
# or
decoder_target[idx]
</code></pre>
<p>make sure the size of the tensors in <code>dim0</code> is sufficiently large for the index.</p>

username: dhsusf
cooked: <p>Thanks for replying. I saw that you’re a bit of a master at these messages and was hoping you’d reply.</p>
<p>I did figure out the problem. It was just mostly understanding the Pytorch Forecasting library. To run that specific loop, the data needs to more than a single time series. I thought it was choosing different starting time_idx from a single time series, but it’s actually looking for idx=idx from multiple time series data.</p>

username: Kaiyoshine
cooked: <p>Hi, I have the exact same error. Can you please explain more and maybe show the fix? Thank you very much.</p>

username: dhsusf
cooked: <p>Hi, I’m very new at this, literally have only been learning python for the last six months as a self study. so, I don’t think I’m really the best to explain this. I’ll try and explain, but I just wanted to make sure you are also using the Python Forecasting package since I think my experience is very specific to that package.</p>

username: Varad_Apte
cooked: <p>Yes <a class="mention" href="/u/dhsusf">@dhsusf</a>  I am using pytorch forecasting library and getting same error</p>
Kernel crash OOM

username: promach
cooked: <p>For <a href="https://colab.research.google.com/drive/1mEU5TPDOwZztqh2BySqHP1ZU1SwsM62D#scrollTo=iNV_S7NOa8O0&amp;line=59&amp;uniqifier=1" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a> , anyone have idea why the kernel crashes due to OOM ?</p>
<p>I only just added <code>TriangleMultiplicationOutgoing</code> , and it is OOM.</p>
Convert model to ONNX with CUDA C++ custom code

username: trand2k
cooked: <p>Hi all,<br>
I try to convert pytorch model to ONNX but my onnx model only output a const. In pytorch model have some layer using CUDA c++ function build by pybind11. have any one have same issue?<br>
Can you help me in this case<br>
Thank all,</p>
How to mask edge_index with train_mask

username: Nimeshika_Udayangani
cooked: <p>How can I mask the edge_index so that it only contains edge connections of train nodes before passing in to a GCN model, in order to prevent data leakage between test nodes and train nodes.</p>
<pre><code class="lang-auto">import torch
import torch_geometric
from torch_geometric.nn import GCNConv

# Assuming you have a graph with node features and training labels
edge_index = torch.tensor([[0, 1, 1, 2],
                           [1, 0, 2, 1]], dtype=torch.long)
x = torch.tensor([[-1], [0], [1]], dtype=torch.float)
y = torch.tensor([1,1,1], dtype=torch.long)

train_mask = [True, True, False] # Boolean mask indicating the training nodes

data = torch_geometric.data.Data(x=x, y=y, edge_index=edge_index)
# Apply mask to input data
x_train = data.x[train_mask]
edge_index_train = **???**


# Create the GCNConv layer
conv = GCNConv(in_channels, out_channels)
# Perform forward pass only on training nodes
out = conv(x_train, edge_index_train)

# Perform further computations with the output
</code></pre>
<p>For e.g. How to replace “???” in above code?</p>
Vectorizing double for loop

username: Tigran_Ishkhanov
cooked: <p>Is there a way to vectorize the “for” loops in the following scenario:</p>
<p>The variable “A” is a 2-dim tensor of shape (N, M).<br>
The variable “B” is a 2-dim tensor of shape (M, M).</p>
<p>for k in range(N):</p>
<pre><code>    for i in range(M):
        for j in range(M):
            B[i, j] = A[k, i] - A[k, j]
</code></pre>

username: ptrblck
cooked: <p>Based on the code it seems you are overwriting the intermediate results in <code>B</code> with the last value of <code>k</code>, so you could also remove the <code>for k in range(N)</code> loop.<br>
To get rid of the loop you could <code>unsqueeze</code> the <code>A</code> tensor and use broadcasting.<br>
Here is a code snippet showing the same results:</p>
<pre><code class="lang-python">N, M  = 2, 3
A = torch.randn(N, M)
B = torch.randn(M, M)

for k in range(N):
    for i in range(M):
        for j in range(M):
            B[i, j] = A[k, i] - A[k, j]

C = torch.zeros(M, M)
for i in range(M):
    for j in range(M):
        C[i, j] = A[-1, i] - A[-1, j]

print(B - C)
# tensor([[0., 0., 0.],
#         [0., 0., 0.],
#         [0., 0., 0.]])

D = A[-1].unsqueeze(1) - A[-1].unsqueeze(0)

print(B - D)
# tensor([[0., 0., 0.],
#         [0., 0., 0.],
#         [0., 0., 0.]])
</code></pre>

username: Tigran_Ishkhanov
cooked: <p>Thank you!<br>
That helped.</p>
How to correctly iterate dataset indices

username: peony
cooked: <p>I have this code for my weighted sampler:</p>
<pre><code class="lang-auto">loader_iter = iter(train_dataset)
    for _ in range(50):
        idx, (frames, label, file_name), _ = next(loader_iter)
        class_weight = class_weights[label]
        sample_weights[idx] = class_weight
sampler = WeightedRandomSampler(sample_weights, num_samples=len(sample_weights), replacement=True)
</code></pre>
<p>but I am getting this error:<br>
<code>TypeError: 'int' object is not iterable</code></p>
<p>then I try to wrap the loader iter in range, like this:</p>
<p><code>idx, (frames, label, file_name), _ = range(next(loader_iter))</code></p>
<p>so I get an error <code>TypeError: 'tuple' object cannot be interpreted as an integer</code></p>
<p>then I tried to wrap with enumerate:<br>
<code>idx, (frames, label, file_name) = enumerate(next(loader_iter)</code></p>
<p>but I am getting <code>ValueError: too many values to unpack (expected 2</code>)</p>
<p>Any ideas for me? Thank you in advance</p>

username: ptrblck
cooked: <p>The general logic looks correct and also works for me:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.arange(10), torch.arange(1, 11))

dataset_iter = iter(dataset)
for _ in range(10):
    data, target = next(dataset_iter)
</code></pre>
<p>It’s unclear how your <code>train_dataset</code> is defined so you might need to debug its <code>__getitem__</code> method and check which line of code raises the error.</p>
What's the relationship between oneDNN and caffe2 in pytorch

username: mahuichao
cooked: <p>Hello everyone, recently I am learning the source code of pytorch. From python to the c/c++ implementation, and I saw one quantized layer goes to oneDNN library.<br>
I have thought pytorch use caffe2 as it’s c/c++ implementation, so why there is a oneDNN?bz this library is faster? bz this library support onnx better?<br>
thanks in advance.</p>

username: ptrblck
cooked: <p>PyTorch uses different backends to accelerate its workloads such as e.g. cuDNN for NVIDIA GPUs and oneDNN for Intel CPUs.<br>
Caffe2 was merged into the PyTorch backend a while ago to share common backend implementations but is deprecated now.</p>

username: mahuichao
cooked: <p>Thanks for replying , very clear.</p>
Accessing and plotting data from custom autograd.function - best practice

username: Nerrror
cooked: <p>Hey,</p>
<p>I have a couple of custom torch.autograd.functions that perform some bigger computations within the .forward and .backward methods. For visualization, it would be very nice to plot some of the intermediate results which are computed in the forward and backward methods. So far, whenever I was doing some calculations I just added the scripts that create the plots directly into the .forward and .backward methods but that is quite inflexible and seems suboptimal. Whats the best practice here?</p>
<p>This is how I did the plotting so far, changing the plots is definitely suboptimal since whenever I want to change anything in the plotting, I have to go into the class and change the code there:</p>
<pre><code class="lang-auto">class some_function(torch.autograd.Function):

     @staticmethod
     def forward(ctx, input1,  input2):
          output1 = comp1(input1, input2)
          ctx.save_for_backward(input1)
          ctx.save_for_backward(input2)
          return output1

     @staticmethod
     def backward(ctx, grad_output1):
          input1, input2 =  ctx.saved_tensors()
          intermediate_result = comp2(grad_output1, input1)
          plt.plot(input1, intermediate_result)
          grad_input2 = comp3(intermediate_result, input2)
          return None, grad_input2
</code></pre>

username: ptrblck
cooked: <p>Your approach looks correct since it seems you need to access the intermediates in the backward call. If you would only want to plot the data or weight gradients you should be able to use backward hooks. However, these hooks wouldn’t allow you to access the intermediates.</p>
Output from nn.MultiheadAttention does not match calcs per Attention paper

username: bogdan_ka
cooked: <p>Hi,<br>
I’m trying to make sure I understand what nn.MultiheadAttention() is doing exactly. I have a very simple example below, where I would expect manual result and nn.MultiheadAttention() result to align but they do not. Is it normalization of the scores? Something else? What am I missing?</p>
<pre><code class="lang-auto">from torch.nn import functional as F
import torch
torch.set_printoptions(sci_mode=False)
torch.manual_seed(3) 

# interaction embedding             
mm=np.array([[1,1,1],[1,1,1],[1,1,1],[1,1,0]])
# random matrix for linear projection of mm
m=np.array([[2,3,1],
            [2,3,2],
            [2,3,3]])
# query, key and value matrix, all the same
q=np.dot(mm,m)


# manual calc
scores=q@q.transpose()
scores=torch.tensor(scores).float()
attn_mask = torch.triu(torch.ones((4,4)), diagonal=1)
attn_mask[attn_mask == 1] = -float("Inf")
src_mask=attn_mask
normalizer=q.shape[1]**.05
my_attention_scores=torch.tensor(F.softmax(scores/normalizer+src_mask, dim=-1))@q

# torch way
multihead_attn = nn.MultiheadAttention(3, 1, batch_first=False)
qt=torch.tensor(q).float()
builtin_attn_scores, attn_output_weights = multihead_attn(qt, qt, qt, attn_mask=src_mask)
builtin_attn_scores, my_attention_scores

</code></pre>
<p>Output looks like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/736884cf84493651889d31755bb1207362f22fce.png" alt="image" data-base62-sha1="gsWQ3ljU6R8yr2J33byWdk8YusK" width="667" height="204"></p>

username: eqy
cooked: <p>One thing that looks strange is that your implementation does not appear to do any projection to the input QKV matricies. Note that the PyTorch implementation applies projection matricies to each of the Q, K, and  V: <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">MultiheadAttention — PyTorch 2.0 documentation</a>. I believe that the paper also does this, but it is kind of buried in the text (see the paragraph at the bottom of page 4).</p>
<p>You can check if using the weight matricies of the PyTorch layer in your manual calculation matches the results with the <code>q_proj_weight</code>, <code>k_proj_weight</code>, and <code>v_proj_weight</code> attributes.</p>

username: bogdan_ka
cooked: <p>Thank you for the response <a class="mention" href="/u/eqy">@eqy</a> . My understanding of the projection is that it’s just a multiplication of the embedding matrix by some randomly initiated matrix. So in the above I thought of <code>mm</code> as an embedding and <code>m</code> as that random matrix. So np.dot(mm, m) should be equivalent to the linear projection. But now I realize also that my random matrix m, which is not even random :-),  could be very different from the matrices initiated by the nn.MultiheadAttention() which would explain differences.</p>
<p>In hindsight I think it’s a silly question, of course they would be slightly different unless I made sure that  all the matrices are the same. Appreciate you pointing that out.</p>
Documentation for torch::lstm?

username: bokonon
cooked: <p>I am writing to find the location of the documentation for <code>torch::lstm</code>. I looked up the function on the <a href="https://pytorch.org/cppdocs/api/namespace_torch.html" rel="noopener nofollow ugc">CPP docs for torch namespace</a>, and I could not find it.</p>
<p>Any pointers to it would be greatly appreciated. I am trying to adapt code from the book <em>Hands On Machine Learning with C++</em>, and there is mention of it.</p>

username: bokonon
cooked: <p>I found an equivalent function in <code>at::lstm</code>, and the documentation is <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1afae760b93f1c7276a5f9e8d66f047aaa.html" rel="noopener nofollow ugc">here</a> in the cpp docs.</p>
Handle false positives in object detection

username: Zillur_Rahman
cooked: <p>I encountered an issue while trying to design a simple 2D Object detector of my own. I have ground truth bounding box coordinates and class labels. My idea is I will get bounding box predictions and I will try to match each box to the ground truth. My question is when my model predicts a false object, how would I compute the loss for that? For example, my model could not detect the object on the top left corner; instead it detects a false object beside it. Similarly, it could not detect the object on the bottom left corner too. How can I handle these situations while calculating IoU Loss so that model learns? I tried to understand how other sota models like YOLO handles that, but could not figure that out.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386.png" data-download-href="https://discuss.pytorch.org/uploads/default/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386" title="Untitled Design"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386_2_500x500.png" alt="Untitled Design" data-base62-sha1="rSxFwelYW0q1uC6UPs5QuendTRc" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386_2_500x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386_2_750x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c365147bfe86f7d9c6b6d86d8aa5be42ca28f386_2_1000x1000.png 2x" data-dominant-color="FBFAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled Design</span><span class="informations">1080×1080 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
ROCm --index-url access denied

username: tuxontour
cooked: <p>Hello everyone,</p>
<p>when accessing the ROCm --index-url I get AccessDenied. The CUDA --index-url is accessible.<br>
Is this intended?</p>

username: Jeffrey_Smith
cooked: <p>I really wish this would get some attention.<br>
As a result of this issue, the instructions on <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a> for ROCm are broken.</p>
<p>Some quick testing I did, with “curl -I <a href="https://download.pytorch.org/%7Burl_path%7D" rel="noopener nofollow ugc">https://download.pytorch.org/{url_path}</a>”, and the HTTP response received:<br>
whl  -  200 (OK - reasonable)<br>
whl/cpu  -  200 (OK - reasonable)<br>
whl/cu117  -  200 (OK - reasonable)<br>
whl/cu118  -  200 (OK - reasonable)<br>
whl/rocm5.4.2  -  403 (Forbidden - BROKEN)<br>
whl/blahblahblah  -  403 (Forbidden - reasonable)</p>
<p>EDIT: I guess this was not as big an issue as I originally thought.  Apparently <strong>by adding a / at the end</strong>, all endpoints work that should work.  It still seems odd that rocm5.4.2 and it alone is broken when the / is NOT specified, but it is not keep the <code>pip install</code> command from working.</p>

username: plinnie
cooked: <p>I have the same issue. Please fix this!</p>
<p>To install the <em>current</em> latest version use (tested on python 3.11):</p>
<blockquote>
<p>pip install torch==2.0.1+rocm5.4.2 torchvision==0.15.2+rocm5.4.2 torchaudio==2.0.2 --index-url <a href="https://download.pytorch.org/whl/rocm5.4.2/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/rocm5.4.2/</a></p>
</blockquote>
<p>(note versions are wrong on the site and won’t work)</p>
Torch.compile, Triton cuda capability

username: overdrivek
cooked: <p>Hello everyone,<br>
Since i am working with PINN, the new capabilities of pytorch2.0 is very interesting.<br>
On making the transition to pytorch 2.0, the torch.compile seemed to work fine but on calling the model i get the following error,</p>
<p>“torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised RuntimeError: Found NVIDIA GeForce GTX 1080 Ti which is too old to be supported by the triton GPU compiler, which is used as the backend. Triton only supports devices of CUDA Capability &gt;= 7.0, but your device is of CUDA capability 6.1”</p>
<p>Is my GPU too old for torch2.0? I have cuda 12.1 installed and 11.8 in the conda environment.<br>
Regards</p>

username: marksaroufim
cooked: <p>Yes unfortunately your GPU is too old, Triton itself really needs newer GPUs to shine so I believe they added this error so users of older GPUs wouldn’t get slowdowns on their models which would be even more frustrating</p>
How to achieve distributed deep learning framework for computer vision?

username: Jaeseon
cooked: <p>I am Ph.D. student in Economics, and currently working on project of deep learning-based object detection from satellite imagery.<br>
I am struggling to get an idea to conduct distributed deep learning on my current task. I have access to PySpark 2.4.0 and PyTorch 1.9.0 at my institute. I was wondering if there is useful resources or tutorials that can help me apply these tools to my work.<br>
Specifically, I’m interested in learning about the common approaches used in distributed deep learning frameworks for image processing or computer vision tasks, especially when it comes to working with Pytorch.<br>
Plus, I also would like to know whether PySpark is good or common method to load image data from HDFS for building object detection models, and when I conver Pyspark.dataframe data format to proper data format for PyTorch, does distributed or parallelized nature remain?</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html?utm_source=distr_landing&amp;utm_medium=intermediate_ddp_tutorial"><code>DistributedDataParallel</code> tutorial</a> might be a good starter as it would walk you through the usage of <code>DDP</code> and should be relatively easy to apply for any model. You could then check more advanced techniques of model sharding if needed.</p>
Gradient Checkpointing and Splitting the Model Across Several GPUs

username: Andyy
cooked: <p>So, I have a model that I want to fully shard across two GPUs - i.e. put the first half of the layers on one GPU and the other half on the second, and then move the output of the first half from the first GPU to the second in each forward pass.</p>
<p>One can imagine the model as follows (a simplified example):</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, n):
        self.A1 = nn.Linear(n, n).to(0)
        self.A2 = nn.Linear(n, n).to(0)
        self.B1 = nn.Linear(n, n).to(1)
        self.B2 = nn.Linear(n, n).to(1)

    def forward(self, x):
        x = nn.functional.relu(self.A2(nn.functional.relu(self.A1(x)))
        x = x.to(1)
        return nn.functional.relu(self.B2(nn.functional.relu(self.B1(x)))
</code></pre>
<p>This seems to work without issues by itself - however, the issues seem to arise when I am trying to use gradient checkpointing -  the memory consumption on each of both GPUs becomes significantly higher than the memory consumption when doing the whole training on just one of these GPUs with gradient checkpointing (i.e. when the whole model is situated on just one GPU so that there are two times more parameters on it as compared to the “distributed” case).</p>
<p>Theoretically, however, I think that the memory consumption on each of the two gpu’s should be about twice lower than the memory consumption on just one GPU (this is the case without gradient checkpointing). Could someone please help me understand what could be the reason for this discrepancy?</p>
Dice scores metric

username: Umar
cooked: <p>Why is it that my dice scores is above 1?</p>
<p>intersection = torch.sum(targets * predictions)<br>
union = torch.sum(targets) + torch.sum(predictions)<br>
dice_score = (2 * intersection + EPSILON) / (union + EPSILON)</p>
<pre><code>        dice_score_total += dice_score.item() * data.size(0)
        num_samples += data.size(0)

    dice_score_average = dice_score_total / num_samples
</code></pre>

username: ptrblck
cooked: <p>I guess your targets might not be one-hot encoded or your predictions might not contain probabilities.</p>
To pass Different mask to each batch element in nn.TransformerEncoder's forward function

username: hwaseem04
cooked: <pre><code class="lang-auto">class PositionalEncoding(nn.Module):

    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 5000):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(max_len, 1, d_model)
        pe[:, 0, 0::2] = torch.sin(position * div_term)
        pe[:, 0, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x):
        """
        Arguments:
            x: Tensor, shape ``[seq_len, batch_size, embedding_dim]``
        """
        x = x.permute(1,0,2)
        x = x + self.pe[:x.size(0)]
        return self.dropout(x).permute(1,0,2)
    
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        encoder_layer = nn.TransformerEncoderLayer(d_model=256, nhead=8, batch_first=True)
        self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
        self.pos = PositionalEncoding(256, max_len=1000) # sequence length
        self.cls1 = nn.Parameter(torch.rand(1, 256)) # 1, enc_dim
        self.cls2 = nn.Parameter(torch.rand(1, 256))  

    def forward(self, x):
        x = self.pos(x)  
        M = ...  # defined on some logic and need to have dimension as (4, 9, 9) as stated below
        x = self.transformer_encoder(x, mask=M.logical_not(), src_key_padding_mask=None)#pad.logical_not())
        
        return x

</code></pre>
<p>Say batch size is 4.<br>
For some reason I am trying to apply different mask for each input, i.e 4 mask for 4 input.</p>
<p>One such example of the mask looks like (assume sequence length as 9, so 9 by 9 matrix for mask)(some kind of restrictive self attention)</p>
<pre><code class="lang-auto">tensor([[1., 1., 1., 1., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>When I try to pass 4 such different mask of dimension (4, 9,9), I get error as<br>
<code>RuntimeError: The shape of the 3D attn_mask is torch.Size([4, 9, 9]), but should be (32, 9, 9).</code></p>
<p>What should I do to achieve what I am expecting.</p>
Captum, Integrated gradients and Deep Metric Learning

username: DrC
cooked: <p>Hi everyone,</p>
<p>I started looking into the Captum tutorial, and I realize that the Integrated Gradients need a label for the computation.</p>
<p>Here is my problem: I use a Deep Metric Learning model, which means that I don’t have a classification layer but an AveragePooling (or MaxPooling) layer at the end that gives me a feature vector of length n.</p>
<p>Then the “classification” is simply the cosine similarity between the test embedding and a set of reference embeddings.</p>
<p>My question is: can captum’s IG be used in a scenario like this ?</p>
<p>Thanks</p>
Padding semantics of convolution_backward

username: gengrill
cooked: <p>Hi! I have a question about backward conv padding. Basically, I’m following <a href="https://github.com/pytorch/pytorch/blob/6cc0f1c20c2f87a6c7b0e4abd5419e5007920999/torch/nn/grad.py#L160" rel="noopener nofollow ugc">this</a> to construct:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; grad_out = torch.rand([8,256,5,7,5])
&gt;&gt;&gt; image = torch.rand([8,128,10,14,10], requires_grad=True)
&gt;&gt;&gt; kernel = torch.rand([256,128,3,3,3], requires_grad=True)
&gt;&gt;&gt; print(torch.ops.aten.convolution_backward(grad_out, image, kernel, None, (2, 2, 2), (1, 1, 1), (1, 1, 1), False, [0], 1, (True, False, False))[0].shape)
torch.Size([8, 128, 10, 14, 10])
</code></pre>
<p>Seeing that <a href="https://github.com/pytorch/pytorch/blob/593642d1d8d0edec05eaeeca38bb2b8be9b73639/aten/src/ATen/native/Convolution.cpp#L728" rel="noopener nofollow ugc">convolution_backward calls check_shape_forward</a> I assumed it followed forward conv padding semantics, but I cannot reconcile the spatial dimensions of the output of this op with either <a href="https://pytorch.org/docs/main/generated/torch.nn.Conv3d.html" rel="noopener nofollow ugc">Conv3d</a> or <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose3d.html" rel="noopener nofollow ugc">ConvTransposed3d</a> padding:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; (10+2*1-1*(3-1)-1)//2 + 1
5
&gt;&gt;&gt;&gt; (10-1)*2-2*1+1*(3-1)+1
19
</code></pre>
<p>The fact that the out_padding is seemingly never set doesn’t really help either <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> any pointers to what’s going on or what I’m missing?</p>