akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert from nn.linear to nn.rnn?

kim_junyeong: <p>How can I make the output form of nn.rnn into [torch.Tensor] like nn.linear?</p>
<p>My MLP code is</p>
<pre><code class="lang-auto">class sample(nn.Module):
    def __init__(self):
        super(sample, self).__init__()
        self.n = nn.Linear(1, 64)
        self.nn = nn.Linear(64, 3)
    def forward(self, t, is_train = False, y = None):
        # t = torch.Size([128,1]) / t = &lt;class 'torch.Tensor'&gt;
        ot = self.n(t)
        ott = self.nn(ot)
        return ott
</code></pre>
<p>and I tried</p>
<pre><code class="lang-auto">class sample(nn.Module):
    def __init__(self, input_size, hidden_dim, num_layer):
        super(NNfunc, self).__init__()
        self.input_size = input_size
        self.hidden_dim = hidden_dim
        self.num_layer = num_layer
        self.rnn = nn.rnn(1, 3, 1)
    def forward(self, t, is_train = False, y = None):
        output = self.rnn(t) 
        return output
</code></pre>
<p>but I think this is wrong,<br>
I want to make [output] in the form of [torch.tensor], how should I modify the code?</p>

srishti-git1110: <p>Hi,<br>
Can you please elaborate on your requirement more?<br>
As of now, I could only understand that you want the output of the RNN block to be of type <code>torch.tensor</code> that I would think it already is.</p>

J_Johnson: <p>The docs provide an example of how to use an RNN at the bottom of the page:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.RNN.html</a></p>
<pre><code class="lang-auto">rnn = nn.RNN(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
output, hn = rnn(input, h0)
</code></pre>
<p>Note, this is a batch size of 3(dim=1). If you want the batch dim to be first, you can set <code>batch_first=True</code>.</p>

kim_junyeong: <p>Thank you for your kind answer!!!</p>
Sending Bags of images to model

Haidari_Iskandarzoda: <p>Hello everybody.<br>
I am trying to send the bags of instances with shape <strong>torch.Size([8, 6, 3, 224, 224])</strong><br>
to model but getting this error:</p>
<blockquote>
<p>RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [8, 6, 3, 224, 224]</p>
</blockquote>
<p>8 → batch size, aka number of bags<br>
6 → number of instances per bag</p>
<p>if i reshape it to <strong>torch.Size([48, 3, 224, 224])</strong> the whole idea of bags dissapear.</p>
<p><strong>How can i send bags to model?</strong></p>

eqy: <p>What kind of model are you using (or the downstream task) and how does it use “bags” or sequences of images?</p>

Haidari_Iskandarzoda: <p>I am using ResNet18. Actually i didn’t chaged the model itself, yet. I am new to this and don’t have any idea what to do in order the model receives the bag.</p>
<p>The idea is that, if the one of the inctaces(images) in bag are positive the whole bag will be classified as positive, if not negative. I need to receive the bag of features and use the aggregation method to get 0 or 1 output</p>

eqy: <p>In that case you would want to do something like reshape the 5D input to 4D, and then reshape the model output again afterwards for aggregation.</p>
<p>Something like <code>input.reshape(-1, 3, 224, 224)</code> and <code>output.reshape(#bags, -1)</code>.</p>

Haidari_Iskandarzoda: <p>Thank you, i will try.</p>
Does Pytorch benefit from Nvidia GPU direct storage?

Jimmy_Yeh: <p>Following <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions" rel="noopener nofollow ugc">the guide for installing the Cuda library</a>, step 2.5. Install MLNX_OFED says that if you intend to use GPUDirectStorage (GDS), you must install the CUDA package and MLNX_OFED package.</p>
<p>Searching for GPUDirectStorage (GDS), I found that it accelerates data access for GPU. However, I wonder if Pytorch will benefit from installing this.</p>

ptrblck: <p>Yes, you could use GDS e.g. via <a href="https://github.com/NVIDIA/DALI"><code>DALI</code></a> in PyTorch (and other frameworks).</p>
How does fake_quantize_per tensor works?

Rami_Ismael: <p>I was wondering what operation occurs in this program. Does this program does rounding with STE or without STE.</p>
<pre data-code-wrap="Python"><code class="lang-plaintext">                X = torch.fake_quantize_per_tensor_affine(
                    X, self.scale.item(), int(self.zero_point.item()), self.quant_min, self.quant_max)
</code></pre>
<p>The PyTorch Quantization Process is the same of from this github repo <a href="https://github.com/wimh966/outlier_suppression/blob/ae5b5b48e781cd55631128d8ecd746198e6839e4/quant_transformer/quantization/util_quant.py" class="inline-onebox" rel="noopener nofollow ugc">outlier_suppression/util_quant.py at ae5b5b48e781cd55631128d8ecd746198e6839e4 · wimh966/outlier_suppression · GitHub</a></p>
<pre data-code-wrap="Python"><code class="lang-plaintext">def fake_quantize_per_tensor_affine(x, scale, zero_point, quant_min, quant_max):
    x_int = round_ste(x / scale) + zero_point
    x_quant = torch.clamp(x_int, quant_min, quant_max)
    x_dequant = (x_quant - zero_point) * scale
    return x_dequant
</code></pre>
<p>I am unclear.  How does Pytorch do?</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/rami_ismael">@Rami_Ismael</a> , yes, STE is used in the derivative of <code>fake_quantize_per_tensor_affine</code>.</p>
Accelerate backward in DDP REINFORCE

ArchieGertsman: <p>I am running on one 24-core CPU and one A100 GPU. Each subprocess runs the following (stripped down) code:</p>
<pre><code class="lang-auto"># general setup
dist.init_process_group('gloo', rank=rank, world_size=num_envs)
torch.set_num_threads(1)

# set up local model and optimizer
local_model = DDP(model, device_ids=[device])
optim = Adam(local_model.parameters(), lr=.001)

'''
save the trajectory from an episode.

- action_lgprobs[i] is the log 
  probability of the action sampled 
  at step i
- entropies[i] is the entropy of the
  distribution returned by the model
  at step i
- rewards[i] is the reward assigned to
  the i'th action by the environment

action_lgprobs and entropies are 
attached to the computational graph.
'''
action_lgprobs, entropies, rewards = \
    run_episode(
        env,
        local_model,
    )

returns = compute_returns(rewards, discount)

# compute baselines
baselines = returns.clone()
dist.all_reduce(baselines)
baselines /= num_envs
    
advantages = returns - baselines

# compute loss
policy_loss = -advantages @ action_lgprobs
entropy_loss = entropy_weight * entropies.sum()
loss = policy_loss + entropy_loss

# update model
optim.zero_grad()

torch.cuda.synchronize()
start = time.time()
loss.backward() # slow!
torch.cuda.synchronize()
end = time.time()
print('BACKWARD TIME:', end - start)

optim.step()
</code></pre>
<p>I am finding the performance of <code>loss.backward()</code> to scale rather poorly with the number of subprocesses. E.g. with only one subprocess, it takes ~3s for <code>backward</code> to finish, while with 16 subprocesses it takes ~24s to finish on all of them (each subprocess prints the same duration). I’m wondering, is there anything I can do to speed it up? Any ideas will be appreciated.</p>

ArchieGertsman: <p>Instead of using DPP, would it work to</p>
<ol>
<li>aggregate the loss using <code>dist.all_reduce(loss)</code>
</li>
<li>only run <code>loss.backward</code> on rank 0</li>
<li>broadcast the gradients to all other ranks</li>
</ol>
<p>?<br>
Would the reduce operation preserve the computational graphs from all the ranks? It’s seems much faster to do a single <code>backward()</code> call, if possible.</p>
How can I plot loss with epoch?

Cihat_Kaya: <p>I have a model like this:</p>
<pre><code class="lang-auto">PCConvLstmNet(
  (conv): Sequential(
    (0): Conv1d(1, 4, kernel_size=(7,), stride=(3,))
    (1): BatchNorm1d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
    (3): Conv1d(4, 8, kernel_size=(7,), stride=(3,))
    (4): BatchNorm1d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU()
    (6): Conv1d(8, 16, kernel_size=(7,), stride=(3,))
    (7): BatchNorm1d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU()
  )
  (lstm): GRU(16, 16, batch_first=True)
)
</code></pre>
<p>how can I plot loss with epoch? (I am new in pytorch)</p>

ptrblck: <p>You could take a look at <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network">this tutorial</a> which shows how you can print the loss. Note that the current code snippet uses a condition using the steps in the <code>DataLoader</code> so you might want to move the <code>print</code> statement into the epoch loop.</p>
Trouble converting CNN from keras to pytorch

Hayden_Johnson: <p>I have created a CNN using the keras library which takes inputs which are 1D-arrays with 37000 data points each and with an output of 10 fully-connected nodes. I defined what I thought was an equivalent model in pytorch, however my results are significantly different. Does anyone see any mistakes in my implementation? The keras model is defined as the following:<br>
model_aq = Sequential()<br>
model_aq.add(Conv1D(32, 3, activation=“relu”, input_shape=(37000, 1)))<br>
model_aq.add(Conv1D(32, 3, activation=“relu”))<br>
model_aq.add(Conv1D(32, 3, activation=“relu”))<br>
model_aq.add(Flatten())<br>
model_aq.add(Dense(200, activation=“relu”))<br>
model_aq.add(Dense(10))<br>
model_aq.compile(loss=“mse”, optimizer=“RMSProp”, metrics=[tf.keras.metrics.RootMeanSquaredError()])</p>
<p>The model I created using pytorch is defined as:<br>
class ModelAQ(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ModelAQ, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv1d(1, 32, kernel_size=3, stride=1, padding=1)<br>
self.conv2 = nn.Conv1d(32, 32, kernel_size=3, stride=1, padding=1)<br>
self.conv3 = nn.Conv1d(32, 32, kernel_size=3, stride=1, padding=1)<br>
self.flatten = nn.Flatten()<br>
self.dense1 = nn.Linear(32 * 36994, 200)<br>
self.dense2 = nn.Linear(200, 10)</p>
<pre><code>def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.conv3(x)
    x = self.flatten(x)
    x = self.dense1(x)
    x = self.dense2(x)
    return x
</code></pre>
<p>pytorch_model_aq = ModelAQ()</p>
<p>To train the pytorch model I used the following:<br>
criterion = nn.MSELoss(reduction=‘sum’)<br>
def train(model_inp, num_epochs = num_epochs):<br>
<span class="hashtag">#optimizer</span> = optim.RMSprop(model_inp.parameters(), lr=learning_rate)<br>
optimizer = optim.RMSprop(model_inp.parameters())<br>
for epoch in range(num_epochs):<br>
running_loss = 0.<br>
for inputs , labels in train_iter:<br>
outputs = model_inp(inputs)<br>
loss= criterion(outputs, labels)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
running_loss += loss.item()<br>
optimizer.step()<br>
if epoch % 1 == 0:<br>
print(f’Epoch {epoch+1}/{num_epochs} running accumulative loss {running_loss:.3f}')</p>

ptrblck: <p>Your PyTorch model misses the relu activations so either use an <code>nn.ReLU</code> module or use the functional API via <code>x = F.relu(x)</code>. I’m also unsure how the shape is defined in Keras via <code>input_shape=(37000, 1)</code>. Is this shape defining the shape as <code>[batch_size, features]</code> or is the batch size missing and is the shape defined as <code>[sequence_length, channels]</code>?</p>
Using torch.compile() with DDP

JamesDickens: <p>I’ve been trying to check out the new torch.compile() feature for a vision transformer and its working pretty well for one gpu, but when I use distributed data parallel, I get a pickling error as soon as I call mp.spawn with my training loop process.</p>
<p>I get the error:<br>
AttributeError: Can’t pickle local object ‘convert_frame.._convert_frame’</p>
<p>Are there any resources/tutorials to use the new compile feature with DDP training?</p>
One-hot encoded data as an input to non convolutional NN

7125: <p>I’m writing a FFNN and have a problem with my one-hot encoded data.</p>
<p>The network should work with 28000 records of 1000 characters long strings encoded to 21 one-hot categories. The problem is that i want my model to be working on a tensor of size 1000x21, not on a one of a size 21000.</p>
<pre><code>    self.fc1 = nn.Linear(input_size, hidden_size)
    self.fc2 = nn.Linear(hidden_size, hidden_size)
    self.fc3 = nn.Linear(hidden_size, output_size)
</code></pre>
<p>I want to utilize the above architecture with an input size of 1000x21.<br>
I’m working with data loader, this is what i do to work with batches of data:</p>
<pre><code>    x = x.view(x.size(0), -1) 
    x = self.fc1(x)
</code></pre>

eqy: <p>I’m not sure</p>
<pre><code class="lang-auto">    x = x.view(x.size(0), -1) 
</code></pre>
<p>would make sense here, as if it does have shape 21000 as you said, the view would result in a shape of <code>21000, 1</code> with this implementation.</p>
<p>If indeed <code>1000</code> is the correct batch dimension, then you would to specify at least one of the dimensions explicitly in the call to <code>view</code> (either 1000, or 21, or both).</p>
How to efficiently swap a region between two image tensors?

almeidaraul: <p>I have two tensors of same shape, namely 3-channel images (say 3x32x32) and want to swap a region between the two across all channels (say the upper left 2x2 matrix).</p>
<p>I can access this region with indexing (tensor[:, :2, :2]), so I know it’s easy to create a copy of one of the regions and perform the swap manually.</p>
<p>My question is, is there a more efficient way to do this?</p>
Problem with functorch.hessian

AUM: <p>I am calculating the hessian of a ( 2,2) linear model using <code>functorch.hessian</code> as follows</p>
<pre><code class="lang-auto">model = torch.nn.Linear(2,2).to(device)
inputs = torch.rand(1,2).to(device)
criterion = torch.nn.CrossEntropyLoss()
target=torch.ones(len(inputs), dtype=torch.long).to(device)

func, func_params = functorch.make_functional(model) 

def loss(params):
    out = func(params, inputs) 
    return criterion(out, target)  

H=functorch.hessian(loss)(func_params)  

</code></pre>
<p>Since I have 2 inputs and 2 outputs so we have basically 4 parameters I was expecting to see the second derivative with respect to these 4 parameters but the output looks different or not understandable. For example, the output of the code above is the following:</p>
<pre><code class="lang-auto">((tensor([[[[ 0.0217,  0.0701],
            [-0.0217, -0.0701]],
  
           [[ 0.0701,  0.2266],
            [-0.0701, -0.2266]]],
  
  
          [[[-0.0217, -0.0701],
            [ 0.0217,  0.0701]],
  
           [[-0.0701, -0.2266],
            [ 0.0701,  0.2266]]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;),
  tensor([[[ 0.0718, -0.0718],
           [ 0.2321, -0.2321]],
  
          [[-0.0718,  0.0718],
           [-0.2321,  0.2321]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;)),
 (tensor([[[ 0.0718,  0.2321],
           [-0.0718, -0.2321]],
  
          [[-0.0718, -0.2321],
           [ 0.0718,  0.2321]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;),
  tensor([[ 0.2377, -0.2377],
          [-0.2377,  0.2377]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;)))

</code></pre>
<p>Does anyone have an idea what’s going on? and how can I calculate the trace of the hessian matrix in this case?</p>
Using the whole GloVe pre-trained embedding matrix or minimize the matrix based on the number of words in vocabulary

Nikos_Spanos: <p>I have created a neural network for sentiment analysis using bidirectional LSTM layers and pre-trained GloVe embeddings.</p>
<p>During the training I noticed that the <code>nn.Embedding</code> layers with the freezed embedding weights uses the whole vocabulary of GloVe:</p>
<p>(output of the instantiated model object)<br>
<code>(embedding): Embedding(400000, 50, padding_idx=0)</code></p>
<p>Also the structure of the <code>nn.Embedding</code> layer:<br>
<code>self.embedding = nn.Embedding.from_pretrained(embedding_matrix, freeze=True, padding_idx=self.padding_idx)</code></p>
<p>, where <code>embedding_matrix = glove_vectors.vectors</code> object and glove_vectors =   <code>torchtext.vocab.GloVe(name='6B', dim=50)</code> <a href="https://pytorch.org/text/stable/vocab.html#torchtext.vocab.GloVe" rel="noopener nofollow ugc">source</a></p>
<p>400,000 is the shape of glove_vectors object (meaning 400,000 pre-trained words in total).</p>
<p>Then I noticed that the training of the LSTM neural network took approximately 3 to 5 minutes per epoch. Which is quite too long for only 150,000 trainable parameters. And I was wondering if this had to do with the use of the whole embedding matrix with 400,000 words or it’s normal because of the bidirectional LSTM method.</p>
<p>Is it worth to create a minimized version of the GloVe embeddings matrix from the words that only exist in my sentences or using the whole GloVe embeddings matrix it does not affect the training performance?</p>

ptrblck: <aside class="quote no-group" data-username="Nikos_Spanos" data-post="1" data-topic="169955">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_spanos/48/44970_2.png" class="avatar"> Nikos_Spanos:</div>
<blockquote>
<p>And I was wondering if this had to do with the use of the whole embedding matrix with 400,000 words or it’s normal because of the bidirectional LSTM method.</p>
</blockquote>
</aside>
<p>I would recommend to profile the actual workload and check which part of the training might be the slowest before changing the embedding or any other part.<br>
To do so you could try to use the native profiler or e.g. Nsight Systems.</p>

Nikos_Spanos: <p>Thanks for the suggestion will do. So, as a first opinion the whole GloVe embedding vocabulary may be used. It doesn’t affect the results or the performance of the training if custom embedding matrix with less words is used.</p>
Dataloader __getitem__ idx control when using csv file

YejinPu: <p>I am trying to load an image file using the following form of the csv file.<br>
This is a simplified representation of my huge csv file in the same format.</p>
<p><a href="https://i.stack.imgur.com/O2WCF.png?" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/O2WCF.png?" alt="enter image description here" width="690" height="294"></a></p>
<p>I actually understand the basic method of sequentially loading images using rows of csv files by code “iloc[idx]”.</p>
<p>However, like the attached image, it is difficult to sequentially load images containing as many images as “number of images” in the terminal folder written in the “file location.”</p>
<p>According to the attached image, the idx of <strong>getitem</strong>_ points to 16 images in the “./data/asd” folder sequentially, and then to the “./data/vbn” folder so that the next 16 images can be imported sequentially.</p>
<p>It’s a code that I’ve worked on so far, so I’d appreciate any advice you can give me here.</p>
<pre><code>class CustomDataset(torch.utils.data.Dataset):

def __init__(self, root, csv_file):
    self.root = root
    self.df = pd.read_csv(csv_file)

def __len__(self):
    #The sum of "Number of Images" values will be the total length of the image dataset.
    return len(self.df.iloc[1:,1].sum())

def __getitem__(self, idx): 
    #This is where I have trouble controlling idx.
    img_name = os.path.join(self.root, self.df.iloc[idx, 2] + '.dcm')
    image = io.imread(img_name)
    image = torch.tensor(image)

    return image
</code></pre>
How to add tensors of different shapes?

joey.s: <p>Hi,</p>
<p>I would like to know if it is possible to add a tensor of size [8, 55, 110] and a tensor of size [8, 20, 40] to be [8, 75, 150]. I tried using:</p>
<blockquote>
<p>torch.cat</p>
</blockquote>
<p>but the issue is:</p>
<blockquote>
<p>All tensors must either have the same shape (except in the concatenating dimension) or be empty.</p>
</blockquote>

KFrank: <p>Hi Joey!</p>
<aside class="quote no-group" data-username="joey.s" data-post="1" data-topic="169961" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9fc348/48.png" class="avatar"> joey.s:</div>
<blockquote>
<p>I would like to know if it is possible to add a tensor of size [8, 55, 110] and a tensor of size [8, 20, 40] to be [8, 75, 150].</p>
</blockquote>
</aside>
<p>It’s not clear that your desired operation makes sense.</p>
<p>What would you expect the result of “adding” these tensors together to be?</p>
<p>You start with <code>8 * 55 * 110  +  8 * 20 * 40</code> individual numbers, but you<br>
want your result to consist of <code>8 * 75 * 150</code> individual numbers – many more.<br>
Where are all these extra numbers supposed to come from and what should<br>
they be equal to?</p>
<p>Best.</p>
<p>K. Frank</p>
How to execute pytorch 1.0 code on cuda 11

niuyy9026: <p>Hi,</p>
<p>I am running a code that was developed by Python 3.6.8 and PyTorch 1.0 and CUDA 9.2. Here is the link to the code: <a href="https://github.com/kensakurada/sscdnet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - kensakurada/sscdnet: Semantic Scene Change Detection Network (CSCDNet + SSCDNet)</a></p>
<p>But my current workstation has the Nividia driver which is NVIDIA-SMI 460.106.00   Driver Version: 460.106.00   CUDA Version: 11.2. Just wonder how to execute this code in this environment? And I could not downgrade the Cuda version for this machine as I share this machine with other users.  Any help will be appreciated.</p>

ptrblck: <p>Your NVIDIA driver should be backwards-compatible. You also won’t be able to build such an old PyTorch release with CUDA 11.x, so your best bet is to either update PyTorch or to use the 1.0 release with its original CUDA 9 runtime.</p>
Is there a host memory leak in cudnn_convolution_backward?

rinkujadhav2013: <p>I’m creating a custom convolution layer (using PyTorch 10.2) and experiencing a memory leak (on host<br>
not device) whenever I use stride=2. So far, I’ve isloated the memory leak to calling <code>cudnn_convolution_backward</code> (basically <a href="https://github.com/pytorch/pytorch/blob/v1.10.2/aten/src/ATen/native/cudnn/ConvShared.cpp#L393" rel="noopener nofollow ugc">this</a> code) with mask <code>(True, True)</code>. If I use the mask <code>(False, True)</code> (i.e. not call conv backward for input) then the leak goes away. It also goes away when I use <code>stride=1</code>. I’m using the function in Python via <code>torch.utils.cpp_extension.load</code>. Is this a known issue? I couldn’t find anything on Github.</p>

ptrblck: <p>I’m not aware of a memory leak in any <code>cudnn</code> call right now. Are you seeing the same “leak” using PyTorch calls or only using your custom extension?</p>

rinkujadhav2013: <p>There’s no leak if I use the regular PyTorch <code>Conv2d</code> layer.</p>

rinkujadhav2013: <p>It seems like it’s not in the cudnn function itself but rather somehow in transferring data to host.<br>
There’s not leak if I do</p>
<pre><code class="lang-auto">grads_host[:,:,top:bottom,:].copy_(grad_input_from_cudnn)
</code></pre>
<p>But there’s a leak if I instead execute</p>
<pre><code class="lang-auto">grads_host[:,:,top:bottom,:] += grad_input_from_cudnn.to(torch.device('cpu'))
</code></pre>
<p>I need to use the latter but I don’t know what’s wrong with this line. Any tips?</p>

ptrblck: <p>The second code accumulates the <code>CPUTensor</code> <code>grad_input_from_cudnn</code> to <code>grad_host</code> and thus also stores the entire attached computation graph with it, which would be expected behavior and not a leak.<br>
You might want to check if this is the case and if you would need to <code>detach</code> the tensor before the accumulation (which would depend on the actual use case).</p>

rinkujadhav2013: <p>I thought in <code>torch.autograd.Function</code>’s backward function, I wouldn’t need to detach anything but I tried it and it doesn’t solve the issue. The memory still keeps going up with each step.<br>
Also, I’m still not experiencing any leak for <code>stride=1</code>, only when stride is greater than 1.<br>
My temporary workaround is to transfer the grads to GPU, perform addition there and then use <code>copy_</code> to transfer them back. But that’s of course computationally more expensive.</p>

rinkujadhav2013: <p>Just found out that the leak is slower if I do</p>
<pre><code class="lang-auto">grads_host = grads_host.detach()
</code></pre>
<p>but still no clue what the actual problem is.</p>

rinkujadhav2013: <p>Can I somehow turn off computational graph construction in the <code>forward</code> and <code>backward</code> functions of <code>torch.autograd.Function</code>? I thought this happens anyway in those functions but apparently not.</p>
How will the backward pass be affected by applying conditionals on the output?

Sidharth_R1: <p>Hi,<br>
I have a network which is outputing coordinate pairs that is used to update a tensor that is then used for calculation.</p>
<pre><code class="lang-auto">x, y - &gt; output tensors from a torch model that are differentiable for autograd
mask = torch.zeros(size)
for i in range(something):
    if(x[i]&gt;x[i+1] and &lt;conditionals based on x and y&gt;):
        mask[i] = 1.
</code></pre>
<p>I have a loss that is calculated on the basis of this mask. But since the mask is freshly created it is detached from the computational graph.<br>
Is there a way to work around this issue or is this fully non-differentiable?<br>
Thanks</p>

MitchellKT: <p>Do you actually want the mask to be differentiable w.r.t x?<br>
If not, you can wrap the mask generation with torch.no_grad().</p>
<p>Otherwise, the binary ‘compare’ operator is not differentiable, even when you think about it mathematically.<br>
Maybe you can provide more details on what you are trying to achieve?</p>

Sidharth_R1: <p>My plan was to calculate a loss function utilizing the mask that is generated by this code. But since the mask is the only piece from the model being used in the loss, there was no updates being made to the model parameters due to the mask being detached from the graph. The loss is a function of this mask and the input.</p>
PyTorch won't run via GPU (CUDA)

vilad: <p>I’m trying to run the code using the GPU (cuda), but it gives me an error.</p>
<p>I decided to check the result:</p>
<p>As I understand it, I just can’t use the GPU…</p>
<p>Here’s my video card just in case: <strong>NVIDIA GeForce GT 630M</strong></p>
<p>Used: <code>nvcc --version</code><br>
Output:</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:40:05_Pacific_Daylight_Time_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0
</code></pre>
<p>Used:</p>
<pre><code class="lang-auto">available_gpus = [torch.cuda.device(i) for i in range(torch.cuda.device_count())]
print("List Device:\t", available_gpus)
</code></pre>
<p>Output: <code>List Device:     []</code><br>
<strong>::!!!NEXT!!!::</strong><br>
Used:</p>
<pre><code class="lang-auto">device = torch.device("cuda" if (torch.cuda.is_available()) else "cpu")
print("Device:\t\t", device)
print("CUDA GPU:\t", torch.cuda.is_available())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Device:          cpu
CUDA GPU:        False
</code></pre>
<p><strong>::!!!NEXT!!!::</strong><br>
used:</p>
<pre><code class="lang-auto">torch.cuda.get_device_name(0)
# and
torch.cuda.current_device()
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
</code></pre>
<p><strong>Here are some more errors when running other code.:</strong></p>
<pre><code class="lang-auto">UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  ..\c10\cuda\CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() &gt; 0
Warning: caught exception 'CUDA driver initialization failed, you might not have a CUDA gpu.', memory monitor disabled
</code></pre>
<p><strong>Please help, explain in detail what to do, specifically for me: “Download this, then write it, install it like this, write it like this, and so on.”</strong></p>
<p><strong>Thank you in advance.</strong></p>

ptrblck: <p>Based on the initial error messages you might have installed the CPU-only binary, but nevertheless your GPU is too old.<br>
The GT 630M is from the Fermi family and uses compute capability 2.1 which is not supported anymore as the PyTorch binaries ship for compute capabilities 3.7-8.6.</p>

vilad: <p>And I have no way to use the GPU?</p>

ptrblck: <p>No, since Fermi (compute capability 2.1) was dropped in CUDA 9.0 while we are currently using 11.6-11.8.</p>

Dana_Ludwig: <p>I’ve googled that AssertionError, and in every case I’ve seen, it always was triggered by the same command:  ‘current_device()’.  My install was exactly pasted from the PyTorch installation instructions:</p>
<p>conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia</p>
<p>Could the bug be in ‘current_device()’ instead?</p>

ptrblck: <p>No, the bug is not in any PyTorch-related code and the Fermi CUDA architecture is just not supported anymore as previously described.</p>
Three dimension array value check

cbd: <p>I have array “A”  and “B” of size [1,3,10,10]. Variable B all value is 0.8 . I want to check all <strong>three dimension</strong> of A 10x10. if all are greater than 0.8 than i want to replace value 0 in 3 dimension position in variable B.</p>
<p>For, example A[0,0,0,0]=<strong>0.9</strong>,A[0,1,0,0]=<strong>0.95</strong>,A[0,2,0,0]=<strong>0.90</strong> then  B[0,0,0,0]=0,B[0,1,0,0]=0,B[0,2,0,0]=0 but if A[0,0,0,0]=<strong>0.75</strong>,A[0,1,0,0]=0.95,A[0,2,0,0]=0.90 then B should not be change.</p>
<p>B[B&lt;A]=0 will do for each array in all dimension. I want to replace only when all three dimension is greater than 0.8.</p>

ptrblck: <p>Could you check if something like this would work or if I misunderstood your use case?</p>
<pre><code class="lang-python">A = torch.randn(1, 3, 10, 10) * 2
B = torch.ones_like(A)

idx = (A &gt; 0.8).all(1).unsqueeze(1).expand(-1, 3, -1, -1)
B[idx] = 0.
</code></pre>

cbd: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="169722">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">A = torch.randn(1, 3, 10, 10) * 2
B = torch.ones_like(A)

idx = (A &gt; 0.8).all(1).unsqueeze(1).expand(-1, 3, -1, -1)
B[idx] = 0.
</code></pre>
</blockquote>
</aside>
<p>Yes. Solution works for me. Thanks</p>
Why do we use .detach() for fake data in the discriminator?

shrbrh: <p>The code for stargan is <a href="https://github.com/yunjey/stargan/blob/master/solver.py" rel="noopener nofollow ugc">here</a>. In line 244, the discriminator prediction for real data is computed. In line 250, the discriminator prediction for fake data is computed but with .detach(). Why do we need to use .detach() when using zero.grad() will eventually remove all previously computed gradients? Also, why do we use it only for fake data and not the real data?</p>
<p>.</p>

srishti-git1110: <p>Hi,<br>
The purpose of using detach and zero_grad differ.</p>
<p>detach is used to literally detach a tensor from its current computation graph.</p>
<p><code>x.detach()</code> hence returns a new tensor that isn’t attached to any comp graph and doesn’t require gradient. Importantly note that the new tensor shares the same storage with the previous one.</p>
<p>zero_grad simply only zeros out the gradients of the parameters (of the model) it is called on. The tensors in the computation graph aren’t affected.</p>
<p>As for your question, detach is used when the previous calculation history of a tensor wouldn’t be required anymore for the calculation of any gradients and hence we are able to save memory by detaching it from its graph.</p>
<p>zero_grad is used to prevent accumulation of gradients from several backward calls.</p>

shrbrh: <p>Thank you for your reply.<br>
I am a little confused as to why the discriminator uses .detach() with fake data only. Shouldn’t it use for both real and fake data distribution?</p>

srishti-git1110: <p>Hi, this is more of an architectural question and I am not into CV, but here’s what I think:</p>
<p>I think you are talking about this part of the code -</p>
<pre><code class="lang-auto"># Compute loss with fake images.
            x_fake = self.G(x_real, c_trg)
            out_src, out_cls = self.D(x_fake.detach())
            d_loss_fake = torch.mean(out_src)
</code></pre>
<p>As I could understand it, fake data is generated using the generator part of the GAN network.</p>
<p>The goal of the discriminator is to act as a classifier to classify real and fake images. Now, since during the discriminator training only this classifier needs to be trained (and not the generator part), the fake data needs to be detached from its computation graph as its graph involves the parameters of the generator.</p>
<p>Also, for real images I don’t think there’s a need to call detach as those do not have any computation graph associated with them.</p>

shrbrh: <p>Thanks a lot. This makes things so much clearer.</p>
The error in Graph neural networks for node classification

zhengyang_xu: <p>I have several graphs and I use these graphs to train a model hoping to identify the category of each node.I have three categories, and here are the formulation of one of my graph(using the data object):</p>
<pre><code class="lang-auto">Data(x=[100,64],pos=[100,2],edge_index=[2,546],y=[100],edge_weight=[546])
</code></pre>
<p>x is the embeding of every nodes, here have 100 nodes and each node have a 64-dimension feature, pos is the coordinate of each node. A two-dimensional vector in edge_index represents an edge between two points. And the edge_weight is the weight corresponding to each edge.</p>
<p>here is my GCN model:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.nn import GCNConv

# GCN model with 2 layers 
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = GCNConv(data.num_features, 16)
        self.conv2 = GCNConv(16, int(data.num_classes))

    def forward(self):
        x, edge_index = data.x, data.edge_index
        x = F.relu(self.conv1(x, edge_index))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

data =  data.to(device)

model = Net().to(device)
</code></pre>
<p>and here is my training code:</p>
<pre><code class="lang-auto">for epoch in range(epoches):
    for img_path in img_paths: #in order to build the graph
         data=build_graph(img_path)
         model.train()
         optimizer.zero_grad()
         output=GCNmodel(data)
         F.nll_loss(output,target).backward()
         optimizer.step()
</code></pre>
<p>but using this code, every node in one graph have the same output, especially have the negative num.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7ce63f617be0c73fc5e065500cfc967d48e29dd7.png" alt="截屏2023-01-09 11.58.36" data-base62-sha1="hOUvAwViTlr3pijij4ysF0ZJU5p" width="461" height="177"></p>
<p>Note:The category 1 and 2 of each picture is balanced, the category 3 in every image is samller. But the data should not be the main reason for this result</p>
<p>It confuse me  lots of time, I am very grateful, if you can help me.</p>
Should I permute the input's dimensions for the first layer, which is Conv1D?

Dev2150: <p>The input layer shape I have used is (Batches X HistoricalDataSets X InputTypes) (e.g. 64x50x15)<br>
By InputTypes, I mean different functions whose values I’m going to use as an input<br>
By HistoricalDataSets, I mean most recent set of values from the different functions.</p>
<p>What shape is best for the first layer of Conv1D to work with? Should I permute it so that it is (Batches X Input Types X HistoricalDataSets) ?</p>
<p>I use 2 layers of Conv1D + MaxPooling + ReLU</p>

J_Johnson: <p><code>Conv1d</code> layers are typically used in sequences. By sequence, I mean where consecutive values carry information in their positional relationship(i.e. if you shuffled the order around, you’d lose important information). The third dim should be the sequence length.</p>
<p>The second dim should be used for channels. By channel, I mean that you could reorder channels and not lose relevant information because they do not have a positional relationship with one another.</p>
<p>Each trainable kernel will convolve along the sequence dim, and separate kernels will be used to convolve each channel.</p>

Dev2150: <p>Thank you<br>
So input_channels=input_types…<br>
Strangely, compared to before, the initial training and testing score is worse, however it takes way more actions instead of doing nothing. I hope for the best</p>
Difference between aot_function and aot_module in AOT Autograd?

UmeTFE: <p><a href="https://pytorch.org/functorch/nightly/aot_autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/functorch/nightly/aot_autograd.html</a></p>
<p>Thanks for your work, now I want to try a tiny example in pytorch 2.0…</p>
<p>I have some questions,</p>
<ol>
<li>
<p>Does aot_module perform compilation like aot_function? If I want to compile the forward graph and backward graph of a PyTorch DNN, should I call aot_module or aot_function?</p>
</li>
<li>
<p>In the link: <a href="https://pytorch.org/functorch/stable/notebooks/aot_autograd_optimizations.html" class="inline-onebox" rel="noopener nofollow ugc">AOT Autograd - How to use and optimize? — functorch 1.13 documentation</a></p>
</li>
</ol>
<p>You stated “AOT Autograd provides simple mechanisms to compile the extracted forward and backward graphs through deep learning compilers, such as NVFuser, NNC, TVM and others.”, how can I send the graphs to TVM? Can you give me some idea or tiny example?</p>
<ol start="3">
<li>On colab, I tried to use aot_module to perform compilation, but the acceleration effect is very limited. Is there something wrong with my code?</li>
</ol>
<p>My code is:</p>
<p>from functorch.compile import aot_function, aot_module, draw_graph<br>
import torch.fx as fx<br>
import torch<br>
from functorch.compile import ts_compile<br>
import numpy as np</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 6, 5)<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, 5)<br>
self.fc1 = nn.Linear(16 * 5 * 5, 120)<br>
self.fc2 = nn.Linear(120, 84)<br>
self.fc3 = nn.Linear(84, 10)</p>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = torch.flatten(x, 1) # flatten all dimensions except batch
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
<p>net = Net()</p>
<p>nf = aot_module(net, fw_compiler=ts_compile, bw_compiler=ts_compile)</p>
<p>trainloop(nf)…</p>
<p>Thanks in advance!</p>
PytorchStreamReader failed locating file constants.pkl: file not found

hedeya1980: <p>Hello,</p>
<p>I want to use YOLOv5 for object detection Using C++ and libtorch. I’m using Windows 10, and VS 2019.</p>
<p>However, the following code:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;

int main(int argc, char* argv[]) try
{
    torch::jit::script::Module module;
    try
    {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load("C:/Users/hedey/source/repos/Road_Defects_Detector/YOLOv5s.pt");
    }
    catch (const c10::Error&amp; e)
    {
        std::cerr &lt;&lt; "error loading the model\n" &lt;&lt; e.msg();
        std::system("pause");
    }
    return 0;
} 
</code></pre>
<p>is causing the following error:</p>
<pre><code class="lang-auto">error loading the model
PytorchStreamReader failed locating file constants.pkl: file not foundPress any key to continue . . .
</code></pre>

mhbassel: <p>I think this issue is related <a href="https://github.com/pytorch/pytorch/issues/47917" class="inline-onebox" rel="noopener nofollow ugc">LibTorch cannot load PyTorch exported model · Issue #47917 · pytorch/pytorch · GitHub</a></p>
<p>Maybe you are trying to load a non-TorchScripted model.</p>

hedeya1980: <p>You’re correct <a class="mention" href="/u/mhbassel">@mhbassel</a>. I converting to torch script via tracing as described in <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#converting-to-torch-script-via-tracing" rel="noopener nofollow ugc">this link</a>, and it worked. Thanks a lot.</p>
Accuracy decreasing after optimize_for_mobile

Vidit_Agarwal: <pre data-code-wrap="traced_script_module"><code class="lang-nohighlight">
traced_script_module_optimized = optimize_for_mobile(traced_script_module)```

The traced_script_module gives an accuracy of 98% but for optimised one the accuracy falls down to 6%. How can I overcome this?</code></pre>

cccclai: <p>Is it a quantized model? cc: <a class="mention" href="/u/kimishpatel">@kimishpatel</a></p>

Vidit_Agarwal: <p>Yes. The model was dynamically quantized then optimised</p>

mmisiur: <p>I experienced exact same case:<br>
Model that is not quantized and optimized is perfectly ok, however when I quantize model and then optimize its output is complete chaos. I think that is architecture dependent, bc I didn’t reproduced this problem for example of DeiT.</p>

kimishpatel: <p>Can you provide a simpler repro for this?</p>
<p>One thing I can suggest is to check the memory format of output tensor for both models. If one is in NHWC format vs NCHW format, that may make a difference, if you are directly consuming the tensor without regards for the memory format.</p>
Moving custom modules between devices

jfrancis79: <p>I am having trouble understanding how moving custom modules between devices works in Pytorch.<br>
I have to implement some custom logic on a device move.<br>
Example:</p>
<pre><code>class Fred(nn.Module):
    def __init__(self):
        super().__init__()
    
    def to(self, device):
        print("Fred::to")
    
    def _apply(self, device):
        print("Fred::_apply")


class Bert(nn.Module):
    def __init__(self):
        super().__init__()
        self.fred = Fred()
    
    def to(self, device):
        print("Bert::to")
        super().to(device)


bert = Bert()
bert.to("cpu")
</code></pre>
<blockquote>
<blockquote>
<blockquote>
<p>Bert::to<br>
Fred::_apply</p>
</blockquote>
</blockquote>
</blockquote>
<p>I can put my logic into the Fred::_apply method; but my question is is that good practice?<br>
I had imagined that the Bert.to call would recursively call the to methods on all the member modules, ie end up calling Fred.to, but this does not appear to happen.</p>
<p>My concern is that overriding _apply is just a “hack” using internal PyTorch code and may break in future releases. Is there a more pytorchonic (ie good style) way of handling this?</p>
<p>Thank you,<br>
Julian.</p>

ptrblck: <p>The internal <code>to()</code> method calls <code>self._apply(convert)</code> as seen <a href="https://github.com/pytorch/pytorch/blob/ae0c4c4c295b1b4d6ac94f3c665bc39c7db052cd/torch/nn/modules/module.py#L1115-L1132">here</a>. <code>_apply</code> calls then:</p>
<pre><code class="lang-python">for module in self.children():
    module._apply(fn)
</code></pre>
<p>so you might take a look at these methods and try to add your custom logic to them.</p>
<p>I also agree that overriding internal methods might easily break, but also your use case dictates to override the <code>to()</code> method so I’m unsure if there would be a cleaner way.<br>
Maybe the <a href="https://pytorch.org/docs/stable/notes/extending.html">Extening PyTorch docs</a> migth be helpful.</p>

jfrancis79: <p>Thank you for your reply.</p>
<p>My specific use case is also related to issue <span class="hashtag">#7795</span> (codification API for distributions), which doesn’t seem to have gained much traction, although I recognise there can be some subtleties, especially related to parameters which are updated.</p>
<p>I have just read the PyTorch documentation suggests nn.Modules should support device placement on construction, so if I follow that consistently it should remove my need to get the .to function to work.</p>
<p>Thank you for explaining the _apply mechanism.</p>