akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Complex functions : "exp does not support automatic differentiation for outputs with complex dtype."

Simon_C: <p>Hi, it seems that my very elementary complex exponential does not support Autograd :</p>
<pre><code class="lang-auto">z = torch.tensor([1.1-2j], requires_grad=True)
f = torch.exp(z)
</code></pre>
<p>gives me a RunTimeError :</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-31-69abc9207ce8&gt; in &lt;module&gt;
      1 z = torch.tensor([1.1-2j], requires_grad=True)
----&gt; 2 f = torch.exp(z)

RuntimeError: exp does not support automatic differentiation for outputs with complex dtype.
</code></pre>
<p>How can I use Autograd with complex functions in general, and with the elementary complex exponential in particular ?</p>
<p>Thank you,<br>
S</p>

albanD: <p>Hi,</p>
<p>In preparation for the 1.7 release and to avoid issues, we added error messages for all the functions that were not yet audited for complex autograd.<br>
We are working on auditing the formulas and re-enabling them.</p>
<p>cc <a class="mention" href="/u/anjali411">@anjali411</a> do we have an issue describing the process if people want to help here?</p>

Simon_C: <p>Thank you.<br>
Is there a list of the complex functions that already support autograd ?</p>

albanD: <p>Yes you can find the list here: <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/gen_variable_type.py#L151-L164">https://github.com/pytorch/pytorch/blob/master/tools/autograd/gen_variable_type.py#L151-L164</a></p>
<p>You can find the PRs adding new support by doing a search like that: <a href="https://github.com/pytorch/pytorch/pulls?q=is%3Aopen+is%3Apr+label%3A%22module%3A+complex%22">https://github.com/pytorch/pytorch/pulls?q=is%3Aopen+is%3Apr+label%3A"module%3A+complex"</a></p>
<p>Also if you need another op that no one is implementing, feel free to send a PR and it will be available on the next nightly build after it is merged <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

shaohua: <p>Maybe you can use torch.cos(z) + 1j * torch.sin(z) instead.</p>

alidiak: <p>Hey albanD, I am using the stable Pytorch 1.7.1 build and when I try testing functions that are listed under the GRADIENT_IMPLEMENTED_FOR_COMPLEX the ‘does not support automatic differentiaion’ error still occurs for many of them. Should I instead use the nightly build to access these complex autogradients? Thanks</p>

albanD: <p>It depends where you are checking that constant. If you check it on master, then yes it will correspond to the nightly build.<br>
For the ones supported in 1.7.1, you can check the version that was used for the release here: <a href="https://github.com/pytorch/pytorch/blob/57bffc3a8e4fee0cce31e1ff1f662ccf7b16db57/tools/autograd/gen_variable_type.py#L154" class="inline-onebox">pytorch/gen_variable_type.py at 57bffc3a8e4fee0cce31e1ff1f662ccf7b16db57 · pytorch/pytorch · GitHub</a></p>
Can we set param.grad instead of calling backward, THEN call backward to add additional grads on top?

Sam_Lerman: <p>As the question is asked, e.g.:</p>
<pre><code class="lang-auto">for param in model.parameters():
        param.grad = 3.14

### do some additional compute

loss.backward()

optimizer.step()
</code></pre>
<p>Will this add the new grads from <code>backward()</code> on top of the manually set ones of 3.14? Or will it override the manually set ones?</p>

KFrank: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="1" data-topic="108434" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>Will this add the new grads from <code>backward()</code> on top of the manually set ones of 3.14?</p>
</blockquote>
</aside>
<p>Yes, but remember to call <code>optimizer.zero_grad()</code> <em>before</em> setting<br>
<code>param</code> (otherwise you will zero-out the value you set), and wrap<br>
setting the <code>param</code>s in a <code>no_grad()</code> block:</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer.zero_grad()

with torch.no_grad():
    for param in model.parameters():
        param.grad = 3.14

### do some additional compute, e.g.
loss = my_loss_function (input, target)

loss.backward()

optimizer.step()
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Optimizing regression weights for NN outputs

ichitaka: <p>So I’m basically trying to fit a regression on the relation of the input and output of a neural network model. Then the idea is, that these estimated regression weights should be optimized to some specific target value (let’s say matrix of ones). So the weights are optimized, but have a direct relation to the neural network weights.</p>
<p>I think I have an issue with how I define the weights, as they probably don’t relate to the neural network the way I define them?</p>
<pre><code># Some arbitrary PyTorch model
my_nn = Net()

# Our target regression weights
target = torch.ones(2, 2)

# Optimizer for NN weights
optimizer = optim.Adam(my_nn.parameters(), lr=0.0001)

# weights for regression model
w1 = Variable(torch.randn(2, 2).type(torch.float), requires_grad=True)
# optimizer for regression model
optimizer1 = optim.SGD([w1], lr=0.001, momentum=0.9)

# Arbitrary loss function
criterion = nn.MSELoss()

# Training loop
my_nn.train()
for epoch in range(500):
    print('Running Epoch: ', epoch)
    # zero the parameter gradients
    optimizer.zero_grad()

    # output of model is 2-dim
    y_pred = my_nn(X)

    for nest_epoch in range(1000):
        optimizer1.zero_grad()
    
        y_hat = torch.mm(X, w1)
        loss = criterion(y_hat, y_pred)

        # Need to add retain_graph, as it doesn't work otherwise
        # but it does work, this loss will get close to 0
        loss.backward(retain_graph=True)
        optimizer1.step()

# this loss does not decrease
loss = criterion(target, w1)
print('Loss:', loss.item())
loss.backward()
optimizer.step()</code></pre>
[HELP] FedProx loss implementation

Stefano_Setti: <p>Hi, so I’m trying to implement the FedProx loss from this <a href="https://arxiv.org/abs/1812.06127" rel="nofollow noopener">paper</a>, my idea is to take the BCE loss and edit like this:</p>
<pre><code class="lang-auto">def fedprox_loss(output, target, new_weights, original_weights):
    rss = nn.BCELoss(output, target)
    omega = torch.sum(torch.pow(new_weights - old_weights, 2))
    return rss + omega
</code></pre>
<p>But i don’t know if this works properly and i really don’t know if I need to implement the backward method and how to do it.</p>

ptrblck: <p>If all your inputs are PyTorch tensors, you won’t have to implement the backward pass manually and Autograd will do it for you.<br>
Are you seeing any errors or unexpected behavior?</p>

omarfoq: <p>You forgot to multiply by mu.</p>
Apply torch.inverse to batch of matrix without throwing exception for singular matrix

wasabi: <p>I have a 3d tensor A of shape (b, c, c) and I think that torch.inverse(A) is much faster on GPU then applying torch.inverse(A[i,:,:]) for all i in a loop. The issue is that some of A[i,:,:] might be singular and torch.inverse throws a runtime error. How can I circumvent this while still enjoying fast computation?</p>
JIT performance

rafale77: <p>Spent the better part of the day trying to script a couple of facial detection and recognition models for inference in hope of seeing some performance improvements and was very disappointed.</p>
<ol>
<li>Scripting is pretty restrictive, some very benign code don’t seem to compile properly and forced me to exclude from JIT compilation. In my case it was the handling of a list of tensors erroring as the JIT doesn’t know how to handle such objects.</li>
<li>One of the simpler models actually scripted very easily but… showed only a degradation for the first couple of passes and then identical performance to python (cpython).</li>
</ol>
<p>I was reading through <a href="https://program-transformations.github.io/slides/pytorch_neurips.pdf" rel="noopener nofollow ugc">this</a> and was really hopeful but at this point (pytorch1.7.1/cuda11.2/cudnn8.05) with my relatively common pretrained models, I am not seeing anything positive. Has anyone been able to verify performance improvements? I have experience with lua/luajit which shows drastic speed enhancements.</p>
Problem: Plotting confusion metrics result vary each time

ArshadIram: <p>Hi, I am trying to plot the confusion metric but each time when I run a cell I am getting a different result.<br>
Here is code:<br>
pred_list=torch.zeros(0,dtype=torch.long, device=‘cpu’)</p>
<p>label_list=torch.zeros(0,dtype=torch.long, device=‘cpu’)<br>
with torch.no_grad():<br>
for inputs, labels in train_loader:</p>
<pre><code>  inputs = inputs.to(device)

  labels = labels.to(device)

  outputs = model(inputs)

  _, preds = torch.max(outputs, 1)

 

  # Append batch prediction results

  pred_list=torch.cat([pred_list,preds.view(-1).cpu()])

  label_list=torch.cat([label_list,labels.view(-1).cpu()])
</code></pre>
<h1><a name="confusion-matrix-1" class="anchor" href="#confusion-matrix-1"></a>Confusion matrix</h1>
<p>conf_matrix=confusion_matrix(label_list.numpy(), pred_list.numpy())</p>
<p>print(conf_matrix)</p>
<p>Iteration 1: <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64d5140e13f1ab4dcb3e95882e549ff73a0cf408.png" alt="image" data-base62-sha1="eo0grKfxvzNMki1gnWtjty99Jr2" width="468" height="198"></p>
<p>Iteration 2: <img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be6ddac5511960c12750c556e884838720b7dcd5.png" alt="image" data-base62-sha1="raC548UZY0Dtzg4ZLFQWpHxqFxj" width="457" height="201"></p>
<p>Due to this my classification report is also varies. Any help in this regard is appriciated.</p>

anujd9: <p>Hi. If I understand your problem correctly, you are getting a slightly different confusion matrix values each time you run the complete cell that makes the inference using your trained model.<br>
If my understanding is correct, then one thing to try out in this case would be to set seed, something like this:</p>
<pre><code class="lang-auto">seed = 0
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)
</code></pre>
<p>Setting a seed helps in result reproducibility. You can read more on this here: <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.7.0 documentation</a></p>

ArshadIram: <p>Yes, your understanding is correct. In each iteration when i run a complete cell confusion matrix values slightly updated. Let me try this solution which you shared with me.</p>
Proplems with loss in GAN

viatkin_sasha: <p>I have implemented pix2pix GAN from <a href="https://arxiv.org/pdf/1611.07004" rel="noopener nofollow ugc">scratch</a>. And I ran into a problem with loss on generator. It is rising(<a href="https://i.stack.imgur.com/Ttsko.png" rel="noopener nofollow ugc">visualisation of losses on valid and train</a>). As backbone I have implemented UNet(It is too big to post it here, but if it would be helpful I can insert it).<br>
Why it can be so?</p>
<p>It is my train section:</p>
<pre><code class="lang-auto">def set_grad(model, flag):
    for parameter in model.parameters():
        parameter.requires_grad = flag


for epoch in range(epochs):
        avg_gen_train_loss = 0
        avg_disc_train_loss = 0
        disc_model.train()
        gen_model.train()

        for X_batch, Y_batch in data_tr:

            X_batch, Y_batch = X_batch.to(device), Y_batch.to(device) # data to device
            
            # generate image

            Y_batch_gen = gen_model(X_batch)

            # discriminator

#--------------------------------------------
            set_grad(disc_model, True)
            disc_optim.zero_grad()

            X_Y_gen = torch.cat((X_batch, Y_batch_gen), 1)
            d_gen = disc_model(X_Y_gen.detach())
            loss_gen = torch.nn.BCEWithLogitsLoss()(d_gen, torch.Tensor([0.]).expand_as(d_gen).to(device))

            X_Y = torch.cat((X_batch, Y_batch), 1)
            d_real = disc_model(X_Y)
            loss_real = torch.nn.BCEWithLogitsLoss()(d_real, torch.Tensor([1.]).expand_as(d_real).to(device))
            
            disc_loss = (loss_gen + loss_real) * 0.5
            avg_disc_train_loss += disc_loss
            disc_loss.backward()

            disc_optim.step()
#--------------------------------------------

            # generator

#--------------------------------------------
            set_grad(disc_model, False)
            gen_optim.zero_grad()

            d_gen = disc_model(X_Y_gen)
            loss_gen = torch.nn.BCEWithLogitsLoss()(d_gen, torch.Tensor([1.]).expand_as(d_gen).to(device))

            loss_l1 = nn.L1Loss()(Y_batch_gen, Y_batch)

            loss_gen = loss_gen + lambda_ * loss_l1
            avg_gen_train_loss += loss_gen
            loss_gen.backward()

            gen_optim.step()
#--------------------------------------------
</code></pre>
How to customize the double backward?

Naruto-Sasuke: <p>I have a custom module which aims to try rearranging values of the input in a sophisticated way(I have to extending autograd) .<br>
Thus the double backward of gradients should be the same as backward of gradients, similar with reshape?</p>
<p>If I define in this way in XXXFunction.py:</p>
<pre><code class="lang-python">    @staticmethod
    def backward(ctx, grad_output):
        # do something to rearrange grad_output to get grad_input
        return grad_input
</code></pre>
<p>The double backward seems not correct. Then how to customize the double backward?</p>

SimonW: <p>Define another Function to do backward of this function (and use it in this backward) and define double backward in that Function’s backward.</p>

Naruto-Sasuke: <p>Hi, SimonW, I am still a bit confusing. Could you please give some pseudocode about this? Thank you very much.</p>

amensch: <p>Hi !</p>
<p>You may go for</p>
<pre><code class="lang-python">class CubeFunction(torch.autograd.Function):
    """
    Dummy activation function x -&gt; x ** 3
    """
    @staticmethod
    def forward(ctx, X):
        ctx.save_for_backward(X)
        return X ** 3

    @staticmethod
    def backward(ctx, M):
        X, = ctx.saved_tensors
        return CubeFunctionBackward.apply(X, M)


class CubeFunctionBackward(torch.autograd.Function):

    @staticmethod
    def forward(ctx, X, M):
        ctx.save_for_backward(X, M)
        return M * 3 * X ** 2

    @staticmethod
    def backward(ctx, V):
        X, M = ctx.saved_tensors
        return V * 6 * X * M, V * 3 * X ** 2
</code></pre>
<p>See this gist <a href="https://gist.github.com/arthurmensch/b02c45f3440c9d3de0ef2c0ae5ea1107" rel="nofollow noopener">https://gist.github.com/arthurmensch/b02c45f3440c9d3de0ef2c0ae5ea1107</a> for a complete demo.</p>
<p>I could PR this to the documentation if any developer deems it interesting enough to be exposed.</p>

Naruto-Sasuke: <p>Aha, interesting. I would like to see a PR about this, I don’t know what the developers think about this.</p>

AlphaBetaGamma96: <p>Hey, <a class="mention" href="/u/amensch">@amensch</a> I was wondering if I could ask a question about the <code>grad_output</code> (V) in the doublebackward?</p>
<p>From my understanding, <code>grad_output</code> is the gradient of the loss with respect to the output of the function. So, <code>M</code> is d(loss)/df where f = x^3 and the loss is the output of the function. So in effect, d(loss)/df = 1.</p>
<p>I understand that there’s 2 returned gradients where backward1 = M * 3 * x^2 is differentiated with respect to X, and M and then multiplied by <code>V</code> is order to get the correct 2nd order gradient for each variable. Where <code>V</code> would be the derivative of some loss with respect to backward1. But what is this 2nd order loss?</p>
<p>Hopefully, this makes sense!</p>
<p>Thank you!</p>
Pytorch geometric: get data for individual graphs in a batch

Quang_Tran: <p>I’m new to PyTorch Geometric. I’m processing data in batch and for each batch I forward the data through several layers and finally get <code>w_att</code> (attention weight matrix) of dimension NxN, with N being the total number of nodes of all the graphs in the batch. (Eg, If my batch has 2 graphs that have 3 and 4 nodes, respectively, then N = 3+4=7). The problem is, I now need to get the attention weight matrix for <em>each</em> graph (size num_nodes x num_nodes) individually from <code>w_att</code>. (Using my same example, I need 2 separate attention weight matrices; the first matrix should have dimension 3x3 and the second should have dimension 4x4). What is a good way to obtain these matrices from the encompassing <code>w_att</code>?</p>
Passing part of the NN with one optimizer, and all other with another optimizer

MrPositron: <p>I have a seq2seq with encoder and decoder. The structure of the architecture is as follows:</p>
<pre><code class="lang-auto"># initialize encoder
class Encoder(nn.Module):
    ...
# initialize decoder
class Decoder(nn.Module):
    ...
# initialize seq2seq model
# we pass the decoder and encoder to it
class Seq2Seq(nn.Module):
   ...
</code></pre>
<p>Then, we create the network as follows:</p>
<pre><code class="lang-auto">encoder_net = Encoder(...)
decoder_net = Decoder(...)
model = Seq2Seq(encoder_net, decoder_net, ...)
</code></pre>
<p>In decoder network, there is a quite complicated network, let say on top of the <code>nn.GRU()</code>. Thus, I want to optimize this part with one LR, and everything other that that with another LR. How can I do that?</p>
<p>Can I do it like this?</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.decoder_net.layer_x.parameters(), 'lr':1e-4},
                {'params': model.parameters()}
            ], lr=1e-3, momentum=0.9)
</code></pre>

MrPositron: <p>ups. I can just pass them as a list to the optmizer.</p>
Adding scalar parameter

frank-ccc: <p>dear all,</p>
<p>I am a rookie with pytorch.</p>
<p>I am trying to add a parameter, something like a scalar, to a neural network with I previously defined and added to the optimizer</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model_g.parameters(), lr=learning_rate)

a = torch.tensor([1.0]).type(Tensor)
optimizer.add_param_group({'params': a})
</code></pre>
<p>then I do:</p>
<pre><code class="lang-auto"># Forward pass
g = model_g(input)

# Compute the loss
loss = loss_fn(a, g)

# Finally
optimizer.zero_grad()
loss.backward()    
optimizer.step()
</code></pre>
<p>“g” changes accordingly during the loop but “a” does not (it would if “a” were a neural network). Is there any pretty and clean way to optimize “a” as well.</p>
<p>Thanks a lot</p>

frank-ccc: <aside class="quote no-group" data-username="frank-ccc" data-post="1" data-topic="108348">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frank-ccc/48/32935_2.png" class="avatar"> frank-ccc:</div>
<blockquote>
<p><code>add_param_group</code></p>
</blockquote>
</aside>
<p>Found error needs to add <code>device="cuda"</code></p>
<p><code>a = torch.tensor([1.0], requires_grad=True, device="cuda")</code></p>
Compiling example-app with CMake and Visual Studio on Windows

Antoine_Caillon: <p>Hi,</p>
<p>I’m trying to compile the example-app from the libtorch tutorial using Visual Studio 16 (2019).</p>
<p>The CMakeLists.txt file is the following:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

set(CMAKE_PREFIX_PATH "C:\\Users\\caill\\libtorch\\")

find_package(Torch REQUIRED)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 11)

# The following code block is suggested to be used on Windows.
# According to https://github.com/pytorch/pytorch/issues/25457,
# the DLLs need to be copied to avoid memory errors.
if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET example-app
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:example-app&gt;)
endif (MSVC)
</code></pre>
<p>When building the example, I get the following error:</p>
<pre><code class="lang-auto">C:\Users\caill\source\Repos\caillonantoine\example-app\ninja : error : 'torch-NOTFOUND', needed by 'example-app.exe', missing and no known rule to make it
</code></pre>
<p>I’ve been trying to find a solution to this problem online, without success. Do you by any chance know how to fix this problem? Thanks!</p>

peterjc123: <p>Could you please clean the cache and build the project again?</p>

mih.net: <p>Hi,<br>
I don’t know a nice way how to solve this problem, but here is a dirty solution (and it works!).</p>
<p>Find file ‘biuld.ninja’ . For VS2019 it’s in ‘[your_project_dir]\out\build\x64-Release’ directory.<br>
For VS2017 I’ve found it in ‘[your_win_home_dir]\CMakeBuilds\4483b75d-3038-b138-ad88-323c5cdc1522\build\x64-Release’ directory (or Debug if you trying it).</p>
<p>Then you have manually replace in this text file ‘torch-NOTFOUND’ string to full path to your torch.lib (like ‘C$:\libtorch\lib\torch.lib’ (don’t ask me what ‘$’ mean after C, I don’t know. It all experimental, but it work for me.)<br>
Save ‘build.ninja’ file and now you can build and run your project.</p>
<p>Wonderful, isn’t it?</p>

sayed_ali_hoseini_ba: <p>i so had this problem and solved it . delete catch  and clean all in visual studio and delete cmake_minimum_required(VERSION 3.0 FATAL_ERROR)  from CMakeLists.txt. and then generate catch and build project , then works as well.</p>

carlsummer: <p>I slove it,<br>
you can see <a href="https://github.com/carlsummer/libtorch-demo/tree/main/example-app" class="inline-onebox" rel="noopener nofollow ugc">libtorch-demo/example-app at main · carlsummer/libtorch-demo · GitHub</a>,<br>
set(CMAKE_BUILD_TYPE “Release”)</p>
<p>-DCMAKE_PREFIX_PATH=path\to\opencv\build\x64\vc14\lib;path\to\libtorch -DCMAKE_BUILD_TYPE=Release -G"Visual Studio 14 Win64"</p>
Lin's Concordance Correlation Coefficient as loss function

a.par.s: <p>I am trying to use Lin’s Concordance Correlation Coefficient as loss function but it seems it is not work correctly as the loss value using this cost function does not change meaningfully. Can anyone help me with some advices?<br>
Here is my code:</p>
<p>class ConcordanceCorCoeff(nn.Module):</p>
<pre><code>def __init__(self):
    super(ConcordanceCorCoeff, self).__init__()
    self.mean = torch.mean
    self.var = torch.var
    self.sum = torch.sum
    self.sqrt = torch.sqrt
    self.std = torch.std
def forward(self, prediction, ground_truth):
    mean_gt = self.mean (ground_truth, 0)
    mean_pred = self.mean (prediction, 0)
    var_gt = self.var (ground_truth, 0)
    var_pred = self.var (prediction, 0)
    v_pred = prediction - mean_pred
    v_gt = ground_truth - mean_gt
    cor = self.sum (v_pred * v_gt) / (self.sqrt(self.sum(v_pred ** 2)) * self.sqrt(self.sum(v_gt ** 2)))
    sd_gt = self.std(ground_truth)
    sd_pred = self.std(prediction)
    numerator=2*cor*sd_gt*sd_pred
    denominator=var_gt+var_pred+(mean_gt-mean_pred)**2
    ccc = numerator/denominator
    return 1-ccc</code></pre>

Ranahanocka: <p>I’m not sure, but I have a feeling that using the same function might make the autograd history graph incorrect. Instead of using self.mean try calling torch.mean (same for var, sqrt, etc - basically erase the entire init) and see if you still get the problem…</p>

a.par.s: <p>Thanks for your statement. But, my code was true. There was something wrong with my model.</p>

ZhangMeiXian: <p>I try to use this loss function, but I found that the value of ccc would be negative…I want to know the result of your method.Thanks!</p>
Model's not getting updated

Akhil_Koilada: <p>I am trying to use a model based on Graphs but the weights arent getting updated and the output remains the same even after several iterations and the loss is also not getting changes.Below is my implementation of the model.If someone can find the problem, It would be of great help.</p>
<pre><code class="lang-auto">class GCN_Layer(nn.Module):

  def __init__(self,in_features,out_features,bias=True):

    super(GCN_Layer,self).__init__()

    self.in_feaures=in_features

    self.out_features=out_features

    self.weight=Parameter(torch.cuda.FloatTensor(out_features),requires_grad=True)

    if bias:

      self.bias=Parameter(torch.cuda.FloatTensor(out_features),requires_grad=True)

    else:

      self.register_parameter('bias', None)

  def forward(self,input,adj_til,D_til_inv):

    result= (D_til_inv.float() @adj_til.float() @ D_til_inv.float() @input.float())

    return torch.matmul(result,self.weight.reshape(self.weight.shape[0],1))+self.bias

import scipy.linalg

class GCN(nn.Module):

  def __init__(self,n_feat,n_hidden,n_classes):

    super(GCN, self).__init__()

    self.gcn1=GCN_Layer(n_feat,n_hidden)    

    self.gcn2=GCN_Layer(n_hidden,1024)

    self.linear=nn.Sequential(nn.Linear(1024,5,bias=True),

                              nn.Softmax())

    self.relu=nn.ReLU()

  def forward(self,x,adj,features):

    # Degree Matrix

    D_til = torch.zeros(adj.shape[0],adj.shape[1],dtype=float)

    for i in range(D_til.shape[0]):

      D_til[i,i] = torch.sum(adj[i,:],dtype=float)

    # Degree Matrix power -1/2

    D_til_inv = torch.from_numpy(scipy.linalg.fractional_matrix_power(D_til, (-1/2))).cuda()

    out=self.relu(self.gcn1(x, adj,D_til_inv))

    out=self.relu(self.gcn2(out,adj,D_til_inv))

    

    result=torch.matmul(features.float(),out)

    

    out=self.linear(result)

    

    return out`Preformatted text`
</code></pre>
<p>Thank you…</p>

ptrblck: <p>Using other 3rd party libraries such as numpy or scipy will detach these operations from the computation graph, which would be the case in:</p>
<pre><code class="lang-python">D_til_inv = torch.from_numpy(scipy.linalg.fractional_matrix_power(D_til, (-1/2))).cuda()
</code></pre>
<p>If you don’t need or want to calculate the gradients for preceding operations, your code should work fine.<br>
Otherwise you would need to either use PyTorch methods or write a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>

Akhil_Koilada: <p>Thank you…It worked out pretty well</p>
Indexing adj matrix of a knn graph

Y_shin: <p>Say I have a top-k indexing matrix <code>P (B*N*k)</code>, a weight matrix <code>W(B*N*N)</code> and a target matrix <code>A (B*N*N)</code>, I want to get a adjacent matrix that operates as the following loops:</p>
<pre><code class="lang-auto">for i in range(B):
         for ii in range(N):
             for j in range(k):
                 if weighted:
                     A[i][ii][P[i][ii][j]] = W[i][ii][P[i][ii][j]]
                 else:
                     A[i][ii][P[i][ii][j]] = 1
</code></pre>
<p>How to implement it more efficiently and concisely?</p>

ptrblck: <p>You could use a <code>scatter_</code>/<code>gather</code> approach as seen here:</p>
<pre><code class="lang-python">B, N, k = 2, 3, 4

P = torch.randint(0, N, (B, N, k))
W = torch.randn(B, N, N)
A = torch.zeros(B, N, N)
A_ = A.clone()

# loop
for i in range(B):
    for ii in range(N):
        for j in range(k):
            A[i][ii][P[i][ii][j]] = W[i][ii][P[i][ii][j]]

# scatter/gather approach
A_.scatter_(2, P, torch.gather(W, 2, P))

# check
print((A == A_).all())
&gt; tensor(True)
</code></pre>
Potential data race on gpu with tensor advanced indexing

wdyin: <p>When I try to assign the value of tensors <strong>c</strong> to another tensor <strong>a</strong> via some index mappings <strong>b1,b2,…, bi</strong> using advanced indexing, potential data race on GPU is observed when some indexes point to the same entry of <strong>a</strong>. An example code snippet is posted here:</p>
<blockquote>
<p>a = torch.zeros(100,100).cuda()<br>
b = torch.zeros(50,dtype=torch.long).cuda()<br>
c = torch.arange(50).unsqueeze(1).expand(50,100).float().cuda()<br>
a[b]=c</p>
</blockquote>
<p>and the corresponding value is as follows:</p>
<blockquote>
<p>In [17]: c<br>
Out[17]:<br>
tensor([[ 0.,  0.,  0.,  …,  0.,  0.,  0.],<br>
[ 1.,  1.,  1.,  …,  1.,  1.,  1.],<br>
[ 2.,  2.,  2.,  …,  2.,  2.,  2.],<br>
…,<br>
[47., 47., 47.,  …, 47., 47., 47.],<br>
[48., 48., 48.,  …, 48., 48., 48.],<br>
[49., 49., 49.,  …, 49., 49., 49.]], device=‘cuda:0’)</p>
<p>In [15]: a<br>
Out[15]:<br>
tensor([[35., 35., 35.,  …, 34., 34., 34.],<br>
[ 0.,  0.,  0.,  …,  0.,  0.,  0.],<br>
[ 0.,  0.,  0.,  …,  0.,  0.,  0.],<br>
…,<br>
[ 0.,  0.,  0.,  …,  0.,  0.,  0.],<br>
[ 0.,  0.,  0.,  …,  0.,  0.,  0.],<br>
[ 0.,  0.,  0.,  …,  0.,  0.,  0.]], device=‘cuda:0’)</p>
<p>In [16]: a[0]<br>
Out[16]:<br>
tensor([35., 35., 35., 35., 35., 35., 35., 35., 35., 35., 35., 35., 35., 35.,<br>
35., 35., 35., 35., 35., 35., 14., 14., 14., 14., 14., 14., 14., 14.,<br>
14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14.,<br>
14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14., 14.,<br>
34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34.,<br>
34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34.,<br>
34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34., 34.,<br>
34., 34.], device=‘cuda:0’)</p>
</blockquote>
<p>I wonder if there should be some documentation or warnings on this case. The aggregation rule of this implicit tensor scattering on multiple dimensions seems to be undefined.</p>

ptrblck: <p>This is expected behavior and the docs for <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_put_"><code>index_put_</code></a> mention:</p>
<blockquote>
<p>If <code>accumulate</code> is <code>True</code> , the elements in <code>value</code> are added to <code>self</code> . If accumulate is <code>False</code> , the behavior is undefined if indices contain duplicate elements.</p>
</blockquote>
Indexing 4D Tensor With 2D indices

Jack_Rolph: <p>Suppose I have a tensor, <code>a,</code> that I wish to index.</p>
<p><code>a</code> has:<br>
<code>B</code> batches;<br>
<code>C</code> channels;<br>
and dimensions <code>X</code>, <code>Y</code>, <code>Z</code> that contain the values I want to index.</p>
<pre><code class="lang-auto">a.shape
&gt;&gt; torch.Size([B, C, X, Y, Z])
</code></pre>
<p>I have a list of tensors, <code>b</code>, of length <code>B</code>, each containing <code>N</code> indices I want to obtain. Each element of the list may have different values for <code>N</code> ie. <code>N0</code>, <code>N1</code>…</p>
<pre><code class="lang-auto">len(b)
&gt;&gt; B

b[0].shape
&gt;&gt; torch.Size([N0, 3])

b[1].shape
&gt;&gt; torch.Size([N1, 3])
</code></pre>
<p>To be clear, having indexed the tensor, the final result I want is the following:</p>
<pre><code class="lang-auto">len(out)
&gt;&gt; B

out[0].shape
&gt;&gt; torch.Size([C, N0])

out[1].shape
&gt;&gt; torch.Size([C, N1])
</code></pre>
<p>What is the best way of going about this?</p>

ptrblck: <p>I’m unsure how to understand the “values” of <code>N0</code> and <code>N1</code>. Could you post an example code snippet with real shapes and values (even if it’s a slow nested loop)?</p>
How to map a 3D feature volume into a stack of 2D features using pre-defined slicing planes

yft123: <p>I am not sure if this is the right place to ask this technical question. If not, I have also post the same question on <a href="https://stackoverflow.com/questions/65637279/pytorch-map-3d-feature-into-2d-according-to-arbitrary-slicing-planes" rel="noopener nofollow ugc">stack overflow</a>. Please excuse me and answer the question there, if you believe it would be better.</p>
<p>I wonder how I can build a function <code>map_feat(feat, plane_mask)</code> that can map a 3D feature [B, C, D, H, W] into a stack of 2D feature [B, C, H’, W’] based on a few pre-defined slicing planes. Here is a toy example,</p>
<pre><code class="lang-auto">import torch
import numpy as np

def coordinates(voxel_dim, device=torch.device('cpu')):
    nx, ny, nz = voxel_dim
    x = torch.arange(nx, dtype=torch.long, device=device)
    y = torch.arange(ny, dtype=torch.long, device=device)
    z = torch.arange(nz, dtype=torch.long, device=device)
    x, y, z = torch.meshgrid(x, y, z)
    return torch.stack((x.flatten(), y.flatten(), z.flatten()))


def get_plane_mask(verts, n1, n2, n3):
    a =  verts[0] * n1 + verts[1] * n2 + verts[2] * n3
    return a == 1
</code></pre>
<p>I have a 3D feature from network. To make things simple, let’s say batch size = 1 and channel = 5</p>
<pre><code class="lang-auto">voxel_dim = [10, 10, 5]
feat = torch.randn([1, 5] + voxel_dim)
</code></pre>
<p>And I have a few pre-defined slicing planes</p>
<pre><code class="lang-auto">verts = coordinates(voxel_dim)
plane_mask = []
plane_mask.append(get_plane_mask(verts, 0.1, 0.2, 0.))
plane_mask.append(get_plane_mask(verts, 0.2, 0, 0))
plane_mask.append(get_plane_mask(verts, 0, 0, 0.5))
</code></pre>
<p>Next, I wish to index the feature values lying on these planes and interpolate them into a new 3D feature volume, where the last dimension corresponding to the index of the plane masks, and the first 4 dimensions are the [B, C, H’, W’] as described above, i.e. batch size = 1, <span class="hashtag">#channel</span> = 3, height = 8 and width = 8.</p>
<pre><code class="lang-auto">new_feat = torch.zeros([1, 3, 8, 8, len(plane_mask)])
for i in range(len(plane_mask)):
    new_feat[:,:,:,:, i] = map_feat(feat, plane_mask[i])
</code></pre>
<p>This mapping is similar to the CT scan, in which, given a slicing plane, we get a 2D image show the cross section of a 3D volume. It also somehow similar to ROI_pooling I feel.</p>
<p>Any idea to implement this <code>map_feat</code> function is appreciated! It would be even better if the implemented function can do it in one pass without <code>for</code> loop. I can save the plane mask into a sparse tensor.</p>
<p>Currently, I am only considering the case where cross_section is a rectangle, no need to deal with more general cases (triangle, pentagon, etc) at this point.  Thank you in advance.</p>
Creating index tensor inside model

x8808099: <p>I’m using advanced indexing in the forward pass, where an indexing tensor need to be created:</p>
<p><code>torch.arange(batch_size).view(-1,1)</code></p>
<p>By default it seems to be created in cpu, but do not cause any error. Should i specify the device on which it is created, will it then be faster? If so how can I make it in the current device (same as the model)? Or is there a more efficient way to do this?</p>
How can I choose specific output and calculate loss for the that output in pytorch?

111451: <p>I am using a multilingual BERT model from hugging face, and doing some additional pretraining using MLM. However, it is a bit different. I want to get mean value of sequence of masks and calculate the loss.</p>
<p>For example,</p>
<pre><code class="lang-auto">model = AutoModel.from_pretrained('bert-base-multilingual-cased')
input = [MASK][MASK][MASK] is a great guy.
label = [PERSON] is a great guy.
</code></pre>
<p>I want to train the model using the loss between mean value of three [MASK] tokens and a [PERSON] token.</p>
<p>What is the best way to do this?</p>
Do we need to update pretrained embedding?

son: <p>I have been working with pretrained embeddings(GloVe) and suffered memory increases problem. But when I updated code below, memory increase problems didn’t occur.</p>
<p>What I want to ask is that</p>
<ol>
<li>
<p>do we need to update pretrained embedding?</p>
</li>
<li>
<p>Is there any relationship between embedding updates and memory increase problem?</p>
<pre><code> self.embedding = nn.Embedding.from_pretrained(embedding)
 if embedding is not None:
     self.embedding.weight = nn.Parameter(embedding)
 self.variable_lengths= variable_lengths
 self.embedding.weight.requires_grad = False  #memory increases every time when True</code></pre>
</li>
</ol>

ptrblck: <p>If you want to update the embedding layer, the gradients would need to be stored and would thus use memory.<br>
It depends on your use case, if you need/want to fine tune these embeddings or use the pretrained ones.</p>
A variety of distance calculation methods

xiashu: <p>Recently i research all kinds of distance calculation methods,like “Euclidean Distance”," Manhattan Distance"<br>
i know a litte ways</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
a= torch.arange(2,5).view(1,4).float()
print(a)
b=torch.tensor([[2,2],[1,2]]).view(1,4).float()
c=F.pairwise_distance(a,b,p=2)
print(c)
</code></pre>
<p>when “p” get 2 ,it caculates <strong>Euclidean Distance</strong>,when “p"get 1,it caculates <strong>Manhattan Distance</strong>,<br>
but i want know is there any function can quicky caculate " <strong>Hamming Distance</strong>” " <strong>Jaccard Distance</strong>"  " <strong>Mahalanobis distance</strong>" ?</p>
Simple neural network coding for tic-tac-toe

promach: <p>I drafted a <a href="https://gist.github.com/promach/52c5199e98647c694163abd0b3af3dae" rel="noopener nofollow ugc">simple neural network coding</a> for tic-tac-toe using some online references.</p>
<p>However, I am having some problem with neural network input dimension error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://i.imgur.com/douNANS.png" title="" rel="noopener nofollow ugc"><img src="https://i.imgur.com/douNANS.png" alt="" width="690" height="371"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">1922×1034</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>

promach: <p>I have solved the above error for the <a href="https://github.com/promach/mcts/blob/main/Net.py" rel="noopener nofollow ugc">NN training code</a> for tic-tac-toe</p>
<p>Now, how to do  test/inference  ?</p>

promach: <p>The <a href="https://github.com/promach/mcts/blob/main/play.py" rel="noopener nofollow ugc">inference coding</a> gave out repeated wrong results, it seems like the model output trained by the <a href="https://github.com/promach/mcts/blob/main/Net.py" rel="noopener nofollow ugc">training code</a> is wrong. Any idea why ?</p>

promach: <p>now I have test accuracy of 90% on the trained <a href="https://gist.github.com/promach/52c5199e98647c694163abd0b3af3dae#file-net-py" rel="noopener nofollow ugc">NN model</a></p>
<p>But why the <a href="https://gist.github.com/promach/52c5199e98647c694163abd0b3af3dae#file-play-py" rel="noopener nofollow ugc">inference coding (Play.py)</a> failed so badly ?</p>
Create A single channel Target from RGB mask

linem-davton: <p>I have an RGB image which is a segmentation ground truth. It has only two classes, Blue for the background and Red for the foreground. ( Most pixels are transparent)</p>
<p>Now I want to create a mask of size (h,w) where the pixel is 1 if the original corresponding mask’s pixel was blue, 2 if red, and 0 otherwise.</p>
<p>Any idea how do I go about doing this?</p>

ptrblck: <p>You could create a color mapping as described <a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">here</a> to transform the color values to the class indices.</p>
Does `index_add_` work with repeated indexes?

moi90: <p>In <a href="https://discuss.pytorch.org/t/indexing-with-repeating-indices-numpy-add-at/10223/12">this answer</a>, <a class="mention" href="/u/colesbury">@colesbury</a> states:</p>
<blockquote>
<p>Most other in-place indexing functions (like <code>index_add_</code>) have undefined behavior for duplicate indices.</p>
</blockquote>
<p>Is this still the case?</p>
<p>In my test, it worked fine. Can it be relied upon or does it depend on other factors?</p>
<p>The documentation is quiet about this.</p>

ptrblck: <p>I would rely on the posted comment as also given <a href="https://discuss.pytorch.org/t/indexing-with-repeating-indices-numpy-add-at/10223/16">here</a>, since this undefined behavior in <code>index_add</code> matches <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html#detailed-notes">numpy’s behavior</a> for indexing.</p>
Differentiated tensor not being used in a graph

algol: <p>I am trying to take derivatives of a network with respect to its arguments and I am confused about a result I have been getting. For this example my network, <code>model</code>, takes 3 inputs and has two outputs. Here is the code I am running:</p>
<pre><code class="lang-auto">ipt = Variable((batch[0]),requires_grad=True).view(batchSize, batch[0].shape[1])
x = ipt[:,0].unsqueeze(1)
y = ipt[:,1].unsqueeze(1)
z = ipt[:,2].unsqueeze(1)

out = model(ipt).cuda()
A = out[:,0].unsqueeze(1)
B = out[:,1].unsqueeze(1)

A_z = grad(A,z,torch.ones((batch_size, 1),requires_grad=True).cuda(), create_graph=True)[0]
A_y = grad(A,y,torch.ones((batch_size, 1),requires_grad=True).cuda(), create_graph=True)[0]
A_x = grad(A,x,torch.ones((batch_size, 1),requires_grad=True).cuda(), create_graph=True)[0]
</code></pre>
<p>I get the following error on the <code>A_z</code> line:<br>
<code>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</code><br>
This confuses me since this is the only derivative computed so far so it seems like <code>A_z</code> isn’t being added to the graph?</p>

albanD: <p>Hi,</p>
<p>From what I can see, x, y and z are not used to compute the output right? Hence the error that you’re seeing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
You can simply ask for the gradients wrt <code>ipt</code> and then slice that gradient to get the subset you’re interested in <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

algol: <p>So I was wondering about this so I tried:<br>
<code>A_z = grad(out[:,0].unsqueeze(1),ipt[:,2].unsqueeze(1),torch.ones((batch_size, 1),requires_grad=True).cuda(), create_graph=True)[0]</code><br>
and got the same error. Is it important that the full input and output tensors are used and not any slices?</p>

algol: <p>I have a follow up question: what’s the most appropriate way to calculate second derivatives? It sounds like the preferred method is to calculate derivatives with respect to all inputs, and this gets slow when I have to calculate higher order derivatives. Is there a way to calculate derivatives with respect to just one input if I don’t need all of the higher order derivatives?</p>

albanD: <p>When you do <code>ipt[:,2].unsqueeze(1)</code> it creates a new Tensor that is a slice of <code>ipt</code>. Bu that new Tensor is not used to compute the loss and that’s why you see this error.</p>
<blockquote>
<p>Is there a way to calculate derivatives with respect to just one input if I don’t need all of the higher order derivatives?</p>
</blockquote>
<p>You can only compute derivatives wrt to a Tensor. So if all your inputs are in a single Tensor, you have to compute the derivative for the whole thing. But this kind of batch computations are usually not much more expensive than doing a single element.</p>
Is there a way to release gpu memory held by cuda tensors/variables?

alexbellgrande: <p>I’m trying to run a pytorch script in ipython so that I can monkey around with the outputs of my computation graph. The problem that my gpu runs out of memory after I run the same script a few times, forcing me to exit and reenter ipython. In nvidia smi you can see the gpu memory decrease by the same amount after every run until it eventually reaches 0 and throws an error. Pytorch seems to be allocating new gpu memory every time the script is executed instead of reusing the memory allocated in previous runs. Is there a way to forcibly release all gpu memory held by pytorch in between script executions so that I don’t have to constantly exit and reenter ipython? Thanks!</p>
<p>I’ve tried %reset and unfortunately this doesn’t do the trick (not unexpectedly).</p>

smth: <p>We haven’t exposed this functionality of flushing memory at the python level yet.</p>

jef: <p>I want it too. Hope this task is running.</p>

SimonW: <p>The method is available now <a href="http://pytorch.org/docs/master/cuda.html#torch.cuda.empty_cache">http://pytorch.org/docs/master/cuda.html#torch.cuda.empty_cache</a>. Note that you need to delete all references to the variable you want to free before calling this. This requirement is due to safety concern. <a class="mention" href="/u/jef">@jef</a></p>

jmaronas: <p>Well. At least in my pytorch version is not implemented</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
a=torch.cuda.FloatTensor(10,10)<br>
del a<br>
torch.cuda.empty_cache()<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
AttributeError: ‘module’ object has no attribute ‘empty_cache’</p>
</blockquote>
</blockquote>
</blockquote>
<p>I have just installed the last pytorch version from the webpage. Python2.7 cuda8 and pip installation.</p>
<p>I think this should be quickly fixed as I am getting out of memory when running on a 8GB nvidia gpu. I have two networks sharing some parameters. The problem is I get out of memory after some batches… this does not make a lot of sense however as each batch I do same operation. Forward for network 1, forward for network 2 and backward. Each data from the data iterator is copied into a Variable so I can use the nn.Module, but I think this should not allocate more memory. After all the batches nvidia-smi does not show more allocation.</p>
<p>Thanks.</p>

SimonW: <p>Pip install is only the 0.2 version. If you look at the doc, the method is only available at &gt;= 0.3. You will need to either build from source or wait for 0.3 release.</p>

KeithYin: <p>this function is very useful</p>

lbdalmendrayCaseguar: <p>Hi ,<br>
I read your message. I would like to know if the exposed functionality of flushing memory is for C++ Libtorch developers . I am using Libtorch C++ and I cannot find a way to release ALL the CUDA GPU Memory used by a torch::nn::Module . Here I explained better an example : <a href="https://discuss.pytorch.org/t/release-all-cuda-gpu-memory-using-libtorch-c/108303">https://discuss.pytorch.org/t/release-all-cuda-gpu-memory-using-libtorch-c/108303</a></p>
<p>Thanks in advance .</p>
Convert native CUDA data containers to `torch::Tensor`

justanhduc: <p>Hi. Is there any official way to convert directly an array of native CUDA containers like <code>float3</code> to a <code>torch::Tensor</code>? If not, is there any efficient way to to it? Thanks!</p>

ptrblck: <p>You should be able to create a <code>tensor</code> via <code>torch::from_blob</code> with <code>kCUDA</code> as the optional argument.</p>