akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Overfitting to the development data-set

Capo_Mestre: <p>After training my network, I have got similar cost-errors for training and development data-sets, and twice as large cost-error over the test set.</p>
<p>I used the cost-error over the development data-set to tune the hyperparameters.</p>
<p>This looks like the overfitting of the training and the development data-sets.</p>
<p>How can I improve my network, to get the cost-error over the test data-set down to the level of the cost-error over the train/dev data-sets?</p>

ayalaa2: <p>Interesting situation. Essentially, your development set must be a poor generalization of the distribution you’re attempting to model.</p>
<p>For example, suppose I was training a model to classify dogs/cats/birds. It’s possible that both my train-set and dev-set both lack samples of bird of images whereas my test-set is primarily bird images. As a result, we would get a similar behavior as to what you described.</p>
<p>I would suggest revisiting your data splits and consider if the splits made are reasonable. Is there something interesting about test not shown in dev or train? Is there any kind of large class imbalance that could have seeped through? You just really need to examine your data to figure this out.</p>

Capo_Mestre: <p>Thanks a lot for your reply</p>
'Conv3d' object has no attribute "'detach"

antonia: <p>while end_slice &lt; ct_array.shape[0]:</p>
<pre><code>        ct_tensor = torch.FloatTensor(ct_array[start_slice: end_slice + 1]).cuda()

        ct_tensor = ct_tensor.unsqueeze(dim=0).unsqueeze(dim=0)

        outputs = net(ct_tensor)

        count[start_slice: end_slice + 1] += 1

        #probability_map[start_slice: end_slice + 1] += np.squeeze(outputs.cpu().detach().numpy())


        del outputs</code></pre>

antonia: <p>Anyone got a tip for this? This is the code and error it gives:</p>

ptrblck: <p>Based on the error message you are trying to call <code>detach</code> on the module, not the output tensor, so could you check its <code>type</code> before calling this operation?</p>
some problem by use torch::stft with C++ API,but output Shape is different to Python API

FrankyCao: <p>macos, torch version:1.7.1</p>
<p>Python API:</p>
<p>import torch<br>
x = torch.rand([2, 441344], dtype=float);<br>
y = torch.stft(x, 4096)<br>
print(y.shape)<br>
output:torch.Size([2, 2049, 432, 2])</p>
<p>C++ API<br>
auto wav = torch::rand({2, 441344});<br>
auto stft = torch::stft(wav, 4096);<br>
std::cout &lt;&lt; “output sizes:” &lt;&lt; stft.sizes() &lt;&lt; std::endl;<br>
output sizes::[2, 2049, 428, 2]</p>
<p>Why?</p>

FrankyCao: <p>I find that stft note:" librosa’s center and pad_mode arguments are currently only implemented in python because it uses torch.nn.functional.pad which is python-only."<br>
but, C++ API has torch.nn.functional.pad also</p>

ThomasLepercq: <p>Hello, I have had the same issue. I know I am late but it might be useful for someone else :<br>
The C++ version of STFT does not have padding implemented by default whereas the Python version does, in “reflect” mode particularly.</p>
<p>In your case, n_fft = 4096 so the tensor should be padded by 2048 on each side before the STFT.<br>
The “reflect” padding on a tensor like {0,1,2,3,4,5,6,7,8,9} with a padding of 3 on each sides would return : {3,2,1,0,1,2,3,4,5,6,7,8,9,8,7,6}. In C++, a code to do that would be something like :</p>
<pre><code class="lang-auto">torch::Tensor input = {0,1,2,3,4,5,6,7,8,9}
int length = input.sizes()[0]
int pad = 3
torch::Tensor left_pad = input.slice(0,1,pad+1).flip(0)
torch::Tensor right_pad = input.slice(0,length-pad-1,length-1).flip(0)
torch::Tensor output = torch::cat({left_pad,input,right_pad},0)
</code></pre>
<p>You may also need to use "view"s on your Tensors when working with multiple dimensions.<br>
Hope this helps.</p>
Backward memory leak

AreTor: <p>I am currently using the <code>torch.Tensor.backward(retain_graph=True, create_graph=True)</code> method and I have a memory leak.</p>
<p>According to the <a href="https://pytorch.org/docs/1.8.1/autograd.html?highlight=backward#torch.autograd.backward" rel="noopener nofollow ugc">docs</a>, this can be expected, and in order to avoid this, the <code>.grad</code> field of the parameters should be reset. I am performing this with the following snippet:</p>
<pre><code>for param in model.parameters():
    param.grad = None
</code></pre>
<p>however, I am still getting the memory leak. Am I missing something to be reset?</p>
<p>Note that I cannot use the alternative <code>torch.autograd.grad()</code> function, because I have registered some backward hooks, that would not otherwise be called.</p>
Cowardly refusing to serialize non-leaf

silvester: <pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import time
import copy

def producer(queue, event):
    while True:
        for _ in range(10):
            a = torch.ones(2,2).float().cuda()
            idx = torch.ByteTensor([[0, 0], [0, 1]]).cuda()
            a.requires_grad = True
            b = 3 * a

            queue.put(b)
            queue.put(idx)
            event.wait()
            event.clear()
        time.sleep(1000)
        return

def consumer(queue, event):
    while True:
        for _ in range(10):
            idx = queue.get()
            temp = queue.get()
            print("CONSUMER ", idx)
            print("CONSUMER ", temp)
            del idx
            del temp
            event.set()


if __name__ == '__main__':
    mp.set_start_method('spawn')

    queue = mp.Queue()
    queue2 = mp.Queue()

    event = mp.Event()
    p = mp.Process(target=producer, args=(queue, event))
    c = mp.Process(target=consumer, args=(queue, event))
    p.start()
    c.start()

    time.sleep(10)

    p.join()
    c.join()
</code></pre>
<p>Hello, What’s is the problem with pushing the non-leaf variable <code>b</code> to the queue and retrieve it? Any work-around is welcome.</p>
<p>I get the error below:<br>
<em><code>RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</code></em></p>
Specific value in a row and a column from that row and the rest is zero

Sssrnty: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f1f7f984daee070cd8ae496f1a240056b2ccb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1f1f7f984daee070cd8ae496f1a240056b2ccb9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1f1f7f984daee070cd8ae496f1a240056b2ccb9_2_634x500.png" alt="image" data-base62-sha1="weNQt9PaaEfv81Ub9nOK2bVmiTv" width="634" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1f1f7f984daee070cd8ae496f1a240056b2ccb9_2_634x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f1f7f984daee070cd8ae496f1a240056b2ccb9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f1f7f984daee070cd8ae496f1a240056b2ccb9.png 2x" data-dominant-color="F3F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">722×569 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
is there a way to this by not using loops?</p>

eqy: <p>There might be a cleaner way to do it, but a first attempt:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; y = torch.tensor([[2], [7], [1], [2]])
&gt;&gt;&gt; y_idx = torch.zeros(y.shape[0], 10).float()
&gt;&gt;&gt; y_idx.index_put_((torch.arange(4), y.squeeze()), torch.ones(4))
tensor([[0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>

Sssrnty: <p>thank you for this, going to read about index_put_, there should be a lot of solutions but i want to avoid using loops as much as possible.</p>
Why does gradcheck fail for floats?

marcman411: <p>I have implemented by own convolution layer as a learning experience in extending PyTorch. Looking at the printouts for small tensors, my outputs and gradients appear to match those of the native PyTorch convolutions.</p>
<p>When I go to much larger tensors, I have to rely on an sum of absolute differences metric between my implementation’s outputs/grads and the builtin’s ones. I’ve found that while my forward pass outputs and my gradients w.r.t the inputs match the native implementation every time, my gradients w.r.t the weights and biases are often slightly off. The per-element error is fairly small for small tensors (&lt;1e-5), but as the tensor size increases to something like a (1, 512, 128, 256) tensor,  I see per-element avg. errors close to 1e-2 or higher.</p>
<p>When I defined all the input and grad output (i.e. the argument to <code>backward</code>) elements to be whole numbers, the error disappears. Hence I thought it was a precision issue.</p>
<p>Sure enough, my implementation passes <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/gradcheck.py#L128" rel="nofollow noopener"><code>gradcheck</code></a> when all values are whole number <code>float</code>s or <code>double</code>s. It fails (which <code>gradcheck</code> warns will happen), when using decimal <code>float</code>s. Why does this happen?</p>
<p>Weirder still is that the errors when using <code>float</code>s are often the same set values but move around the returned tensor depending on the run (e.g. all the SAD errors are 0.002 or 0.0039 with one set of convolution parameters, or 0.0625, 0.125 and 0.25 with another). I’d thought I was accidentally accessing uninitialized memory in my cuBLAS gemm call, but switching to <code>double</code>s largely resolves the issue.</p>
<p>Am I looking at a precision issue? If so, why does it only affect weight and bias gradients and not input gradients?</p>

SimonW: <p>Yeah, 32 bit floating point usually doesn’t have enough precision to pass numerical gradient checks.</p>

BruceDai003: <p>But I’m checking the gradients for float32 datatypes, so is there like a minimal code changes to the gradcheck and gradgradcheck function API to let it be a proper test API for float32?</p>
Access image pixels from [Nx2] float tensor

soulslicer: <p>I have an image of size [500, 500]</p>
<p>I also have a tensor that was generated by a method of size [N X 2]. I need to access in a differentiable way, the outputs of the indexing of the image. So an element in the [NX2] contains a pixel position as a float, and I want to access that index in the image and get the value. Assuming that pixel is within the image.</p>
<p>Is this actually possible?</p>
Loss function in a Siamese Network

beto16: <p>I’m trying to send two images through a siamese network. It looks fine, but I don’t understand very well the loss function <strong>CrossEntropyLoss</strong>. I’m trying to calculate the similarity between two images, the image could be classified between 5 different classes.<br>
The model returns the results from one image concatenated with the second image</p>
<pre><code class="lang-auto">    def forward(self, input1, input2):
        o1 = self.model(input1)
        o2 = self.model(input2)
        final = torch.cat([o1, o2], 1)

        return final
</code></pre>
<p>When I train the model I pass two inputs to the network * (lateral, medial) *. The model returns the results of one input concatenated with the result of the second input. This result is a vector that contains the five predictions of one image plus the five predictions of the second image. Since I have to classify the images between five classes the label * (target) * will have values between 0 and 4. I don’t understand how the loss functions works because I’ve predictions that go from 0 to 9 but the labels go from 0 to 4. My question is if this will be a problem with the performance of the model and if it’s how could I fix it.</p>
<pre><code class="lang-auto">for lateral, medial, target, file, name in dataloader[phase]:
    lateral, medial, target = lateral.to(device), medial.to(device), target.to(device) 
    optimizer.zero_grad()
    with torch.set_grad_enabled(phase == 'train'):
        output= model(lateral, medial)
        loss = criterion(output, target)
        if phase == "train":
            loss.backward()
            optimizer.step()
</code></pre>
Getting TypeError: list indices must be integers or slices, not tuple

bing: <p>Hi,<br>
I am trying to use a <code>Subset random Sampler</code> in my data-loading flow. But I am getting<br>
<code> "TypeError: list indices must be integers or slices, not tuple"</code>.<br>
Without using <code>Subset Random Sampler</code> my code is working fine.<br>
I am using a custom dataset class and below is the <code>__getitem__</code> function, it returns a tuple, but it is a standard way taken from pytorch tutorials.<br>
Where I am doing wrong as I am not violating the default return of the function?<br>
How to fix it?</p>
<pre><code>def __getitem__(self, index):
    """Generate one sample of data."""
    ID = self.list_IDs[index]
    image = Image.open(os.path.join(self.dir, ID))
    y = self.labels[ID]
    if self.train_transform:
        image = self.train_transforms(image)
    elif self.valid_transform:
        image = self.valid_transforms(image)
    else:
        image = self.test_transforms(image)
    img = np.array(image)
    return img, y
</code></pre>
<p>My data loading pipeline -</p>
<pre><code class="lang-auto">training_set = Dataset(partition['train_set'], labels, root_dir='path', train_transform=True) # noqa
train_sampler = SubsetRandomSampler(training_set)
train_loader = torch.utils.data.DataLoader(training_set, shuffle=False, sampler=train_sampler,pin_memory=True, num_workers=0, batch_size=batch_size,) # noqa

</code></pre>

eqy: <p>It looks like in the documentation for <code>SubsetRandomSampler</code> the constructor expects indices that can be sampled from rather than the actual dataset itself:<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#SubsetRandomSampler" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#SubsetRandomSampler</a></p>

bing: <p>So I do the modification like below, it should work but it is not working still</p>
<pre><code class="lang-auto">train_indices= training_set[:500]
train_sampler = SubsetRandomSampler(train_indices)
</code></pre>

eqy: <p>Right, that is is still giving you data elements rather than indices.<br>
It appears that the expected use is something like<br>
<code>train_sampler = SubsetRandomSampler([i for i in range(500)])</code></p>
ImportError: DLL load failed when switching from CUDA 10.2 to CUDA 11.1

BenAAndrew: <p>Hi everyone,</p>
<p>When compiling my python project (with pyinstaller) I run into the following error:</p>
<pre><code class="lang-auto">ImportError: DLL load failed: The specified module could not be found.
</code></pre>
<p>What’s interesting about this is that it only occurs when I change from <code>torch==1.9.0+cu102</code> to <code>torch==1.9.0+cu111</code> meaning this missing DLL problem is unique to changing from CUDA 10.2 to CUDA 11.1.</p>
<p>Please note that the project works fine if you run it directly with python, so this issue is to do with the correct DLL’s being detected when the app is compiled with pyinstaller.</p>
<p>Does anyone have any insight into what changes between these CUDA versions may be causing this issue?</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=10" title=":smile:" class="emoji" alt=":smile:"></p>
Using two optimizers to train a model with two outputs

slapeeh: <p>Hi,</p>
<p>I have been browsing the forum and could not find answer to my question.<br>
I have a model that has two outputs. At first, I combined the two losses and used one optimizer to calculate the gradients.</p>
<p>Now, I want to try using two separate optimizers. However, the two branches of the model (that result in two outputs) have overlapping parameters. How can I calculate and update the gradients of the model without updating the same parameters twice?</p>

tom: <p>So what do you want to do for the parameters that appears in both?</p>
<p>If you want to update them twice, you don’t really need to do anything except keeping the zero_grad-backward-step bits of each model disentangled. For the first, you probabably want retain_graph or so.<br>
An alternative could be to update each part alternatingly on every second input batch.</p>
<p>That said, I’d probably try with your first approach. From my experience, what is important in such a case is some equilibration, i.e. you likely want the losses to be of the same order or in some relation to each other.</p>
<p>Best regards</p>
<p>Thomas</p>

slapeeh: <p>I just thought of one possible solution (I don’t know whether it is feasible) where I would do the two backward() calls first which would accumulate the computed gradients on the overlapping parameters right?<br>
And then I could call on of the optimizer’s step() function, which would apply the changes on every parameter on that branch (inlcuding the overlapping parameters) and then call zero_grad() so that the other optimizer’s step would not update the overlapping parameters again because they are 0 thanks to the previous zero_grad() call.</p>
<p>Is this a possibility or not in you opinion?</p>

tom: <p>If you want to do it this way, I would just leave out the duplicate parameters in one of the step functions.</p>
<p>But in general, it makes me wonder what you intend to gain over summing the loss. If you do the backprop and step separately and keep the grads for both this means you get the adaptivity of the optimizer (if there is any, e.g. because you use Adam or so) tailored to either loss.<br>
But even then: In the end, you optimize some loss function and you likely implicitly optimize a combination of the two losses, so I wonder what there is to gain to make this implicit rather than explicit.</p>
<p>All that said, you can run the experiments, if the result works better for you, reality beats my limited intuition.</p>
<p>Best regards</p>
<p>Thomas</p>
Inplace operation error: sigmoid and relu

Magjo: <p>Hi,<br>
I have difficulties to understand why this code does not throw error</p>
<pre><code class="lang-auto">model = torch.nn.Linear(512,2)

opt = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.rand((32,512))
outputs = torch.rand((32,2))

opt.zero_grad()
pred = model(inputs)
pred[:,1] = torch.sigmoid(pred[:,1])
loss = torch.nn.functional.mse_loss(pred, outputs)
loss.backward()
opt.step()
</code></pre>
<p>But if I replace torch.sigmoid by torch.nn.functional.relu it throws RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.<br>
What is the hidden reason explaining the difference between the two activation functions ?</p>
<p>Thank you very much</p>

KFrank: <p>Hi Magjo!</p>
<aside class="quote no-group" data-username="Magjo" data-post="1" data-topic="128259" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/9dc877/48.png" class="avatar"> Magjo:</div>
<blockquote>
<p>But if I replace torch.sigmoid by torch.nn.functional.relu it throws RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.<br>
What is the hidden reason explaining the difference between the two activation functions ?</p>
</blockquote>
</aside>
<p>I’m <em>guessing …</em>   (But it’s the weekend, so let me guess away!)</p>
<p>At issue is not only whether you are using an inplace operation,<br>
but also whether the tensor modified by the inplace operation is<br>
used <em>directly</em> in the <code>backward()</code> computation.</p>
<p>The derivative of <code>relu()</code> depends on the sign of its argument<br>
(your <code>pred</code>).  So <code>backward()</code> might do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">if  pred &gt; 0.0:  deriv = 1.0
else:            deriv = 0.0
</code></pre>
<p>and complain if you have modified <code>pred</code> inplace.</p>
<p>But the derivative of <code>sigmoid()</code> can be expressed simply in terms<br>
of <code>sigmoid()</code> itself, so I imagine that <code>sigmoid()</code>'s <code>forward()</code><br>
saves the result of <code>sigmoid()</code> in the context (<code>ctx</code>) that it will use in<br>
its <code>backward()</code>, and having this information obviates the need to use<br>
the original argument to <code>sigmoid()</code> in the <code>backward()</code> computation<br>
(and it’s smart enough not to complain if it doesn’t need it).</p>
<p>Best.</p>
<p>K. Frank</p>

Magjo: <p>Thank you for the answer.</p>
Torch.jit.trace_module with strict = False hardcoding input shape

p1x31: <p>I’m tracing the mapping network between two generators with torch.jit.trace_module<br>
Code example:</p>
<pre><code class="lang-auto">mapping_input =  torch.rand(1, 64, 504, 378).to("cuda"), inst_data   
inputs = {'forward' : encoder_forward_input, 'forward_encoder' : encoder_forward_input, 'forward_decoder' : decoder_forward_input}
inputs_mapping = {'forward' : mapping_input, 'inference_forward' : mapping_input}

traced_model_cuda = torch.jit.trace_module(self.mapping_net, inputs_mapping, strict=False)
torch.jit.save(traced_model_cuda, "scratch_mapping_net_traced_model_forward_strict_off.pt") 
self.mapping_net = torch.jit.load("scratch_mapping_net_traced_model_forward_strict_off.pt")
self.mapping_net.cuda(self.opt.gpu_ids[0])

label_feat_map=self.mapping_net.inference_forward(label_feat.detach(), inst_data)
</code></pre>
<p>I’ve encountered this error when running traced graph and weights on a different image of the same size.</p>
<pre><code class="lang-auto">torch.Size([1, 1, 2016, 1512])
Skip HR_input_2016.png due to an error:
The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/models/NonLocal_feature_mapping_model.py", line 231, in inference_forward
    _120 = torch.slice(x_unfold, 0, 0, 9223372036854775807, 1)
    _121 = torch.slice(_120, 1, 0, 9223372036854775807, 1)
    _122 = torch.view(composed_unfold, [32768, 154])
           ~~~~~~~~~~ &lt;--- HERE
    mask_index5 = torch.to(mask_index, dtype=4, layout=0, device=torch.device("cuda:0"), pin_memory=None, non_blocking=False, copy=False, memory_format=None)
    _123 = annotate(List[Optional[Tensor]], [None, None, mask_index5])

Traceback of TorchScript, original code (most recent call last):
.../Global/models/networks.py(774): inference_forward
.../Global/models/NonLocal_feature_mapping_model.py(197): inference_forward
.../anaconda3/lib/python3.8/site-packages/torch/jit/_trace.py(934): trace_module
.../Global/models/mapping_model.py(415): inference
test.py(190): &lt;module&gt;
RuntimeError: shape '[32768, 154]' is invalid for input of size 1474560

</code></pre>
<p>inference_forward method:<br>
<a href="https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/blob/master/Global/models/NonLocal_feature_mapping_model.py#L187" rel="noopener nofollow ugc">link</a></p>
<p>Nevertheless traced module running fine for the image it was traced with.</p>

p1x31: <p>inst_data is a mask image with one channel passed as a tensor with size</p>
<pre><code class="lang-auto">torch.Size([1, 1, 2016, 1512])
</code></pre>
<p>When I try<br>
<code> mapping_input =  torch.rand(1, 64, 504, 378).to("cuda"), torch.rand(1, 1, 2016, 1512).to("cuda")</code><br>
I’ve got this error instead:<br>
<code>cannot perform reduction function max on tensor with no elements because the operation does not have an identity</code></p>
<p>Is there any way of tracing this?</p>
Torchscript not utilizing OpenMP threads (intra-op)

Bohaterowicz: <p>Hi, we have recently started working on implementing model inference via torchscript in libtorch (C++), we use methods described in the documentation, where we generate a model in python using <code>troch.jit.script()</code>, save it to a file, load in c++ using <code>torch::jit::load()</code> and use it as a <code>torch::jit::Module</code>. One thing we noticed is that when we run our models in C++ this way, we do not get any parallelism from OpenMP (intra-op), or inter-op parallelism (when specifying a <code>torch.jit.fork()</code>, method in our forward pass in the python model). We have verified that we have OpenMP working, and we have set the number of threads using <code>torch::set_num_threads()</code>, and the corresponding get() function returns correct number. When we run our models with 8 intra-op threads we expect around 800% CPU usage, yet we get only 100%. We looked into <a href="https://github.com/pytorch/pytorch/issues/26948" rel="noopener nofollow ugc">this</a> github issue, and even when we run the code that <code>wizardk</code> posted (where he says he gets 2400% CPU usage) we only get 100% (when we specify 1 working thread). We have verified that we get expected CPU usage (800% when specifying 8 intra-op threads) if we recreate the model using a sequential with a bunch of linear modules, so the CPU usage problem only occurs when running torchscript models. Do you have any suggestions, or thoughts why this may be the case?</p>
Seq2Seq RNN for Numeric Data

drbeethoven: <p>Hello,</p>
<p>Is there any implementation online for a seq2seq RNN that takes numeric data and outputs classification between a fixed number of classes? I cannot find it anywhere. If it makes it easier, the numeric data is limited between -3 and 3, and can have limited accuracy (so I could make a finite lookup table if embedding is needed, but im not sure I need to use embedding if it is numeric data anyways).</p>
<p>Thank you very much for all the help.</p>
Update a weight after 15 epoch

Kuldeep_Rana: <p>Dear Sir/Madam,<br>
I am a beginner at the pytorch. I have occurred a problem. I want to update the weight after 15 epoch and keep the same weight till the end of training. How can I do this? Thank you very much.</p>

Samuel_Bachorik: <p>Show code and tell what you mean</p>
<p>EDIT: Or simply make if statement and after your 15th epoch dont call<br>
<code>optimizer.zero_grad()     loss.backward()     optimizer.step()</code></p>
Is there some document about adding a custom JIT backend

edimetia3d: <p>Hi,</p>
<p>We have developed a NN JIT compiler, which is capable to compile a whole deep learning model. And now we want to add some TorchScript style JIT support transparently, like “pytorch/glow/torch_glow” did, so that user could just do a <code>import my_jit</code>, and then all jit stuff will be forward to our backend, pytorch will only act as an frontend that output a TorchScript Graph.</p>
<p>When reading torch_glow’s code, I found it hard to understand how should it work, so I was wondering if there was some document/tutorial about how to impl a custom JIT backend?</p>

tom: <p>You could</p>
<ul>
<li>add an op with a subgraph attribute that would run the subgraph in your system,</li>
<li>register a custom pass moving ops you want to run yourself into the subgraph.</li>
</ul>
<p>This way you can even fall back to running ops you might not support in PyTorch.</p>
<p>Best regards</p>
<p>Thomas</p>

edimetia3d: <p>Hi, Thomas</p>
<p>Thanks,</p>
<p>I’ve noticed this solution, there is also a tutorial <a href="https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch" rel="noopener nofollow ugc">jott - Writing a Toy Backend Compiler for PyTorch</a> talks about it.</p>
<p>So it seems we need to create a dummy_op and a custom post pass that fuse everything in the TorchGraph into this dummy_op , and then, we could do the jit stuff during the dispatch on dummy_op? Is this right?</p>
<p>I also notice that there is a <code>torch::jit::registerPrePass</code>, is it fine to do the fusing at there too? For we also have an IR optimize system, and we might want  to get the raw graph IR. Could this fusing in PrePass<br>
cause any issue?</p>
<p>Another question now is that glow also registerd a new backend <code>torch::jit::backend&lt;TorchGlowBackend&gt;("glow")</code> and a preprocess of this backend <code>torch::jit::backend_preprocess_register("glow", preprocess);</code>. And it seems this backend isn’t used during the transparent JIT, so when should I also implement such a backend, or what is the usecase of the  <code>torch::jit::backend</code>?</p>
<p>Sincerely</p>

tom: <p>This is approximately what I had in mind. (And I do have a branch somewhere that I forget about that tries to make more of the JIT accessible form Python… Sigh.)</p>
<p>There are some passes that you would want, but it probably is OK to register a pre pass, too, if you prefer the IR at that level.</p>
<p>Now that you mention it, using backend probably is the right way to go. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It is a relatively new part of the JIT (from 2020, in particular newer than the tutorial you linked).</p>
<p>Best regards</p>
<p>Thomas</p>

edimetia3d: <p>OK,  I see.</p>
<p>Many thanks.</p>
<p>Sincerely</p>
Does selecting tensor using a mask produces same effect as ignore_index in Cross Entropy loss?

chinmay5: <p>I have a model wherein I need to compute the cross-entropy loss for only a certain index. The <code>node_mask</code> allows for the selection of those indices. There is also <code>ignore_index</code> field in the Cross-Entropy loss class. Since I have the gt for only these masked labels, I thought maybe it would make sense in terms of the overall space efficiency if we just do the masked selection beforehand.</p>
<pre><code class="lang-auto">x = self.linear1(self.node_features)  # W, 200
x = self.linear2(x)  # W, 1024
x = F.relu(x)
x = x[self.node_mask] # W, 14
</code></pre>
<p>However, I am still not sure if this is going to lead to the same effect or I should stick to <code>ignore_index</code> term and expand my Ground truth to a 1024 dimensional vector with -100 at the non-mask indices.</p>
<p>Any help would be highly appreciated.</p>

tom: <p>I think this is a very cool question!</p>
<p>CrossEntropyLoss consists of two things, a LogSoftmax followed by a NLLLoss (Negative Log Likelihood).<br>
Accordingly, there are two parts:</p>
<ul>
<li>
<p>If you select the the rows (you could also use <code>x[:, class_mask]</code> btw if you have fixed classes), the implicit (log) softmax will be only over the selected classes, i.e. they wil sum (or logsumexp) to 1 (0). If you don’t select them, the other items in the tensor will lower the class (log) softmax of the selected ones (unless they’re all -infinity).</p>
<p>One comparison to draw might be to the training of word2vec etc. where the negative sampling is closer in spirit to selecting the classes.</p>
<p>If you wanted as-if behaviour for the tensor without using a mask, you could make a tensor where the not-to-be-considered items are set to -infinity (inplace by assigning or in a new tensor using where, depending on your needs).</p>
</li>
<li>
<p>In the NLL loss optimization only the likelihood of the data, i.e. the probability the model assigns to the true class, is pushed up (by minimizing the negative log likelihood). This means that unless the true class is sometimes one of the indices you want to ignore, it will be all the same regardless what you do. The key difference is that the labeling mus be 0…num_classes-1 so depending on what your target data is initially, you may have to relabel for one solution or the other.</p>
</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
Optimizer wont change x and loss does not change

Samuel_Bachorik: <p>Hi, thanks for help with rotation matrix, now when i got it, iam trying to rotate matrix and then find angle with Pytorch. This is my code - iam not sure if loss function is good but my angle_pred and loss does not change after 100 loops.</p>
<p>Do you have idea where problem is ?</p>
<pre><code class="lang-auto">import torch

x = torch.rand((256,2))
angle = torch.tensor(0.08)
rot = torch.tensor([[torch.cos(angle), -torch.sin(angle)], [torch.sin(angle), torch.cos(angle)]])

x_rot = torch.matmul(x, rot)

angle_pred = torch.nn.Parameter(torch.zeros(1), requires_grad=True)

optimizer = torch.optim.Adam([angle_pred], lr = 0.1)


for step in range(100):
    rot_back = torch.tensor([[torch.cos(-angle_pred), -torch.sin(-angle_pred)], [torch.sin(-angle_pred), torch.cos(-angle_pred)]], requires_grad=True)
    x_pred = torch.matmul(x_rot, rot_back)

    loss = ((x_pred-x)**2).mean()

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(loss)
</code></pre>

tom: <p>There used to be a warning about this, but torch.tensor is bad here as it’ll treat the inputs as numbers rather than connecting them through autograd (and maybe needing the requires_grad=True should have given you a hint <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ). <strong>Edit:</strong> Let me emphasize, that it’s a common enough thing to get this wrong, don’t be discouraged –  I know at least 20 people who think that you can never know enough about autograd.  <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
<p>This (and not because it was aesthetically pleasing, it isn’t) was the reason I used the nested stack in the other thread. If you plug that back in, it’ll work a lot better.</p>
<p>Best regards</p>
<p>Thomas</p>

Samuel_Bachorik: <p>Again, thank you a lot! Respect !</p>
Error in "summary(model, input_size = (100,64,64), device = 'cpu')"

PytorchLearning: <p>Hi all,</p>
<p>I have the following model:</p>
<p>class ModelExample(torch.nn.Module):</p>
<pre><code>def __init__(self):
    super(ModelExample, self).__init__()
    
    self.linear = torch.nn.Linear(100, 1024*4*4)
    
    self.conv1 = nn.Sequential(
                               nn.ConvTranspose2d(in_channels=1024, out_channels=128, kernel_size=4, stride=2, padding=1, bias=False)
                               ,nn.BatchNorm2d(128)
                               ,nn.ReLU(inplace=True)
                              )
                              
    self.conv2 = nn.Sequential(
                               nn.ConvTranspose2d(in_channels=128, out_channels=3, kernel_size=4, stride=2, padding=1, bias=False)
                              )
    self.out = torch.nn.Tanh()

def forward(self, x):
    
    x = self.linear(x)
    x = x.view(x.shape[0], 1024, 4, 4)
    
    x = self.conv1(x)
    x = self.conv2(x)

    x = self.out(x)

    return x
</code></pre>
<p>I tried to make the following code work:</p>
<p>model = ModelExample()<br>
summary(model, input_size = (100,64,64), device = ‘cpu’)</p>
<p>But it gives me the following error</p>
<p>RuntimeError: size mismatch, m1: [12800 x 64], m2: [100 x 16384] at …\aten\src\TH/generic/THTensorMath.cpp:41</p>
<p>Can someone please tell me where the error is and how to solve it?<br>
Thanks for your help.</p>

Vrushank98: <p>The input that you have passed to summary is a 4D tensor of size ([batch_size, 100, 64, 64]), so before passing that directly to the linear layer, you have to reshape to a 2D Tensor. just add:</p>
<p><code> x = x.view(-1, 100)	//100 = input_channels to the Linear layer</code></p>
<p>before x = self.linear(x) in the forward() and you are good to go.</p>

PytorchLearning: <p><a class="mention" href="/u/vrushank98">@Vrushank98</a></p>
<p>Thanks a lot for your answer.</p>
Pass a generator to torch.nn.init functions

antoche: <p>Hi,</p>
<p>RNG functions like <code>torch.normal</code> and <code>torch.Tensor.normal_</code> allow the caller to pass a <code>generator</code> object. But other functions, like <code>torch.Tensor.uniform_</code>, and all the methods in <code>torch.nn.init</code>, (as well as the modules’ <code>reset_parameters</code>) don’t.</p>
<p>Being able to pass a generator object is needed in cases where determinism and consistency are required. Even though calling the global <code>torch.manual_seed</code> can sometimes be used as a workaround, it doesn’t shield the user from issues such as concurrently initializing independent networks, or a third-party calling that function outside of the user’s control.</p>
<p>What’s the appetite for adding an <code>generator</code> argument to all those functions, the same way <code>torch.Tensor.normal_</code> already does? What other solutions would there be to avoid relying on a global variable?</p>
<p>Thanks,<br>
A.</p>

tom: <p>I think this would be better in an issue on github, but prima facie it makes sense to me.<br>
Note that the modules have the broader issue of having not-so-good-default-init choices. (There is an issue on it and a stale PR or so.)</p>
<p>Best regards</p>
<p>Thomas</p>

antoche: <p>Ok, happy to log this on GitHub. Thanks for the response.</p>
Rotation matrix

Samuel_Bachorik: <p>Hi, lets think matrix x = Torch.rand((256,2)). How can I rotate this matrix in PyTorch ? For example by 5 degrees. How to create rotation matrix for this ?</p>
<p>Thank you</p>

tom: <p>So you have 256 points in 2d?</p>
<pre><code class="lang-python">import math
x = torch.rand(256, 2)
phi = torch.tensor(5 * math.pi / 180)
s = torch.sin(phi)
c = torch.cos(phi)
rot = torch.stack([torch.stack([c, -s]),
                   torch.stack([s, c])])

x_rot = x @ rot.t()   # same as x_rot = (rot @ x.t()).t() due to rot in O(n) (SO(n) even)
</code></pre>
<p>or somesuch.</p>
<p>Best regards</p>
<p>Thomas</p>

Samuel_Bachorik: <p>Thank you a lot, this helped me!</p>

Samuel_Bachorik: <p>If I can have one more question what is the difference betwen-</p>
<pre><code class="lang-auto">x_rot = x @ rot.t()
</code></pre>
<p>and</p>
<pre><code class="lang-auto">x_rot = torch.matmul(x,rot)
</code></pre>

tom: <p><code>@</code> and <code>torch.matmul</code> are identical, but if you leave out the <code>.t()</code>, you’ll change the direction of rotation (because <code>.t()</code> is the inverse rotation), I thought that the version with <code>.t()</code> might be the more canonical (because the convention would seem to be to multiply from the left but that doesn’t work with the batch dimension coming first). Combined this is the funny math geek comment in the code snippet. <img src="https://discuss.pytorch.org/images/emoji/apple/nerd_face.png?v=10" title=":nerd_face:" class="emoji" alt=":nerd_face:"></p>

Samuel_Bachorik: <p><a class="mention" href="/u/tom">@tom</a> Hi, one more thing. Do you have any idea how to plot these two matrices and see rotation on some type of plot ? When I want to do this i cannot just plot Matrix as image right ?</p>

tom: <p>Maybe plotting one or two rotated unit vectors (ie use the two entries of a row of the matrix as x and y) would work.</p>
<p>Edit: Back at the computer (and “eer” about the rotation direction convention…):</p>
<pre data-code-wrap="python"><code class="lang-python">from matplotlib import pyplot
pyplot.figure(figsize=(3, 3))
pyplot.arrow(0, 0, 1, 0, width=0.02, alpha=0.5)
pyplot.arrow(0, 0, 0, 1, width=0.02, alpha=0.5)
pyplot.arrow(0, 0, rot[0, 0].item(), rot[0, 1].item(), width=0.02)
pyplot.arrow(0, 0, rot[1, 0].item(), rot[1, 1].item(), width=0.02)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d987159c1e0bb901a669b0be0bf34cd98caf8f5.png" alt="image" data-base62-sha1="kcBVGK4dm4IWaKodUxoCtNmzZLT" width="204" height="194"></p>
How to evaluate MarginRankingLoss and CosineEmbeddingLoss during testing

danielvella: <p>I am dealing with a Siamese Network for vectorised data and want to apply a Contrastive Loss through the MarginRankingLoss or CosineEmbeddingLoss functions.</p>
<p>This is my training loop, however I do not know how to properly evaluate the outputs from the model during test phase</p>
<p><strong>Training</strong></p>
<pre><code class="lang-auto">for batched_graph_1, batched_graph_2, labels in train_dataloader:
     pred1, pred2 = model(batched_graph_1, batched_graph_2)
     loss = loss_func(pred1, pred2, labels)
     scheduler.step(loss)
     optimizer.zero_grad()
     loss.backward()
     optimizer.step()
</code></pre>
<p>This is what I have so far:</p>
<p><strong>Testing</strong></p>
<pre><code class="lang-auto">for batched_graph_1, batched_graph_2, labels in test_dataloader:
    pred1, pred2 = model(batched_graph_1, batched_graph_2)
    pred = torch.cdist(pred1, pred2, p=2)
    y_pred += pred.tolist()
    y_true += labels.tolist()
</code></pre>
<p>Thank you for your help.</p>
Select different value of sensitivity from ROC curve to test a pre-trained model

Nanni: <p>Hi, I made the ROC curve on the validation_set of my dataset using a pre-trained model (“fpr, tpr, thresholds = roc_curve (…)”). Now I wanted to test this model on new images to be able to see the results of my work for this binary classification. At the moment I just used a “torch.max (output, 1)” to determine which class the input image belongs to. Is it possible to do this classification by modifying the choice according to the type of sensitivity I want?</p>

tom: <p>You have two classes?<br>
In this case you could use <code>output[:, 0] &gt;= thresh</code> if you determined the threshold for class 0 (or change 0 to 1 if you used class 1.</p>
<p>Best regards</p>
<p>Thomas</p>
After spectral_norm grad isNone

Siarhei_D: <p>After usage spectral_norm after backward no grad:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.utils.spectral_norm as spectral_norm
m = nn.Conv2d(1,1,3,1,1)
m(torch.randn((1,1,3,3))).mean().backward()
print(m.weight,m.weight.grad)
</code></pre>
<p>(Parameter containing:<br>
tensor([[[[ 0.1578,  0.1756,  0.3056],<br>
[ 0.1704, -0.0901, -0.1729],<br>
[ 0.0391, -0.2639, -0.1461]]]], requires_grad=True),<br>
tensor([[[[-0.0684,  0.1441, -0.0647],<br>
[-0.1048,  0.0695, -0.1498],<br>
[-0.2293, -0.0627, -0.1013]]]]))</p>
<p>But when i appy spectral_norm: no grad<br>
m = spectral_norm(nn.Conv2d(1,1,3,1,1))<br>
m(torch.randn((1,1,3,3))).mean().backward()<br>
print(m.weight,m.weight.grad)</p>
<p>(tensor([[[[ 0.2949,  0.3651, -0.1934],<br>
[-0.5089, -0.3238, -0.2878],<br>
[-0.3722, -0.1071,  0.3818]]]], grad_fn=), None)</p>

ptrblck: <p><code>SpectralNorm</code> manipulated the weight parameter as seen <a href="https://github.com/pytorch/pytorch/blob/7adb78017a81b656d22864ec97c072519a7dca63/torch/nn/utils/spectral_norm.py#L114-L152">here</a> so you could access the gradient via <code>m.weight_orig.grad</code>.</p>
Training with array type label column

prantik: <p>Hi All,<br>
I have structured training data as a pandas dataframe object. The data looks like below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09b538902372de048012e9fdde5fcc0480780dac.png" alt="image" data-base62-sha1="1nSyNK2Ij4uHEaz9cyJxxKHMusY" width="457" height="134"></p>
<p>The label column is my target and I want to train a convolutional layer with this data. The input values are normalized between 0 to 1. The problem is with the label column. It takes an array of classes.<br>
I am unable to feed this data to a convolution layer. Can anyone please help me with how I can use this data for training a model?</p>
<p>Thanks</p>

ptrblck: <p>I’m not sure I understand the issue correctly, as you wouldn’t pass the targets to the model (or conv layer as mentioned in your post).<br>
Assuming you are working on a multi-label classification, you could create a multi-hot encoded target and use <code>nn.BCEWithLogitsLoss</code> as the criterion. Each active class would then be represented by a 1 in the target, while inactive classes would get a 0.</p>
How to save an image in YCbCr mode

One4ever: <p>Hi,<br>
I have a 3D tensor of an image in YCbCr color space instead of RGB.<br>
I want to save the image using <code>torchvision.utils.save_image</code>, how can I set the mode to YCbCr? Are there any other ways to do this?<br>
Thanks</p>

tom: <p>So under the hood, <code>save_image</code> does is lines of code:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/a839796328cf4f789c9de5da0b3367b742b5a00c/torchvision/utils.py#L132-L136">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/a839796328cf4f789c9de5da0b3367b742b5a00c/torchvision/utils.py#L132-L136" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/a839796328cf4f789c9de5da0b3367b742b5a00c/torchvision/utils.py#L132-L136" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/a839796328cf4f789c9de5da0b3367b742b5a00c/torchvision/utils.py#L132-L136</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="132" style="counter-reset: li-counter 131 ;">
          <li>grid = make_grid(tensor, **kwargs)</li>
          <li># Add 0.5 after unnormalizing to [0, 255] to round to nearest integer</li>
          <li>ndarr = grid.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to('cpu', torch.uint8).numpy()</li>
          <li>im = Image.fromarray(ndarr)</li>
          <li>im.save(fp, format=format)</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<ul>
<li>First it uses <code>make_grid</code> which will arrange a batch of tensors in a grid, giving you a c x h x w tensor even if you passed in b x c x h x w. You may or may not need this.</li>
<li>Then it maps from 0…1 to 0…255, clamps, permutes to h x w x  c and converts to a numpy uint8 array. You would want that line, too.</li>
<li>Third it makes a PIL image from the numpy array. Here <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.fromarray" rel="noopener nofollow ugc">you can add a mode argument</a> to the call to <code>PIL.Image.fromarray</code>.</li>
<li>Then it saves the PIL image, you want that, too.</li>
</ul>
<p>It’s a bit tedious, but probably not too much.<br>
Or you could propose a patch to torchvision to add an optional <code>mode</code> argument to <code>save_image</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

One4ever: <p>Thanks for your accurate and complete answer <img src="https://discuss.pytorch.org/images/emoji/apple/sunflower.png?v=10" title=":sunflower:" class="emoji" alt=":sunflower:"><img src="https://discuss.pytorch.org/images/emoji/apple/ok_hand.png?v=10" title=":ok_hand:" class="emoji" alt=":ok_hand:"></p>