akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchserve interaction with scipy and other packages

username: watchdog90
cooked: <p>Hi, I am working on a project with Torchserve where an already implemented running service (torchserve) needs to be integrated with a numpy-scipy code. In particular, the numpy-scipy code uses some function for generating sparse arrays, solveds some equations etc. In particular, in the part of the code where scipy calls the linalg function (scipy.linalg.cg(…)) I notice a strange behaviour when calling from postprocess function, my local python file. In particular after executing the scipy.linalg.cg(…) function, service stops.<br>
Any idea? Could this be an issue related with the interaction between scipy packages and torchserve? Should I integrate my code in the handler class instead of have it externally and call it from handler class?</p>
NaN validation loss with batch size 1

username: ksmdanl
cooked: <p>I implemented SSD using resnet18 backbone and while the model trains I got NaN loss during the validation on some images.<br>
After further exploration degenerate boxes were found and then I did the tricks by applying some kind of filter that accepts only non degenerate boxes.<br>
This doesnt fix the issue. What happens is on each of these images no bounding boxes are detected so a zero array and the ground truth passed to the loss function and it returns a NaN.<br>
Looking on the images, where the NaNs occur, I see no anomaly on how the data is pre processed, both the images and the labels, i.g. from these images I see, at least for me, reasonable distribution after preprocessing.<br>
Could it be that it traces back to the batch size that I use, which is 1?<br>
Looking forward for the exchange!</p>
Loading a csv containing a series of Graphs, Pytorch Geometric

username: Guiltyfart
cooked: <p>I  have a <code>.csv</code> file which gives the same graph with different node features, as there are 104 nodes, this means that every 104 rows in the <code>.csv</code>, is the same graph but with different values for the node features.<br>
e.g.</p>
<pre><code class="lang-auto">node,feature1,feature2,feature3
0, 1, 2, 3
1, 6, 5, 7
...
104, 0, 68, 1
0, 7, 9, 76
1, 7, 8, 9
</code></pre>
<p>I want to load this <code>.csv</code> in as the multiple (a few thousand) graph states it represents (i.e. the <code>edge_index</code> is the same for all of these graphs, the only difference is the node feature values.)</p>
<p>How would I do this in Pytorch Geometric? (i.e. load it in as a list of <code>Data()</code> objects)</p>
<p>Regards,<br>
Jordan McConnon</p>
Implementing this formula in PyTorch

username: ant66
cooked: <p>I’m implementing a siamese net for depth prediction for 2 images. Both images are fed into an autoencoder, generating estimated feature map / disparity maps. Let’s label those output as d_left and d_right. I’ve came into this loss function that is used for minimizing the error.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64dd6b6adebc73de5d59f4f8655f7bcca209ea24.png" alt="Screenshot_20220220_132531" data-base62-sha1="eoi8s7KUNKLxkKNwFOCWjPhfh7S" width="503" height="75"></p>
<p>I don’t understand it’s concept, because in the fomula above, the left disparity at index i, j is added to index i of right disparity. But that doesn’t make any sense, because disparity is a floating point number and index is an integer. If someone knows the right implementation of this, I would appreciate all help.</p>

username: JuanFMontesinos
cooked: <p>I mean you have 2 images which look at the same scene. Due to the separation of the lenses, an object in the right image is gonna be slightly displaced to the left compared to its position in the left image.</p>
<p>I think this simply tries to compensate that so the object is in the same position when you apply the loss.<br>
So prob  D_r(i+D_l) works in the continuous world. Since images are sampled, in the discrete domain you just need to round or crop that number. Maybe either cropping or rounding is one of this stupid tiny details that matters a lot but don’t have exp in the field</p>

username: ant66
cooked: <p>Thanks a lot, I was aware of this approach but wanted to be sure about it. They didn’t mention anything about it in the paper.</p>
List type has no attribute 'to'

username: Marie_khan
cooked: <p>List type has no attribute ‘to’</p>
<p>print(type(label)) &gt;&gt;&gt; class ‘list’</p>
<p>for data, label in train loader:<br>
data=data.to(device)<br>
&gt;&gt;label=label.to(device)</p>
<p>AttributeError: ‘tuple’ object has no attribute ‘to’</p>
<p>I also tried previous solutions but no success please help</p>

username: Marie_khan
cooked: <p>problem solved, because im using windows so label split should use double \ instead of /.</p>
How to accurately keep track of training and validation loss when using mini-batches?

username: jagandecapri
cooked: <p>I have a large training and validation dataset that does not fit inside the memory. Hence, I am training it in mini-batches using a data generator. I am not sure how to keep track of the losses so that I can plot a graph to see whether my training and validation loss is on the right track. I implemented a (pseudo)code as below. Is my approach correct?</p>
<pre><code class="lang-auto">validate_every = 500
training_loss_list = []
for i in number_of_epochs:
	for batch_train in train_generator:
		iter += 1
		X_train, y_train = batch_train
		...
		output = model(X_train)
		loss = criterion(output, y_train)
		...
		training_loss_list.append(loss.item())

		if iter % validate_every == 0:
			validation_loss_list = []

            # Validation data will be different from everytime next(validation_generator) is called
			for i in range(40):
				X_val_batch, y_val_batch = next(validation_generator)
                                ...
				output = model(X_val_batch)
				loss = criterion(output, y_val_batch).item()
				validation_loss_list.append(loss)
			
			training_loss = mean(training_loss_list)
			validation_loss = mean(validation_loss_list)
			print(f'Training loss: {training_loss} Validation loss: {validation_loss}')

			training_loss_list = []
			
</code></pre>
Loading data in FlatBuffers format

username: ccoa
cooked: <p>Hi,</p>
<p>I’ve got data in <a href="https://google.github.io/flatbuffers/" rel="noopener nofollow ugc">FlatBuffers</a> format that I need to feed to a Pytorch model. I don’t have experience reading/loading such files. Has anyone worked with such files and knows how to load them and work with them properly?</p>
<p>If there’s a nice way to work directly with such data, that’d be great. But if it’s easier to first convert them to json/protobuf, and feed the json/protobuf data (even if it worsens performance to some extent), I’m OK with that (ease of use/coding is important for me).</p>
<p>Thanks for the help!</p>
How to add an install target for libtorch binaries in cmake?

username: victor-shepardson
cooked: <p>this question is about using the <code>install</code> command in CMakeLists to place+link the libtorch shared libraries in a relocatable way. Is there a proper way to do this with the libtorch cmake module?</p>
<p>I figured either</p>
<pre><code class="lang-auto">install(TARGETS ${TORCH_LIBRARIES} DESTINATION /path/to/install)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">install(FILES ${TORCH_LIBRARIES} DESTINATION /path/to/install)
</code></pre>
<p>but each causes an error, because TORCH_LIBRARIES contains both targets and filenames? In this case it is <code>torch;torch_library;/Users/victor/libtorch-1_11_0_dev20220110/lib/libc10.dylib;/Users/victor/libtorch-1_11_0_dev20220110/lib/libkineto.a</code></p>
<p>I think I know how to do this manually on macOS, but I would like cmake to handle it for all platforms + in  a forward-compatible way.</p>
Convolutional Out shape padding

username: alessandro_mondin
cooked: <pre><code class="lang-auto">img = torch.randn(1,3,32,32)
conv = torch.nn.Conv2d(in_channels=3,out_channels=6,kernel_size=4,stride=2,padding=1)
conv(img).size()
</code></pre>
<p>Why does this code does not raise an error? If the height and width are 32, with a kernel size = 4 and a stripe=2, the the second last group of pixels in the first row will be 29-30-31-32 and the last 31-32-33 and one pixel is missing to complete the kernel_size.<br>
Are extra pixels automatically discarded or something else happens?</p>

username: alessandro_mondin
cooked: <p>I thought padding were added only on the right and bottom column while instead is added in all four dimension…now it all makes sense</p>
Where is the source code for `torch.Tensor.contiguous()`?

username: dl4daniel
cooked: <p>When I experiment on language model example of <code>pytorch</code>, the <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L66" rel="nofollow noopener">following use</a> of <code>contiguous</code> seems make no difference without it.</p>
<p><code>data = data.view(bsz, -1).t().contiguous()</code></p>
<p>I checked docs on <code>contiguous</code> <a href="http://pytorch.org/docs/tensors.html?highlight=size#torch.Tensor.contiguous" rel="nofollow noopener">here</a>, but the limited info still makes me question the necessity of use here.</p>
<p>I tried to find <code>tensor.contiguous</code> in github, but couldn’t find it.</p>
<p>Could anyone point to the source code of <code>torch.Tensor.contiguous</code> for me? Thanks a lot!</p>

username: apaszke
cooked: <p>If you check <code>data.strides()</code> you’ll see that they’re different after you call <code>contiguous()</code>. It’s all about how the tensors are stored, you can find more details <a href="https://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memory-layout-of-an-ndarray">in the numpy docs</a>.</p>

username: dl4daniel
cooked: <p>Thanks <a class="mention" href="/u/veril">@Veril</a> <a class="mention" href="/u/apaszke">@apaszke</a> for your replies!</p>
<p>the source code on <code>nn.Countiguous</code> is different from <code>tensor.contiguous</code> right?</p>
<p>I tried to use <code>tensor.stride</code> but I don’t see how it helps to see the differences.  <a class="mention" href="/u/apaszke">@apaszke</a>, could you make a simple code  to demonstrate the use of <code>contiguous</code>?</p>
<p>I found <a href="http://jucor.github.io/torch-doc-template/tensor.html#annotations:hZ9Uvg1PEee1rzPsK-mAsA" rel="nofollow noopener">an example</a> of <code>contiguous</code> on this torch page, but I can’t translate it to pytorch. Will this help you make an example in pytorch?</p>
<p>Thanks a lot!</p>

username: apaszke
cooked: <p><code>nn.Contiguous</code> is part of the legacy package and unless you’re coming from Lua Torch you probably should never touch that code.</p>
<pre><code class="lang-auto">x = torch.randn(5, 4)
print(x.stride(), x.is_contiguous())
print(x.t().strinde(), x.t().is_contiguous())
x.view(4, 5) # ok
x.t().view(4, 5) # fails
</code></pre>

username: dl4daniel
cooked: <p>Thank you so much for your patience and help <a class="mention" href="/u/apaszke">@apaszke</a> and <a class="mention" href="/u/veril">@Veril</a></p>

username: kj7kunal
cooked: <p>x.t().view(4,5) should work, since the x.t() tensor is of size (4,5).<br>
However, x.t().view(5,4) should give the contiguous error.</p>

username: yassineAlouini
cooked: <p>Thanks for the clarification.</p>
<p>I guess there is a minor typo in the second print:</p>
<p><code>print(x.t().strinde(), x.t().is_contiguous())</code></p>
<p>should be</p>
<p><code>print(x.t().stride(), x.t().is_contiguous())</code></p>
Output of classifier as condition of generator in gan

username: Arpit_Garg
cooked: <p>I wanted to give an output of classifier CNN (softmax) as the input of generator in gan as the conditions. But I wanted to treat them as class weights, I am not sure how to do it?</p>
<p>(<a href="https://takuhirok.github.io/CP-GAN/" class="inline-onebox" rel="noopener nofollow ugc">CP-GAN</a>)</p>
Map sorting in python

username: Nil_MSh
cooked: <p>I’ve trained a CNN model and for a neuron x in layer y I have a a tensor like</p>
<blockquote>
<p>tensor(0.4518)</p>
</blockquote>
<p>now I want to sort based on this tensor and find the 5 neuron with most values and I want the layer and id be mapped with this tensor once I sort it.<br>
How should I do this?</p>
Backprop with Multiple Classifier Heads (Linear Layers) with Single CNN (Embedding Network)

username: Alison_Ma
cooked: <p>Hi,</p>
<p>I was curious as to if anyone has experience training multiple classifier heads but updating a single embedding network (CNN). Is there a way to elegantly call model.backwards() for the shared CNN and the single appropriate classifier head? I would prefer to do it inside a single nn.Module because I believe it will make updating the common CNN easier but I am open to other suggestions.</p>
<p>Thank you!</p>
<p>Alison</p>

username: eqy
cooked: <p>How would the classifier heads be selected? If it is ordinary control flow then backprop will only happen through the head that was used.</p>

username: Alison_Ma
cooked: <p>I was thinking that upon calling model.forward() that the appropriate classifier be selected from a list. If this was the case, would backprop only happen through this head?</p>
<p>I plan to cycle through all classifier heads per training epoch</p>

username: eqy
cooked: <p>Yes, I believe backprop would only happen through that particular head. If you want to verify this explicitly you can do something like <code>head.weights.sum()</code> to check which head had its weights updated in a particular iteration.</p>

username: Alison_Ma
cooked: <p>Got it, I’ll give it a try and see! Thanks!</p>
LSTM generating same word multiple time

username: Dhimitrios_Duka
cooked: <p>Hello everyone!<br>
I’m trying to develop a model for image captioning. The architecture that I’m using is a CNN + LSTM. The feature vector generated from the CNN is fed as the hidden and cell state of the LSTM.<br>
The problem that I have is that the LSTM generates the word with the higher frequency in the vocabulary multiple times.(I’m using argmax in the decoder output).<br>
I also tried using the output as a probability distribution but the BLEU score in the case was too low?<br>
Can anyone tell me what the problem can be? If you want I can share the code too.<br>
Thanks in advance!</p>

username: vdw
cooked: <p>There are couple of the usual suspects, but without seeing any code it would be purely guesswork.</p>
Are there any best practices for using torchscript with pytorch-mobile?

username: Ludwig_Friborg
cooked: <p>Are there any best practices for using torchscript with pytorch-mobile? Im currently trying to implement a conv-net with pytorch-mobile on android. The model is very slow in comparison to an identical tensorflow trained tflite model. Perhaps this might have something to do with the torchscript tracing.</p>
<p>Any thoughts would be appreciated</p>

username: Ludwig_Friborg
cooked: <p>I tries freeze the graph using <code>torch.utils.mobile_optimizer.optimize_for_mobile</code> which is new for pytorch 1.6.0. It seems to improve performance a little but not significantly.</p>

username: eagomez
cooked: <p>Hi,</p>
<p>Did you find anything new? We are experiencing a very similar pattern in terms of performance while comparing a TorchScript model against its TFLite counterpart</p>

username: szewaiyuen6
cooked: <p>cc: <a class="mention" href="/u/kimishpatel">@kimishpatel</a> if you have any thoughts on this</p>

username: cccclai
cooked: <p><a href="https://pytorch.org/mobile/home/" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a> is updated recently and can be a good reference.</p>
<p>How do you export the model and how do you use the mobile runtime?</p>
Learning about the PyTorch Implementation

username: BlackBird123
cooked: <p>Hello,</p>
<p>I’m learning about how ML Libraries are implemented and I would like to read about PyTorch. Can someone point me to a resource where I can read about the basic implementation strategy of the library?</p>
<p>I would also love to find something similar for Scikit-Learn.</p>
<p>Thanks!</p>
<p>Emil</p>

username: AlphaBetaGamma96
cooked: <p><a href="https://pytorch.org/tutorials/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/</a></p>

username: BlackBird123
cooked: <p>Thanks but this isn’t what I had in mind. The tutorials will tell you how to use PyTorch. But I want to know how the library itself is implemented. What are the base classes? How are they combined? What is the organization of the code?</p>
Cannot install via pip

username: Nathan_Goedeke
cooked: <p>I went on <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a> and entered the combination that fit my system (Stable, Windows, Pip, Python, CPU). I copy and pasted the command in Command Prompt and got this message:</p>
<pre><code class="lang-auto">&gt;pip3 install torch torchvision torchaudio
WARNING: Ignoring invalid distribution -ip (d:\program files (x86)\python\lib\site-packages)
WARNING: Ignoring invalid distribution -ip (d:\program files (x86)\python\lib\site-packages)
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
WARNING: Ignoring invalid distribution -ip (d:\program files (x86)\python\lib\site-packages)
WARNING: Ignoring invalid distribution -ip (d:\program files (x86)\python\lib\site-packages)
</code></pre>
<p>I don’t understand why that wouldn’t work.<br>
Nathan</p>

username: Nathan_Goedeke
cooked: <p>It seems that pytorch will refuse to install if Python version is greater than 3.79, and I am using Python 3.9. The only way to downgrade Python is by creating a virtual environment with Anaconda. I find that very frustrating because I want to run pytorch on my regular machine, not with this complicated overhead. I welcome any other suggestions.</p>
Best way to implement dilation (without convolution)?

username: ZimoNitrome
cooked: <p>I am trying to make a function/module that takes an array of some shape and pads all pixels by a given amount:</p>
<pre><code class="lang-auto">a = torch.arange(1, 5).expand(4, -1)
print(a)

b = dilate(a)
print(b)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4]])

tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0, 3, 0, 4, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0, 3, 0, 4, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0, 3, 0, 4, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0, 3, 0, 4, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0])
</code></pre>
<p>I don’t believe this functionality exists in pytorch? How would I best implement this without “for” loops?</p>

username: chrudimer
cooked: <p>Hi, I don’t know whether there is a better way to do this, but you could do a transposed convolution (torch.nn.functional.conv_transpose2d) with a 1x1 kernel were there is a 1 in the 1x1 kernel. Via stride you can then choose how many zeros you want in between your pixels. For the padding at the edges, you could then use the pad function. So you could do something like this here:</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt


def pad_zeros_in_between(input_, num_zeros_in_between=1):
    weight = torch.ones((1, 1, 1, 1))
    out = torch.nn.functional.conv_transpose2d(input_, weight, stride=num_zeros_in_between+1)
    out = torch.nn.functional.pad(out, [num_zeros_in_between for i in range(4)])

    return out


input_ = torch.linspace(start=1, end=4, steps=4)
input_ = torch.stack([input_ for i in range(4)])
input_ = torch.unsqueeze(torch.unsqueeze(input_, dim=0), dim=0)

out = pad_zeros_in_between(input_, num_zeros_in_between=1)
    
print(input_)
print(out)

plt.figure()
plt.imshow(torch.squeeze(input_))

plt.figure()
plt.imshow(torch.squeeze(out))

</code></pre>

username: ZimoNitrome
cooked: <p>Hehe I guess that is one way to do it!</p>
<p>I am kind of trying to “re-make” the conv_transpose, but using the actual function in my “new” definition for this part won’t hurt <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Well done, thanks!</p>
Memory usage when doing Monte Carlo sampling during optimization

username: Yonim
cooked: <p>Hi,</p>
<p>I’m working with energy-based models, using contrastive divergence to train my model.</p>
<p>In each iteration, I’m generating samples from my model using the Markov chain Monte Carlo algorithm:<br>
<code>mcmc_samples = generate_samples(weights, n_samples)</code><br>
Then I calculate the energy of these samples and the energy of my train samples. The loss is the delta between the mean of the energies.</p>
<p>The complete training process looks something like this:</p>
<pre><code class="lang-auto">weights = torch.nn.Parameter(...)
optimizer = torch.optim.SGD([weights], lr=lr)

for i in range(iterations):
    mcmc_samples = generate_samples(weights, n_samples)
    mcmc_energies = get_energies(weights, mcmc_samples)

    train_samples_energies = get_energies(weights, samples)

    loss = - mcmc_energies.mean() + train_samples_energies.mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>This works well for small models. For large models I get a cuda out of memory exception when calling <code>generate_samples</code>.</p>
<p>I do not get an exception when I call <code>generate_samples</code> with the same arguments outside the training process, when <code>weights</code> is a tensor and not a parameter.</p>
<p>I understand that this is because more memory is required to store the data later needed for gradient calculation when using parameters. But in this case, the gradient is independent of the <code>generate_samples</code> step.</p>
<p>Is there a way to tell PyTorch to ignore the <code>generate_samples</code> line of code during the gradient calculation?<br>
Maybe there’s a different approach to using PyTorch with Monte Carlo sampling?</p>
<p>Currently, I solved this by computing the gradients on my own and implementing the optimization step as well, but I would like to use PyTorch automatic differentiation and out-of-the-box optimizers.</p>
<p>Thanks!</p>

username: AlphaBetaGamma96
cooked: <p>Are you detaching your samples? Otherwise, Autograd will backprop through your sampling process as well which you don’t want at all (and will lead to an ever-increasing memory footprint)</p>

username: Yonim
cooked: <p>Actually no, that means that inside my sampler I should detach after every operation involving my model parameters?</p>
<p>EDIT:</p>
<p>Thanks <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>! Inside <code>generate_samples</code> I simply detached every tensor resulting from a calculation with my model parameters and it solved the issue.</p>
Transfer learning yolo5 using pytorch

username: Medhat_Elhady
cooked: <p>Hello everyone</p>
<p>I am working on my graduation project, I am stuck in building an object detection model to detect vehicles (cars, buses, bicycles …etc)</p>
<p>I used yolo3, 5, and x, but it’s time-consuming on raspberry pi and even laptop CPU.</p>
<p>is there a way to optimize yolo like transfer learning or should I create from scratch? in both cases, I will be grateful if you shared resources</p>
Pytorch hooks removal

username: Tejan_Mehndiratta
cooked: <p>hi,<br>
How to remove a hook from a layer.<br>
I am doing the below, but it’s incorrect.</p>
<p>Accessing the features of <strong>denselayer1</strong> with <strong>hook1</strong> function.</p>
<pre><code class="lang-auto">denselayer1 = []
def hook1(self, inputs, output):
    denselayer1.append(output)

handle1 = densenet121.features.denseblock1.denselayer1.register_forward_hook(hook1)
_ = densenet121(random_input)

q=denselayer1[0] + torch.tensor(3)
handle1.remove()
</code></pre>
<p>Now, here I am removing the hook.<br>
Then,</p>
<p>Accessing the features of <strong>denselayer2</strong> with <strong>hook2</strong> function.</p>
<pre><code class="lang-auto">denselayer2 = []
def hook2(self, inputs, output):
    denselayer2.append(output)

handle2 = densenet121.features.denseblock1.denselayer2.register_forward_hook(hook2)
_ = densenet121(random_input)
</code></pre>
<p>Now, Accessing the features of <strong>denselayer2</strong> with <strong>hook1</strong> function.</p>
<pre><code class="lang-auto">densenet121.features.denseblock1.denselayer2.register_forward_hook(hook1)
</code></pre>
<p>Now, when I do</p>
<pre><code class="lang-auto">denselayer1[0]==denselayer2[0]
</code></pre>
<p>it comes <code>False</code>.</p>
<p>what am I doing wrong?</p>

username: tom
cooked: <p>I would recommend using <code>print</code> in the hook to print which hook is called and the <code>len</code> of the list you are saving things in to get a better idea of what you are actually doing and when.<br>
My impression is that</p>
<blockquote>
<p>Now, Accessing the features of <strong>denselayer2</strong> with <strong>hook1</strong> function.</p>
</blockquote>
<p>doesn’t do what you think it does.</p>
<p>Best regards</p>
<p>Thomas</p>
Keeping constant value in module on correct device?

username: mattrobin
cooked: <p>What’s the appropriate way to keep a constant that’s part of a module on the correct device. If I define a constant <code>Variable</code> for use in a module, calling the module’s <code>cuda()</code> method will not move the constant to the GPU. However, if I set the constant as a <code>Parameter</code> that does not require gradient, then turning off and on the gradient for the entire module will flip this switch for that one parameter as well (changing it to no longer be a non-trained constant).</p>
<p>As an example, I the following module which is made to scale the input by a trainable positive value (using a trainable exponent):</p>
<pre><code>class MyModule(Module):
    def __init__(self):
        super().__init__()
        self.exponent = Parameter(torch.Tensor([1]))
        self.e = Variable(torch.Tensor([math.e]), requires_grad=False)

    def forward(self, x):
        x = x * (self.e.pow(self.exponent))
        return x
</code></pre>
<p>Here <code>e</code> should remain constant. But if I want to be able to move this module off and on the GPU without trouble, what’s the best way to handle this? I know I could also override the <code>cuda()</code> method of the module, but this still requires having a special case for any constant I use. Is there a better way? Thank you!</p>
<p>Edit: Actually, I’ve found that overriding <code>cuda()</code> for the module does not work well, because if this module is include as part of another module, the larger module does not call the submodule’s <code>cuda</code> method, and so does not trigger the override method.</p>

username: ptrblck
cooked: <p>You could use <code>register_buffer</code> for automatic handling of your constant.</p>
<pre><code>class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.exponent = nn.Parameter(torch.Tensor([1]))
        self.e = torch.Tensor([np.e]).float()
        self.register_buffer('e_const', self.e)

    def forward(self, x):
        x = x * (Variable(self.e_const).pow(self.exponent))
        return x</code></pre>

username: wolterlw
cooked: <p>Since Variable API is now deprecated, how to set <code>requires_grad=True</code> on a buffer or is it unnecessary?</p>

username: ptrblck
cooked: <p>A buffer shouldn’t require gradients. Instead use <code>nn.Parameter</code> if your tensor should require gradients (by default <code>requires_grad</code> will be <code>True</code>). Generally, you can now pass the <code>requires_grad</code> argument to <code>torch.tensor(1, requires_grad=True)</code>.</p>

username: gabrieldernbach
cooked: <p>This way, we have two names for the same thing <code>e</code> and <code>e_const</code>. Is this still the recommended way?</p>

username: ptrblck
cooked: <p>You don’t have to assign <code>self.e</code>, if it’s never used and can just register the buffer once.</p>
nn.Flatten() documentation seems wrong

username: quazi
cooked: <p>The default end_dim is -1, which means the last time dimension is NOT flattened, however if you pass a torch.Size([3, 2, 4, 3]) to nn.Flatten() you get a torch.Size([3, 24]) where 24 is coming from flattening dimensions 1,2 AND 3 returning a tensor size [dim 0, dim 1 x dim 2 x dim 3]. I was expecting a tensor of shape [dim 0, dim 1 x dim 2, dim 3]. Am I reading the doc wrong?? seems pretty clear though and wrong.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3351dba3ec0327f362e8babc3af499888665cd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3351dba3ec0327f362e8babc3af499888665cd3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3351dba3ec0327f362e8babc3af499888665cd3.png" alt="image" data-base62-sha1="nhNCXLK5ANqb940EpX3Yi7S1nHl" width="690" height="90" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">728×95 6.03 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Yuv
cooked: <p>As you can see in the documentation:</p>
<blockquote>
<p>Parameters</p>
<ul>
<li><strong>start_dim</strong> – first dim to flatten (default = 1).</li>
<li><strong>end_dim</strong> – last dim to flatten (default = -1).<br>
(<a href="https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html#torch.nn.Flatten" class="inline-onebox" rel="noopener nofollow ugc">Flatten — PyTorch 2.1 documentation</a>)</li>
</ul>
</blockquote>
<p>Which means the default is to flatten the tensor from the second dimension (start_dim=1) until the last dimension (end_dim= -1) and the first dimension (dim = 0) stays the same.</p>

username: quazi
cooked: <p>but end_dim = <strong>-1</strong> in not the last dim, its the one before that, no? to be clear, nn.Flatten is doing what I want, I’m just saying the documentation is wrong OR confusing at least.</p>

username: quazi
cooked: <p>I think I misread the the equations… [d0, d1, d2, d3,…, dn] → dim -1 would be <strong>dn</strong> as in the doc</p>

username: Yuv
cooked: <p>end_dim = -1 refers to the last dimension</p>
Fluctuating Memory Consumption through training

username: ShaoMinLiu-Holmusk
cooked: <p>Has anybody has the issue of spiking memory usage through training. I am fine-tuning a Sentence BERT model from <code>https://github.com/UKPLab/sentence-transformers</code>.</p>
<p>I have noticed that the memory use of the model will remain below 2Gb for a few epochs, but will suddenly break the training at some point and complains Out of Memory.</p>
<p>I have about 8Gb ram on my GPU, because of this issue, I have to reduce my batchsize and allow the training to use less than 2Gb of ram most of the time just to avoid OOM in case of a memory spike.</p>
<p>This is definitely under utilising resources, I was wondering if anybody faced similar situations before, what could be the reason here?</p>
Model resume resulting to high train loss but same inference loss

username: yuri
cooked: <p>applying the model resume as explained <a href="https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html" rel="noopener nofollow ugc">here</a> led to the same inference error. however, the train resume showed a way higher loss than the loss at the time of model saving. it is worth mentioning that LR was decreased by 10% each time the validation loss was improved. Which means, no lr scheduler was defined<br>
thank you</p>

username: eqy
cooked: <p>Is the learning rate schedule resumed at the same point when training is resumed?</p>

username: yuri
cooked: <p>Thank you for your reply.<br>
I have not defined lr schedule. I just multiply it by 0.9 every time the validation error decreases.</p>

username: eqy
cooked: <p>In that case you might want to store the learning rate (e.g., in something like the state dict) so that the same learning rate is used when training is resumed. As a hack to test if restoring the learning rate when training is stopped helps, you could first try something like adding it to the checkpoint name and then setting this rate manually when training is resumed.</p>

username: yuri
cooked: <p>Exactly, I assigned the same lr value  to optimizer_pd.param_groups[0][‘lr’] and for safety, to optimizer_pd.defaults[‘lr’]. still, I get exactly the same inference loss value but 10 times worst training loss.<br>
kind regards</p>
Efficient Matrix multiplication

username: Kevinzhu
cooked: <p>I have two matrices: <strong>A</strong> with size [D,N,M] and <strong>B</strong> with size[D,M,S]. My target is to multiply two matrices with respect to the dim_1 and dim_2, which is like <strong>A[d,:,:] * B[d,:,:]</strong> for <strong>d from 1:D</strong>. Currently I’m doing it with a for loop. I’m wondering if there is any other method I can use to make this operation more efficient.</p>
<p>Kevin</p>

username: eqy
cooked: <p>You might want to take a look at the docs for matmul: <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html" class="inline-onebox" rel="noopener nofollow ugc">torch.matmul — PyTorch 1.10 documentation</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.tensor([[[2, 2], [2, 2]], [[3, 3], [3, 3]]])
&gt;&gt;&gt; b = torch.tensor([[[1, 0], [0, 1]], [[1, 0], [0, 1]]])
&gt;&gt;&gt; torch.matmul(a, b)
tensor([[[2, 2],
         [2, 2]],

        [[3, 3],
         [3, 3]]])
&gt;&gt;&gt;
</code></pre>
<p>Even more general matmul-style ops can be expressed with einsum:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.einsum.html</a></p>

username: Kevinzhu
cooked: <p>Thanks. Actually I just found this document which is super useful to me.</p>
PyTorch won't detect CUDA 10.2 on Ubuntu 18.04 Jetpack

username: stSch
cooked: <p>Currently working with a NVIDIA Jetson. nvcc outputs,</p>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Sun_Feb_28_22:34:44_PST_2021<br>
Cuda compilation tools, release 10.2, V10.2.300<br>
Build cuda_10.2_r440.TC440_70.29663091_0</p>
<p>I tried setting up PyTorch from this page with the following settings,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26f30fc269d5f4faca3c931d27f017801a126c23.png" data-download-href="https://discuss.pytorch.org/uploads/default/26f30fc269d5f4faca3c931d27f017801a126c23" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26f30fc269d5f4faca3c931d27f017801a126c23.png" alt="image" data-base62-sha1="5yyTQW7yfaqRddZeWBs1i8XP1RN" width="690" height="281" data-dominant-color="F5DDDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">825×337 18.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Despite this, whenever I run torch.cuda.is_available() in python, I get the “False” output. Are there any dependencies or compatibility issues that I may be unaware of? Thanks.</p>

username: marksaroufim
cooked: <aside class="quote no-group" data-username="stSch" data-post="1" data-topic="144435">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ebca7d/48.png" class="avatar"> stSch:</div>
<blockquote>
<p>orch.cuda.is_available()</p>
</blockquote>
</aside>
<p>This post has some good suggestions <a href="https://forums.developer.nvidia.com/t/my-jetson-nano-board-returns-false-to-torch-cuda-is-available-in-local-directory/182498" class="inline-onebox">My jetson nano board returns 'False' to torch.cuda.is_available() in local directory - Jetson Nano - NVIDIA Developer Forums</a></p>
ResNet takes large input size (1000 X 1000)

username: Dadatata-JZ
cooked: <p>Hi folks,</p>
<p>I am wondering if there is any existing generations of ResNet (or any architecture) taking large image inputs, such as 1000 by 1000 without resizing to 224 by 224. I have a few high resolution images and fairly sufficient GPU resources, so I want to train the model on them directly. Please share some thoughts. Thanks in advance.</p>

username: ptrblck
cooked: <p>The current ResNet implementaions are flexible regarding the spatial input size (as long as it’s not too small) and would accept the <code>1000x1000</code> input images (assuming your GPU has enough memory).<br>
I’m not aware of any pretrained models using this resolution so you might want to train the model from scratch (or fine-tune it).</p>
Batch loader with custom indices

username: sri9s
cooked: <p>how to load batches of data with specific indices as input with mapstyle dataset?</p>

username: ptrblck
cooked: <p>Assuming this post is related to <a href="https://discuss.pytorch.org/t/selecting-data-from-dataset-by-indices/55290/10">this cross-post</a>:<br>
if you want to specify the batch indices manually, you could write a custom sampler, compute the indices beforehand, and pass them to the sampler.<br>
Alternatively, using a <code>Subset</code> with the desired indices as its input and disabling shuffling might also work.</p>
How to update only one coordinate of a layer?

username: My_C
cooked: <p>For example, I have a simple network like this</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self,input_shape):
    super(Net,self).__init__()
    self.fc1 = nn.Linear(20, 1)
  def forward(self,x):
    x = self.fc1(x)
    return x
</code></pre>
<p>fc1 should have 20 parameters, if I don’t want to update all of them in a single step, instead, I want to update one  of those parameters in a customized way. How should I do that? Thanks for any help.</p>

username: ptrblck
cooked: <p>Take a look at <a class="mention" href="/u/kfrank">@KFrank</a>’s answer in <a href="https://discuss.pytorch.org/t/is-there-any-way-to-set-the-requires-grad-as-false-of-particular-weights-instead-of-whole-layer/144495/2">this post</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: SemiSimon
cooked: <p>Ok what I think you can try is to call <code>.backward()</code> but don’t call <code>.step()</code> immediately. Instead, you can access the individual gradients generated by <code>.backward()</code> using:</p>
<pre><code class="lang-auto">self.fc1.weight.grad
self.fc1.bias.grad
</code></pre>
<p>And call <code>.step()</code> after you are done or even apply the gradients manually.</p>
Creating 2 Residual Block in the Same Class

username: tanvir_14
cooked: <p><strong>Case 1:</strong></p>
<pre><code class="lang-auto">class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv2 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv2(out)
        return residual + out
		
class MyGCNConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(MyGCNConv, self).__init__()
        layers = []
        for i in range(2):
			layers.append(_ResGraphConv(adj, input_dim, hid_dim, output_dim, p_dropout=p_dropout))
		
        self.myLayers = nn.Sequential(*layers)
		
    def forward(self, x):
        out = self.myLayers(x)
        return out
</code></pre>
<p><strong>Case 2:</strong></p>
<pre><code class="lang-auto">class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv2 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv3 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv4 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv2(out)
        out = residual + out
        y = out
        residual = out
        out = self.gconv3(y)
        out = self.gconv4(out)
        return residual + out
		
class MyGCNConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(MyGCNConv, self).__init__()
		self.myLayers = _ResGraphConv(adj, input_dim, hid_dim, output_dim,p_dropout=p_dropout)
		
    def forward(self, x):
        out = self.myLayers(x)
        return out
</code></pre>
<p>Is there any difference between these two and Is there any disadvantage of using case 2?</p>
<p>I want to create 2 residual block in the same class.</p>

username: ptrblck
cooked: <p>Both approaches look equal in their execution, so you could pick whichever fits your use case best.</p>

username: tanvir_14
cooked: <p>Thank you so much for the answer.</p>
What's up with the gradient of torch.linalg.norm?

username: ndronen
cooked: <p>I’d expect the gradient of the L2 norm of a vector of ones to be 2. The gradient is as I expect when I roll my own norm function (<code>l2_norm</code> in mwe below). The gradient is not what I expect when I call <code>torch.linalg.norm</code>. Is there some implementation detail about <code>torch.linalg.norm</code> that I need to know in order to understand what it thinks the gradient should be?</p>
<blockquote>
<p>expected grad of l-2 norm: tensor([2., 2., 2.])<br>
grad of l-1 norm: tensor([3., 3., 3.])<br>
grad of l-2 norm: tensor([0.5774, 0.5774, 0.5774])<br>
grad of l-3 norm: tensor([0.4807, 0.4807, 0.4807])</p>
</blockquote>
<pre><code class="lang-auto">import torch

x = torch.ones(3, requires_grad=True)

def l2_norm(x):
    """Derivative of L2 norm is 2x."""
    return torch.sum((x * x.detach()) ** 2)

l2_norm(x).backward()
print(f"expected grad of l-2 norm: {x.grad}")
x.grad.zero_()

for p in [1, 2, 3]:
    torch.linalg.vector_norm(x, ord=p).backward()
    print(f"grad of l-{p} norm: {x.grad}")
    x.grad.zero_()
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Your definition of the L2-norm is incorrect, hence why you’re getting a different result to PyTorch. Here’s the correct definition,</p>
<pre><code class="lang-auto">import torch

x = torch.ones(3, requires_grad=True)

def l2_norm(x):
    """Derivative of L2 norm is 2x."""
    #return torch.sum((x * x.detach()) ** 2) #wrong
    return x.pow(2).sum(-1).sqrt()           #correct

l2_norm(x).backward()
print(f"expected grad of l-2 norm: {x.grad}")
x.grad.zero_()

for p in [1, 2, 3]:
    torch.linalg.vector_norm(x, ord=p).backward()
    print(f"grad of l-{p} norm: {x.grad}")
    x.grad.zero_()
</code></pre>
<p>This returns</p>
<pre><code class="lang-auto">expected grad of l-2 norm: tensor([0.5774, 0.5774, 0.5774])
grad of l-1 norm: tensor([1., 1., 1.])
grad of l-2 norm: tensor([0.5774, 0.5774, 0.5774])
grad of l-3 norm: tensor([0.4807, 0.4807, 0.4807])
</code></pre>
<p>as expected! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><strong>TL;DR</strong> - you forgot the sqrt on your L2-norm definition. Also, there’s no need to detach the 2nd <code>x</code> either!</p>

username: ndronen
cooked: <p>Thanks for finding that bug. I still have a question: Why is torch’s gradient different than <code>2x</code>? I’m assuming that e.g. <a href="https://math.stackexchange.com/questions/883016/gradient-of-squared-2-norm" rel="noopener nofollow ugc">this</a> applies.</p>

username: AlphaBetaGamma96
cooked: <p>That’s the gradient of the <strong>square</strong> of the norm, not the gradient of the norm itself. So they’re different functions you’re differentiating, and hence have different answers.</p>

username: ndronen
cooked: <p>Yup, just figured that out. I wasn’t looking at the definition closely enough. Thanks!</p>
Do matmul between lists

username: J.Chry
cooked: <p>Hi, I am looking for a function could do this:</p>
<pre><code class="lang-auto">a=torch.rand(1)
b=torch.rand(1)
c=torch.rand(2,3)
d=torch.rand(3,4)

Func( [a,c], [b,d]) = [matmul(a,b), matmul(c,d)]
</code></pre>
<p>It could be done by iteration, but takes a lot time for long list <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> .<br>
Is there some func can handle this ?</p>

username: AlphaBetaGamma96
cooked: <p>You can use list comprehension and a zip statement</p>
<pre><code class="lang-auto">import torch
 
a=torch.rand(1)
b=torch.rand(1)
c=torch.rand(2,3)
d=torch.rand(3,4)
 
def func(list1, list2):
  return [a@b for a, b in zip(list1, list2)]

list1=[a,c]
list2=[b,d]
 
func(list1, list2)
"""
returns [tensor(0.0130), tensor([[0.6270, 1.0470, 0.2362, 0.7666],
        [0.9061, 1.3999, 0.2975, 1.0312]])]
"""
</code></pre>
How do I map the tensor

username: ck172
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f85e90f154c9943e7267c24f3909b28b5f929d9d.png" alt="Screenshot from 2022-02-18 11-51-22" data-base62-sha1="zrb1e2N9ssE3nheinrQDwKHFkO9" width="297" height="279"></p>
<p>the left two tensors are the source value and index of each value at target tensor. I wanted to map the source value to the target tensor based on its index. e.g. a1’s index is 3 so a1 went to the third position target sensor. How can I do this in pytorch? Any help will be apprecaited!</p>

username: eqy
cooked: <p>It sounds like this is basically implemented by <code>scatter_add_</code> depending on how exactly you want the 1-based indexing/ignoring values mapped to <code>0</code> to work.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.arange(10).float() + 1
&gt;&gt;&gt; a
tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.])
&gt;&gt;&gt; index = torch.tensor([3, 3, 0, 0, 0, 0, 0, 0, 0, 1])
&gt;&gt;&gt; target = torch.zeros(11)
&gt;&gt;&gt; target.scatter_add_(0, index, a)[1:]
tensor([10.,  0.,  3.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
</code></pre>
Equivalent of np.ptp()?

username: Alan_Wang
cooked: <p>Title basically. Is there an equivalent of <a href="https://numpy.org/doc/stable/reference/generated/numpy.ptp.html" rel="noopener nofollow ugc">np.ptp</a> in Pytorch?</p>

username: kklemon
cooked: <p>I think there is no direct equivalent, but you can easily implement the function with</p>
<pre><code class="lang-python">def ptp(t, axis):
  return t.max(axis).values - t.min(axis).values
</code></pre>

username: Carlos_Segura
cooked: <p>To match Numpy’s semantics, then the following is equivalent to <code>numpy.ptp()</code>;</p>
<pre><code class="lang-python">def ptp(input: Tensor, dim: Optional[int] = None, keepdim=False) -&gt; Tensor:
    if dim is None:
        return input.max() - input.min()
    return input.max(dim, keepdim).values - input.min(dim, keepdim).values

x = torch.tensor([[4, 9, 2, 10], [6, 9, 7, 12]])
ptp(x, 1, True), ptp(x, 1), ptp(x, 0), ptp(x)
...
(tensor([[8],
         [6]]),
 tensor([8, 6]),
 tensor([2, 0, 5, 2]),
 tensor(10))
</code></pre>