akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Please help with implementing GradCAM

Tofzik19: <p>Hi everyone,</p>
<p>I have been trying to implement the PyTorch GradCAM <a href="https://github.com/jacobgil/pytorch-grad-cam" rel="noopener nofollow ugc">here</a></p>
<p>My code is below:</p>
<pre><code class="lang-auto"># Construct the CAM object once, and then re-use it on many images:
cam = GradCAM(model=effnet_model, target_layers=target_layers, use_cuda=True)

# In this example, we want to generate a GradCAM visualization for each image in the input tensor,
# so we set targets to be a list of length 64, where each element is a BinaryClassifierOutputTarget with
# a target class of 0
targets = [BinaryClassifierOutputTarget(1) for _ in range(64)]

grayscale_cam = cam(input_tensor=input_tensor, targets=targets)

# In this example, grayscale_cam has 64 images in the batch, one for each image in the input tensor
# We will iterate over each image and generate a visualization for it
for i in range(grayscale_cam.shape[0]):
    grayscale_cam_i = grayscale_cam[i, :]
    visualization = show_cam_on_image(rgb_img[i], grayscale_cam_i, use_rgb=True)

</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>My input tensor code looks this way:</p>
<pre><code class="lang-auto">from pytorch_grad_cam import GradCAM, HiResCAM, ScoreCAM, GradCAMPlusPlus, AblationCAM, XGradCAM, EigenCAM, FullGrad
from pytorch_grad_cam.utils.model_targets import BinaryClassifierOutputTarget
from pytorch_grad_cam.utils.image import show_cam_on_image


target_layers = [effnet_model.features[-1]]
input_tensor,label = next(iter(train_loader))
input_tensor,label = torch.tensor(input_tensor), torch.tensor(label)
input_tensor.to(device), label.to(device)

</code></pre>
<p>Please any thoughts on the problem? Thanks</p>

Tofzik19: <p>Just wanted to bump this in case anyone had ideas. Thanks</p>
Question about BatchNorm2d

Jack_Shi: <p>A naive question about how BatchNorm2d works. Say I have an input size of</p>
<pre><code class="lang-auto">im = torch.rand(1000,1,2,2)
print(im.shape)
torch.Size([1000, 1, 2, 2])
</code></pre>
<p>i.e. a batch size of 1000 2x2 matrices.<br>
Now say I initialize a BatchNorm2d layer and apply to my randomly generated tensor</p>
<pre><code class="lang-auto">m = nn.BatchNorm2d(1, affine=False,eps=0)
im = m(im)
</code></pre>
<p>where 1 corresponds to the number of channels, or the <code>num_features</code> in BatchNorm2d. I would expect <code>torch.mean(im,0)</code> to be a 2x2 matrix that is essentially 0 and <code>torch.var(im,0)</code> to be a 2x2 matrix of 1. However, when I run the simple few lines above, I get</p>
<pre><code class="lang-auto">In [102]: torch.mean(im,0)
Out[102]: 
tensor([[[ 0.0650, -0.0563],
         [ 0.0067, -0.0154]]])
</code></pre>
<p>and</p>
<pre><code class="lang-auto">In [103]: torch.var(im,0)
Out[103]: 
tensor([[[0.9965, 1.0232],
         [1.0066, 0.9700]]])
</code></pre>
<p>Those results are not what I have expected. I’m sure I misunderstand how BatchNorm2D works, but where did I make the mistake?</p>

ptrblck: <p>You can see a manual (and slow) reference implementation in <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">this code snippet</a> which I’ve created some time ago.<br>
As can be seen there, the proper check would be:</p>
<pre><code class="lang-python">im.mean([0, 2, 3])
Out[55]: tensor([-5.6028e-08])

im.var([0, 2, 3], unbiased=False)
Out[58]: tensor([1.0000])
</code></pre>

Jack_Shi: <p>I see, that makes sense. Thank you for the reference!</p>
Training RNN with indirect loss function

hao_chen: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a44fb1b51f5554c9bb28f28ec56656fa4189a75a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a44fb1b51f5554c9bb28f28ec56656fa4189a75a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a44fb1b51f5554c9bb28f28ec56656fa4189a75a_2_690x229.png" alt="image" data-base62-sha1="nrz32PWczLy0DjNzhiaco15UM8W" width="690" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a44fb1b51f5554c9bb28f28ec56656fa4189a75a_2_690x229.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a44fb1b51f5554c9bb28f28ec56656fa4189a75a_2_1035x343.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a44fb1b51f5554c9bb28f28ec56656fa4189a75a_2_1380x458.png 2x" data-dominant-color="BCBB03"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1889×627 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am facing a similar problem with a post ( <a href="https://discuss.pytorch.org/t/how-to-backpropagate-a-black-box-generated-cost-value/60422">How to backpropagate a black box generated cost value - autograd - PyTorch Forums</a> ). However, the blackbox is 90% known in advnace.</p>
<p>I apply RNN (LSTM/ or GRU or TFT) to generate driving signals, however, there is no label data(target signal) to calculate loss function directly. However, there is one-to-one relationship (pre-known with certain unknown noises) between the driving signals and the response signal. The response signal has target data to generate loss function.</p>
<p>I am wondering to applied decoder (another LSTM to model the relation-ship between driving signal and the response), however, this might increase the complexity (since the relationship between driving signal and the response is 90% known in advance).</p>
<p>Now I am learning the autograd of Pytorch, hoping that the parameters of RNN could be auto-corrected by the indirect loss function. I try to make the response signal to be (requires_grad=True), however, it does not update the model at all, since the loss keeps all the same during training.</p>
<p>Any suggestion?<br>
Best regards.</p>
Possible to do weight updates in forward pass?

ChristianV: <p>Is there a way to have each layer in a network class do a weight update during its forward function? I am developing local learning algorithms and they need to update weights at every layer, and they don’t require gradient descent.</p>
<p>I dont see anything about doing this on the documentation. Is there a way to do this, or would I be better off with a different framework?</p>

ptrblck: <p>I’m not sure if I completely understand your use case, so feel free to correct me in case I’m missing something.</p>
<aside class="quote no-group" data-username="ChristianV" data-post="1" data-topic="168105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christianv/48/55617_2.png" class="avatar"> ChristianV:</div>
<blockquote>
<p>and they don’t require gradient descent.</p>
</blockquote>
</aside>
<p>I assume this means you are never calling <code>.backward()</code> on any tensor associated with the model’s output tensor(s). In this case, your parameter update logic would compute the actual updates or new parameters via another approach.<br>
If so, you should be able to manipulate the parameters in the <code>forward</code> directly by copying the new data to the corresponding parameter. Something like this would work:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x):
    ...
    new_param = self.calculate_new_parameter(self.param)
    with torch.no_grad():
        self.param.copy_(new_param)
    ...
</code></pre>
<p>Note that I explicitly wrap the copy into a <code>no_grad()</code> block so that Autograd will ignore it.<br>
If you are not interested in Autograd at all, you could also disable it globally via <code>torch.autograd.set_grad_enabled(False)</code> and would not need this context manager anymore.</p>

ChristianV: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="168105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>explicitly wrap the copy into a <code>no_grad()</code> block so that Autograd will ignore it.<br>
If you are not interested in Autograd at all, you could also disable it globally via <code>torch.autograd.set_grad_enabled(False)</code> and would not need this context manager anymore.</p>
</blockquote>
</aside>
<p>Thanks for the quick reply. I think this is what I wanted to do. Basically, calculating new params plus outputs and then updating the weight param and still have the output go into the next layer.</p>
<p>I am researching Hebbian learning methods and have been working with my own scuffed framework. Thanks for the input.</p>
Separate last fully connected layer weights

siabdullah4: <p>Is there any way of separating the final fully connected layer weights after a few local epochs of training?  How could I append them into a vector?</p>

siabdullah4: <pre><code class="lang-auto">def train(self, is_print = False):
        self.net.to(self.device)
        self.net.train()
        
        optimizer = torch.optim.SGD(self.net.parameters(), lr=self.lr, momentum=self.momentum, weight_decay=0)
        
        epoch_loss = []
        for iteration in range(self.local_ep):
            batch_loss = []
            for batch_idx, (images, labels) in enumerate(self.ldr_train):
                images, labels = images.to(self.device), labels.to(self.device)
                self.net.zero_grad()
                #optimizer.zero_grad()
                log_probs = self.net(images)
                loss = self.loss_func(log_probs, labels)
                loss.backward() 
                        
                optimizer.step()
                batch_loss.append(loss.item())
            


            epoch_loss.append(sum(batch_loss)/len(batch_loss))
         
        
        return sum(epoch_loss) / len(epoch_loss)
</code></pre>
How to lower the last dimension of a Tensor?

kynthesis: <p>Hi everyone, I have an immature question.</p>
<p>For example, I got a tensor with the size of: torch.Size([2, 1, 80, <strong>64</strong>]).<br>
I need to turn it into another tensor with the size of: torch.Size([2, 1, 80, <strong>16</strong>]).<br>
Are there any right ways to achieve that? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I don’t think there is a single “right way” to achieve this as the proper approach would of course depend on your use case.<br>
You could try these approaches and I’m sure I might miss others:</p>
<ul>
<li>slice the tensor</li>
</ul>
<pre><code class="lang-python">y = x[..., :16]
print(y.shape)
# torch.Size([2, 1, 80, 16])
</code></pre>
<ul>
<li>index it with a stride of 4</li>
</ul>
<pre><code class="lang-python">y = x[..., ::4]
print(y.shape)
# torch.Size([2, 1, 80, 16])
</code></pre>
<ul>
<li>use any pooling (max, avg etc.) layer (the same would also work using adaptive pooling layers):</li>
</ul>
<pre><code class="lang-python">pool = nn.MaxPool2d((1, 2), (1, 4))
y = pool(x)
print(y.shape)
# torch.Size([2, 1, 80, 16])

pool = nn.AdaptiveAvgPool2d(output_size=(80, 16))
y = pool(x)
print(y.shape)
# torch.Size([2, 1, 80, 16])
</code></pre>
<ul>
<li>or manually reduce the last dimension with any reduction op (sum, mean, max, …)</li>
</ul>
Pytorch Geometric Temporal: confusion concerning input shape of node features required for recurrent graph convolutional layers

dbisig: <p>I’ve recently started to experiment with Pytorch Geometric Temporal. I’m trying to train a model consisting of recurrent graph neural network layers (e.g. GConvGRU or GConvLSTM) on the MTM Dataset with the purpose of eventually performing sequence continuation. I’m now stuck with what seems my incomprehension how to properly configure these layers for specific shapes of the input features.</p>
<p>At the moment, the constructor of the layer is called with an input channel count of 3 (corresponding to the dimension of the joint positions), an arbitrary output channel count such as 16, and a similarily arbitrary filter size of 2.</p>
<p>It seems that no matter how a prepare the shape of the input features, whenever I call the forward function of this layer, I receive an error message that either points to a mismatch between matrices or an index that is out of bounds.</p>
<p>When using a feature shape of 3, 21, 128 (which is what the dataset returns by default when obtained for 128 frames), the error message is as follows:</p>
<p>File “C:\Users\dbisig\anaconda3\envs\pytorch112_geomtemp\lib\site-packages\torch_geometric\nn\dense\linear.py”, line 118, in forward<br>
return F.linear(x, self.weight, self.bias)<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (63x128 and 3x16)</p>
<p>When using a feature shape of 21, 3, 128 (by moving axis 0 and 1) the error message is as follows:</p>
<p>File “C:\Users\dbisig\anaconda3\envs\pytorch112_geomtemp\lib\site-packages\torch_scatter\scatter.py”, line 21, in scatter_sum<br>
return out.scatter_add_(dim, index, src)<br>
RuntimeError: index 3 is out of bounds for dimension 0 with size 3</p>
<p>Can somebody explain to me how to do this properly? Or point me to example code that works with the MTM Dataset?</p>
<p>Thanks a lot and best regards</p>
<p>Daniel</p>
Contradiction between Val_loss and Val_PSNR

shaki: <p>I have a question about the EDSR code (it’s a single image super resolution method). Why there’s a contradiction between Val_loss and Val_PSNR in training? in some epochs, from one epoch to the next epoch when Val_loss decreases, we expect Val_PSNR to increase, but this does not happen and vice versa. Is this normal? what’s the reason? If not, what is the solution?</p>
<p>I tried to implement PSNR and Loss function (mae)  in such a way that they are compatible with each other, but The problem was not solved.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> I will be glad to know your opinion.</p>
<p>ps: My code is not in Pythorch. I just want to know your opinion, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Vectorization and broadcasting

milan_kalkenings: <p>i have two tensors:<br>
<code>alpha.size() -&gt;[3]</code><br>
<code>image.size() -&gt; [16, 3, 28, 28]</code>.</p>
<p>I want to elementwise add <code>alpha[i]</code> to all elements in <code>image[16, i, 28, 28]</code>.</p>
<p>what is the (in best case most efficient) vectorized way to do so?</p>

tiramisuNcustard: <p>In general, you can add two n-dimensional vectors where n_1 == n_2. Element-wise addition where n_1 != n_2 is not possible, unless you pad the shorter vector with 0’s and force n_1 == n_2.</p>

milan_kalkenings: <p>i want to broadcast alpha[i] to the size [16,1,32,32] and add it to the ith channel of the image. don’t want to pad here. is there a vectorized way to achieve that?</p>

KFrank: <p>Hi Milan!</p>
<aside class="quote no-group" data-username="milan_kalkenings" data-post="1" data-topic="168093" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/milan_kalkenings/48/42748_2.png" class="avatar"> milan_kalkenings:</div>
<blockquote>
<p>I want to elementwise add <code>alpha[i]</code> to all elements in <code>image[16, i, 28, 28]</code>.</p>
</blockquote>
</aside>
<p>Yes, you may use broadcasting to “vectorize” your element-wise addition.<br>
You just need to use <code>unsqueeze()</code> to add the singleton dimensions that<br>
will then get broadcast:</p>
<pre data-code-wrap="python"><code class="lang-python">image + alpha.unsqueeze (-1).unsqueeze (-1)
</code></pre>
<p>Equivalently, I sometimes find the index-with-<code>None</code> syntax stylistically<br>
more readable, depending on the use case:</p>
<pre data-code-wrap="python"><code class="lang-python">image + alpha[:, None, None]
# or
image + alpha[None, :, None, None]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Torchaudio modules

hossein: <p>Hello everyone,<br>
I have several questions regarding touchaudio modules:</p>
<ol>
<li>There is a hyperparameter named n_fft in transform classes (for example stft, spectrogram,…). What is this exactly?! (this quantity is mostly referred to as freq bins). Is it the number of data points (from sound wave) that can be transformed?</li>
<li>If win_length &gt; n_fft, there might be zero padding. But what will be the case for the opposite situation?</li>
<li>When I perform <code>T.spectrogram</code> on a vector (sound wave), it returns a matrix. The last dimension of this matrix is n_frames. What is this and how can we compute this number?</li>
</ol>
<p>Many thanks already.</p>
CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Ali_Elhenidy20222: <p>I train Deformable DETR on crowd human dataset and got this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b5940a806ffd4136be1fc49c0b57aa726f9b7b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b5940a806ffd4136be1fc49c0b57aa726f9b7b6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b5940a806ffd4136be1fc49c0b57aa726f9b7b6.png" alt="image" data-base62-sha1="6btKKf6b5c0XKIU7EqJfkZZzo34" width="690" height="339" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1084×533 31.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help …?!</p>

ptrblck: <p>Based on the stacktace and error message it seems that an indexing operation is failing. Rerun the code via <code>CUDA_LAUNCH_BLOCKING=1</code> and check which operation exactly fails. Then check the min. and max. values of the index tensor as well as the shape of the tensor which should be indexed.</p>

Ali_Elhenidy20222: <p>I run the code through command line not a jupyter and this is the training command<br>
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25</p>
<p>How I could add the CUDA_LAUNCH_BLOCKING=1 flag</p>

ptrblck: <p>Add this env variable in front of the terminal command as:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25
</code></pre>
<p>or <code>export</code> it in your current terminal:</p>
<pre><code class="lang-auto">export CUDA_LAUNCH_BLOCKING=1
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25
</code></pre>

Ali_Elhenidy20222: <p>I am running the code on windows platform (anaconda), so i guess this bash command isn’t supported.<br>
is there any alternative ?</p>

sloth612: <p>maybe you can use this code</p>
<pre><code class="lang-auto">os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
How can I use stn as crop and resize?

be7f984b2f2e66ba7969: <p>I want to use stn to crop a patch and resize to the specific size.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1689b0bd1f773b4e6e66a3a17bb180e7a3d46039.png" alt="image" data-base62-sha1="3dnvhQ3v7Wt4AweiMEv1vqkRKg9" width="490" height="252"></p>
<p>This is STN, we can draw and visualize the cropped bounding box.</p>
<p>given the transformation matrix, similar effect  can be done in opencv</p>
<p>original image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ef3d250f0cd2918ee9d5dd9c6306739ec5af2e52.jpeg" alt="0001_01" data-base62-sha1="y8p7hBzJcwTyPuY5ATrguyyg7KO" width="112" height="112"></p>
<pre><code class="lang-auto">src_rgb = cv2.imread('./0001_01.jpg')
Affine_Mat_w = [1., 0, -32]
Affine_Mat_h = [0, 1., -32]
M = np.c_[ Affine_Mat_w, Affine_Mat_h].T
res = cv2.warpAffine(src_rgb, M, (48, 48))
</code></pre>
<p>crop and resize</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a4080cee7e5e386d45cfd2bd3cdafe296d222a57.jpeg" alt="r" data-base62-sha1="np5yiT1GyxZ9gmvTMmFXrwkhIvJ" width="48" height="48"></p>
<p>Although pytorch provide  grid_sample and affine_grid, but it does not act like opencv</p>
<pre><code class="lang-auto">src_rgb = Image.open('./0001_01.jpg')
trans = transforms.ToTensor()
trans1 = transforms.ToPILImage()
grid = torch.nn.functional.affine_grid(M, (1, 3, 48, 48))
y = torch.nn.functional.grid_sample(trans(src_rgb).unsqueeze(dim=0).float(), grid.float())
plt.imshow(convert_image_np(y))
plt.show()
</code></pre>
<p>it’s blur and leave many black area.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2a9c32125f15adebd3fc1957aff497a620c67c99.png" alt="Figure_1" data-base62-sha1="64WHtH17DvBqDto7EcXuN2tDuPL" width="640" height="480"></p>
<p>Any idea?</p>

harryhan618: <p>Have you figure out how to crop the image with <code>F.affine_grid</code> ?</p>

edgarriba: <p>you can use crop_and_resize from kornia<br>
<a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.crop_and_resize" class="onebox" target="_blank" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.crop_and_resize</a></p>

whiletrue: <p>but is using this function still differentiable?</p>

edgarriba: <p>You can backprop gradients, yes</p>
Session crashed for an unknown reason

nastaranmarzban: <p>Hi, hope you’re doing well.<br>
Although I updated my torch version and cuda by</p>
<p>!pip install -q torch-scatter -f <a href="https://data.pyg.org/whl/torch-1.13.0+cu116.html" rel="noopener nofollow ugc">https://data.pyg.org/whl/torch-1.13.0+cu116.html</a><br>
!pip install -q torch-sparse -f <a href="https://data.pyg.org/whl/torch-1.13.0+cu116.html" rel="noopener nofollow ugc">https://data.pyg.org/whl/torch-1.13.0+cu116.html</a><br>
!pip install torch-geometric</p>
<p>, my session crashed when I import torch_geometric.</p>
<p>Would u please help me? where is the problem?</p>

ptrblck: <p>Try to rerun the <code>import</code> statement from your terminal with <code>gdb</code> and post the stacktrace here:</p>
<pre><code class="lang-python">gdb --args python script.py
...
run
...
bt
</code></pre>

nastaranmarzban: <p>excuse me, I did not understand what do you mean by</p>
<p>gdb --args python script.py<br>
…<br>
run<br>
…<br>
bt</p>
<p>If you have time, would u please explain it a bit?</p>

ptrblck: <p>These are commands that you should execute in your (Linux) terminal to create the backtrace of the crash in your application.</p>

nastaranmarzban: <p>Thanks, my problem is solved</p>

ptrblck: <p>What was the issue and how did you solve it?</p>

nastaranmarzban: <p>To be honest, I don’t know where the problem was…</p>
<p>I opened a new notebook and copied all the things from the previous notebook into the new one and I saw it worked correctly in the new notebook…<br>
So I decided to close my google colab for 1 hour. After that I ran my previous notebook and I saw it works well even in it.<br>
I really don’t know what had happened, not to mention, that I restarted the run time  and even closed the google colab many times, but it didn’t work well, surprisingly, it worked well after 1 hour that I had closed my google colab. It’s a bit strange.</p>

nastaranmarzban: <p>hi<br>
I wanna say how to find my problem, I’ve just found it.<br>
I’ve used !pip uninstall torch scatter and !pip uninstall torch sparse<br>
and then install pytorch geometric, this time it works well.</p>
Add a new layers to a trained model

tom7: <p>I have a (example) model shown in the left of a picture. The model is already trained, and I have a model_state_dict file.</p>
<p>I actully would like to add new layers to this trained model, which is shown in the right of the picture, where I added new output by concatinating two different features by scaling the dimension of the second feartures.</p>
<p>if I write code, which is something like</p>
<pre><code class="lang-auto">    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.features1(x)
        out1 = x
        scale= out1.shape[2] # take a scaling factor

        x = self.features2(x)
        out2 = F.interpolate(x, (scale), mode='nearest') # scale the size of features
        outA = torch.cat([out1, out2], dim=1) # concat two kinds of features

        x = self.features3(x)
        #return x
        return x, outA
</code></pre>
<p>So, <strong>I’d like to prepare the new model (right one) without any training.</strong> I guess that the new layers in the model do not include any new trainable paramters, where the scaling parameter is required, which is already determined by the dimentions of the features.</p>
<p>So, I guess that the model_state_dict file trained with the former model (left one) could be loaded into the new model although the name of the layer must be modified properly.</p>
<p>Then, I guess that the model could get availble.</p>
<p>Is this correct? Are there any useful information on this kind of transformation/displacement?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/398c0df20cb99249771d9d0d5bebfef690bad678.png" data-download-href="https://discuss.pytorch.org/uploads/default/398c0df20cb99249771d9d0d5bebfef690bad678" title="スクリーンショット 2022-12-11 003716"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/398c0df20cb99249771d9d0d5bebfef690bad678_2_500x500.png" alt="スクリーンショット 2022-12-11 003716" data-base62-sha1="8d5hqIox4SlRwYwq04hZQTpkPtm" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/398c0df20cb99249771d9d0d5bebfef690bad678_2_500x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/9/398c0df20cb99249771d9d0d5bebfef690bad678.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/398c0df20cb99249771d9d0d5bebfef690bad678.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">スクリーンショット 2022-12-11 003716</span><span class="informations">652×652 67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your description sounds correct. The <code>state_dict</code>s would be compatible as long as no submodule or parameter names are changes. Adapting the <code>forward</code> method to your needs is also valid and doesn’t break the <code>load_state_dict</code> usage.</p>

tom7: <p>I see, thank you for your comment, I’ll try it.</p>
Can ImageFolder be used to load binary image data?

tiramisuNcustard: <p>As the title says, is it possible to load binary image data with ImageFolder? The documentation doesn’t mention anything in specific about this. Thanks in advance.</p>

ptrblck: <p>Yes, this is possible and the <code>ImageFolder</code> will then return target values in <code>[0, 1]</code>.<br>
Depending on your used criterion you might need to double check if the shape is correct and manipulate it if needed.</p>
Error with jit script

kaoutar: <p>i am trying to save a model after training using JIT, but it throws me this error:</p>
<pre><code class="lang-auto">NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 147
    def forward(self, *inputs, **kwargs):
                                ~~~~~~~ &lt;--- HERE
        with torch.autograd.profiler.record_function("DataParallel.forward"):
            if not self.device_ids:
</code></pre>
<p>here’s the model,</p>
<pre><code class="lang-auto">class custom_model(nn.Module):
    def __init__(self):
        super().__init__()
        self.base = base_model
        self.mean_max_pool = MeanMaxPool(hidden_size)
        self.dropout = nn.Dropout(0.1)
        self.fc = nn.Linear(hidden_size*3,inter_size[0])
        self.out = nn.Linear(inter_size[0],n_classes)

    def forward(self, ids,mask):
        x = self.base(ids, mask)[0]
        x = self.mean_max_pool(x,mask)
        x= self.fc(x)
        x=self.dropout(x)
        out=self.out(x)
        return out

</code></pre>
<pre><code class="lang-auto">model = custom_model()
model.to(device)
model= nn.DataParallel(model,device_ids=[0, 1])
train_function(model)

model_scripted=torch.jit.script(model)
model_scripted.save('model_scripted.pt') 
</code></pre>

ptrblck: <p>Could you try to <code>script</code> the model before wrapping it into <code>DataParallel</code>?</p>

kaoutar: <p>Thank you for replying, it throws a different error:</p>
<pre><code class="lang-auto">UnsupportedNodeError: GeneratorExp aren't supported:
  File "/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1538
        activations".
        """
        return any(hasattr(m, "gradient_checkpointing") and m.gradient_checkpointing for m in self.modules())
                   ~ &lt;--- HERE
</code></pre>

ptrblck: <p>It seems the error is raised by the <code>transformers</code> package and I don’t know what known scripting limitations there are, so you might need to ask in the HuggingFace discussion board.</p>
Questions about optimizer behavior in the model with moduledict

ChineseBest: <p>Hi, I wonder if I use my optimizer in a data structure containing module dict (for example)</p>
<p>{key1:m1, key2:m2, key3:m3}</p>
<p>and my training process is like:</p>
<pre><code class="lang-auto">for epoch in ...:
    for key in keys:
        output = model\[key\](x) #that is because for one key, I only train certain layer in this model.
        loss = f(output, true)
        loss.backward()
        optimizer.step()
</code></pre>
<p>It is ok for me to train all the layers in this model for one epoch? Moreover, I think it is also different to use this approach:</p>
<pre><code class="lang-auto">for epoch in ...:
    loss = 0
    for key in keys:
        output = model\[key\](x) #that is because for one key, I only train certain layer in this model.
        loss += f(output, true)
    loss.backward()
    optimizer.step()
</code></pre>
<p>I am confused about the difference existing in this process. Thanks.</p>

ptrblck: <p>Your code is not properly formatted so I don’t know when the <code>backward</code> calls are used.<br>
However, note that optimizers with internal states (e.g. <code>Adam</code>) would update parameters even if their <code>.grad</code> attributes are set to zero if a valid internal state is available.<br>
You might thus want to consider deleting the <code>.grad</code> attributes via <code>optimizer.zero_grad(set_to_none=True)</code> to avoid it.</p>

ChineseBest: <p>Hi, I modified my data structure to a right format. Now I will try your method.</p>
RPC workers fail to connect on different machines in the same intranet

MengqiYe: <pre><code class="lang-auto"># On worker 0:
import torch
import torch.distributed.rpc as rpc
import os

from torch.distributed.rpc import BackendType

os.environ['MASTER_ADDR'] = '10.8.9.106'
os.environ['MASTER_PORT'] = '10000'
os.environ['GLOO_SOCKET_IFNAME'] = 'br0'
os.environ['TF_SOCKET_IFNAME'] = 'br0'

options = rpc.TensorPipeRpcBackendOptions(
    num_worker_threads=4,
    rpc_timeout=0,
    init_method='tcp://127.0.0.1:10000'
)


# On both workers:
@torch.jit.script
def my_script_add(t1, t2):
    return torch.add(t1, t2)

rpc.init_rpc("worker0", backend=BackendType.TENSORPIPE, rank=0, world_size=2, rpc_backend_options=options)

# RuntimeError: ECONNREFUSED: connection refused
ret = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(2), 3))
rpc.shutdown()
</code></pre>
<pre><code class="lang-auto"># On worker 1:
import torch
import torch.distributed.rpc as rpc
import os

from torch.distributed.rpc import BackendType

os.environ['MASTER_ADDR'] = '10.8.9.106'
os.environ['MASTER_PORT'] = '10000'
os.environ['GLOO_SOCKET_IFNAME'] = 'enp6s0'
os.environ['TF_SOCKET_IFNAME'] = 'enp6s0'
os.environ['NCCL_SOCKET_IFNAME'] = 'enp6s0'

options = rpc.TensorPipeRpcBackendOptions(
    num_worker_threads=4,
    rpc_timeout=0,
    init_method='tcp://10.8.9.106:10000'
)


# On both workers:
@torch.jit.script
def my_script_add(t1, t2):
    return torch.add(t1, t2)


rpc.init_rpc("worker1", backend=BackendType.TENSORPIPE, rank=1, world_size=2, rpc_backend_options=options)

rpc.shutdown()
</code></pre>
<p>Both workers connect successfully on the same machine.<br>
But workers fail to connect over the simplest intranet.<br>
On worker0, it says “RuntimeError: ECONNREFUSED: connection refused”<br>
On worker1, it says “RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [10.8.9.106]:21667”, what confuses me is this peer at a random port. Here 21667 changes every time.</p>
<p>I set port larger than 1023, so it’s not the permission thing.<br>
I also tries gRPC in go programming, which is not relevent much, but it passed between two machines successfully.</p>

mndl1: <p>Hi, did you manage to solve this in the end?</p>
Could not run 'aten::as_strided' with arguments from the 'Metal' backend

maddythakker: <p><strong>Could not run ‘aten::as_strided’ with arguments from the ‘Metal’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::as_strided’ is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Functionalize, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy].</strong></p>
<p>Is there an exhaustive list of operators that are supported with Metal backend?</p>
Questions about creating a shared neural network among different models or layers

ChineseBest: <p>Hi, I intend to create a newtork with different pathes and I believe I can select them to determine which part of the nerwork I intend to train. Here is my current design:</p>
<p>class GCNEncoder_Multiinput(torch.nn.Module):<br>
def <strong>init</strong>(self, out_channels, graph_list, label_list):<br>
super(GCNEncoder_Multiinput, self).<strong>init</strong>()<br>
self.activ = nn.ReLU()</p>
<pre><code>    conv_dict = {}
    for i in graph_list:
        conv_dict[i.show_index] = TransformerConv(i.x.shape[1], out_channels, heads = 2).to(device)
    self.convl1 = conv_dict
    
    conv_dict_l2 = {}
    conv_dict_l3 = {}
    tissue_specific_list = list(set(label_list))
    
    for i in tissue_specific_list:
        conv_dict_l2[i] = TransformerConv(out_channels*2, out_channels).to(device)
        conv_dict_l3[i] = TransformerConv(out_channels, out_channels).to(device)
    self.convl2 = conv_dict_l2
    self.convl3 = conv_dict_l3
        
    
def forward(self, x, edge_index, show_index):
    x = self.convl1[show_index](x, edge_index)
    x = self.activ(x)
    x = self.convl2[show_index.split('__')[0]](x, edge_index)
    x = self.activ(x)
    return self.convl3[show_index.split('__')[0]](x, edge_index)
</code></pre>
<p>However, if I intend to pass the parameters to the optimizer, I received such a error:<br>
ValueError: optimizer got an empty parameter list<br>
How to address this problem? Thanks a lot.</p>

srishti-git1110: <p>Hi,<br>
As I can see, <code>convl1</code> through <code>convl3</code> aren’t getting registered as tunable parameters (these apparently are just python dictionary objects).</p>
<p>Potentially, checkout <a href="https://pytorch.org/docs/stable/generated/torch.nn.ParameterDict.html#parameterdict" rel="noopener nofollow ugc">ParameterDict</a> that behaves similar to a regular python dictionary but ensures that the parameters/layers it contains are properly registered.</p>

ChineseBest: <p>Thanks a lot, I will try it to see its function!!!</p>

ChineseBest: <p>Btw, here I think I should use moduleDict.</p>
How to serialize in TorchScript Beta distribution?

brook: <p>In order to be able to compile with Torchtrace  beta distribition,<br>
am using this code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">class Algo2(torch.nn.Module, ABC):
    def  __init__(self,  ):
        self.aa = [1]*10
        self.bb = [2]*10

    def get_beta_samples(self, algoid=0) -&gt; Tensor:
        one     = torch.ones(1)        
        samples = []
        kk      = 0
        for i, j in zip(one + self.aa[algoid], one + self.bb[algoid]):
            samples.append( torch._sample_dirichlet(torch.tensor([i, j]))[:1] )               
            kk = kk +1

        samples = torch.cat(samples, 0)
        return samples


X1= torch.tensor(6)
trace_model = torch.jit.trace_module(model1, {'get_samples': X1,})
                        
</code></pre>
<p>Output is always the same.</p>
<p>However it says:</p>
<pre><code class="lang-auto">TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.


</code></pre>
<p>How to fix the issue ?</p>
<p>One cannot use torch.distribution.beta  for TorchScript.</p>
Custom Android Yolov5

zozolina123: <p>I am trying to use my trained yolov5 model with the android app example here:<br>
<a href="https://github.com/pytorch/android-demo-app/tree/master/ObjectDetection" rel="noopener nofollow ugc">Android Object Detection Example</a></p>
<p>I figured i have to modify the values inside</p>
<blockquote>
<p>PrePostProcessor.java</p>
</blockquote>
<p>so i updated them according to my model</p>
<pre><code class="lang-auto">    // model input image size
    static int mInputWidth = 640;
    static int mInputHeight = 640;

    // model output is of size 25200*11
    private static int mOutputRow = 25200; // as decided by the YOLOv5 model for input image of size 640*640
    private static int mOutputColumn = 11; // left, top, right, bottom, score and 6 class probability
    private static float mThreshold = 0.50f; // score above which a detection is generated
    private static int mNmsLimit = 15;
    ........................................................
    Result result = new Result(cls, outputs[i*11+4], rect);
</code></pre>
<p>This is how my input and output look like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10b93b88ab2049bb594eb6def68c2fd74712f744.png" alt="inputonx" data-base62-sha1="2nWuU1ccgsKA5Ay4IPV5adTVqsc" width="361" height="285"><br>
My result view always places all predictions in the corner of the image (I cant upload multiple pictures)</p>
<p>Is there anyting else i should change?</p>

xta0: <p>cc <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> could you help take a look? Thanks!</p>

jeffxtang: <p>The changes look good… Can you upload your model here so I can give it a try? Thanks.</p>

zozolina123: <p>Sure, thank you.<br>
<a href="https://drive.google.com/file/d/16AewM-dyZDbUrXiVrPT6LuNZX68e7CPi/view?usp=sharing" rel="noopener nofollow ugc">Model</a></p>
<p>Also, i did get it to work using tflite and this repo:<br>
<a href="https://github.com/zldrobit/yolov5" rel="noopener nofollow ugc">Tflite YOLOv5 Android</a></p>

jeffxtang: <p>With your model and the same changes you made (but the original classes.txt file), I got “airplane 0.34” for the 1st test image, no results for the 2nd one, and “car 0.70” for the 3rd one. Are these the same as what you see? If not can you please upload your test result on the 1st image? I haven’t been able to see what other changes need to be made yet…</p>

Najir_Pacal: <p>Hello, can you please guide me how to convert my YOLOv5 custom model using that pytorch lite interpreter? So that i can integrate it in Android Studio using that demo app also that came from github.</p>
Multi-label Classification with ordinal value

Jae0143: <p>I just faced a problem that I never experienced. My goal is to <strong>predict the number of times they will eat food in cafeteria from the rating (1-5) of each menu</strong> there are total of 34 menus. When I implemented logistic regression or feedforward network to tackle it, it did not work as both the accuracy and loss did not decrease.</p>
<p>Loss function: Cross entropy</p>
<p>for logistic regression, I have 3 hidden layer.</p>
<p>One think I am suspecting is that I have too less data, which is around 200 points. This is kind of prototype but could you guys maybe give some suggestions on tacking this issue.</p>

callmebroogy: <p>Hello Jae,</p>
<p>I have similar task too, right now im just doing simple multiple output regression with MSE as a loss function. How did your research going till now? Could you share what you’ve done in this kind of task?</p>
<p>Thanks</p>

MarkW: <p>I think the discussion in <a href="https://discuss.pytorch.org/t/create-a-variation-of-cross-entropy-loss-with-per-pair-of-classes-weights/157369" class="inline-onebox">Create a Variation of Cross Entropy Loss with Per Pair of Classes Weights</a> might be useful.</p>
Counting FLOPs of a batch normalization layer

H_Jahan: <p>Could you please let me know how I can count the number of flops related to the batch normalization layer theoretically?</p>
<p>FLOPs: Note that s is lowercase, which is the abbreviation of FLoating point OPerations (s stands for plural), which means floating point arithmetic, understood as calculation amount. It can be used to measure the complexity of the model. The evaluation of the complexity of neural network models should refer to FLOPs, not FLOPS.</p>
RuntimeError: "max_cpu" not implemented for 'ComplexFloat'

Nika-Soltani-Tehrani: <p>Hello,<br>
I have a tensor of shape <code>torch.Size([128, 1, 480])</code> and I want to get the maximum value of the tensor. The tensor has float complex numbers and I am running my code on CPU, but the code results in the following error:<br>
RuntimeError: “max_cpu” not implemented for ‘ComplexFloat’<br>
It seems that the <strong>max</strong> function is not implemented for running on <strong>CPU</strong> yet. Is there any alternatives to use instead?<br>
Thanks in advance.in</p>

J_Johnson: <p>You’ll need to define what you mean by max value for complex numbers. You have a few options:</p>
<ol>
<li>Max real value;</li>
<li>Max imaginary value;</li>
<li>Max absolute value or magnitude;</li>
<li>Max complex angle.</li>
</ol>
SURF not exist in opencv-contrib-python

saluei: <p>SIFT and SURF are examples of algorithms that OpenCV calls “non-free” modules. These algorithms are patented by their respective creators, and they are free to use in academic and research settings and installed with this command:</p>
<p>pip install opencv-contrib-python</p>
<p>However after installation, using SURF algorithm cause the error:</p>
<pre><code class="lang-auto">OpenCV(3.4.11) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-neg5amx3\opencv_contrib\modules\xfeatures2d\src\surf.cpp:1029: error: 
(-213:The function/feature is not implemented) This algorithm is patented and is excluded in this
configuration; Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function 
'cv::xfeatures2d::SURF::create'
</code></pre>
<p>the error exist in version 4.6.0.66 of  opencv-contrib-python. this is my code to use SURF</p>
<pre><code class="lang-auto">SURF = cv.xfeatures2d.SURF_create()
kp,des = FREAK.compute(img_gray,kp)
</code></pre>

ptrblck: <p>Your question seems to be unrelated to PyTorch, so I would recommend to post it in an OpenCV discussion board or their GitHub repository.</p>
<p>Based on the error message I guess you might need to rebuild OpenCV with the mentioned option to enable these patented algorithms as the public version doesn’t seem to ship with it (again I guess also due to licensing issues).</p>

edgarriba: <p>Agree with <a class="mention" href="/u/ptrblck">@ptrblck</a> , you should go to opencv forums. However, if you like pytorch and opencv you can use Kornia re-implementation of sift found in Kornia</p>
<p><a href="https://kornia.readthedocs.io/en/latest/feature.html#kornia.feature.SIFTDescriptor" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/feature.html#kornia.feature.SIFTDescriptor</a></p>
Model parameters doesn't change when training

Can_Nguyen: <p>Hey guys,</p>
<p>I’m working with a simple model which has to train 1 value only, named <strong><code>self.my_weights</code></strong> in my below code. The only new thing in my experiment is that I have to round that value when forwarding the input.</p>
<p>In my observation, that value doesn’t change anything when training. Assume the loss function is working. Does my “rounding values” action make that tensor un-trainable? Do I have any wrong implementations? Pls help!</p>
<p>This is an example of my network:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.backbone = MobileNetv3(pretrain=True)  #This self.backbone's weights are frozen when training
        self.backbone.classifier = nn.Sequential()
        self.my_weights = nn.Parameter(torch.ones(1, requires_grad=True))

    def forward(self, input):
        feature = self.backbone(input)
        repeat_weights = torch.round(self.features_weights).to(torch.int64)

        final_features = feature.repeat(1, repeat_weights[0]))

        final_features = F.normalize(final_features, dim=1)
        return final_features
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Can_Nguyen" data-post="1" data-topic="167987">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/can_nguyen/48/24555_2.png" class="avatar"> Can_Nguyen:</div>
<blockquote>
<p>Does my “rounding values” action make that tensor un-trainable?</p>
</blockquote>
</aside>
<p>Yes, for two reasons:<br>
Rounding a value will create a step function. While the output will still be attached to the computation graph, the gradient will be zero almost everywhere:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.linspace(0, 10, 1000)
y = torch.round(x)

plt.plot(x.numpy(), y.numpy())
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/089cb2282e9fc5c7b3970077dd8f7e765f25102c.png" alt="image" data-base62-sha1="1ebxqP1lDYZUCGcFLH8UZGMiwKo" width="368" height="248"></p>
<p>Afterwards you are transforming the output to an integer type (<code>torch.int64</code> in this case), which will explicitly detach the result from the computation graph since only floating point tensors are differentiable.</p>

Can_Nguyen: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your magnificent explanation!</p>
<p>I’m working around to find a method to make it works. My purpose is to train a repeat value as the parameter for <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.repeat.html" rel="noopener nofollow ugc">torch.repeat()</a> function.</p>
<p>Unfortunately, <code>torch.repeat()</code> requires an <code>int</code> number for its parameter, and the <code>self.my_weights</code> in my code has to be <code>float</code> tensor to be trainable.</p>
<p>It seems to be unrealizable here!</p>