akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Where is torch.cudnn_affine_grid_generator?

Keke: <p>I want to modify the affine_grid_generator function in torch/nn/_functions/vision.py from 2D to 3D. But I can’t find where is torch.cudnn_affine_grid_generator function. My pytorch version is 0.4.0 and python version is 3.6. The source code is shown below. Thanks in advance!</p>
<p>def affine_grid_generator(theta, size):<br>
if theta.data.is_cuda:<br>
if not cudnn.enabled:<br>
raise RuntimeError("AffineGridGenerator needs CuDNN for "<br>
“processing CUDA inputs, but CuDNN is not enabled”)<br>
if not cudnn.is_acceptable(theta.data):<br>
raise RuntimeError(“AffineGridGenerator generator theta not acceptable for CuDNN”)<br>
N, C, H, W = size<br>
return torch.cudnn_affine_grid_generator(theta, N, C, H, W)<br>
else:<br>
return AffineGridGenerator.apply(theta, size)</p>
<p>def affine_grid_generator_3d(theta, size):<br>
return AffineGridGenerator_3d.apply(theta, size)</p>

tom: <p>It’s in the C++ modules (ATen to be precise).</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/AffineGridGenerator.cpp" rel="nofollow noopener">aten/src/ATen/native/cudnn/AffineGridGenerator.cpp</a></li>
<li>found via <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml" rel="nofollow noopener">aten/src/ATen/native/native_functions.yaml</a>.</li>
</ul>
<p>Personally, I tend to use the “rgrep” command on the PyTorch source tree to find functions.</p>
<p>Best regards</p>
<p>Thomas</p>

Mohamed_Hassan: <p>Hi <a class="mention" href="/u/keke">@Keke</a><br>
Did you solve this issue. I want to do exactly the same.</p>

teaghan: <p>I am also interested in a solution to this. <a class="mention" href="/u/keke">@Keke</a> did you happen to build a 3D affine grid generator?</p>

oldriver-wang: <p>I have the same question! How did you do!</p>

gauenk: <p>This looks like it points to cudnn code, so I can’t access the source. Does anyone know where the actual code is stored, maybe for the cpu version? I want to see the code itself for my experiments.</p>

tom: <p>Yeah, well. The question was about cudnn_affine_grid_generator.</p>
<p>Without CuDNN, by applying the same method to search, it turns out that <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/AffineGridGenerator.cpp" rel="noopener nofollow ugc"><code>/aten/src/ATen/native/AffineGridGenerator.cpp</code></a> is the code that is used, independent of CPU vs. CUDA.</p>
How to validate and select a 'best' model?

morphism: <p>Hello. everyone.</p>
<p>In deep learning and PyTorch, to select the ‘best’ model, what is the best strategy to validate a model??<br>
K-fold cross validation?<br>
Monte Carlo cross validation?</p>
<p>I will appreciate much if you can explain and introduce the validation process to me<br>
Thank you in advance and have a nice day</p>
How to get hidden values of hidden layers

kenny.smith1339: <p>I am new to pytorch, and i am trying to implement Resnet, and get the hidden layer of the input in every layer.<br>
How can I do that?<br>
The problem is that after adding layers to nn.sequential. I don’t have a way to get the hidden of each layer (not each block, but each layer inside each block)</p>

J_Johnson: <p>One way to get the values is by iterating over the model. I usually do something like this:</p>
<pre><code class="lang-auto">temp=[]
for layer in model.parameters():
    temp.append(layer)

print(temp) 
</code></pre>
<p>If layers are in blocks, you may need to add a second nested iterator.</p>
Slope of the Loss function

NikosKr: <p>To get the slope of the Loss function one has to get the derivative. I ve tested the following code with no success.<br>
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time<br>
for i in range(training_iter+1):<br>
optimizer.zero_grad()<br>
x_ = train_x.detach().requires_grad_(True)<br>
output = model(train_x)<br>
loss = -mll(output,train_y)<br>
dy_dx = torch.autograd.grad(loss.mean(), x_,allow_unused=True)<br>
loss.backward()</p>
<p>loss.backward(retain_graph=True) does not resolve the issue</p>

marvosyntactical: <p>If you want to access the derivative dl/dx (the jacobian)) you can access x_.grad after loss.backward(), as this attribute is set after specifying requires grad = True and backpropping. I did not test your code with the detach() on x_ and would simply write x_ = train_x.requires_grad_(True); as x_ needs to be in the graph anyways.</p>

albanD: <p>If you call autograd.grad or backward multiple times, then in the first call, you have to specify <code>retain_graph=True</code>.<br>
In your sample above, that means changing the autograd.grad call to:<br>
<code> dy_dx = torch.autograd.grad(loss.mean(), x_,allow_unused=True, retain_grad=True)</code></p>

NikosKr: <p>Sorry for late reply as my gmail missed Pytorch notifications.<br>
Thanks for your help.<br>
The  following worked<br>
for i in range(training_iter+1):<br>
optimizer.zero_grad()<br>
train_x= train_x.requires_grad_(True)<br>
output = model(train_x)<br>
loss = -mll(output,train_y)<br>
loss.backward(retain_graph=True)<br>
dy_dx = torch.autograd.grad(loss.mean(), train_x,allow_unused=True, retain_graph=True)</p>
Control Tensor Memory Allocation?

Hiperdyne19012: <p>Is there a way to control how memory is allocated, deleted and reallocated for Tensors? Is very any doc or guide on how Torch memory allocation happens behind the scenes? I am on Linux, using Libtorch.</p>

dfalbel: <p>The CPU memory allocator is implemented here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/e26fccc22b1f1fa28bceb076567d3e53c92a4c77/c10/core/CPUAllocator.cpp#L100-L120" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/e26fccc22b1f1fa28bceb076567d3e53c92a4c77/c10/core/CPUAllocator.cpp#L100-L120" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/e26fccc22b1f1fa28bceb076567d3e53c92a4c77/c10/core/CPUAllocator.cpp#L100-L120</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="100" style="counter-reset: li-counter 99 ;">
<li>struct C10_API DefaultCPUAllocator final : at::Allocator {</li><li>  DefaultCPUAllocator() {}</li><li>  ~DefaultCPUAllocator() override {}</li><li>  at::DataPtr allocate(size_t nbytes) const override {</li><li>    void* data = alloc_cpu(nbytes);</li><li>    profiledCPUMemoryReporter().New(data, nbytes);</li><li>    return {data, data, &amp;ReportAndDelete, at::Device(at::DeviceType::CPU)};</li><li>  }</li><li></li><li>  static void ReportAndDelete(void* ptr) {</li><li>    if (!ptr) {</li><li>      return;</li><li>    }</li><li>    profiledCPUMemoryReporter().Delete(ptr);</li><li>    free_cpu(ptr);</li><li>  }</li><li></li><li>  at::DeleterFnPtr raw_deleter() const override {</li><li>    return &amp;ReportAndDelete;</li><li>  }</li><li>};</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>AFAICT you can control how memory is allocated by implementing another <code>at::Allocator</code> and registering it.</p>
Access Violation(c10_cuda.dll)

sid1: <p>Hi, I am new to c++ frontend. I am trying to implement DDPG algorithm using c++ frontend(Release) and Mujoco. The code is a bit complex but in brief I am doing something like this.</p>
<pre><code class="lang-auto">#include "stdio.h"
#include "stdlib.h"
#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;vector&gt;

using namespace std;

int main()
{
    torch::Device device(torch::kCPU);
    if (torch::cuda::is_available()) 
    {                
        device = torch::Device(torch::kCUDA);
        cout&lt;&lt;device&lt;&lt;endl;
    }
    double sensordata[]={0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5};

    for(auto epoch=0; epoch&lt; 10000; epoch++)
    {
        vector&lt;float&gt; v1(sensordata,sensordata+15);
        cout&lt;&lt;"trying to create a tensor for: "&lt;&lt;epoch&lt;&lt;" time"&lt;&lt;endl;
        torch::Tensor a = torch::tensor(v1).to(device);
        cout&lt;&lt;"success"&lt;&lt;endl;

        for(auto time=0; time&lt;10000; time++)
        {
            vector&lt;float&gt; v2(sensordata,sensordata+15);
            torch::Tensor b = torch::tensor(v2).to(device);
            //do something
            a=b;
        }
    }
    return 0;
} 
</code></pre>
<p>At any random epoch iteration the tensor “a” fails to create and I get an error message: Finished in "x"s with exit code 3221225477<br>
Upon running it on visual studio I get the error message: “Unhandled exception at 0x00007FFC979C0990 (c10_cuda.dll) in example-app.exe: 0xC0000005: Access violation reading location 0x0000000000000019.”<br>
Unfortunately The error cannot be replicated using the above code.<br>
In the actual code the “sensordata” array is initialized by mujoco simulator. I suspect that mujoco uses multithreading on CPU in background and libtorch does asynchronous initialization of the tensors on GPU, so maybe the tensor “a” is tried to be initialized before “sensordata” array can be initialized.<br>
I tried using “cudaDeviceSynchronize()” and “.to(device,/<em>non_blocking=</em>/true)” but ended up getting the same error at a random epoch.</p>
Gradient flow after conversion from tensor to numpy

cbd: <p>To use one function i need to convert pytorch tensor to numpy and want to back propagate gradient after calculating loss which is in numpy array. Is there any way to backpropagate error as i used the “detach” on tensor?</p>

ptrblck: <p>Autograd won’t be able to create the computation graph for the numpy opertations, so you would have to write a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> and implement the <code>backward</code> method manually.</p>
Is my resnet Implementation Right

LostAtlas: <p>Last week i implemented a Resnet Block for my SRGAN code and i was doubtful about one aspect of the implementation.</p>
<blockquote>
<p>When you pass the input through the residual skip connection, do you detach that from the computation graph ( Like var.detach() in Pytorch) or do you let it persist in the graph?</p>
</blockquote>
<p>If so, does this mean the during the backward pass, the input’s weights would be influenced by both the next layer connection and the skip connection?<br>
Below is the code for my implementation in Pytorch:</p>
<pre><code class="lang-auto">class ResnetBlock(nn.Module):
  def __init__(self,kernel_size,stride,n_filters,in_channels):
    super(ResnetBlock,self).__init__()
    self.conv1 = nn.Conv2d(in_channels=in_channels,kernel_size=kernel_size,stride=stride,out_channels=n_filters,padding=1,bias=False)
    self.bn = nn.BatchNorm2d(n_filters)
    self.prelu = nn.PReLU()
    self.conv2 = nn.Conv2d(in_channels=n_filters,kernel_size=kernel_size,stride=stride,out_channels=n_filters,padding=1,bias=False)

  def forward(self,x):
    xp = torch.clone(x).detach() #Detached this variable xp and passed it through the skip connection
    x = self.prelu(self.bn(self.conv1(x)))
    x = xp.add(self.prelu(self.bn(self.conv2(x))))

    return x
</code></pre>

ptrblck: <p>I’m not familiar with the usage in SRGAN, but in ResNets the <code>identity</code> used in the skip connections is not detached from the computation graph as seen e.g. <a href="https://github.com/pytorch/vision/blob/09ce6758ad6ab3177f3fdec3b1bb7c7033b16712/torchvision/models/resnet.py#L67-L83">here</a>.</p>
Receiving 'nan' parameters after first optimization step

muhammadirfanzafar: <p>I am using a 5 layers fully connected neural network with tanh() activation function. I using it in PINN model, which has worked fine for several times before. It’s not this time.</p>
<p>When i use<code> torch.autograd.set_detect_anomaly(True)</code>, following error message appears.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-21-268a4603ec9c&gt; in &lt;module&gt;
----&gt; 1 loss.backward()
      2 optimizer.step()

~/.conda/envs/nr_powerai36/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196 
    197     def register_hook(self, hook):

~/.conda/envs/nr_powerai36/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

RuntimeError: Function 'ReciprocalBackward' returned nan values in its 0th output.
</code></pre>
<p>I am not sure how to tackle this error. What exactly is ‘ReciprocalBackward’ function?</p>

ptrblck: <p><code>ReciprocalBackward</code> points towards a division by the tensor:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
y1= 1/x
y2 = torch.reciprocal(x)

print(y1 == y2)
&gt; tensor([True])
print(y2)
&gt; tensor([-1.2178], grad_fn=&lt;ReciprocalBackward&gt;)
</code></pre>
<p>I assume the used tensor might be zero (or close to zero), which would yield <code>Inf</code> as the output and thus also an invalid gradient.<br>
Could you check your model for these operations and make sure the used values are in a reasonable range?</p>
View the images by slicing NCHW to [n0, C, h0, W0]

Praveen_K: <p>Here I’m trying to view my images after performing some <code>DataLoader</code> operation.</p>
<pre><code class="lang-auto">class Classification(Dataset):
    def __init__(self, df,  length, transform=None):
        self.df = df
        self.data_len = len(self.df)
        self.len = length
        self.transform = transform
        

    def __getitem__(self, index):
        data_idx = index % self.data_len
        X = Image.open(self.df['file_path'][data_idx])
        y = torch.tensor(int(self.df['class_name'][data_idx]))
        if self.transform:
            X = self.transform(X)
        
        
        return X, y
        
    def __len__(self):
        return self.len
		
length=2
training_set = Classification(df, length, transform=train_transform)
train_loader = DataLoader(training_set, batch_size=5)

for batch_idx, (inputs, labels) in enumerate(train_loader):   
    print(inputs.shape)     #torch.Size([2, 3, 224, 224])
    inputs = np.squeeze(inputs, axis=0) 
    inputs = inputs.permute(1, 2, 0)
    plt.figure()
    plt.imshow(inputs.numpy())
    plt.show()
</code></pre>
<p>The above code fails because <code>np.squeeze</code> does not work with <code>dim=0</code>, as the value is 2. How can I slice 1 image at a time and plot using for loop ?<br>
I want to check all images. If <code>length=25</code>, then i would like to view 25 images.</p>

ptrblck: <p>Instead of <code>squeeze</code>, which tries to remove a dimension with <code>size=1</code>, you could index the images in the batch in a loop:</p>
<pre><code class="lang-python">for batch_idx, (inputs, labels) in enumerate(train_loader):   
    print(inputs.shape)     #torch.Size([2, 3, 224, 224])
    for idx in range(inputs.size(0)):
        input = inputs[idx]
        input = input.permute(1, 2, 0)
        plt.figure()
        plt.imshow(input.numpy())
        plt.show()
</code></pre>
Torch.cuda.device_count() shows only one gpu in MIG setting

hanbit: <p>Hi, I have an a100 machine and the configuration for MIG is following.</p>
<pre><code class="lang-auto">GPU 0: A100-SXM4-40GB (UUID: GPU-b428bd3e-1cd2-38b1-833a-bae2ac1edf60)
  MIG 7g.40gb Device 0: (UUID: MIG-GPU-b428bd3e-1cd2-38b1-833a-bae2ac1edf60/0/0)
GPU 1: A100-SXM4-40GB (UUID: GPU-67129a8b-fff4-5944-eada-c4f04ef3871b)
  MIG 7g.40gb Device 0: (UUID: MIG-GPU-67129a8b-fff4-5944-eada-c4f04ef3871b/0/0)
GPU 2: A100-SXM4-40GB (UUID: GPU-59efe21f-857a-87de-3d87-74527465b26a)
  MIG 7g.40gb Device 0: (UUID: MIG-GPU-59efe21f-857a-87de-3d87-74527465b26a/0/0)
GPU 3: A100-SXM4-40GB (UUID: GPU-0723161b-142b-001e-e2e5-ba267d31ba0f)
  MIG 7g.40gb Device 0: (UUID: MIG-GPU-0723161b-142b-001e-e2e5-ba267d31ba0f/0/0)
GPU 4: A100-SXM4-40GB (UUID: GPU-0973e5f1-5abc-26c7-d298-baf3bf29c8d0)
GPU 5: A100-SXM4-40GB (UUID: GPU-bfa1ca29-5339-06d7-b9bb-ea503e10f793)
GPU 6: A100-SXM4-40GB (UUID: GPU-9b67e30c-1635-be44-9015-aa650b1a6342)
GPU 7: A100-SXM4-40GB (UUID: GPU-d52b7cce-5485-136e-3f6d-c8184f40bdef)
</code></pre>
<p>and <code>torch.cuda.device_count()</code> shows only one gpu. I need exact <code>world_size</code> to use DDP. How could I solve it?</p>

ptrblck: <p>This is expected behavior, as MIG doesn’t support multiple compute instances subscription for processes.<br>
If no GPUs are in MIG mode, all devices should be visible. Otherwise, if one of the devices is in MIG mode, the GPUs in non-MIG are invisible.</p>
Backpropagation accumulate gradient on a Tensor

Harsh_Pathak: <p>Hi All,<br>
By default, say if a tensor x is used multiple times in a computation graph, and we do back-propagation on this graph, then x.grad stores a sum of the gradients coming from each outgoing edge of x. As an example,</p>
<pre><code class="lang-auto">import torch
x = torch.ones((1), requires_grad = True)
a = torch.ones((1), requires_grad = True)
b = a + x
c = b + x
d = c + x
d.backward()
print(x.grad) // will be 3 (1+1+1)
</code></pre>
<p>However, my use-case requires that I need to fill x.grad with gradient coming from the branch d = c + x (meaning only d to x immediate edge). However, by default, the gradient will also flow from c to x and b to x as well.</p>
<p>Also note, that I cannot detach x while calculating b and c. My use-case involves an LSTM where I am calculating a loss and back-propagation at each time.</p>
<p>Hence, Is there a way to concatenate gradients for x, something like x.grad = [1,1,1] instead?</p>
Faster R-CNN input question

diningeachox: <p>Hi Everyone,</p>
<p>This is my first post here and I’m new to pytorch in general. So I’m wondering if you guys can help me understand the Faster R-CNN documentation: <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.7.1 documentation</a></p>
<p>I’m trying to train an object detection model for heart conditions/anamolies on chest x-rays. I expect a large percentage of the data will be healthy people (i.e. no objects to detect). In this case, what should be the bounding box? As I understand, the torchvision model still expects a nontrivial bounding box even if no object is present, is that right? If so, would I just take the bounding box to be the whole image? Or something else?</p>
<p>Thank you in advance for any tips or assistance.</p>

JamesDickens: <p>The faster rcnn training process expects a percentage of positive anchors as well as negative anchors of the region proposal batch fed to the roi heads, and if the bounding box dicts are empty it will throw an error if I remember correctly.</p>
<p>I think you will have to tweak the Generalized RCNN code as well as the RPN code in torchvision to deal with the scenario of having no anchors with an IOU of a ground truth bounding larger than the threshold. The RPN will still output a batch of region proposals whose objecteness scores are closest to 1.</p>
<p>I encountered this issue since the MS COCO 2014 object detection dataset has some images with no boxes. Most pre-processing I saw simply discarded those images, but in your case that probably doesn’t make sense since you want to be able to output something like “no anomalies” for an image, not to mention the fact that in order to eliminate false positives your model needs a decent amount of healthy person images.</p>
<p>What you want ultimately for an image with no anomalies is for the region proposal network to learn, in these scenarios, that all anchors should have objectness value to 0, and for any of the region proposals output by the rpn, that the classifier head should predict the background class for all of them (where the regression loss for both the rpn and the roi heads are disabled in this case).</p>
<p>Alternatively, what would be an interesting experiment would be to have either</p>
<ol>
<li>
<p>A cascade approach of two models in which the first model computes a binary classification task to see if the image has ANY issues. From there a second model could take images that pass through a confidence threshold from the previous model and compute faster-rcnn like object detection</p>
</li>
<li>
<p>A third network head to the faster-rcnn model that works for the above binary classification task, this is probably not a good idea but it would be fun to try.</p>
</li>
</ol>
<p>I definitely wouldn’t recommend using the whole image as a bounding box. I guess you would give it the label of 0, the background class.</p>
DDP training slow during first epoch

Praveen_Srinivasan: <p>Hi,<br>
I am training a model with DDP w/ 8 GPUs (one process per GPU) and the DataLoader with 8 data loading workers. Training is slow during the first epoch and speeds up significantly immediately starting the second epoch and onwards. Has anyone seen this before?</p>

Dwight_Foster: <p>Yes this is not unusual. The first epoch is always the slowest because everything needs to load onto the gpu at first. After that the model and gradients are loaded on the gpu already so it is much faster.</p>

Praveen_Srinivasan: <p>Do you mean the first batch item? I’m talking about the first pass through the entire dataset.</p>

ptrblck: <p>Is your system using some caching for the data loading?<br>
I’ve seen servers in the past, which load data from the network and are thus using a built-in caching mechanism to avoid the network transfer once all samples are loaded, and load the cached files from a fast SSD installed in the server.</p>

Dwight_Foster: <p>I am talking about the first pass through the entire dataset too. This is the first time the data is loaded onto the gpu so it should take the longest. Or at least it does for me.</p>

Praveen_Srinivasan: <p>No, based on timing I think the issue is in the synchronization between workers during forward and backward.</p>

Praveen_Srinivasan: <p>It turns out the issue was <code>torch.backends.cudnn.</code> <code>benchmark=True</code> - the model has variable length tensors which led to a slowdown. With that set to false, the issue was resolved.</p>
PyTorch weight sharing siamese

amsalehoof: <p>Hello there!<br>
I want to have a simple Siamese CNN which uses a <strong>“Shared weight”</strong> feature extractor. I have implemented the network like this:</p>
<pre><code class="lang-auto">class CnnSiamese(nn.Module):
	def __init__(self):
		super(CnnSiamese, self).__init__()
		self.feature_extractor = nn.Sequential(OrderedDict([
			('conv1', nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3, padding=1)),
			('maxpool1', nn.MaxPool2d(2)),
			('relu1', nn.ReLU()),
			('conv2', nn.Conv2d(in_channels=10, out_channels=20, kernel_size=3, padding=1)),
			('maxpool2', nn.MaxPool2d(2)),
			('relu2', nn.ReLU()),
			('conv3', nn.Conv2d(in_channels=20, out_channels=30, kernel_size=3, padding=1)),
			('dropout', nn.Dropout2d()),
			('maxpool3', nn.MaxPool2d(2)),
			('relu3', nn.ReLU()),
		]))
	
	self.gap = nn.AdaptiveAvgPool2d((1, 1))
	self.fc = nn.Linear(30, 2)
	
	def forward(self, x1, x2):
		x1 = self.feature_extractor(x1)
		x2 = self.feature_extractor(x2)
		x = torch.abs(x1 - x2)
		
		x = self.gap(x)
		x = x.view(x.shape[0], -1)
		x = self.fc(x)
		return x
</code></pre>
<p>The problem is that when I test the model with a single image and the copy of it as the input for the model, I expect that the output of <code>self.feature_extractor(x1)</code> and <code>self.feature_extractor(x2)</code> will be the same but I have got different tensors.</p>
<p>It seems that whenever I call <code>self.feature_extractor</code> on an image, a new instance is created so if I call it twice on a single image, I got different tensors.</p>
<p>How Can I solve this pronlem?</p>
<p>Thanks in advance.</p>

ruotianluo: <p>Because there are dropouts which are stochastic. Set the model to eval() mode, you should get identical tensors.</p>
Linker Error MacOSX lib torch

qhollist: <p>I’m trying to compile a C++ program that essentially runs a simulation for a metal lattice, and requires the evaluation of a machine learning model that I’ve trained on pytorch. I’m attempting to load the pre-trained model, and have had success in downloading the lib torch library. My CmakeLists.txt file is below:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.16)
project(PytorchTest)

set(CMAKE_CXX_STANDARD 14)

set(Torch_DIR /Users/quinnhollister/CLionProjects/Langevin/libtorch/share/cmake/Torch)
find_package(Torch REQUIRED)
find_package(OpenMP)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")

add_executable(PytorchTest main.cpp)
if(OpenMP_CXX_FOUND)
    target_link_libraries(PytorchTest OpenMP::OpenMP_CXX)
endif()
target_link_libraries(PytorchTest "${TORCH_LIBRARIES}")
</code></pre>
<p>The strange thing is that when I run:<br>
$ Cmake …<br>
everything works, no issue. But when I end up trying to actually compile the program:<br>
$ make<br>
*I run into issues with the linker. My error message is below:</p>
<p><em>[ 50%] Building CXX object CMakeFiles/PytorchTest.dir/main.cpp.o</em><br>
<em>[100%] Linking CXX executable PytorchTest</em><br>
<em>ld: unknown option: --no-as-needed</em><br>
<em>clang: error: linker command failed with exit code 1 (use -v to see invocation)</em><br>
<em>make[2]: *** [PytorchTest] Error 1</em><br>
<em>make[1]: *** [CMakeFiles/PytorchTest.dir/all] Error 2</em><br>
<em>make: *** [all] Error 2</em></p>
<p>I’m running MacOSX, and using the compilers cc and c++.<br>
I can see that the error has to do with the linker not knowing that specific flag “–no-as-needed”, but I have no idea how to suppress that flag, or if it hints at a larger issue. Any help would be appreciated.</p>
<p>Thanks!</p>

glaringlee: <p><a class="mention" href="/u/qhollist">@qhollist</a><br>
Can you create an issue at <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a><br>
Add ‘build’ tag on it, someone there will take a look at.</p>

remykarem: <p>Ensure that you downloaded the correct libtorch library for the correct platform. I faced this problem because I downloaded the library meant for Linux (should be macOS).</p>
What does perturb mean in the context of the numerical Jacobian?

mortonjt: <p>I’m looking for clarification on what exactly <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/gradcheck.py#L616" rel="noopener nofollow ugc">“perturbation”</a> means in the context of calculating the numerical Jacobian.</p>
<p>To give a bit of context, I’ve been trying to put together some custom autograd functions where I can compute Hessians.  Below is a skeleton of the code that I’m working with</p>
<pre><code class="lang-python">import torch
def _upward_pass(x):
    # stuff happens - this is the "forward" calculation
    return Vt   # some output value
def _downward_pass(Et):
    # more stuff happens - this is the "backward" calculation
    return E  # some derivative of x, i.e. dVt / dx * Et, I think, ...
def _adjoint_upward_pass(dx):
    # compute directional gradient
    return Vdt
def _adjoint_backward_pass(E):
    # compute hessian, d^2Vt / dx^2
    return Ed

# the actual autograd function
class ForwardFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        Vt  = _upwards_pass(x)
        ctx.save_for_backward(x)
        return Vt
    @staticmethod
    def backward(ctx, Et):
        x = ctx.saved_tensors
        E = ForwardFunctionBackward.apply(x, Et)
        return E

class ForwardFunctionBackward(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, Et):
        E = _downwards_pass(Et)
        return E
    @staticmethod
    def backward(ctx, dx):
        Vtd = _adjoint_upwards_pass(dx)
        Ed = _adjoint_downwards_pass(E)
        return Ed
</code></pre>
<p>As I put this codebase together, I’ve been testing the outputs with <code>gradcheck</code> and <code>gradgradcheck</code>.  I got <code>gradcheck</code> working, but <code>gradgradcheck</code> will randomly fail, either returning all zeros in the numerical output or all zeros in the analytical output (depending on the inputs).  So sometimes my function is “constant”, where no perturbations will change the output of the function.</p>
<p>Part of the issue that makes development slow is my struggle understanding the inputs and outputs of all of the functions.  My current challenge is understanding what a “perturbation” means in the context of calculating the numerical Jacobian – are the inputs to the function “perturbed” in order to compute gradients?  Specifically, should I expect some delta term to be added to <code>dx</code> during the <code>gradgradcheck</code> run?</p>
<p>Any thoughts on understanding the internals of autograd will be greatly appreciated (in addition to any insights on how to debug this sort of thing).</p>
How to change PyTorch faster-rcnn losses?

tornadots: <p>**<em>Hi experts,</em><br>
<em>&gt; *<br>
***     I try to make transfer learning on fasterrcnn_resnet50_fpn pretrained model. My dataset is highly imbalanced (323,3332,127 samples for 3 classes). I think I can overcome this problem with manipulation of the classifier loss. But I couldn’t find a way to change the NLL loss to focal loss. How can I manipulate the losses of the pretrained model.</em></p>
<p>Thanks,<br>
Abdullah</p>
Best alternative to torch.tensor_split without upgrading to 1.8

Gabriel_Simmons: <p>Hi all,</p>
<p>Looking for the exact functionality provided by <a href="https://pytorch.org/docs/master/generated/torch.tensor_split.html" rel="noopener nofollow ugc">torch.tensor_split</a> in v1.8 without having to upgrade to 1.8 <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<pre><code class="lang-auto">predicted_ids = torch.tensor([100, 100, 31090, 100, 100, 100, 31090])
indices = torch.tensor([2,6])
np.array_split(predicted_ids,indices)
# returns [tensor([100, 100]), tensor([31090,   100,   100,   100]), tensor([31090])]
</code></pre>
<p>I would like the last line of the above, except via pytorch operations so I can avoid breaking the computational graph.</p>
<p>Is there a succinct way to do this?</p>

googlebot: <p>isn’t it the same as torch.split(x, int_list). You can do int_list=indices.tolist() if you have a reason to feed in a tensor of integers.</p>

Gabriel_Simmons: <p><a class="mention" href="/u/googlebot">@googlebot</a> Thanks. From what I could tell, torch.split accepts chunk sizes rather than indices. For example to split <code>x</code> at indices 5, 10, and 15 I think you would do <code>torch.split(x,[5,5,5])</code>. So I just ended up doing</p>
<p><code>chunk_sizes = [len(x) for x in np.array_split(predicted_ids,indices)]</code><br>
<code>desired_output = torch.split(predicted_ids,chunk_sizes)</code></p>
<p>If there is a simpler way I would definitely be interested in hearing!</p>

googlebot: <p>Ah, right. Then something like following would calculate sizes</p>
<pre><code class="lang-auto">np.diff(np.asarray([0]+[2,3]+[100])).tolist()
&gt;[2, 1, 97]
</code></pre>
<p>i.e. append total size, prepend zero if you have split points instead of indexes, take deltas.</p>

Gabriel_Simmons: <p>Of course, thanks! Also saw <a href="https://discuss.pytorch.org/t/equivalent-function-like-numpy-diff-in-pytorch/35327/2">this solution</a> (essentially the same) just using indexing.</p>
Is there any nice pre-defined function to calculate precision, recall and F1 score for multi-class multilabel classification?

ADONAI_TZEVAOT: <p>I have a multi-class multi-label classification problem where there are 4 classes (happy, laughing, jumping, smiling) and each class can be positive:1 or negative:0. An input can belong to more than one class . So let’s say that for an input x , the actual labels are [1,0,0,1] and the predicted labels are [1,1,0,0]. So how to calculate the precision, recall and f1 score for this fine grained approach? Are there any predefined methods to do this?</p>

AladdinPerzon: <p>I found <a href="https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/73246" rel="noopener nofollow ugc">this</a> and changed it slightly to adapt it to your question. Haven’t done any rigorous testing so you should probably do that.</p>
<pre><code class="lang-auto">def F1_score(prob, label):
    prob = prob.bool()
    label = label.bool()
    epsilon = 1e-7
    TP = (prob &amp; label).sum().float()
    TN = ((~prob) &amp; (~label)).sum().float()
    FP = (prob &amp; (~label)).sum().float()
    FN = ((~prob) &amp; label).sum().float()
    #accuracy = (TP+TN)/(TP+TN+FP+FN)
    precision = torch.mean(TP / (TP + FP + epsilon))
    recall = torch.mean(TP / (TP + FN + epsilon))
    F2 = 2 * precision * recall / (precision + recall + epsilon)
    return precision, recall, F2

y_true = torch.tensor([[1,0,0,1]])
y_pred = torch.tensor([[1,1,0,0]])
print(F1_score(y_pred, y_true))
</code></pre>
<p>Edit:<br>
Found that PyTorch Lightning does have implementation for <a href="https://pytorch-lightning.readthedocs.io/en/stable/metrics.html?highlight=F1%20#precision-recall-func" rel="noopener nofollow ugc">precision-recall</a> and <a href="https://pytorch-lightning.readthedocs.io/en/stable/metrics.html?highlight=f1#f1-score-func" rel="noopener nofollow ugc">f1 score</a>, perhaps you could use that.</p>

rayryeng: <p>Thanks for this.  Take note that since you are using <code>sum</code> to sum over the whole tensor, <code>torch.mean</code> would be superfluous as you’d be taking the mean over a single value.  You can remove the call to <code>torch.mean</code> for the precision and recall and just compute the single values on their own.</p>
Select sub-tensors via indices from torch.topk

MFajcik1: <p>I cannot seem to produce a nice implementation of indexing with a different list of indices with different elements in each dimension. This is a common use-case, I bet there is some nice solution to that:</p>
<p>Assume  the following:</p>
<pre><code class="lang-auto">import torch

# I have following quantities
batch_size = 6
L = 100
K = 10
d = 300

# I have a matrix of vectors
# batch X L x d
R = torch.randn(batch_size, L, d)

# I have a matrix of scores (each element of this is a score assigned to vector from matrix R via some transformation)
# batch_size x L
scores = torch.randn(batch_size, L)

# I get the top indices of each
# batch x K
topk_positions = torch.topk(scores, K, dim=1)[1]

# example of topk_positions contents
# tensor([[108,  16,  15, 107,  22, 106,  19,  11,  20,  99],
#         [118, 113, 167, 112, 108, 171, 111, 117, 138,  18],
#         [ 43,  54,  44,  35,  36,  40,  55,  51,  10,  47],
#         [ 89,  84,  78,  83,  82,  92,  67,  72,  86,  85],
#         [ 57,  46,  58,  50,  47,  38,  39,  43,  54,  13],
#         [ 46,  57,  47,  38,  39,  54,  58,  43,  13,  50]])

# now I would like to index these positions into matrix R, so I will get the vectors corresponding to the top elements
# the resulting matrix should be of shape batch_size x K x d

R_best = ?

# it should be equivalent to
R_best_ugly = torch.zeros((batch_size, K, d))
for i, batch_size in enumerate(topk_positions):  # over batch
    R_best_ugly[i] = R[i][topk_positions[i]]

assert R_best.allcose(R_best_ugly)
</code></pre>

MFajcik1: <p>Finally, solved it thanks to my colleague!</p>
<p><code>R_best = R.gather(1,topk_positions.unsqueeze(2).expand(batch_size,K,d))</code></p>
Extracting subtensors from tensors

GeoffNN: <p>I’m trying to index a tensor <code>t</code> to compute norms of subtensors included in <code>t</code>.<br>
Suppose that <code>t</code> is of shape <code>(8, 3, 6, 6)</code>, and that I have an indexing tensor <code>idx</code> of shape <code>(12, 3)</code>. <code>idx</code> contains the coordinates of 12 elements I want to extract from <code>t</code> for each element in the batch. I then compute the norm of the subtensor:</p>
<p>I’ve tried <code>t[idx]</code>, <code>t[:, idx]</code> which both don’t work. I have one solution which works :</p>
<pre><code class="lang-auto">idx_ = (...,) + tuple(idx.T)
norm = torch.linalg.norm(t[idx_], dim=-1)
</code></pre>
<p>But when profiling this, forming <code>idx_</code> is slower than computing the norm, and it seems quite unwieldy and unnatural. Would there be any faster and more natural way to perform this indexing?</p>

ptrblck: <p>Could you explain how the <code>idx</code> tensor should be used in indexing <code>t</code>?<br>
Since <code>dim1</code> of <code>idx</code> is of size 3, would you like to apply each of these values to <code>dim1</code> to <code>dim3</code> in <code>t</code>?</p>

GeoffNN: <p>Yes, exactly. The 3 values are the indexes in <code>dim1</code> to <code>dim3</code>.<br>
The resulting tensor should be of shape <code>(8, 12)</code> give or take some squeezing.</p>

GeoffNN: <p>Would another representation for <code>idx</code> make this work?</p>

ptrblck: <p>If I understand the use case correctly, this should work:</p>
<pre><code class="lang-python">x = torch.randn(8, 3, 6, 6)
idx = torch.cat((torch.randint(0, 3, (12, 1)), torch.randint(0, 6, (12, 2))), 1)

res = x[:, idx[:, 0], idx[:, 1], idx[:, 2]]
</code></pre>
UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead

ky_Pa: <p>Hi,<br>
I encountered a problem when I changed <code>nn.Upsample</code> to <code>nn.functional.interpolate</code>.<br>
I want to put <code>nn.functional.interpolate</code> into nn.Sequential by building <code>Interpolate</code> class, but an error occurred.How can I modify the code?</p>
<pre><code class="lang-auto">class Interpolate(nn.Module):
    def init(self, scale_factor, mode):
        super(Interpolate, self).init()
        self.interpolate = nn.functional.interpolate
        self.scale_factor = scale_factor
        self.mode = mode
    def forward(self, x):
        x = self.interpolate(x, scale_factor=self.scale_factor, mode=self.mode)
        return x
</code></pre>
<pre><code class="lang-auto">        self.decoder_s = nn.Sequential(
            Interpolate(scale_factor=7, mode='bilinear'),
</code></pre>
<pre><code class="lang-auto">TypeError: __init__() got an unexpected keyword argument 'scale_factor'
[Finished in 2.9s with exit code 1]</code></pre>

1chimaruGin: <aside class="quote no-group" data-username="ky_Pa" data-post="1" data-topic="110308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/eada6e/48.png" class="avatar"> ky_Pa:</div>
<blockquote>
<pre><code class="lang-auto">def init(self, scale_factor, mode):
        super(Interpolate, self).init()
</code></pre>
</blockquote>
</aside>
<p>You are missing <code>__init__</code> method.</p>
<pre><code class="lang-auto">class Interpolate(nn.Module):
    def __init__(self, scale_factor, mode):
        super(Interpolate, self).__init__()
        self.interpolate = nn.functional.interpolate
        self.scale_factor = scale_factor
        self.mode = mode
    def forward(self, x):
        x = self.interpolate(x, scale_factor=self.scale_factor, mode=self.mode)
        return x
</code></pre>

ky_Pa: <p>This is indeed the problem. Thanks for your help!</p>
How to concat 2 tensors?

mgloria: <p>This question has been asked with slight variations but I could not really get an answer from my (simple) problem.</p>
<pre><code class="lang-auto">all_labels = torch.Tensor([]) # torch.Size([0])
for batch_idx, (images, labels) in enumerate(train_loader, 1):
    all_labels = torch.cat((all_labels, labels)) #labels is torch.Size([32])
</code></pre>
<p>RuntimeError: Expected object of scalar type Float but got scalar type Long for sequence element 1 in sequence argument at position <span class="hashtag">#1</span> ‘tensors’</p>
<p>What is happening here? I would like to understand the meaning of this error message and the reason of the problem. I basically just want to append the labels for each batch to the all_labels.</p>

ptrblck: <p>If you are calling <code>torch.Tensor</code> (uppercase T in <code>Tensor</code>), you are creating an empty <code>FloatTensor</code>.<br>
You could avoid this error using by defining the empty tensor as a <code>LongTensor</code>:</p>
<pre><code class="lang-python">all_labels = torch.tensor([]).long()
for _ in range(5):
    all_labels = torch.cat((all_labels, torch.tensor([1])))
</code></pre>
<p>However, I would recommend to store the <code>labels</code> in a plain <code>list</code> and then convert it to a tensor afterwards, which should be faster than your current approach.</p>

mgloria: <p>Thanks a lot! I am also leaving this link which I found very interesting <a href="https://jdhao.github.io/2017/11/15/pytorch-datatype-note/" rel="nofollow noopener">https://jdhao.github.io/2017/11/15/pytorch-datatype-note/</a></p>

THL701: <p>I’m having a similar issue … but can’t seem to get my code to work (trying your suggested approach). (trying to implement Linear Probe: <a href="https://github.com/openai/CLIP" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openai/CLIP: Contrastive Language-Image Pretraining</a>)</p>
<p>text = []<br>
for example in dataset:<br>
text.append(example[“label”])<br>
text = torch.cat(text).cpu().numpy()</p>
<p>results in “TypeError: expected Tensor as element 0 in argument 0, but got numpy.int64”</p>
<p>text = []<br>
for example in dataset:<br>
text.append(example[“label”])<br>
text = torch.tensor(text)<br>
text = torch.cat(text).cpu().numpy()</p>
<p>Results in “TypeError: cat(): argument ‘tensors’ (position 1) must be tuple of Tensors, not Tensor”</p>
<p>Thoughts?</p>

ptrblck: <p>Based on your second code snippet it seems <code>text</code> is already a tensor, so I’m unsure, why you would want to use <code>torch.cat</code> on it again.<br>
Anyway, the inputs to <code>torch.cat</code> should be a <code>tuple</code> of tensors, so you would have to use <code>torch.cat((text,))</code>.<br>
However, this won’t change the shape of <code>text</code>, since it’s already a single tensor.</p>
How does transpose work in Pytorch?

LostAtlas: <p>I was doing a transpose of tensors of rank 3 and according to transpose rule for rank 2 tensors which follow simple matrix transpose rule Aij = Aji. But when i transposed a rank 3 tensor i ended up with a different output given below.<br>
Can someone explain to me how it is working under the hood?</p>
<pre><code class="lang-auto">a = torch.tensor(
        [[[1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.]],

        [[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]],

        [[1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.]]])
</code></pre>
<p>Printing the tensor and the shape gives the following:</p>
<pre><code class="lang-auto">print(a)
print(a.shape)
tensor([[[1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.]],

        [[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]],

        [[1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.]]])
torch.Size([3, 3, 3])

</code></pre>
<p>Transposing with a.T we get</p>
<pre><code class="lang-auto">tensor([[[1., 0., 1.],
         [1., 0., 1.],
         [1., 0., 1.]],

        [[1., 0., 1.],
         [1., 0., 1.],
         [1., 0., 1.]],

        [[1., 0., 1.],
         [1., 0., 1.],
         [1., 0., 1.]]])
</code></pre>
Slicing gives incorrect result, if the values are the same

protoneqt: <p>I want to make a slice of a tensor with the same value.</p>
<pre><code class="lang-auto">import torch
dist = torch.ones((10, 20), dtype=torch.int32)
print(dist[[1, 2], [3, 4]])
</code></pre>
<p>It returns <code>tensor([1, 1], dtype=torch.int32)</code>. But it should return <code>tensor([[1, 1], [1, 1]], dtype=torch.int32)</code>. How do I make it to return this result?</p>

KFrank: <p>Hi protoneqt!</p>
<aside class="quote no-group" data-username="protoneqt" data-post="1" data-topic="110269" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a5b964/48.png" class="avatar"> protoneqt:</div>
<blockquote>
<p>But it should return <code>tensor([[1, 1], [1, 1]], dtype=torch.int32)</code>. How do I make it to return this result?</p>
</blockquote>
</aside>
<p>I’m not sure exactly what you want or are expecting.  But maybe the following<br>
helps answer your question:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; dist = torch.arange (30).reshape ((5, 6))
&gt;&gt;&gt; dist
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17],
        [18, 19, 20, 21, 22, 23],
        [24, 25, 26, 27, 28, 29]])
&gt;&gt;&gt; dist[[1, 2], [3, 4]]
tensor([ 9, 16])
&gt;&gt;&gt; dist[1:3, 3:5]
tensor([[ 9, 10],
        [15, 16]])
</code></pre>
<p>I would call <code>dist[[1, 2], [3, 4]]</code> <em>indexing</em> and <code>dist[1:3, 3:5]</code> <em>slicing,</em><br>
but I’m not really sure what the proper terminology is.</p>
<p>Best.</p>
<p>K. Frank</p>
Is it means first layer also re-trained?

sampa: <p>I have converted the convolution layer from 3 channels to 4 channels. Is it means the first layer is also re-trained using target images?<br>
.```<br>
model = models.resnet18(pretrained=True)<br>
model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)<br>
num_ftrs = model.fc.in_features<br>
model.fc = nn.Linear(num_ftrs, num_classes)</p>
<pre><code class="lang-auto"></code></pre>

ruotianluo: <p>It should be.<br>
The easiest way to verify is just to print the model.conv1.weight and see if it changes when training.</p>