akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implement custom kernel in gPyTorch

username: michaelSederlin
cooked: <p>Hi,</p>
<p>I am working on a project where I am using Gaussian Process regression to estimate the transition and measuerment functions of a system to be used in a Kalman Filter.</p>
<p>In order to do this I need to use the Jacobian of the learned models. This comes down to taking the derivative of the GP mean functions (and potentially covariance function). However, I’m not sure how to implement a GP model with custom  kernel using gPyTorch.</p>
<p>Please let me know if I need to clarify anything.</p>
<p>Best,<br>
Michael</p>
Expressing nn.Module parameter as a function of another tensor

username: Dmytro_Danevskyi
cooked: <p>I’m working on some stuff related to what is described in the recent Meta Pseudo Labels paper (<a href="https://arxiv.org/abs/2003.10580" rel="nofollow noopener">https://arxiv.org/abs/2003.10580</a>). The process involves updating the weights of one network in a differentiable way and then using the same network (or probably a copy of it) with updated weights to compute another output.</p>
<p>The problem is, it seems that it’s not that straightforward to do this as I initially thought…</p>
<p>My current code (one training iteration) looks like this:</p>
<pre><code class="lang-auto">teacher_logits = teacher(unsupervised_batch)
student_logits = student(unsupervised_batch)
student_loss = kldiv(
    torch.log_softmax(student_logits, dim=1),
    torch.softmax(teacher_logits, dim=1)
)
student_loss.backward(create_graph=True)
teacher_optimizer.zero_grad()

# Here I want to obtain a new student but continue to track the gradients.
# Normally I would use p2.data -= lr * p1.grad.
# Updated student is just a copy of the original student.
for p1, p2 in zip(student.parameters(), updated_student.parameters()):
    p2 -= lr * p1.grad    # this expectedly fails as p2 is a leaf variable

student_logits = updated_student(student_data)
student_logits.squeeze_(dim=1)
student_loss = ce(student_logits, student_labels)
student_loss.backward()

teacher_optimizer.step() 
</code></pre>
<p>I could make it work if I write the student in a functional fashion, such that <code>student(x, params)</code> is a function that takes a batch of data points and the parameters. While this is doable for small networks, that would be super inconvenient to do so for larger models. I wish I could run the <code>updated_student</code> with the updated weights in a differentiable fashion and stay with regular PyTorch nn.Module paradigm.</p>
Making Infinity Addition Idempotent

username: circa
cooked: <pre><code class="lang-auto">r1 = torch.randn(5)
r2 = torch.tensor([-float('inf') for k in range(5)])
dp = 0.
for i in range(r1.shape[0]):
    dp+=(r1[i]*r2[i])
</code></pre>
<pre><code class="lang-auto">dp
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">tensor(nan)
</code></pre>
<p>So it seems adding -inf multiple times gives a nan. This creates a problem while implementing masked attention.</p>
<p>Is there a way in which adding -inf multiple times still gives me -inf ? I am not looking for a hack involving some conditional expression. Perhaps, I am looking for a solution which makes -inf idempotent with respect to addition.</p>

username: KFrank
cooked: <p>Hi Circa!</p>
<aside class="quote no-group" data-username="circa" data-post="1" data-topic="80746" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/circa/48/22337_2.png" class="avatar"> circa:</div>
<blockquote>
<pre><code class="lang-auto">r1 = torch.randn(5)
</code></pre>
</blockquote>
</aside>
<p>It is likely that not all of the elements of <code>r1</code> have the same sign.<br>
Try replacing your original first line with:</p>
<pre data-code-wrap="python"><code class="lang-python">r1 = torch.abs (torch.randn(5))
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code>dp+=(r1[i]*r2[i])
</code></pre>
<pre><code class="lang-auto"></code></pre>
</blockquote>
</aside>
<p>Let’s say that <code>r1[0]</code> is positive.  Then after the first iteration, <code>dp</code> will be<br>
equal to <code>-inf</code>.  If <code>r1[1]</code> happens to be negative, in the second iteration<br>
you will be calculating <code>(-inf) + inf = nan</code>, as it should be.  (And once<br>
<code>dp</code> becomes <code>nan</code>, it stays <code>nan</code>.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">dp
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">tensor(nan)
</code></pre>
</blockquote>
</aside>
<p>Note, if you run your test many times (with different random values<br>
for <code>r1</code>, you will occasionally get <code>-inf</code> and <code>inf</code> for the result, instead<br>
of <code>nan</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>So it seems adding -inf multiple times gives a nan.</p>
</blockquote>
</aside>
<p>The short answer is that you are not adding <code>-inf</code> multiple times.<br>
You are usually adding <code>-inf</code> to <code>inf</code> somewhere in your loop.</p>
<p>Best.</p>
<p>K. Frank</p>

username: circa
cooked: <p>Ahh…I see.<br>
That solved it.<br>
Thanks a lot.</p>
Visualize Convolutional layer of Alexnet

username: Sania
cooked: <p>Hello All,<br>
I am newbie into this field. I was trying to visualize the feature map/activation map of Alexnet.<br>
I am working on cifar-10 dataset and was trying to visualize the kernels and feature map.</p>
<p>AlexNet(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))<br>
(1): ReLU(inplace)<br>
(2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
(4): ReLU(inplace)<br>
(5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(7): ReLU(inplace)<br>
(8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(9): ReLU(inplace)<br>
(10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(11): ReLU(inplace)<br>
(12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(6, 6))<br>
(classifier): Sequential(<br>
(0): Dropout(p=0.5)<br>
(1): Linear(in_features=9216, out_features=4096, bias=True)<br>
(2): ReLU(inplace)<br>
(3): Dropout(p=0.5)<br>
(4): Linear(in_features=4096, out_features=2048, bias=True)<br>
(5): ReLU(inplace)<br>
(6): Linear(in_features=2048, out_features=1024, bias=True)<br>
(7): ReLU(inplace)<br>
(8): Linear(in_features=1024, out_features=10, bias=True)<br>
)<br>
)</p>
<p>I have few questions on visualization.</p>
<ol>
<li>How do I visualize feature map of Alexnet?</li>
<li>Is there any supporting code for visualizing the convolutional layers of Alexnet?</li>
</ol>
<p>Thanks in advance…</p>

username: ptrblck
cooked: <p>You could use forward hooks to get the activation maps and visualize them as described <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/2">here</a>. The topic also includes another post to <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/4">visualize kernels</a>.</p>
Fasterrcnn_resnet50_fpn parallelization warning

username: Magolor
cooked: <p>When training PyTorch built-in model fasterrcnn_resnet50_fpn on a custom dataset using <code>nn.DataParallel</code>, I received the following warning:</p>
<pre><code class="lang-auto">UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
</code></pre>
<p>By <a href="https://discuss.pytorch.org/t/mse-loss-gpu-warning/24782">this</a> post, I realized that the problem is, fasterrcnn_resnet50_fpn is using loss function inside the forward function.</p>
<p>So what should I do to fix this? I tried to ignore it and divide the final loss by num_workers. However, it seems that the model is not parallelized and all work was done by GPU 0 (maybe this is another irrelevant problem).</p>

username: Westerby
cooked: <p>You probably already found it, but this model won’t work with <code>nn.DataParallel</code>. Train with Distributed like here: <a href="https://github.com/pytorch/vision/blob/master/references/detection/train.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/detection/train.py</a></p>
Faster R-CNN Loss Units of Measure

username: rendres
cooked: <p>Hello there everyone. I am really new to all of this and trying to figure things out one by one. I’ve been able to put together a Faster R-CNN model and I’m tracking loss over eachiteration of the batch. What I’m puzzled by and what I can’t seem to find any information on is the unit of measure of the loss I keep printing out. I know this might seem trivial to everyone but it’s not exactly self evident at all. What exactly does a loss of 0.006 mean. Just trying to make sense of this all. Thanks for any help you can offer me.</p>

username: ptrblck
cooked: <p>If you are following the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">tutorial</a>, you should get multiple loss values.<br>
Which loss are you referring to?</p>

username: rendres
cooked: <p>So as I understand it, each epoch is a complete cycle through my dataset and each batch is a smaller subset of that to help manage memory. The loss I am getting looks like it might be an average loss over the batch that was just processed. The thing I don’t seem to quite understand is the unit of measure of that loss. What exactly is that loss.</p>
Computation graph of setting weights

username: sait
cooked: <p>I need a clarification of code written for some code in FastAI2 library.</p>
<p>this is the code WeightDropout written in FastAI2 library.</p>
<pre><code class="lang-auto"> class WeightDropout(Module):
        "A module that warps another layer in which some weights will be replaced by 0 during training."

        def __init__(self, module, weight_p, layer_names='weight_hh_l0'):
            self.module,self.weight_p,self.layer_names = module,weight_p,L(layer_names)
            for layer in self.layer_names:
                #Makes a copy of the weights of the selected layers.
                w = getattr(self.module, layer)
                delattr(self.module, layer)
                self.register_parameter(f'{layer}_raw', nn.Parameter(w.data))
                setattr(self.module, layer, F.dropout(w.data, p=self.weight_p, training=False))
                if isinstance(self.module, (nn.RNNBase, nn.modules.rnn.RNNBase)):
                    self.module.flatten_parameters = self._do_nothing

        def _setweights(self):
            "Apply dropout to the raw weights."
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))

        def forward(self, *args):
            self._setweights()
            with warnings.catch_warnings():
                #To avoid the warning that comes because the weights aren't flattened.
                warnings.simplefilter("ignore")
                return self.module.forward(*args)

        def reset(self):
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer, 
    F.dropout(raw_w.data, p=self.weight_p, training=False))
            if hasattr(self.module, 'reset'): self.module.reset()

        def _do_nothing(self): pass
</code></pre>
<p>where above code randomly drops weights in <code>weight</code> matrix of hidden layers.I am primarily interested in ,</p>
<pre><code class="lang-auto"> def _setweights(self):
                "Apply dropout to the raw weights."
                for layer in self.layer_names:
                    raw_w = getattr(self, f'{layer}_raw')
                    setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))
</code></pre>
<p>my question is that, does this operation of changing weights is recorded in gradient computation.</p>

username: ptrblck
cooked: <p>I don’t think it will be recorded, as the <code>.data</code> attribute is used, which will prevent Autograd to track this operation.<br>
This is generally not recommended and the block should probably be wrapped in a <code>with torch.no_grad()</code> block. However, I’m not familiar with the detailed implementation, and you might get a better answer in the FastAI forum. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Train large models in GPUs

username: surya00060
cooked: <p>I want to train VGG and ResNet from scratch. These models are too big to fit in a single GPU. Always get CUDA out of memory for batches sizes of 128 or 256. Is there any way I could train these models? Should I use multiple GPUs to train with each GPU processing small batch size?</p>
<p>Thanks in advance for any solutions.</p>

username: ptrblck
cooked: <p>If you have multiple GPUs, you could use e.g. <code>DistributedDataParallel</code> to chunk the batch so that each model (and device) will process a smaller batch size.</p>
<p>Alternatively, you could lower your batch size or use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

username: Eta_C
cooked: <ul>
<li>
<p><strong>[Recommend]</strong> For PyTorch &gt;= 0.4, you can use <code>torch.utils.checkpoint</code>. It will compute the model piece by piece.</p>
</li>
<li>
<p><strong>[Impaired performance]</strong> You can also lower your batch size, multiple <code>forward</code>, one <code>backward</code>. If you use multiple GPUs, maybe <a href="https://nvidia.github.io/apex/parallel.html?highlight=sync#apex.parallel.SyncBatchNorm" rel="nofollow noopener">NVIDIA Apex SyncBatchNorm</a> could help you to correcte the bad consequences.</p>
</li>
<li>
<p><strong>[Impaired performance]</strong> Training a mixed precision model is also a way to solve your problem.</p>
</li>
</ul>

username: Oscar_Rangel
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/eta_c">@Eta_C</a>  can you elaborate mode on checkpoint and if you have an example how to code it ?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb">Here</a> is a notebook showing an example usage (which is quite old by now, but should still show the proper usage of it). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Mean of the vector tensor from the same class in a differentiable way

username: Abdelpakey
cooked: <p>Hi All,</p>
<p>I have a list of tensors and I have say 500 classes, I want to calculate the mean vector for each class in a differentiable way.</p>
<p>here is my code:<br>
y is the labels<br>
H =torch.randn (800,800)</p>
<pre><code>C = 500
mean_vectors_list = []
mean_vectors_list.append([torch.mean(H[y == cl], dim=0, dtype=torch.float32) for cl in 
range(C)])
</code></pre>
<p>The problem is some times H tensor doesn’t have some classes as it’s a randomly selected batch<br>
that gives me Nan.<br>
How to use .scatter_add_ to produce the desired output with cuda</p>
How to find each class AP

username: Thecoldheart_Wolf
cooked: <p>I got mAP from using evaluate(model, testDatasetLoader, device=device) which give<br>
class1+class2+class3+class4 test dataset = 777 img<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.736<br>
but when I try to use evaluate(model, testDatasetLoader, device=device) with each class dataset I got<br>
class1 test dataset = 213 img<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.043<br>
class2 test dataset = 188 img<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.663<br>
class3 test dataset = 176 img<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.495<br>
class4 test dataset = 200 img<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.988</p>
<p>after I calculate (0.043+0.663+0.495+0.988)/4=0.54725 which not equal to mAP from class1+class2+class3+class4 test dataset</p>
<p>How can I find AP for each class correctly?<br>
PS: evaluation function from engine.py</p>
Super slow data_loader

username: secg95
cooked: <p>I wrote a custom data loader but its training buckle spent a lot of time iterating over the data, loader (217 seconds per batch (64) ). I will like to know if by any chance i have committed a mistake writing the data set. In addition i used the nn.random_split for val_loader creation.</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset
import pandas as pd
from constants import IPY
from CTR.cross_net.constants import trainable

class Dataset_rtb(Dataset):
    
    def __init__(self, file_path):
        self.data = pd.read_csv(file_path, sep='\t')
        self.index =  self.data[IPY["request_id"]].dropna()
        self.data = self.data[['click'] + trainable]
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, index):
        # load image as ndarray type (Height * Width * Channels)
        # be carefull for converting dtype to np.uint8 [Unsigned integer (0 to 255)]
        # in this example, i don't use ToTensor() method of torchvision.transforms
        # so you can convert numpy ndarray shape to tensor in PyTorch (H, W, C) --&gt; (C, H, W)
        features = []
        #row_cont = self.data[["slotwidth", "slotheight"]].iloc[index]
        row = self.data[trainable].iloc[index]
        for i in range(len(trainable)):
            one_hot_tensor = torch.tensor(int(row[i]))
            features.append(one_hot_tensor)
    
        #features.append(torch.tensor(row_cont))
        label = torch.tensor(self.data['click'].iloc[index])
            
        return features, label```</code></pre>

username: futscdav
cooked: <p>I don’t know what your <code>trainable</code> is or what its length is, but creating many tensors each time you want a sample of data will inherently be slow. If you need to create a bunch of one-hot tensors, consider using <code>scatter_</code>.</p>
Batch multiplication for scalar and second-order tensor

username: Abueidda
cooked: <p>Hi,</p>
<p>How can I multiply a scaler with a 2D matrix? Please see the example below:</p>
<pre><code class="lang-auto">batch_size = 128
a = torch.randn(batch_size, 3, 3)
b = torch.randn(batch_size, 1)

c = torch.bmm(a,b)
</code></pre>

username: futscdav
cooked: <p>Don’t need a mm operation for that, use broadcasting</p>
<pre><code class="lang-auto">batch_size = 128
a = torch.randn(batch_size, 3, 3)
b = torch.randn(batch_size, 1)

c = a * b.view((-1, 1, 1))
</code></pre>
Using two optimizers for the same model (1 full precision, 1 binarized)

username: CuauSuarez
cooked: <p>Hello everyone,</p>
<p>I am currently working on a binarized optimizer for CNNs. Right now I am using a model which has both full precision and binarized weights (not in the same module).</p>
<p>What I am trying to do is something like this:</p>
<pre><code class="lang-auto">bin_params = []
fp_params = []

for name, param in model.named_parameters():
     if "bin" in name:
          bin_params.append(param)
     else:
          fp_params.append(param)
</code></pre>
<p>This for creating two different sets of parameters. Then, I define my optimizers</p>
<pre><code class="lang-auto">fp_optimizer = optim.SGD(fp_params, lr = lr, momentum = momentum, weight_decay = weight_decay)
bin_optimizer = BinaryOptimizer(binary_params, 1.0)

optimizer = MultipleOptimizer(fp_optimizer, bin_optimizer)
</code></pre>
<p>Where</p>
<pre><code class="lang-auto">class BinaryOptimizer(Optimizer): 

    def __init__(self, params, scale=required):     
        if scale is not required and scale &lt;= 0.0:
            raise ValueError("Invalid scale value: {}".format(lr))
            
        defaults = dict(scale=scale)
        
        super(BinaryOptimizer, self).__init__(params, defaults)
        
        
    def __setstate__(self, state):
        super(BinaryOptimizer, self).__setstate__(state)
        
        
    def step(self, closure=None):
        loss = None
        if closure is not None:
            loss = closure()
            
        for group in self.param_groups:
            scale = group["scale"]
            
            for p in group["params"]:
                if p.grad is None:
                    continue
                    
                grad = p.grad
                
                flips = # Some value
                                
                p.mul_(flips)
                
        return loss
</code></pre>
<p>My problem is that when <code>p.mul_(flips)</code> runs, I have the following error:</p>
<pre><code class="lang-auto">RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>I don’t know where the problem is, do you see it?</p>
<p>Thank you</p>
Loss function with Numpy libraries

username: Hmrishav_Bandyopadhy
cooked: <p>Hello all,</p>
<p>I am trying to set up a loss function that involves image transform with the skimage library. The skimage library is, however completely in Numpy and I have had no luck converting the arrays into tensors so far. Me converting the tensors to arrays would require me to break the computational graph with detach(). Is there any way I can use the skimage library without detaching the tensors as the use of the library is imperative to my work?</p>
<p>Any help will be highly appreciated.</p>

username: albanD
cooked: <p>hi,</p>
<p>I’m afraid you won’t be able to use skimage and pytorch’s autograd. To work, the autograd needs definition of the derivatives for everything that happens. So we only support pytorch ops.</p>
<p>If you know how to write the backward for that part though, you can create a custom Function and define the backward yourself. See doc <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a> for more details.</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="80734">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>If you know how to write the backward for that part though, you can create a custom Function and define the backward yourself.</p>
</blockquote>
</aside>
<p>Yeah I can try with that. For more reference, I am currently trying to take in some tensors and with the help of those trying to transform the image. I haven’t written backward ever and have a vague intuition this won’t work, but can I use the tensors to transform the Image–then compare the image and the label with a loss and return loss.backward() in backward ? It would still break the link between the input tensors and the Image right?</p>
Copy a block of layers from a model

username: anshu957
cooked: <p>Hi everyone,</p>
<p>I’m new to the forums, so correct me if I have not framed the problem correctly.</p>
<p>I’m in a situation where I have to make a copy of a specific block of the network for some further calculations (like Jacobian)</p>
<p>My network looks like this:</p>
<pre><code class="lang-auto">class AutoCanonical(torch.nn.Module):
    def __init__(self, d_in, d_hidden, d_out, activation_fn, device, experiment):
        super(AutoCanonical, self).__init__()
        self.layers = torch.nn.ModuleList()

  ## Block for converting provided coordinates to canonical coordinates
        self.layer1 = torch.nn.Linear(d_in, 50)
        self.layers.append(self.layer1)

        self.layer2 = torch.nn.Linear(50, 50)
        self.layers.append(self.layer2)

        self.layer3 = torch.nn.Linear(50, d_in)
        self.layers.append(self.layer3)
        ## -------------------------------------------------

        ## Hamiltonian block
        self.layer4 = torch.nn.Linear(d_in, 200)
        self.layers.append(self.layer4)

        self.layer5 = torch.nn.Linear(200, 200)
        self.layers.append(self.layer5)

        self.layer6 = torch.nn.Linear(200, 1)
        self.layers.append(self.layer6)
        ## -------------------------------------------------

  def to_canonical(self, x):
        y1 = self.nonlinear_fn(self.layer1(x))
        y2 = self.nonlinear_fn(self.layer2(y1))
        #out = self.nonlinear_fn(self.layer3(y2))
        out = self.layer3(y2)
        return out

    def Ham(self, x):
        y1 = self.nonlinear_fn(self.layer4(x))
        y2 = self.nonlinear_fn(self.layer5(y1))
        #nn_fn = torch.nn.LeakyReLU(0.9) 
        out = self.layer6(y2)
        return out

 def forward(self, x):
        # set_trace()
        can_coords = self.to_canonical(x) # convert to canonical variables
        H = self.ham(can_coords)        
        return  H
</code></pre>
<p>I would like to make a copy of <code>self.to_canonical</code> block and would like to use it calculate it’s jacobian. How do I make a copy of that ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Given that all the modules are on the same module, you will have to work with the whole Module.<br>
You can make a simply copy of it here by doing</p>
<pre><code class="lang-auto">new_model = AutoCanonical()
new_model.load_state_dict(current_model.state_dict())
</code></pre>

username: anshu957
cooked: <p>Thanks!</p>
<p>Last night, I stumbled upon <a href="https://github.com/pytorch/pytorch/issues/415" rel="nofollow noopener">this</a> issue. Looks like one can also make a copy using : <code> new_model = copy.deepcopy(AutoCanonical())</code> . Am I right ?</p>

username: albanD
cooked: <p>Yes that would work as well.</p>
Does torch.einsum optimize order?

username: chausies
cooked: <p>For example, <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html" rel="nofollow noopener">numpy’s einsum</a> has an “order” and an “optimize” parameter, which lets you either specify exactly the order to eliminate dimensions, or whether or not/how elimination order should be specified. For example, optimize=‘greedy’ just uses a simple and fast greedy algorithm to find a good ordering.</p>
<p>Does <a href="https://pytorch.org/docs/stable/torch.html?highlight=einsum#torch.einsum" rel="nofollow noopener">torch.einsum</a> optimize the elimination ordering at all?</p>

username: tom
cooked: <p>No. It has been on my list of things to do when I have time ever since I implemented einsum, but it never happened.  You’d make a lot of friends if you implemented that. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The (third party) <a href="https://github.com/dgasmith/opt_einsum" rel="nofollow noopener">opt_einsum</a> package supports PyTorch, so you could use that, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: chausies
cooked: <p>I’m not brave enough to try my hand at it &gt;.&lt;. But if you want my armchair suggestion: at least implementing the greedy approach (even defaulting to it) would be great, as it gets you basically 95% percent of the way from being prohibitively slow to blazing fast.</p>
<p>Thanks so much for making the einsum function in the first place!</p>

username: Carsten_Ditzel
cooked: <p>I am using einsum and am interested:</p>
<p>How much is the loss in speed?</p>

username: chausies
cooked: <p>The computation time can be orders of magnitude worse on a bunch of every-day cases. The computation order that einsum uses needs to be at least greedily optimized (which takes hardly any time). I underestimated this fact before, and it gave me an enormous performance hit on what would’ve been a simple computation.</p>
<p>It would be prudent to perhaps test in numpy at least to see if the default einsum computation order is fine (as compared to the optimized order that numpy will let you use).</p>
Custom operator segfault with CUDA 10.2 and PyTorch 1.5

username: tbroad
cooked: <p>I have just installed up CUDA10.2 and PyTorch using pip. I am ubuntu 18.04 and using Python 3.6.9.</p>
<p>I have built a series of custom operators using these instructions: <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#building-the-custom-operator" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#building-the-custom-operator</a></p>
<p>The only difference I have made is to the CMake file is changing the line:</p>
<pre><code class="lang-auto">target_compile_features(dilate PRIVATE cxx_range_for)
</code></pre>
<p>To:</p>
<pre><code class="lang-auto">target_compile_features(dilate PRIVATE cxx_std_14)
</code></pre>
<p>As the latest version of PyTorch uses the std library for c++14 so that I had to make this change to get it to compile.</p>
<p>The custom operators compile but they segfault when I try to use them in the code I am running. It segfaults with both the matching libtorch library downloaded from the PyTorch website (<a href="https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.5.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.5.0.zip</a>) and it also segfaults when I compile using the libtorch libraries in the PyTorch package installation folder: <a href="https://discuss.pytorch.org/t/segmentation-fault-when-loading-custom-operator/53301/8" class="inline-onebox">Segmentation fault when loading custom operator</a></p>
<p>I was also having this issue when using CUDA 10.1 on this installation of ubuntu so I am not sure what the problem is. I had this working with CUDA 10.1 on an old installation of Ubuntu (with PyTorch 10.4) but unfortunately I have lost access to this install and I have not been able to recreate the environment. I also have been unable to get this to work using nvidia-docker.</p>

username: tbroad
cooked: <p>If anyone has any suggestions of the best way to set up a new Ubuntu environment to get the custom operators to work that would be much appreciated also.</p>

username: tbroad
cooked: <p>Ah I finally figured out the issue. It had nothing to do with the version of CUDA or Ubuntu. I was getting a segfault because I was massing in a cuda tensor and then try and access the memory with a CPU OpenCV Mat. Converting the tensor to CPU before passing it to opencv fixed the issue.</p>
Pytorch TypeError - eq() received an invalid combination of arguments

username: icelandszt
cooked: <p>I’m working on a text classification problem with BERT. When training on the local machine everything works just fine, but when switching to the server, I get the following error:</p>
<pre><code class="lang-auto"> &lt;ipython-input-28-508d35ac5f5f&gt; in flat_accuracy(preds, labels)
          5     pred_flat = np.argmax(preds, axis=1).flatten()
          6     labels_flat = labels.flatten()
    ----&gt; 7     return np.sum(pred_flat == labels_flat) / len(labels_flat)
          8 
          9 # Function to calculate the f1_score of our predictions vs labels
    
    TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
     * (Tensor other)
          didn't match because some of the arguments have invalid types: (numpy.ndarray)
     * (Number other)
          didn't match because some of the arguments have invalid types: (numpy.ndarray)
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto"> def flat_accuracy(preds, labels):
        pred_flat = np.argmax(preds, axis=1).flatten()
        labels_flat = labels.flatten()
        return np.sum(pred_flat == labels_flat) / len(labels_flat)
</code></pre>
<p>Torch version on local machine: <code>1.4.0</code></p>
<p>Torch version on the server: <code>1.3.1</code></p>
<p>Any help would be greatly appreciated!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes, we used not to support operations between numpy arrays and torch Tensors. This was added in recent versions.</p>
Why my input tensor does need to allocate so much memory?

username: antonio
cooked: <p>I followed <a href="https://discuss.pytorch.org/t/how-to-know-the-memory-allocated-for-a-tensor-on-gpu/28537">this previous thread</a> to measure how much memory is being allocated for the input tensor to my CNN. After applying the formula <code>tensor.element_size() * tensor.nelement()</code> I discovered that each example allocates 1.18 Mb. The dimension of this tensor is (batch_size, num_images, input_height, input_width, n_channels) = [1, 6, 128, 128, 3] and dtype is float32. These 6 images have a disk size from 4 to 7 Kb at most. It is normal that such an input tensor needs to allocates so much memory? Thx</p>

username: albanD
cooked: <p>Hi,</p>
<p>When we send things to the disk, we always compress them.<br>
But when they are on memory, to be able to use them efficiently, we cannot keep them compressed. So yes it is expected that they take more time in memory than on disk.</p>

username: antonio
cooked: <p>Good to hear that. I was afraid my input was too big to train the CNN in a reasonable amount of time. Thanks</p>
How can you train your model on large batches when your GPUs can only hold couple of batches

username: Oscar_Rangel
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am writing this message for you as you always have helped me with very good answers.</p>
<p>I am doing Kaggle competitions but I always run on the problem that I can run bigger batch size and get really bad results.</p>
<p>I have two 2080 ti with 11 Gig of memory and trying to run images 300x300 with batch size 8 give me very bad results and with 16 it always tells me that CUDA ran out of memory…</p>
<p>Can you help me, please</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you have two GPUs, you can use the <code>DataParallel</code> module to use both GPUs which will allow you to double the batch size.</p>
<p>Another approach is to compute the whole gradient in multiple steps:</p>
<pre><code class="lang-auto"># effective_batch = 4 * batch
processed_batch = 0
for batch in dataloader:
  loss = get_loss(batch)
  loss.backward()
  processed_batch += 1
  if processed_batch == 4:
    opt.step()
    opt.zero_grad()
    processed_batch = 0
</code></pre>

username: Oscar_Rangel
cooked: <p><a class="mention" href="/u/alband">@albanD</a>, I tried that, the biggest batch size is 8… for the code you posted it is done using  torch.utils.checkpoint to trade compute for memory, but I dont know how to implement it…</p>
<p>Thanks.</p>

username: albanD
cooked: <p>checkpoint is different from what I posted.</p>
<pre><code class="lang-auto">batch_size_you_want = 64
max_batch_size = 8 # create dataloader with this

processed_samples = 0
for batch in dataloader:
  loss = get_loss(batch)
  loss.backward()
  processed_samples += max_batch_size
  if processed_samples &gt;= batch_size_you_want:
    opt.step()
    opt.zero_grad()
    processed_samples = 0
</code></pre>
<p>This code will do training as if you were using a batch size of 64. Without ever using more memory than if you were using a batch size of 8.</p>
<p>checkpointing is a bit different. You will need to modify you model’s forward pass to wrap groups of operations in a <code>torch.utils.checkpoint</code>. This will allow you to free the memory used by the intermediary buffers that are between the ops that you gave to the checkpoint.</p>
Wide Self Attention

username: omer_sahban
cooked: <p>In the current implementation of nn.MultiheadAttention, model_dim has to be equal to num_heads * kqv_dim, is it possible to break this limitation without reimplementing MHA? for example, I want the input and output of multi-head attention to be 512 dimentions, while having 24 heads with 64dim each head. And if that’s not possible, could someone recommend me a good implementation of multi-head attention?<br>
I used this one <a href="https://github.com/pbloem/former/blob/master/former/modules.py" rel="nofollow noopener">https://github.com/pbloem/former/blob/master/former/modules.py</a> to train a small BERT style language model on wikitext103, but it’s not to converging for some reason.</p>

username: omer_sahban
cooked: <pre><code class="lang-auto">class MHSA(nn.Module):
  def __init__(self,
         emb_dim,
         kqv_dim,
         num_heads,
         residual=True,
         norm=True,
         dropout=0):
    super(MHSA, self).__init__()
    self.emb_dim = emb_dim
    self.kqv_dim = kqv_dim
    self.num_heads = num_heads
    self.residual = residual
    self.dropout = nn.Dropout(dropout)
    if norm:
      self.layer_norm = nn.LayerNorm(self.emb_dim)
    else:
      self.layer_norm = nn.Identity()

    self.w_k = nn.Linear(emb_dim, kqv_dim * num_heads)
    self.w_q = nn.Linear(emb_dim, kqv_dim * num_heads)
    self.w_v = nn.Linear(emb_dim, kqv_dim * num_heads)
    self.w_out = nn.Linear(kqv_dim * num_heads, emb_dim)

  def forward(self, x):
    b, t, _ = x.shape
    e = self.kqv_dim
    h = self.num_heads

    x = self.dropout(x)

    keys = self.w_k(x).view(b, t, h, e)
    values = self.w_v(x).view(b, t, h, e)
    queries = self.w_q(x).view(b, t, h, e)

    keys = keys.transpose(1, 2)
    queries = queries.transpose(1, 2)
    values = values.transpose(1, 2)

    keys = keys / (e ** 0.25)
    queries = queries / (e ** 0.25)

    dot = queries @ keys.transpose(2, 3)
    dot = F.softmax(dot, dim=-1)

    out = dot @ values
    out = out.transpose(1,2).contiguous().view(b, t, h * e)
    out = self.w_out(out)
    if self.residual:
      out = out + x
    out = self.layer_norm(out)
    return out
</code></pre>
<p>Could someone point out where the problem is?</p>
Multiprocessing not working on Pytorch on MacBook

username: Megh_Bhalerao
cooked: <p>I am training a model on my local MacBook. My dataloader looks something like this:</p>
<p><code>data_train_source = MNISTSourceTrain("path1","path2")</code><br>
<code>source_train_loader = DataLoader(data_train_source,batch_size=64,shuffle=True,num_workers = 0) </code></p>
<p>When I set <code>num_workers=0</code>, the program runs. But, for any value of <code>num_workers&gt;0</code>, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.
        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom

        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
</code></pre>
<p>However when run the program with num_workers=2 or 4 or any other other number greater than 0, on any other OS i.e. Linux Ubuntu it works just fine.</p>
<p>Can someone please elaborate why does this happen? Or any suggestions how do I do something to stop it?</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentionned in the error, does your program has the proper <code>if __name__ == '__main__':</code> guard?</p>

username: Megh_Bhalerao
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a><br>
Yes, it works after inserting the <code>if __name__ == 'main'</code> guard. But, I am just curious why does it not work otherwise? Also, what does <code>num_workers=0</code> mean? How can the number of processes be zero when fetching the batches from the data loader?</p>

username: albanD
cooked: <p>This is a quirk of python’s multiprocessing I’m afraid: <a href="https://stackoverflow.com/questions/20360686/compulsory-usage-of-if-name-main-in-windows-while-using-multiprocessi">https://stackoverflow.com/questions/20360686/compulsory-usage-of-if-name-main-in-windows-while-using-multiprocessi</a></p>
How to convert model format from PyTorch to tflite?

username: syarudhi
cooked: <p>I want to convert trained model from PyTorch to tflite.<br>
I saved trained model bellow.<br>
torch.save(net.to(“cpu”).state_dict(), ‘mobilenet_v2.pth’)</p>
<p>How to convert model format from PyTorch to tflite?</p>
<p>python 3.5.6<br>
pytorch 1.3.1<br>
torch 1.4.0<br>
torchvision 0.4.2<br>
tensorflow 2.0.0</p>

username: David_Reiss
cooked: <p>We don’t officially support this.  It might be possible by using ONNX.</p>

username: glenn.jocher
cooked: <p>Yes, first export to ONNX, then onward to the format of your choosing.</p>

username: mcPytorch
cooked: <p>I tried this, but get the problem of pytorch and onnx working witch NCHW Tensor order, while tensorflow / tflite expects NHWC.<br>
Has anyone been able to solve this?<br>
I was trying to convert a vanilla mobilenet V2 model and ran into this issue as explained <a href="https://stackoverflow.com/questions/61679908/pytorch-convert-2d-cnn-model-to-tflite" rel="nofollow noopener">here on stackoverflow</a></p>
<p>My hunch is that Mobilenet V2 is too new for onnx…<br>
Here is my Configuration:</p>
<pre><code class="lang-auto">torch                1.6.0.dev20200508 (needs pytorch-nightly to work with mobilenet V2 from torch.hub)
tensorflow-gpu       1.14.0
onnx                 1.6.0              
onnx-tf              1.5.0 
</code></pre>

username: vferrer
cooked: <p>Try <a href="https://github.com/nerox8664/pytorch2keras" rel="nofollow noopener">https://github.com/nerox8664/pytorch2keras</a> Then, export it to tflite. Six months ago, I managed to export it. However, I recommend you to try pytorch mobile. It’s almost as fast as tflite but completely flawless.</p>
Prediction error while generating confusion matrix

username: maxiuw
cooked: <p>Hi, after trying a couple of solutions I am getting two errors. I have my model.to(device) defined and all the simulation runs correctly. The problem appears just while evaluation of the code with a confusion matrix in this piece of code:</p>
<h1>heatmap of confusion matrix</h1>
<p>truelabels = []<br>
predictions = []<br>
model.eval()<br>
print(“Getting predictions from test set…”)<br>
for data, target in test_loader:<br>
data = data.cpu().numpy()<br>
# data.to(device=device, dtype=torch.cuda.float)<br>
# data, target = data.to(device).cpu(), target.to(device).cpu()<br>
# data = torch.from_numpy(data).float().to(‘cuda’)<br>
# target = target<br>
for label in target.numpy():<br>
truelabels.append(label)<br>
for prediction in model(data).data.argmax(1):</p>
<pre><code>    predictions.append(prediction)
</code></pre>
<p>I m getting either this. I tried to put it data.to(cuda),<br>
then I also tried to put data.cpu.numpy(), but nothing really works… I would be greatful for any help</p>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same.<br>
or<br>
TypeError: conv2d(): argument ‘input’ (position 1) must be Tensor, not numpy.ndarray</p>

username: maxiuw
cooked: <p>SOLVED:</p>
<p>In calss function:<br>
def forward(self,x):</p>
<h1>adding</h1>
<p>x =x.cuda()</p>
<p>and adding<br>
for prediction in model(data).data.**cpu().**numpy().argmax(1):</p>
<pre><code>    predictions.append(prediction)</code></pre>
To install torch with OpenBLAS on Windows10

username: Adder
cooked: <p>I installed torch with pip, and I found that it didn’t detect the correct BLAS.<br>
I just wonder if there is any way to easily install a binary file use OpenBLAS instead MKL, or I should build it myself?</p>
<p>Besides, if I use OpenBLAS as the libary, could the installed torch with MKL work well?</p>
<pre><code class="lang-auto">#
# Name                    Version                   Build  Channel
blas                      2.16                   openblas    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libopenblas               0.3.9                hcfe7411_0    conda-forge
</code></pre>
<pre><code class="lang-auto">In [4]: torch.__config__.show().split("\n")
Out[4]:
['PyTorch built with:',
 '  - C++ Version: 199711',
 '  - MSVC 191627039',
 '  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191125 for Intel(R) 64 architecture applications',
 '  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)',
 '  - OpenMP 200203',
 '  - CPU capability usage: AVX2',
</code></pre>

username: peterjc123
cooked: <aside class="quote no-group" data-username="Adder" data-post="1" data-topic="80553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e79b87/48.png" class="avatar"> Adder:</div>
<blockquote>
<p>if I use OpenBLAS as the libary, could the installed torch with MKL work well?</p>
</blockquote>
</aside>
<p>It will use MKL instead and doesn’t rely on any external packages.</p>
<aside class="quote no-group" data-username="Adder" data-post="1" data-topic="80553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e79b87/48.png" class="avatar"> Adder:</div>
<blockquote>
<p>I just wonder if there is any way to easily install a binary file use OpenBLAS instead MKL, or I should build it myself?</p>
</blockquote>
</aside>
<p>You’ll need to compile it yourself.</p>
Input image shape for resnet18 architecture

username: MJN
cooked: <p>can the input image shape be 28x28 for training or testing renet18? I read that input image shape should be multiples of 32x32 but is that really required or does present resnet18 take care by itself?<br>
I trained the resnet18 on MNIST data set their shape is 28x28. But i did not face any problem while training or testing, even though the shape is not a multiple of 32x32. Is this okay or do i need to train the model again by resizing the image to 224x224?</p>

username: futscdav
cooked: <p>If it’s not a multiple of 32, you will lose some values in the intermediate layers due to having odd sizes of tensors. That may or may not be a problem for the problem you are solving, but in general, it would be better to resample the images to at least 32x32 if you wish to use unmodified resnet18 architecture.</p>

username: MJN
cooked: <p>thanks for the answer. So there is no adaptive pooling layer in resnet which makes image shape to a multiple of 32x32 as in fast ai (<a href="https://forums.fast.ai/t/why-mnist-works-with-resnet-without-throwing-error/57855" rel="nofollow noopener">https://forums.fast.ai/t/why-mnist-works-with-resnet-without-throwing-error/57855</a>)?</p>

username: futscdav
cooked: <p>The adaptive pooling only happens right before the classifier part of the network and just serves to average the size down to a single feature vector. Issue with something like 28x28 is that since there are several layers with <code>stride=2</code>, you will be disproportionally losing values out of the right side of the image. Again, it probably won’t be an issue for MNIST, because the image is mostly uniform background near the sides anyway.</p>

username: MJN
cooked: <p>It is clear now, thanks!</p>