akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is diluted conv net requires more computation?

ljj7975: <p>So I have two networks one <code>with</code> diluted conv2d layer and one <code>without</code> diluted conv2d layer.<br>
When I run non-diluted network, I only see 16% of Volatile GPU-Util from GPU usage and it’s progress is decent, (meaning fast).<br>
However, when I just add dilation (dilation=True), it actually uses 100% Volatile GPU-Util and the progress is much slower.</p>
<p>I just got curios if this behavior is expected because it is such a huge difference for a single flag change</p>

ptrblck: <p>The time spent on the device (and thus the performance) not only depends on the theoretical number of operations, but also on the implemented algorithm. E.g. if you are using cudnn, you might see a difference between highly optimized kernels for common use cases and “uncommon” cases.</p>
<p>I’m not sure, if you are using cudnn for your workloads and thus cannot comment on the applied kernels internally for your convs.</p>
Pytorch on Macbook pro with AMD eGPU?

benkolber: <p>Hello everyone!</p>
<p>I’m using a MacBook pro (2019 Catalina) connected to a Radeon Vega eGPU via a thunderbolt 3 cable (razer core x). The eGPU appears in my system report, but I’m finding it very hard to run PyTorch models on it. There seems to be no clear answer.</p>
<p>I am aware of OpenCL, as well as some hackish way to get Nvidia cards to work with Pytorch on mac’s, but I was wondering if anyone has a novel solution given my setup?</p>
<p>If not, I would love to hear other suggestions for ways to run my models in a fast and efficient way. The corona virus got me working from home, and it really takes ages for each model to run, I am quite desperate at this point!</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Stacking augmented samples in dataloader

Khadija_Khaldi: <p>I have a network, that takes two augmented images (xi,xj) of the same sample x. I stack the two images in one input in the <strong>getitem</strong> function.<br>
Now, in my training function, how can I unstack the images (xi,xj) for each input in each batch, before feeding them to the model?<br>
Thank you so much!</p>

charan_Vjy: <p>You could try stacking the images in a way that allows you to easily retrieve out the 2 images, as shown below.</p>
<pre><code class="lang-auto">import torch
batch_size = 16
img_h,img_w = 64,64
img_1 = torch.randn(batch_size,1,img_h,img_w)
img_2 = torch.randn(batch_size,1,img_h,img_w)
### Use this method to stack the inputs in the get item method
concat_inp = torch.cat((img_1,img_2),dim = 1)
### The repective images can be obtained by indexing
inp_1 = concat_inp[:,0]
inp_2 = concat_inp[:,1]
</code></pre>
Сhange weights in the net

slavavs: <p>I have</p>
<pre><code class="lang-auto">class Network(torch.nn.Module):
    def __init__(self, input_size, out_size):
        super(Network, self).__init__()
        self.input_size = input_size
        self.out_size = out_size
        self.lin = nn.Linear(self.input_size, self.out_size, bias = False)
        nn.init.zeros_(self.lin.weight)
</code></pre>
<pre><code class="lang-auto">net = Network(input_size, output_size)
</code></pre>
<p>i need <code>net.lin.weight = net.lin.weight + delta</code></p>
<p>I get an error: TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</p>
<p>How can I fix this?</p>

ptrblck: <p>As the error message suggests, wrap the new tensor in <code>nn.Parameter</code>:</p>
<pre><code class="lang-python">with torch.no_grad():
    net.lin.weight = nn.Parameter(net.lin.weight + delta)
</code></pre>
HUGE loss function

Oleg: <p>I solved the standard task of predicting the price of a house, but my loss function does not decrease after few steps. I tried to change the learning speed, add and delete hidden layers, change the activation function. In general I did all the standard actions but the loss function is still huge<br>
Here is my code <a href="https://colab.research.google.com/drive/1LnPijEDV1sy7S6oSf9N0WzyykqbRhXxh" rel="nofollow noopener">https://colab.research.google.com/drive/1LnPijEDV1sy7S6oSf9N0WzyykqbRhXxh</a><br>
Here is dataset<br>
<a href="https://drive.google.com/file/d/1TLq0fJZJQ7Ne8ajL5C6cc8oLP4g66fkW/view?usp=sharing" rel="nofollow noopener">https://drive.google.com/file/d/1TLq0fJZJQ7Ne8ajL5C6cc8oLP4g66fkW/view?usp=sharing</a><br>
HUGE thanks, if u find my mistake and explain to me what’s wrong</p>

fs4ss1: <p>Hello.</p>
<p>The magnitude of your target is very high, this implies that your loss value is high.</p>
<p>If you analyze your code you can see the loss value, after the ending of training, aroud 70000.</p>
<p>If you do  sqrt(70000),  you got an mean error of 294 in prices, what is reasonable.</p>
Torch.cuda.is_available() == True using Python 3.7.1, but False on Python 3.6.9

m-k-S: <p>Hi,</p>
<p>I’m trying to run some code on a VPS that I don’t have many permissions for (and I won’t be able to get requisite permission). For some reason, I am not allowed to install packages for the Python 3.7.1 installation on this machine, but I can do so just fine for the Python 3.6.9 installation.</p>
<p>I have gotten my repository set up properly with all required packages, and my code runs, but when I try to train my model, it runs on the CPU. Upon further investigation, I found that opening a Python 3.6.9 interpreter and running <code>torch.cuda.is_available()</code> returns False. However, opening a Python 3.7.1 interpreter and running the same command returns True. Furthermore, in the 3.6.9 interpreter, <code>torch.backend.cudnn.enabled</code> returns True. Is there any way I can fix this? I am running <code>torch 1.4.0</code>, <code>nvidia-smi v410.104</code>, and <code>nvcc release 9.1, V9.1.85</code>.</p>
<p>Thanks!</p>

ptrblck: <p>How did you install both PyTorch versions for these different Python versions?<br>
Did you install the initial CUDA version in your Python3.7 environment?</p>
Convert pickle to .pth

kbtorcher: <p>I have a .pkl file from a Model Zoo. Is it possible to convert it into .pth file?</p>

jmandivarapu1: <p>One way to do it is</p>
<ol>
<li>Flatten that .pkl file</li>
<li>Initialize a new model</li>
<li>Reload the flatten weights</li>
<li>Save then t .pth<br>
(This only works if you can initialize the model with the same shapes)</li>
</ol>

Belmondo: <p>Could you point some tutorial<br>
I have production-ready .pkl model (with GPT2) and i need fine-tune it with a new dataset, but not sure how to do that</p>
Setting up unique custom filters in Conv2D for each image?

angadi: <p>Hello</p>
<p>I know how to set up a custom filter for conv2D but, How do i set up unique filters for each image in my batch. For example  if I have 10 images in my batch, and let’s say I want to apply k filters of size 3 x 3 to each image, Now I want to be able to feed unique K filters to each of the 10 images something like 10 x k x num_channels x 3 x 3. How do I do that?</p>

JuanFMontesinos: <p>Just define 10 different convolutions in the main nn.Module and redirect each sample to each convolution. Later on collect them and concatenate them back.</p>
Torchvision rpn anchor_sizes can't be modified

Deltap: <p>Hi,<br>
I inherited torchvision.models.detection.rpn.RegionProposalNetwork to add some customized features to my Faster R-CNN(also based on torchvision), and I found that when editing the anchor sizes of anchor generator, there will be an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 17, in &lt;module&gt;
    train()
  File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 10, in train
    best_path = source_train()
  File "/home/shiqian/zhangzh/FAFRCNN/source_train.py", line 78, in source_train
    losses, detections, image_sizes, original_image_sizes = faster_rcnn_mp(images, targets)
  File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 213, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 382, in forward
    proposals = self.box_coder.decode(pred_bbox_deltas.detach(), anchors)  # Tensor at GPU4
  File "/home/shiqian/.local/lib/python3.5/site-packages/torchvision/models/detection/_utils.py", line 168, in decode
    rel_codes.reshape(sum(boxes_per_image), -1), concat_boxes
RuntimeError: shape '[1670004, -1]' is invalid for input of size 6779988
</code></pre>
<p>I just modified the initial Faster R-CNN initialization of</p>
<pre><code class="lang-auto">        if rpn_anchor_generator is None:
            anchor_sizes = ((32,), (64,), (128,), (256,), (512,))
            aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
            rpn_anchor_generator = AnchorGenerator(
                anchor_sizes, aspect_ratios
            )
</code></pre>
<p>to</p>
<pre><code class="lang-auto">        if rpn_anchor_generator is None:
            anchor_sizes = ((128,), (256,), (512,))
            aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
            rpn_anchor_generator = AnchorGenerator(
                anchor_sizes, aspect_ratios
            )
</code></pre>
<p>as my GPU can’t handle so many anchors.<br>
Anybody could help me out?</p>

Deltap: <p>Looks like it is continuously missing some fixed ratio of anchors:<br>
I tried three times, the number above is respectively:<br>
1158948  4705188  missing 17349 * 4<br>
1079568  4382736  missing 16116 * 4<br>
1063692  4318476  missing 15927 * 4<br>
constantly missing 0.01497 of the anchors.</p>

NDana: <p>I’m encountering the same issue myself. I believe you also need to modify the construction of <code>roi_heads</code>, as it likely is expecting to recommend boxes on more scales than your RPN generator is expecting. But I haven’t entirely figured out what I’m doing wrong yet. I’ll post if I discover the error</p>
The second model on the same GPU slows down inference. Why?

Jan_Vainer: <p>I have two models. One model generates spectrograms and the other uses the spectrogram to generate audio. Now, the first run is fast for both models, but in the second run, the first model slows down significantly. Why? My loop looks as follows</p>
<pre><code class="lang-auto">t = time.time()
with torch.no_grad():
    spectrograms = model1(phonemes)
print(time.time() - t)  # 35 ms

t = time.time()
with torch.no_grad():
    audio = model2(spectrograms)
print(time.time() - t) # 7 ms

t = time.time()
with torch.no_grad():
    spectrograms = model1(phonemes)
print(time.time() - t)  # 2640 ms
</code></pre>
<p>Why should be the second call to model1 so slow? The slowness persists for consecutive runs.<br>
It does not make sense to me, that the first run is so fast and the consecutive runs so slow.<br>
If i drop the second model and only call the first model repeatedly, th eproblem persists, but the duration of the consecutive calls drops to around 133 ms.</p>
<p>Any ideas would be appreciated!</p>

Jan_Vainer: <p>Alright, it was caused by not using torch.cuda.synchronize().<br>
I had to call this command before calling the timer and before calculating the final duration. Only then I started getting consistent timing.</p>
Weight Quantization

amin_sabet: <p>I’m trying to implement the fixed point version of VGG 16. I want to start with the pre-trained VGG 16 with the floating point weight precision, then I wand to add a quantization layer  before each convolutional layer which quantized the floating point weights into fixed point format (e.g., 8bits ) before multiplied by feature map in conolutional layers. My quantization function is   :</p>
<p>wq = clip(round(w/stp), a,b)</p>
<p>where w,wq,stp, a and b are floating point weight, quantized weight, step size , min value and max value, respectively.</p>
<p>Then I want to fine tune my model with the quantized wieght.<br>
So far, I have defined a new layer as quantization layer which accept the floating point weight as the input and returns the quantized value of weight. here is my questions:</p>
<ol>
<li>Is the the best way to impeliment fixed point network?</li>
<li>Do I need to define a new backward method to be used during the traning(fine tuning) process ?</li>
<li>How can I feed the weight of each convolution layer to the quantization layer ? or How can I use this layer in my model architecture?</li>
</ol>
<pre><code class="lang-auto">class Linear_Quantization(nn.Module):
  def __init__(self):
    super().__init__()
    self.bit_width=8
    self.step_size = Parameter(torch.Tensor(1))
  
  def forward(self, x):
    x = torch.round(x/self.step_size) 
    x = torch.clamp(x, min = -2**(self.bit_width-1) , max = (2**(self.bit_width-1)-1))
    x = x*self.step_size
    return x
</code></pre>

Sairam954: <p>Hey Amin,<br>
Were you able find answers for these, I am trying to solve similar issue it would be helpful if you can share the information</p>

amin_sabet: <p>Hey Samira,</p>
<p>Recently, Pytorch has provided a wide range of quantization approaches. You can find more details <a href="https://pytorch.org/docs/stable/quantization.html" rel="nofollow noopener">here</a>.</p>
<p>Regards<br>
Amin</p>

Sairam954: <p>Thank you, Amin<br>
I was looking to quantize a model to 4 bit, currently pytorch has it implemented for 8 bit. I was trying to create my own Qconfig and provide the FakeQuantization with minlevel and maxlevel. Do you think this is the right way of achieving it?</p>

mg1371: <p>I am trying to do the same thing Sairam954. Has it worked for you that way? Technically, that makes sense.</p>
Convert torch._C.Graph to torch.jit.Graph

Sharath_R: <p>Hi,<br>
Is there any way to convert torch._C.Graph to torch.jit.Graph. Or is there a way generate a dot file from the torch._C.Graph. I would like to create a dot file from tracing like in <a href="https://github.com/szagoruyko/pytorchviz/blob/master/torchviz/dot.py" rel="nofollow noopener">make_dot_from_trace</a> function. But it takes only torch.jit.Graph as input.</p>
<p>When I ran my model through <a href="https://pytorch.org/docs/stable/jit.html" rel="nofollow noopener">torch.jit.trace(model, input)</a>; I  got a object whose .graph was of type torch._C.Graph which throws error when I keep it as a parameter for. make_dot_from_trace function.</p>

wanchaol: <p>Hi so we don’t have a <code>torch.jit.Graph</code> object to begin with I think, all graph objects are <code>torch._C.Graph</code>, so I think if pytorchviz is using <code>torch.jit.Graph</code> that could be already a issue, you probably should submit a issue or a fix to the pytorchviz repo instead.</p>
Half precision: ignoring a buffer

sebastienwood: <p>Hi,</p>
<p>I’m trying to switch a model to <code>.half()</code>. However, I defined a buffer which keeps tracks of the number of activation for a convolution layer that is used repeatedly in the network. Since the number of activations grows to be very large, half precision for this particular buffer overflow.</p>
<p>The code looks like so :</p>
<pre><code class="lang-auto">class convmodule(nn.Module):
  def __init__(self):
    self.register_buffer('fan_in', torch.Tensor([0]))
    (...)

  def update_fan_in(self, qty):
        self.fan_in += qty

# The forward hook
def count_act(self, input, output):
    """Hook to update the size of feature maps going through convmodule
    """
    with torch.no_grad():
        if str(output.device)=='cuda:0' or str(output.device)=='cpu': 
            self.update_fan_in(output.numel())
</code></pre>
<p>Is there a way to let this particular buffer stay in full precision while putting the rest of the tensors in half ?</p>
<p>(I was thinking of switching this buffer to pure python integer to avoid it being cast to half, but it messes with DataParallel – any idea is welcome ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ).</p>

sebastienwood: <p>I tried at initialisation :</p>
<pre><code class="lang-auto">net = net.to(device).half()
for layer in net.modules():
    if isinstance(layer, nn.BatchNorm2d) or isinstance(layer, convmodule):
        layer.float()
        if isinstance(layer, convmodule): layer.fan_in.float()
</code></pre>
<p>I’m not sure if it’s legit but it’s a bit better on GPU RAM occupancy.</p>
Trainable HardShrink threshold

Ajinkya_Ambatwar: <p>Hi, how can the behavior of torch.nn.Hardshrink (found here <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Hardshrink" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Hardshrink</a>) be implemented with a learnable <code>lambd</code> parameter?</p>
<p>We’ve tried wrapping it as nn.Parameter as shown below but it doesn’t seem to work.</p>
<pre><code class="lang-auto">class foo(nn.Module):
    def __init__(self, eps = None):
        super(LossySparsifier, self).__init__()
        # initialize eps as trainable parameter
        if eps == None:
            self.eps = nn.Parameter(torch.tensor(0.1))
        else:
            self.eps = nn.Parameter(torch.tensor(eps))
            
        self.eps.requires_grad = True

    def forward(self, x):
        return torch.hardshrink(x, self.eps.data) # can't give self.eps directly here
</code></pre>
<p>We’ve also tried implementing it  ourselves using clamp function as follows:</p>
<pre><code class="lang-auto">class foo(nn.Module):
    def __init__(self, eps = None):
        super(LossySparsifier, self).__init__()
        # initialize eps as trainable parameter
        if eps == None:
            self.eps = nn.Parameter(torch.tensor(0.1))
        else:
            self.eps = nn.Parameter(torch.tensor(eps))
            
        self.eps.requires_grad = True

    def forward(self, x):
        e = torch.abs(self.eps)
        y = (x - e).clamp(min = 0)
        y[y != 0] = y[y != 0] + e
        z = (-x - e).clamp(min = 0)
        z[z != 0] = z[z != 0] + e
        return y - z
</code></pre>
<p>But even here, the gradient wrt eps seems to go to 0. Shouldn’t this not happen as autograd has support for indexing? The grad_fn for y after the indexing is done is <code>grad_fn=&lt;IndexPutBackward&gt;</code>. How is this implemented in pytorch?</p>
<p>Any help on how to proceed will be appreciated. Thank you.</p>

albanD: <p>Hi,</p>
<p>Autograd has support for indexing but cannot compute gradients when they don’t exist.<br>
In particular, given that lambda is a threshold, if you ask for the gradient wrt lambda, you will get a function that is 0 almost everywhere. So you cannot learn lambda by gradient descent-like methods.</p>

Ajinkya_Ambatwar: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, do you know any way by which we could set this up correctly? Maybe some sort of manual gradient method</p>

albanD: <p>If you have a better formula for your gradient, you can use a custom Function to specify what the backward should compute (see doc <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a>).<br>
But not sure what that formula would be as the gradient is just 0 here.</p>
Detach a intermediate variable without recomputing whole graph

sbaio: <p>Hi,</p>
<p>I have a situation that looks like this:</p>
<pre><code class="lang-auto">model = resnet50()
classifier = nn.Linear(1000, 20)

x = model(input)
out1 = classifier(x)
out2 = classifier(x.detach()) # Computation to avoid
</code></pre>
<p>where the classifier is expensive and would like to avoid recomputing out2_d = classifier(x.detach())</p>
<p>Can we use out1 to backward with x detached ?</p>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>You mean that you want to get the gradients for the elements in the classifier but not the ones in model?<br>
If s, you can use <code>autograd.grad</code> to specify what you want the gradients for: <code>grads = autograd.grad(loss, classifier.parameters())</code>.</p>

sbaio: <p>Thank you for the quick response, actually my example was simplifying my case too much. Let me provide a better example.</p>
<p>I’m not trying to compute the gradient of the loss w.r.t to my classifier parameters only.</p>
<pre><code class="lang-auto">x1, x2 = model(x)
out = heavy_module(x1, x2)
loss = Loss1(out)
loss.backward()

out_d = heavy_module(x1.detach(), x2)
loss_d = Loss2(out_d)
loss_d.backward()
</code></pre>
<p>Can we avoid the second heavy_module computation ? while having access to both out and out_d for different losses.</p>
<p>Thank you</p>

albanD: <p>You won’t be able to have two Tensors if you do a single forward <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Why can’t you reuse <code>out</code> to compute <code>loss_d</code>?</p>

sbaio: <p>If I use <code>out</code> to compute <code>loss_d</code>, then x1 won’t be detached and  Loss2 will have gradients at the parameters that produced x1 (which I want to avoid).</p>

albanD: <p>Hi,</p>
<p>You would need two completely different graphs here I’m afraid. So you won’t be able to do that without re-doing the forward.</p>
How to use a network B to obtain a tensor and use it to replace the weight of a layer of network A, and this back propagation process will train A and B

syiswell: <h3>How to use a network B to obtain a tensor and use it to replace the weight of a layer of network A and this backpropagation process will train A and B。</h3>
<p>I make a code, which uses the weight of a layer of network A as input into network B. then B output a tensor and I use it to replace the weight of a layer of the network A. But I find the weight only needs the type of nn.Parameter() and I convert the tensor to Parameter type, but I find the weight of network B does not get an update. Can you help me please!</p>
<p>It’s noted that I want to train the weight of network B by the loss of the network A.</p>
Edge case with register_hook

dfalbel: <p>This may be a fairly rare edge case, but wanted to confirm if this should be possible with the c++ api.<br>
Suppose I have the following program calling <code>backward</code> of a different graph inside the backward <code>hook</code> of another one from a different thread.</p>
<p>I always need to call backward in a different thread because the main thread must be available to execute the hooks, that in my case go back to a an interface that only supports calls from the main thread.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;thread&gt;

int main()
{

    auto x = torch::randn(1, torch::requires_grad());
    auto y = 2 * x;

    std::function&lt;void(torch::Tensor)&gt; hook1 = [](torch::Tensor grad) {
        std::cout &lt;&lt; "hey" &lt;&lt; std::endl;
    };
    x.register_hook(hook1);

    auto a = torch::randn(1, torch::requires_grad());
    auto b = 2 * a;

    std::function&lt;void(torch::Tensor)&gt; hook2 = [&amp;](torch::Tensor grad) {
        std::cout &lt;&lt; "hello" &lt;&lt; std::endl;
        std::thread t([&amp;]() {
            y.backward();
        });
        t.join();
    };

    a.register_hook(hook2);

    std::thread t2([&amp;]() {
        b.backward();
    });
    // wait for hooks

    t2.join();

    return 0;
}
</code></pre>

albanD: <p>Hi,</p>
<p>Yes this will work. Do you see any issue with it?</p>

dfalbel: <p>Hi <a class="mention" href="/u/alband">@albanD</a></p>
<p>Yes, sorry! forgot to mention that this enters a lock state where it seems that torch is waiting a mutex unlock in order to run the backward operation inside the hook.</p>

albanD: <p>Ho right.<br>
The thing is that your hook actually waits on the other backward to finish because it waits on the the other thread.<br>
The thing is that because the hook is blocked waiting on this, another thread cannot use run backward (this current thread can though).<br>
So you either want to run this other backward in the same thread as the hook. Or not block the hook waiting on that backward.</p>

dfalbel: <p>OK! Thanks for your reply, I’ll figure out the best way to not spin up a new thread for each backward call.</p>
Average Pooling with Outlier Ignored?

soulslicer: <p>I’d like to use Average Pooling to downsample a depth map, but only including the points that are close to each other.</p>
<p>For example, if a pooling neighbourhood consists of</p>
<p>[4 4.1, 3.9 3.8] then we take the mean of the whole thing</p>
<p>but if it consists of</p>
<p>[4 4.1 3.9 0.1], then we should ignore 0.1 and take the mean of the rest</p>
<p>Does anything like that exist?</p>
Differentiable way of "Replace By Value" in pytorch

Ajinkya_Ambatwar: <p>Hello everyone,</p>
<p>I am trying to figure out is there any “replace by value” way in pytorch that is differetiable.<br>
For eg</p>
<pre><code class="lang-auto">a = torch.tensor([1.5, 2, 1.5, 3]) 
</code></pre>
<p>And I want to replace 1.5 with 0.</p>
<p>Is there a way that can be done and with the autograd supporting it?</p>
<p>Thanks!!</p>

tom: <p>Probably something like</p>
<pre><code class="lang-python">eps = 1e-7
z = torch.zeros((), device=a.device, dtype=a.dtype)
a_new = torch.where((a - 1.5).abs() &lt; eps, a, z)
</code></pre>
<p>probably is what you want.<br>
Note that <code>a == 1.5</code> might not be a good condition to test for due to the accuracy limits of floating-point computation.<br>
You can assign the new thing to <code>a</code> again if you want.</p>
<p>Best regards</p>
<p>Thomas</p>

Ajinkya_Ambatwar: <p>Hi <a class="mention" href="/u/tom">@tom</a>, I tried implementing what you have mentioned.</p>
<pre><code class="lang-auto">threshold=nn.Parameter(torch.tensor(1.5))
threshold.requires_grad=True
eps=1e-7
z=torch.zeros((),dtype=a.dtype)
a_new=torch.where((a-threshold).abs()&lt;eps,a,z)
</code></pre>
<p>Now I try to do the backward pass</p>
<pre><code class="lang-auto">a_new.sum().backward()
</code></pre>
<p>But I am getting RuntimeError:<br>
<code>element 0 of tensors does not require grad and does not have a grad_fn</code></p>

Ajinkya_Ambatwar: <p>Oh sorry, I had to declare z as the nn.Parameter and make <code>requires_grad=True</code> for z.<br>
That solved the problem.<br>
But<br>
<code>threshold.grad</code><br>
returns 0. So I am unclear if the gradient is being computed for threshold even though its <code>requires_grad</code> parameter is True. If not, am  I interpreting it wrong?<br>
Is there a way we can have a learnable threshold?</p>

albanD: <p>Well if you write down the gradient formula for the threshold here, you will just get 0 almost everywhere. So it is expected to return 0 <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
LSTM predicting inputs possible solution

Topsoil: <p>I have an LSTM for time seris predictions, and its problem is that it just “predicts” input values</p>
<p>Is calculating loss not only with labels (label is input sequence but shifted 1 step further) but with inputs too? So model would try not only to minimize difference between output and labels, but also maximize difference between output and input?</p>
<p>Something like loss = loss_function(y_pred,labels) + 1 / loss_function(y_pred,labels)</p>
<p>For some reason with this configuration it doesnt work, but is this a good/functional idea?</p>
Does .item() affect performance?

braindotai: <p>Hi there, as far I know we can fill our ram by not releasing the pointer to the computational graph, for example-</p>
<pre><code class="lang-python">losses = []
#...training on batch...
losses.append(loss)
</code></pre>
<p>The problem above is that loss still has a copy of the graph!!! So its recommended to do as follows in order to release it-</p>
<pre><code class="lang-python">losses.append(loss.item())
# not losses.append(loss)
</code></pre>
<p>My question is if we call <code>.item()</code> on loss for let’s say 2 to 3 times at each batch training step, then is it gonna affect the performance? Or are there any other issues that it might have?</p>

charan_Vjy: <p>.item() ensures that you append only the float values to the list rather the tensor itself. You are basically converting  a single element tensor value to a python number. This should not affect the performance in any way.</p>

richard: <p>as <a class="mention" href="/u/charan_vjy">@charan_Vjy</a> mentioned, <code>item()</code> is pretty inexpensive. Are you running into some situation where it is affecting the performance?</p>

braindotai: <p>Thanks <a class="mention" href="/u/charan_vjy">@charan_Vjy</a>, <a class="mention" href="/u/richard">@richard</a>. I just read somewhere that .item() has to do something related to the computational graph, so I thought it might affect performance and memory in some way. I ran some tests and found no difference as such (at least with small models like AlexNet).</p>

richard: <p><code>.item()</code> turns a Tensor into a Python number. Autograd can only keep track of computations on Tensors, so using the result of <code>.item()</code> effectively creates a new value that is not a part of the computation graph.</p>
Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Odin_Silverfang: <p>I moved my model to gpu</p>
<pre><code class="lang-auto">net.to(device)
</code></pre>
<p>but when i try to test the model , as per the<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">Pytorch tutorial</a></p>
<pre><code class="lang-auto">dataiter = iter(testloader)
images, labels = dataiter.next()

imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

outputs = net(images)  #This line throws the error
</code></pre>
<p>what is the issue ???</p>

JuanFMontesinos: <p>You need to pass images to device also.</p>

sai_tharun: <p>pass images to device by using below code</p>
<pre><code class="lang-auto">dataiter = iter(testloader)
images, labels = dataiter.next()
images,labels = images.to(device),labels.to(device)
</code></pre>

Odin_Silverfang: <p>Thanks man , it worked</p>

PARISA: <p>I have a problem. i mounted my data on google drive and when i want to train my model it has an issue same as below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fc8f46dc453c92bf7c27e77e1607fde24b89aea.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fc8f46dc453c92bf7c27e77e1607fde24b89aea" title="Screenshot (797)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc8f46dc453c92bf7c27e77e1607fde24b89aea_2_690x306.png" alt="Screenshot (797)" data-base62-sha1="kvYOXz4B2cL95BVhi4aM8VHb2zM" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc8f46dc453c92bf7c27e77e1607fde24b89aea_2_690x306.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc8f46dc453c92bf7c27e77e1607fde24b89aea_2_1035x459.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc8f46dc453c92bf7c27e77e1607fde24b89aea_2_1380x612.png 2x" data-dominant-color="F4F3F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (797)</span><span class="informations">1551×689 83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Make sure to push the input data to the device using:</p>
<pre><code class="lang-python">input = input.cuda()
# or
input = input.to('cuda')
</code></pre>

PARISA: <p>thankyou so much. it helped.</p>
<p>‫‪ptrblck via PyTorch Forums‬‏ &lt;‪noreply@discuss.pytorch.org‬‏&gt; در تاریخ دوشنبه ۶ آوریل ۲۰۲۰ ساعت ۸:۱۰ نوشت:‬</p>
Question about torchvision.utils.make_grid ()

SangYC: <p>Hello,</p>
<p>In the tutorials of “VISUALIZING MODELS, DATA, AND TRAINING WITH TENSORBOARD”,Use iterator to view data，as follow:</p>
<blockquote>
<blockquote>
<blockquote>
<p>testiter = iter(testloader)<br>
images, labels = testiter.next()<br>
images.shape<br>
Out:<br>
torch.Size([4, 1, 28, 28])</p>
</blockquote>
</blockquote>
</blockquote>
<p>Why after the function torchvision.utils.make_grid (), the output becomes like this.：</p>
<blockquote>
<blockquote>
<blockquote>
<p>torchvision.utils.make_grid(images).shape<br>
Out:<br>
torch.Size([3, 32, 122])</p>
</blockquote>
</blockquote>
</blockquote>
<p>I don’t understand why the channel changed from 1 to 3.</p>
<p>Thank you.</p>
Does triplet loss seem very problematic?

lycaenidae: <p>Hi, I am using triplet loss for my training data. At the beginning of the training why it seems so bad? I am wondering whether this behaviour can be considered as a bad loss? Thanks in advance</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14ba94accd0110099540b373570a41ab788b70b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/14ba94accd0110099540b373570a41ab788b70b9" title="Screenshot 2020-04-05 at 20.48.17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14ba94accd0110099540b373570a41ab788b70b9_2_690x408.png" alt="Screenshot 2020-04-05 at 20.48.17" data-base62-sha1="2Xnip52xya9dPKf32v5h6bVKECB" width="690" height="408" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14ba94accd0110099540b373570a41ab788b70b9_2_690x408.png, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14ba94accd0110099540b373570a41ab788b70b9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14ba94accd0110099540b373570a41ab788b70b9.png 2x" data-dominant-color="F6ECEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-04-05 at 20.48.17</span><span class="informations">990×586 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Clarify cpu() function

eranb: <p>Hi everyone,<br>
can someone explain to me, if I have some tensor and I’m using GPU, and I want to convert the tensor to NumPy  without gradient, example:<br>
img_var.detach().cpu().numpy()</p>
<p>img_var is tensor, so my question is if I’m only writing img_var.cpu().numpy() it’s not the same as what I write above.</p>
<p>Thanks!</p>

yash1994: <p>Hi,<br>
Let’s see what each method does:</p>
<p>1.<code>tensor.detach()</code>: creates a tensor that shares storage with  <code>tensor</code>  that does not require grad. You should use  <code>detach()</code>  when attempting to remove a tensor from a computation graph, and  <code>clone</code>  as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from.</p>
<p>2.<code>tensor.cpu()</code>: moves the tensor back to memory accessible to the CPU.</p>
<p>3.<code>tensor.numpy()</code>: converts tensor to numpy array <code>np.array</code>.</p>
<p>So <code>img_var.cpu().numpy()</code> is not same as <code>img_var.detach().cpu().numpy()</code> because it will move tensor to cpu without creating a storage shared copy of it. And would result in error for next forward pass of your graph computation.</p>

eranb: <p>OK,<br>
Thank you very much.</p>
Unpackbits from tensor

ManosMagnus: <p>Hello people,</p>
<p>I am trying to compute bit error rate as metric. Currently I am using numpy and more specific <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.unpackbits.html" rel="nofollow noopener">np.unpackbits(x)</a></p>
<p>I am wondering if there is a similar operation in pytorch to take advantage of gpu.</p>
<pre><code class="lang-auto">    def __call__(self, output, target):
        t_target = target.detach().cpu().numpy().astype(np.uint8)
        t_output = output.detach().cpu().numpy().astype(np.uint8)

        target_bits = np.unpackbits(t_target).reshape(
            -1, 8)

        output_bits = np.unpackbits(t_output).reshape(
            -1, 8)

        output_bits = output_bits.reshape(-1)
        target_bits = target_bits.reshape(-1)

        absolute = np.abs(
            output_bits.astype(np.int) - target_bits.astype(np.int))

        errors = np.sum(absolute)

        return errors
</code></pre>
Simple question reg reshaping/broadcasting

Miyako: <p>Hi there,</p>
<p>Sorry for the question, but I am still a rookie and I’m trying to understand how it works here:</p>
<p>I have a tensor x with shape (64,10) so 64 rows and 10 columns.<br>
Checking shape for the below denominator I get a shape 64, why it is not equal to (64,1) and I need to reshape my tensor while using a softmax function?</p>
<p>torch.sum(torch.exp(x), dim=1)  -&gt; shape 64</p>
<p>def softmax(x):<br>
return torch.exp(x)/torch.sum(torch.exp(x), dim=1).view(64, 1)</p>
<p>Thanks a lot!</p>

paganpasta: <p>Hi,</p>
<p>Softmax is essentially normalising the values in the range <code>[0, 1]</code> The input shape and output shape would be the same if you are using a softmax function. For reference you can check the documentation and implementation of sotmax <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Softmax" rel="nofollow noopener">here</a></p>

theevann: <p>The <code>torch.sum</code> function will by default remove the dimension on which you apply the sum:</p>
<pre><code class="lang-python">torch.sum(torch.exp(x), dim=1)  # shape (64)
</code></pre>
<p>To keep that dimension, you can use the <code>keepdim</code> optional parameter:</p>
<pre><code class="lang-python">torch.sum(torch.exp(x), dim=1, keepdim=True)  # shape (64,1) 
</code></pre>

Miyako: <p>Hi,</p>
<p>I understand what softmax is doing, but what I do not understand here is the shape of tensor in denominator. Why it’s 64 not (64,1) hence I need to reshape it to (64,1).</p>
<p>Thanks!</p>

Miyako: <p>Thank you so much, this is what I asked about! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Combinations of a set of tensors and GPU optimization

tani: <p>Hello everyone,</p>
<p>I am a Pytorch beginner and I would like to do the <strong>combinations of a set of 1D tensors with different shapes</strong> and then concatenate the results in a single output.<br>
I coded this, which give me the desired output:</p>
<pre><code class="lang-auto">import torch

# Input example
sub_data_idx = torch.tensor([2,  5,  5,  0,  4,  1,  4,  5,  3,  2,  1,  0,  3,  3,  0,  0,  2,  1]).cuda()
data_idx = torch.tensor([ 0,  1,  2,  3,  4,  5]).cuda()

# initialization data_combinations_pair for torch.cat
data_combinations_pair = torch.zeros([1, 2]).type(torch.LongTensor).cuda()

# Loop over the data indices each corresponding to a 1D tensor each with a different length
for i in data_idx:
   sub_data_item = (sub_data_idx == i.item()).nonzero().flatten()
   data_combinations = torch.combinations(sub_data_item)
   data_combinations_pair=torch.cat([data_combinations_pair, data_combinations])

# erase initialization
data_combinations_pair = data_combinations_pair[1:]
</code></pre>
<p><strong>My problem:</strong> avoid the “for loop” which is time consuming when using GPU.</p>
<p><strong>Is there a pytorch function that can do that?</strong><br>
<strong>Does anyone have an idea how to avoid this “for loop”?</strong></p>
<p>I thought about using these Pytorch functions inside a Numba kernel function but I also do not know if torch.combinations is a device or a kernel function.</p>
<p>Thank you in advance for all your help and for Pytorch developers.</p>
<p>Best regards,</p>
Statistics for whole dataset

poojitharamachandra: <p>how to calculate statistics (mean, variance, skewness,kutosis)on whole dataset . loading huge dataset at once is a problem. is there any trick for this?</p>
<p>i could not find in built functions for skewness , kutosis for tensors.<br>
how to move numpy nd array to gpu in that case?</p>

jubick: <p>You can create gpu tensor from numpy array like this:</p>
<pre><code class="lang-auto">array = torch.from_numpy(some_np_array).float().to(device)
</code></pre>
<p>and calculate like this:</p>
<pre><code class="lang-auto">mean = torch.mean(array)
diffs = array - mean
var = torch.mean(torch.pow(diffs, 2.0))
std = torch.pow(var, 0.5)
zscores = diffs / std
skews = torch.mean(torch.pow(zscores, 3.0))
kurtoses = torch.mean(torch.pow(zscores, 4.0)) - 3.0 
</code></pre>

poojitharamachandra: <p>thank you!<br>
but what is d in diffs = d - mean ?</p>

jubick: <p>I’m sorry it’s supposed to be array. I will edit answer</p>

poojitharamachandra: <p>thanks!<br>
for few datasets, the skewness and kurtosis values are nan. could you please tell me why is that? what is the statistical significance of that?</p>

jubick: <p>It could be alot of reasons. Like std == 0 =&gt; zscores = inf and so on. I recommend you to print each number to make sure which one is NaN</p>

jubick: <p>If this solved your problem could you please mark it as a solution?</p>

poojitharamachandra: <p>yes. thanks for your support!</p>
How to choose pytorch

Imene.b: <p>Hy, thanks for accepting me with you<br>
i am trying to write a code with python witch detect objects in images, and i am confused if i should use cpu pytorch or gpu pytorch,<br>
i have a computer i7 , 8 cores, with 12 G in my RAM<br>
and a GPU card GeForce GTX 750 Ti with 2G memory<br>
i would like to tel you that the data set is very large and i am looking for the speediest way to run my code.<br>
tanks.</p>

vdw: <p>Without any details, it’s difficult to tell for sure. Processing on GPU is of course much faster but the 2GB might be a bottleneck. The problem is less the dataset size – you can always reduce the batch size – but the complexity of your network, i.e., the number of (trainable) parameters.</p>
<p>I would just give it a try. Install PyTorch with GPU support, create and test your network with a small sample dataset on the CPU (any error messages are usually more helpful compared to GPU). If this seems to work, try to move it on the GPU to see if it works and how the performance looks like.</p>
<p>The good thing is that moving the network and the data onto the GPU is very easy with PyTorch, so you don’t loose anything here.</p>

Imene.b: <p>tank you for your reply, i will try, but can you tel me how to move network between GPU and CPU<br>
have you any documentation please</p>

vdw: <p>There are different methods. That’s how I do it:</p>
<p>First, check if the GPU is available, otherwise fall back to CPU (I only have 1 GPU so I can fix it to “cuda:0”)</p>
<pre><code>device = torch.device("cuda:0" if use_cuda else "cpu")
</code></pre>
<p>For the network model:</p>
<pre><code>model = MyCoolModel(...)
model.to(device)
</code></pre>
<p>For the data</p>
<pre><code>test_tensor = torch.ones(32)
test_tensor.to(device)
</code></pre>
<p>You only have to make sure that all your data and the model are moved to the same device. I common error is that the model is on the GPU but the current batch (still) on the CPU. Having basically the <code>.to(device)</code> at your model and all the tensors should take care of that.</p>

Imene.b: <p>tank you very much,<br>
excuse me because i am novice in programming, but one last question: if i install pytorch cpu how can i chose between: pytorch  cpu_0 or cpu_1 or cpu_2</p>

vdw: <p>As far as I know, there’s only “cpu” as device. But PyTorch seems to support parallelism on the CPU; have a look at <a href="https://discuss.pytorch.org/t/does-pytorch-support-parallelism-across-multiple-cpus/1631">this thread</a>, for example.</p>
GPU memory leak during iteration of LSTMcell

vincentzlt: <p>I am trying to implementing the following two toy codes:</p>
<pre><code class="lang-auto">lstm = nn.LSTM(512, 512, 1, True, True, 0, False)

input = torch.rand(128, 260, 512)
output, hidden = lstm(input)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">lstm_cell=nn.LSTMCell(512, 512, True)

input = torch.rand(128, 260, 512).unbind(1)
h=c=torch.zeros(128, 512)
for input_step in input:
    h,c=lstm_cell(input_step, (h, c))
</code></pre>
<p>The when the sequence lengths is set to, for example, <code>260</code>, the second implementation consumes far more GPU memory than the first one.</p>
<p>Anyone know the reason? How can I optimize it? I need to add some conditions inside the for loop of the second implementation.</p>
<p>I also found similar <a href="https://discuss.pytorch.org/t/seq2seq-rnn-decoder-gpu-memory-leaks-every-time-step/13304">post</a> way back but nobody has given good solutions.</p>
Trying to load a pre-trained vision model

anirudhgarg: <p>Hello,</p>
<p>I am trying to load a pre-trained vision model (resnet18) from the pth file (rensnet18-xxxx.pth) on my local disk. I see this only contains the state dictionary.</p>
<p>I understand to load this into a model I have to initialize the Model class and then do a load_state_dict on an instance of the model class as specified here: <a href="https://pytorch.org/docs/master/notes/serialization.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/serialization.html</a></p>
<p>However what parameters should I use to initialize the Model class. How can I find them out ?</p>
<p>Thanks<br>
Anirudh</p>

ptrblck: <p>You can just create an instance of the model and use the default initialization, since you’ll load the <code>state_dict</code> anyway and replace all parameters and buffers with the pretrained ones.</p>
Expected input batch_size (4) to match target batch_size (1)

Sreekar: <pre><code class="lang-auto">sequence_length = 2
input_size = 3
hidden_size = 8000
num_layers = 2
# num_classes = len(mentions)
batch_size = 300
num_epochs = 1000
learning_rate = 0.01

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')


class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        # self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)

        # Decode the hidden state of the last time step
        return out


model = RNN(input_size, hidden_size, num_layers).to(device)
print(model)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, batch in enumerate(train_loader):
        features = batch[0]
        labels  = batch[1]
        features = features.reshape(4,10,3) # only if features's shape is not this already, put the value of n here!!
        labels = labels.reshape(1,4,10) # only if labels's shape is not this already
        outputs = model(features)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))
</code></pre>