akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Systematic Way of Computing Layer Dimensionality

username: BaruchG
cooked: <p>Hi everyone,<br>
I’m pretty new to pytorch and building neural nets in general and have been slowly making my way through the cs231n course.  One thing that I struggle with (and if I’m any indicator, so do a lot of newbies) is computing the input/output dimensions of each layer so that the network can be glued together properly.  I’ve seen many questions here about figuring out how to connect a convolutional layer to a linear layer (or other types of layers) and things of that sort, with one off answers to each person on how to go about computing it.<br>
Is there some kind of calculator out there that I can use to tell me what the input and output size of all  of my layers is supposed to be without having to compute it each time?  If this is not possible, why not?<br>
Thanks</p>

username: tom
cooked: <p>This is not done by the network because the conv layers don’t have information what input shapes you will run through them.<br>
The usual thing I do (and I think I might not be the only one) is just to run some data through it and print the shape of the result (or just take the error and adjust based on that).</p>
<p>Best regards</p>
<p>Thomas</p>
Change learning of only one layer

username: HEIMDAL13
cooked: <p>Hello,</p>
<p>I have seen that is possible to set diferent learnign rates for diferent layers using Per-parameter options. But in my case, I have a lot of diferent modules, and I only want to set a diferent learnign rate for just a layer.</p>
<p>If tried something like this:</p>
<pre><code class="lang-auto">self._optimizer = optim.SGD([{'params': model.parameters()},{'params': model.fc2.parameters(), 'lr': 5e-3}], lr=1e-2)
        
</code></pre>
<p>But then I ger the following error: ValueError: some parameters appear in more than one parameter group</p>
<p>So, how could I create a list with the parameters I want from a specific layer and then the other parameters left without having duplicate parameters?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You could try to filter out your special layer:</p>
<pre><code class="lang-python">optim.SGD([{'params': [param for name, param in model.named_parameters() if 'fc2' not in name]}, {'params': model.fc2.parameters(), 'lr': 5e-3}], lr=1e-2)
</code></pre>

username: HEIMDAL13
cooked: <p>Thank you! That is just what I was looking for.</p>

username: Paralysis
cooked: <p>Is it possible to change the learning rate of a specific layer during training? I don’t see any “name to parameter mapping” in optimizer.</p>

username: ptrblck
cooked: <p>You would have to get the parameters of this specific layers. Is my example in this thread not working for your use case?</p>

username: Paralysis
cooked: <p>Thanks for the reply. If I understand correctly, the example code is the initialization of an optimizer. But I mean, when I already have an optimizer, and after several epochs, I want to change the learning rate of a specific layer only, let’s say layer “conv2.2”, is there a way to directly change it in the optimizer, given the layer’s name?</p>

username: recursion
cooked: <p>To do this for a specific layer after several epochs, you should be able to combine <a class="mention" href="/u/ptrblck">@ptrblck</a>’s solution with <code>torch.optim.lr_scheduler.LambdaLR</code>. You can supply a list of lambdas, one for each parameter group in the original optimizer. Put the update you want to make in the corresponding lambda. More info and the first below example can be found in the docs. <a href="https://pytorch.org/docs/stable/optim.html" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Assuming optimizer has two groups.
&gt;&gt;&gt; lambda1 = lambda epoch: epoch // 30
&gt;&gt;&gt; lambda2 = lambda epoch: 0.95 ** epoch
&gt;&gt;&gt; scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
&gt;&gt;&gt; for epoch in range(100):
&gt;&gt;&gt;     train(...)
&gt;&gt;&gt;     validate(...)
&gt;&gt;&gt;     scheduler.step()
</code></pre>
<p>To add to that, here is an example of a per-epoch lambda, with changes at epochs 10, 30, and 50:</p>
<pre><code class="lang-auto"># Multiply the original LR by 0.5 after epoch 10, etc.
lambda_epoch = lambda e: 1.0 if e &lt; 10 else (0.5 if e &lt; 30 else 0.25 if e &lt; 50 else (0.1))
</code></pre>
Error while downloading PascalVOC dataset using torchvision

username: vihas_makwana
cooked: <p>while running:               torchvision.datasets.VOCDetection("./",download=True,transform=transforms.Compose([transforms.ToTensor()]))<br>
I got this error:<br>
584         while True:<br>
585             try:<br>
–&gt; 586                 return self._sock.recv_into(b)<br>
587             except timeout:<br>
588                 self._timeout_occurred = True</p>
<p>ConnectionResetError: [Errno 104] Connection reset by peer</p>

username: ptrblck
cooked: <p>The connection just timed out. Could you retry the download?</p>

username: vihas_makwana
cooked: <p>The error still appears.<br>
I also tried to download it from <a href="http://host.robots.ox.ac.uk/pascal/VOC/" rel="nofollow noopener">http://host.robots.ox.ac.uk/pascal/VOC/</a> but it’s also showing Error104</p>

username: ptrblck
cooked: <p>The host site is indeed down, so nothing you can do from your side.</p>
Matrix transform for TransformedDistribution?

username: krylea
cooked: <p>Hey,</p>
<p>I’m trying to make use of TransformedDistribution to handle situations such as using the reparameterization trick to backpropagate through sampling from a multivariate normal, but I am noticing a problem. There doesn’t seem to be any Transform in torch.distributions that matches just a simple matrix multiplication of the form y = Ax + b. I thought AffineTransform would work, but it appears to be just a pointwise multiplication. Is there really no native implementation for this? I would have thought this would be the primary use case of TransformedDistribution. Do I need to just implement my own matrix transform?</p>
Time difference between models forward method and model(input) call

username: Knievel
cooked: <p>I have network with a rather complicated forward pass that includes several for-loops and I accounted that for the 7 seconds one forward pass needed to complete. These 7 seconds where measured with:</p>
<pre><code class="lang-auto">start = time.process_time()
output = network(input_batch)
end = time.process_time()
</code></pre>
<p>However if I take the time inside the forward method of the model (<code>start</code> before the first statement and <code>end</code> just before the return) it results in a time of around 0.03 seconds.</p>
<p>Where does this difference come from and can I do something to reduce it?</p>

username: ptrblck
cooked: <p>If you are executing the forward pass on the GPU, you should add <code>torch.cuda.synchronize</code> before starting and stopping the timer, as CUDA operations are executed asynchronously.<br>
Currently you might time the kernel launch times or some other operations, which create a synchronization point.</p>

username: Knievel
cooked: <p>I’m currently only using CPU, as I didn’t get my custom component to efficiently run on the GPU yet.</p>

username: ptrblck
cooked: <p>That’s interesting. Could you post an executable code snippet so that we could have a look?</p>

username: Knievel
cooked: <p>I actually made a stupid mistake in my time tracking in the forward method. Now everything works properly and the times match up.</p>
<p>Thanks anyway for the help and the GPU tip!</p>
Inference time and TFLOPS

username: tuxa
cooked: <p>I am currently looking into the <strong>half-precision</strong> inference time of different CNN models using the torch.autograd.profiler using two different GPUs:</p>
<ol>
<li>Nvidia RTX 2080 Ti (26.90 TFLOPS) - done locally (better CPU)</li>
<li>Nvidia T4 (65.13 TFLOPS) - done in the cloud</li>
</ol>
<p>It took me by surprise that the 2080 Ti is <strong>significantly</strong> faster (half the time or less), independent of batch size, input resolution, and architecture even though it has less than half the TFLOPS.</p>
<p>Does anyone know why?</p>
<pre><code class="lang-auto">import torch
import segmentation_models_pytorch as smp # pip install git+https://github.com/qubvel/segmentation_models.pytorch

runs = 10
res = 512
bs = 8
is_half = True

m = smp.Unet(encoder_name='resnet101', encoder_weights=None)
m.eval()
m.cuda()
m.half()

t = torch.rand((bs, 3, res, res)).cuda()
t = t.half()

if is_half:
    m.half()
    t = t.half()

# warm up
with torch.no_grad():
    m(t)

cpu_time_ms = 0
cuda_time_ms = 0
for i in range(runs):
    with torch.no_grad():
        with torch.autograd.profiler.profile(use_cuda=True) as prof:
            m(t)
        cpu_time_ms += prof.self_cpu_time_total / 1000
        cuda_time_ms += sum([evt.cuda_time_total for evt in prof.key_averages()]) / 1000

cpu_time_ms /= runs * bs
cuda_time_ms /= runs * bs

print('res={}x{} cuda={:.1f}ms cpu={:.1f}ms'.format(res, res, cuda_time_ms, cpu_time_ms))


</code></pre>
<p>Example:</p>
<pre><code class="lang-auto">unet with resnet101 as backbone and batch size 8

t4
res=128x128 cuda=11.3ms cpu=3.0ms
res=256x256 cuda=14.5ms cpu=2.8ms
res=512x512 cuda=50.4ms cpu=7.3ms

rtx 2080 ti
res=128x128 cuda=7.5ms cpu=1.7ms
res=256x256 cuda=8.6ms cpu=1.8ms
res=512x512 cuda=21.1ms cpu=3.0ms

</code></pre>
Got 'cudnn RNN backward can only be called in training mode' after using model.train()

username: wwwangzhch
cooked: <p>Hi, I’m running into the following error:</p>
<blockquote>
<p>Cudnn RNN backward can only be called in training mode</p>
</blockquote>
<p>I’m pretty sure I call model.eval() before backward(), My codes are attached below</p>
<pre><code class="lang-auto">    def episode(self, deterministic=False):
        if self.noise:
            self.noise.reset_states()

        state, _ = self.env.reset()
        steps = 0

        log_probs = []
        rewards = []

        agent_wealth = [1.]
        market_wealth = [1.]
        while True:
            self.actor.eval()
            action = self.actor(torch.from_numpy(state[None, ...]).to(self.args.device))
            action.squeeze_(0)
            if self.noise and not deterministic:
                action += torch.from_numpy(self.noise.sample())

            weights, log_p = self.get_weights(action)

            next_state, reward, done, info = self.env.step(weights)
            log_probs.append(log_p)
            rewards.append(reward)

            agent_wealth.append(info['portfolio_value'])
            market_wealth.append(info['market_avg_return'] * market_wealth[-1])

            steps += 1
            state = next_state
            self.total_steps += 1
            if done and not deterministic:
                # --------update policy ---------
                self.actor.train()

                agent_sr = SharpeRatio(agent_wealth)
                market_sr = SharpeRatio(market_wealth)

                policy_gradient = []
                for log_ps in log_probs:
                    policy_gradient.append(- log_ps * (agent_sr - market_sr))

                self.optimizer.zero_grad()
                policy_gradient = torch.stack(policy_gradient).sum()
                policy_gradient.backward()
                self.optimizer.step()
                # ---------------------------

            if done:
                break
        return sum(rewards), steps

    def get_weights(self, action):
        values, indices = torch.topk(action, self.args.G)
        proportions = torch.softmax(values, dim=-1)
        weights = np.zeros_like(action.detach().cpu().numpy())
        for i, indice in enumerate(indices):
            weights[indice.item()] = proportions[i].detach().cpu().numpy()
        log_p = torch.log(proportions)
        return weights, log_p
</code></pre>
<p>the episode method executes the interacting and learning procedure of the agent, and the get_weights method get the real weight pass to the environment according to the network’s output, as you can see, I called model.train() method before policy_gradient.backward(), so anyone can tell me where is the problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the problem is that you forward in eval mode. So the backward of that operation is not implemented.<br>
You need to forward in train model if you want to be able to backward through that op.</p>
Why torch.sparse.cuda() is much faster than torch.cuda()?

username: flyerjia
cooked: <p>The opt torch.sparse.cuda() is much faster than torch.cuda(), why?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think <code>torch.cuda()</code> is a function? Could you be clearer on what you compare and how please?</p>

username: flyerjia
cooked: <p>Thanks for your replying.</p>
<p>a=torch.FloatTensor()</p>
<p>b=torch.sparse.FloatTensor()</p>
<p>b.cuda() is much faster than a.cuda() even though a.size() is roughly equal to b.size().</p>
<p>发自我的小米手机</p>
<p>在 Alban D via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a>，2019年10月28日 22:08写道：</p>

username: albanD
cooked: <p>Hi,</p>
<p>How many non-zero elements are in the sparse Tensor? The whole point of the sparse tensor is to only save the non-zero values, so there is potentially much less things to transfer to the gpu.</p>

username: flyerjia
cooked: <p>Hi, i have another question. When i use sparse tensor cuda() in dfferent models, the same tensor takes different time (one is based on nn.Module, another is an C++ cuda extension i wrote). What can affect  the time ?</p>

username: flyerjia
cooked: <p>I have known the reason. The cuda() is asynchronous.</p>

username: flyerjia
cooked: <p>I have a kernel that needs one array of floats (for input) and one array of ints (for labels)? How to do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure what the problem is. Just pass these as arguments? There is no restriction about types.<br>
Do you have a code sample that shows what you’re trying to do?</p>
Gradient scaling in federated learning

username: sherdencooper
cooked: <p>Hi, I am using torch.distributed to do federated learning. What should I do if I want to scale some workers’ gradient since their data are more valuable than others? Thx a lot in advance!</p>

username: Yanli_Zhao
cooked: <p>tensor.register_hook(customHook) may work, you need to write customHook to modify grad of the tensor.</p>
<p>but as far as I know customHook should be a function of grad only. For your case, you want to make customHook to be a function of grad and workerRank as well?</p>

username: sherdencooper
cooked: <p>I only want to make customHook to be a function of grad and I haven’t used this before. Does <em>hook</em> only influence the backward process?</p>

username: Yanli_Zhao
cooked: <p>yes, only for backward pass</p>
How does OneCycleLR and optimizer works?

username: cthnguyen
cooked: <p>Hello,</p>
<p>I made a little mistake in my code (I think at least …), where I put the following :</p>
<pre><code class="lang-auto"># Optimizer
wd = 0.01
optimizer = optim.Adam(model.parameters(), lr=10e-4, weight_decay=wd)

# LR scheduler
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-4, steps_per_epoch=len(train_loader), epochs=epochs, pct_start=0.3) # only available on pytorch v1.3.0
</code></pre>
<p>Notice that in the optimizer I put a learning rate of 10e-4, whereas I put the max_lr on OneCycleLR to 1e-4. So in this situation what does it do? I am a little bit confuse, could anyone light me up please?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
RuntimeError no grad accumulator for a saved leaf error

username: 111175
cooked: <p>Hey guys, these lines incur the error in jit mode. Do you have any idea?</p>
<pre><code class="lang-auto"> matching_scores = torch.matmul(bike_key_out.permute(0, 3, 4, 2, 1), taxi_key_out.permute(0, 3, 4, 1, 2))

bt_t_x = torch.matmul(matching_scores, taxi_x.permute(0, 3, 4, 2, 1)).permute(0, 4, 3, 1, 2)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "multi_expt.py", line 329, in &lt;module&gt;
    main()
  File "multi_expt.py", line 298, in main
    (bike_loss + taxi_loss).backward()
  File "/home/jindeng/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/jindeng/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: No grad accumulator for a saved leaf!
</code></pre>

username: tom
cooked: <p>I don’t think those are the leaves the error talks about. Could you try to reduce your code to an reproducing example?</p>
<p>Best regards</p>
<p>Thomas</p>

username: albanD
cooked: <p>Hi,</p>
<p>It might be related to <a href="https://github.com/pytorch/pytorch/issues/19769" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/19769</a><br>
We would need a small code sample to reproduce this to be sure.</p>

username: 111175
cooked: <p>I think I find out where the problem is. I unbind a tensor to a list of subtensors, and iteratively retrieve each subtensor to conduct further operation. If I substitue it by direct indexing on the original tensor, the error is gone.</p>

username: 111175
cooked: <p>Thanks for your help, I think I find out the reason of my issue.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you share what you changed to fix this?</p>
How to check whether model loading is successful in version 1.2

username: jasjuang
cooked: <p>According to the release note <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">here</a>,</p>
<pre><code class="lang-auto">using torch::jit::script::Module;

std::shared_ptr&lt;Module&gt; m = torch::jit::load("my_model.py");
m-&gt;forward(...);
</code></pre>
<p>becomes</p>
<pre><code class="lang-auto">using torch::jit::script::Module;

Module m = torch::jit::load("my_model.py");
m.forward(...);
</code></pre>
<p>what I used to do is add an error checking like the below</p>
<pre><code class="lang-auto">if (m == nullptr) std::cerr &lt;&lt; "failed to load model" &lt;&lt; std::endl;
</code></pre>
<p>How can I do it in version 1.2? There doesn’t seem to be a <code>m.empty()</code> or something like that, and I couldn’t find any doxygen like documentation anywhere that have this information either. Please let me know. Thanks.</p>

username: dambo
cooked: <pre><code class="lang-auto">// Loading your model
const std::string s_model_name = argv[1]; // resnet50_model.pth NOT py.
torch::jit::script::Module module = torch::jit::load(s_model_name, torch::kCUDA);
</code></pre>

username: cyanM
cooked: <aside class="quote no-group" data-username="jasjuang" data-post="1" data-topic="54389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jasjuang/48/13084_2.png" class="avatar"> jasjuang:</div>
<blockquote>
<p>torch::jit::script::Module</p>
</blockquote>
</aside>
<p>I meet the same issue， have you found some way to check this？</p>
Multi-label Dice loss

username: beitian
cooked: <p>class DICELossMultiClass(nn.Module):</p>
<pre><code>def __init__(self):
    super(DICELossMultiClass, self).__init__()

def forward(self, output, mask):
    num_classes = output.size(-1)
    dice_eso = 0
    for i in range(num_classes):
        probs = torch.squeeze(output[:, :, :, :, i], -1)    # [batch_size, D, H, W, channel]
        target = torch.squeeze(mask[:, :, :, :, i], -1)     # one-hot :[batch_size, D, H, W, channel]

        num = probs * target
        num = torch.sum(num, 3)
        num = torch.sum(num, 2)
        num = torch.sum(num, 1)

        # print( num )

        den1 = probs * probs
        # print(den1.size())
        den1 = torch.sum(den1, 3)
        den1 = torch.sum(den1, 2)
        den1 = torch.sum(den1, 1)

        # print(den1.size())

        den2 = target * target
        # print(den2.size())
        den2 = torch.sum(den2, 3)
        den2 = torch.sum(den2, 2)
        den2 = torch.sum(den2, 1)

        # print(den2.size())
        eps = 0.0000001
        dice = 2 * ((num + eps) / (den1 + den2 + eps))
        # dice_eso = dice[:, 1:]
        dice_eso += dice

    loss = 1 - torch.sum(dice_eso) / dice_eso.size(0)
    return loss
</code></pre>
<p>I used the above function to computed the multi-label dice loss(the target label is encoded by one-hot), but I don’t know whether it is correct because I got some negative value for loss. I think the main reason is that one of channels is for background, which is set to 1 for background pixels,this is different from the binary semantic segmentation.<br>
What can I do for this ?</p>
Masked Convolutions

username: Burak_Uzkent
cooked: <p>I want to implement a convolutional operation that skips spatial positions of the feature maps/images conditionally on a binary mask. In other words, this operation will skip the sliding window across the feature maps of a layer if the corresponding mask for the same sliding window is all zero. One way of doing this is multiplying the outputs with respect to the binary mask. However, I prefer to skip convolutions on the pixels of the input layer where the pixel in the corresponding binary mask is 0. You can find a rough pseudo-code of this operation below.</p>
<p>for rw in range(n_rows):<br>
for cl in range(n_cols):<br>
for cn in range(n_channels):<br>
if mask(rw, cl) == 0:<br>
continue<br>
else:<br>
conv(img(rw-1:rw+1, cl-1:cl+1, :), filter)</p>
<p>I will appreciate it if you can point me to a solution in PyTorch.</p>

username: chengjuzhou
cooked: <p>Have you checked the masked conv implement in mmdetection:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/open-mmlab/mmdetection/tree/master/mmdet/ops/masked_conv" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/10245193?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/open-mmlab/mmdetection/tree/master/mmdet/ops/masked_conv" target="_blank" rel="nofollow noopener">open-mmlab/mmdetection</a></h3>

<p>Open MMLab Detection Toolbox and Benchmark. Contribute to open-mmlab/mmdetection development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The above masked_conv can work, but the speed is very slow compared with normal nn.Conv2d.</p>
Quantize conv layer with bias

username: dtcrong
cooked: <p>I have a resnet-based network. Trying to convert it to INT8 but met accuracy loss using static quant.<br>
Tried QAT but met with assert error “Only support fusing Conv2d that does not have bias”</p>
<p>Look into the model structure and found all the conv layers have bias:<br>
nn.Conv2d(in_planes, out_planes, kernel_size, stride=stride, bias=True)</p>
<p>why should conv without bias? is it possible to use fake quant to train it?</p>
<p>can I only modify the fusing code in qat.conv_fuse.ConvBn2d to do fuse conv bias ?</p>

username: raghuramank100
cooked: <p>When you are fusing a conv with a batch norm, there is no separate bias term in the conv. This is because, batch norm already has a trainable bias parameter which serves the same purpose. The reference resnet implementation at: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L24" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L24</a> does not have bias as part of the conv.</p>
torchvision.datasets.FakeData

username: Maria_Fernanda_De_La
cooked: <p>I want to use torchvision.datasets.FakeData and have a training and testing set. What is the best way to achieve this? If I create two of these, how do I ensure they are generating the same classes?</p>

username: ptrblck
cooked: <p><code>FakeData</code> uses the argument <code>num_classes</code> to return the dummy targets.</p>
<p>I’m not sure, why you would need a training and test dataset, as your model won’t learn anything useful and <code>FakeData</code> is usually used for debugging/testing.</p>
How Dataloader gets images from custom dataset

username: mfcs
cooked: <p>Hi,<br>
I am working in a project that I  have to make a dataset class. I already know that this class must has three methods: <strong>init</strong>(), <strong>getitem</strong>() and <strong>len</strong>().<br>
The <strong>getitem</strong>() method recieve as a parameter an index of an image, for example. So, all preprocessing needed for the image is performed and in the end this preprocessed image is returned by the <strong>getitem</strong>() method.</p>
<p>My question is: if the dataloader is configured with batch=100. The <strong>getitem</strong>() method is called 100 times getting 100 images to generate the batch? I am asking this because the method <strong>getitem</strong>() receive just one index of an image. So, in my understanding this method returns only one image.</p>
<p>This is it.</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

username: tom
cooked: <p>Yes, this is true. There also are <a href="https://pytorch.org/docs/stable/data.html#iterable-style-datasets" rel="nofollow noopener">Iterable Datasets</a> for cases where that isn’t appropriate.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: Using backticks (`<code>__getitem__</code>`) you can get the double-underbars.</p>

username: mfcs
cooked: <p>Thanks for the reply!!<br>
So, can I consider that the Dataloader calls the <code>__getitem__()</code> method several time as the number of the batch size? For example, batch_size = 100, the Dataloader will call the method 100 times to make the batch? Am I right?</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

username: tom
cooked: <p>Yes. For the usual datasets, <code>__getitem__</code> will be called a 100 times.<br>
The list of 100 results are then fed to the collate_fn to make a batch.</p>

username: mfcs
cooked: <p>Ahh ok!!<br>
Thanks!!<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Entropy of a Multidimensional Tensor

username: rohaldbUni
cooked: <p>I’m looking to compute the discrete entropy (not cross-entropy) of a multidimensional tensor in my loss function. Consider the tensor to have shape <code>BATCH_S x D</code>, where each data point is a <code>D</code> dimensional vector, and I have <code>BATCH_S</code> of them. The <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)" rel="nofollow noopener">formula</a> requires me to compute the probability of seeing each data point, and I can’t seem to find a way to do this while retaining the ability to do a backward pass. To do so, all I need is the frequency of each element because I can divide that by the sum of the frequencies.</p>
<p>There’s <a href="https://pytorch.org/docs/stable/torch.html#torch.histc" rel="nofollow noopener">torch.histc</a>, but that doesn’t work for multidimensional data, and neither does <a href="https://pytorch.org/docs/stable/torch.html#torch.bincount" rel="nofollow noopener">torch.bincount</a>. There’s also <a href="https://pytorch.org/docs/stable/torch.html#torch.unique" rel="nofollow noopener">torch.unique</a> which has the <code>return_count</code> optional param, but that’s has no <code>grad_fn</code>.</p>
<p>Any help is appreciated! Thanks</p>

username: ru6928
cooked: <p>did you find a solution to this? I have the same question <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: rohaldbUni
cooked: <p>No they are inherently non-differentiable functions as they require binning (and uses an indicator function). Take a look at <a href="https://arxiv.org/pdf/1803.00974.pdf" rel="nofollow noopener">this paper</a> which looks at a continuous approximation of mutual information, which is a function of entropy. You can approximate the indicator function (required for binning) with a triangular kernel function, but you will need to derive your own backward function.</p>
Backward of InstanceNorm2d

username: ChanggongZhang
cooked: <p>Dear all,<br>
I have a very simple question about the gradient flowing backward through the InstanceNorm2d layer.<br>
Here are my test codes:</p>
<pre><code class="lang-auto">x = torch.arange(0., 8).reshape((2, 1, 2, 2))
x.requires_grad = True
instaceN = nn.InstanceNorm2d(1, affine=False, eps=0.0, track_running_stats=False)
instaceN.weight = nn.Parameter(torch.Tensor([1.0]))
instaceN.bias = nn.Parameter(torch.Tensor([0.0]))
y = instaceN(x)
y.register_hook(print)
z = y.sum()
z.backward()
print(x.grad)
</code></pre>
<p>But the output of x.grad is all 0s. I am wondering how this could happen?<br>
If I change InstanceNorm2d to BatchNorm2d, x.grad is sill all 0s.<br>
But if I change InstanceNorm2d to Sigmoid, x.grad has none-zero grads.</p>
<p>Could someone explain this weird phenomenon?<br>
Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure about the exact formula for instance norm, but in your code sample, you can move everything to double then use <code>torch.autograd.gradcheck(instaceN, (x,))</code> to check the gradients using finite differencing and the check passes.<br>
So these are the expected gradients. I think the Jacobian is not 0 but, the sum() operation on y makes you compute <code>1^T J</code> which sums the columns, and this sum is 0.</p>

username: SimonW
cooked: <p>First of all, a lot of your code is unnecessary. You don’t need to repeat default parameters, and weight and bias aren’t used with affine=False (default). Cleaning it up, we have</p>
<pre><code class="lang-auto">x = torch.arange(0., 8).reshape((2, 1, 2, 2))
x.requires_grad = True
instaceN = nn.InstanceNorm2d(1, eps=0.0)
y = instaceN(x)
z = y.sum()
z.backward()
print(x.grad)
</code></pre>
<p>Back to your problem. <code>\sum</code> of any channel in an IN output will always be zero. So for your specific <code>z</code> formula, any input <code>x</code> maps to all zeros, and then <code>x</code> has all zero gradient naturally.</p>
How to preserve GT image texture when upsampling with transposeConv2d?

username: Or_Rimoch
cooked: <p>Hi all,</p>
<p>My input is  a concatenated embedding vector,<br>
I apply series of (transposeConv2d-&gt;BN-&gt;Relu) X5 to get an image size matrix with one channel.</p>
<p>After “supervised” training I get an image which is very similar to the general distribution of the GT  image, although it slightly smooth, and the texture isn’t preserved locally.<br>
I tried to train with variety of hyper-parameters and number of samples - all return the same visual<br>
bad image quality.</p>
<p>Should I use skip connections to solve the texture issue? any relevant idea?<br>
I’d be happy to hear any suggestions to improve this vanilla Decoder architecture.</p>
<p>Thanks,<br>
Or</p>
Best way to track pos-neg or neg-pos zero crossings in 3D tensor?

username: Yuerno
cooked: <p>Hi all. If I have a 3D tensor (say 32x32x32) and I’m indexing along a dimension such that I’m dealing with a 2D 32x32 tensor at each index, I was wondering what the best way would be to track zero crossings of values? Would an approach like <a href="https://codereview.stackexchange.com/questions/45458/finding-a-zero-crossing-in-a-matrix" rel="nofollow noopener">listed here for Numpy</a> work with tensors as well? Thanks!</p>

username: JuanFMontesinos
cooked: <p>Convolve the 3dsignal twice with the proper kernel to compute the derivatives and you can go from there</p>
How to compute the joint-histogram in pytorch?

username: jiawei_chen
cooked: <p>how to compute the joint-histogram between two tensor in pytorch?<br>
i know there is a func: torch.histc() to compute the histogram, but i also want to use the joint-histogram to calculate the mutual information for loss-backward, can anyone implement this code?</p>

username: GM_039
cooked: <p>Hi jiawei_chen,<br>
Did u find any solution? Any help will be appreciated.</p>
<p>Thank you</p>

username: ru6928
cooked: <p>any solution?</p>
<p>I have been looking for a np.histogram2D equivalent in pytorch</p>

username: ru6928
cooked: <p>any solution?</p>
<p>I have been looking for a np.histogram2D equivalent in pytorch</p>
How am I supposed to pass a network module to a function?

username: msqiqi10
cooked: <p>I want to calculate the derivative to the input vector, and adding noise to it.<br>
so how am I supposed to pass a trained network module to a function?<br>
And can I just do one forward pass, then use torch.autograd.backward to get the derivatives input.grad.<br>
Is there any more elegant way for this?</p>

username: albanD
cooked: <p>Hi,</p>
<blockquote>
<p>So how am I supposed to pass a trained network module to a function?</p>
</blockquote>
<p>I’m not sure to understand the question. You can pass your model as any other parameter to the function.</p>
<blockquote>
<p>Is there any more elegant way for this?</p>
</blockquote>
<p>You can use <code>torch.autograd.grad()</code> to get the gradients wrt the input directly. (And avoid populating the .grad fields of all the weights in your model).</p>

username: msqiqi10
cooked: <p>Thanks for your reply.</p>
<blockquote>
<p>You can use  <code>torch.autograd.grad()</code>  to get the gradients wrt the input directly. (And avoid populating the .grad fields of all the weights in your model).</p>
</blockquote>
<p>I checked the usage of <code>torch.autograd.grad()</code>, but I’m still confused.<br>
Suppose there is a input vector input, and a loss L, I want to calculate the grad on the input rather than the whole network wrt L. Can I write <code>input_grad = torch.autograd.grad(L, input)</code>?</p>
<p>After getting the grad using<br>
<code>input_grad = torch.autograd.grad(L, input)</code><br>
, I got<br>
<code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code><br>
when I tried to update the network using<br>
<code>optimizer_d.zero_grad()</code><br>
<code>d_loss.backward()</code><br>
<code>optimizer_d.step()</code>.</p>
<p>Why is this happening and how am I suppose to fix this? Thank you.</p>

username: msqiqi10
cooked: <p>To my understanding <code>input_grad = torch.autograd.grad(L, input)</code> calculates the grads wrt input, and <code>d_loss.backward()</code>, <code>optimizer_d.step()</code> are updating weights with their grads. But there is a counter counting how many times you have derived from the network, and it’s not allowed in Pytorch?</p>
<p>In my case, if I don’t want to update the trained network and just want to get the grads wrt input, I can used torch.autograd.grad without problem above?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This error happens because we try to reduce memory usage and so free all the temporary buffer when you do a backward pass (because in most cases you only do one).<br>
If you plan on doing more than on (using backward or grad), all but the last one should be called with <code>retain_graph=True</code>.</p>
Leaf tensor but without grad after bachward

username: Oshrin
cooked: <pre><code class="lang-auto">import torch

a = torch.rand(5, 5, requires_grad=False)
b = torch.rand(5, 5, requires_grad=False)
c = torch.rand(5, 5, requires_grad=True)

f = a + b
g = b + c

y = f + g
y.backward(torch.ones_like(y))

print(f.is_leaf, f.grad_fn, f.grad)

</code></pre>
<p>output: True None None</p>
<p>Why f doesn’t have its grad? It is a leaf variable.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83c5088d63fff40dcb6681803e2c0639c3835026.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/83c5088d63fff40dcb6681803e2c0639c3835026" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83c5088d63fff40dcb6681803e2c0639c3835026_2_690x232.jpeg" alt="image" data-base62-sha1="iNGGPgPgoqudYuu6tU2ixrKVSKO" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83c5088d63fff40dcb6681803e2c0639c3835026_2_690x232.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83c5088d63fff40dcb6681803e2c0639c3835026_2_1035x348.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/3/83c5088d63fff40dcb6681803e2c0639c3835026.jpeg 2x" data-dominant-color="F3F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1140×384 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>It is but it does not require gradients. So no gradients are computed for it.</p>

username: crowsonkb
cooked: <p>Try manually setting <code>f</code> to require gradients:</p>
<p><code>f.requires_grad_(True)</code></p>
Training network with 4 GPUs performance is not exactly 4 times over one GPU why?

username: shivsd
cooked: <p>Training neural network with 4 GPUs using pyTorch, performance is not even 2 times (btw 1 &amp; 2 times) compare to using one GPU. From Nvidia-smi we see GPU usage is for few milliseconds and next 5-10 seconds looks like data is off-loaded and loaded for new executions (mostly GPU usage is 0%). Is there any way in pyTorch to improve the data upload and offload for the GPU execution or is this normal ?</p>
<p>I’m running DCNN network with COCO dataset.</p>

username: surya00060
cooked: <p>Using lmdb database datasets would give good increase in performance.<br>
Using 4 GPUs will not give 4X speedup.(Amdahl’s law). It’s usual that performance with 4 GPUs is same as single GPU. In distributed training, there’s lot of communication between GPUs and CPU which hinders performance gains.</p>
Indexing Tensor

username: koelscha
cooked: <p>I have two tensors and want to access one of them using the other one as index.<br>
a.shape = [batch_size, num_channels, height, width]<br>
b.shape = [batch_size, num_channels, 2]</p>
<p>I would now like to use the 2 as x and y coordinates of the feature maps of a, but I don’t know how to do that… Can anyone give me a hint?<br>
The expected output would be of size [batch_size, num_channels]</p>
<p>Thanks in advance</p>

username: koelscha
cooked: <p>I guess, I could do it by expanding the indices and calling gather twice, as follows:</p>
<pre><code>x = b[:,:,0]
x = x.view(4,16,1,1).expand(4,16,256,1)
y = b[:,:,1]
y = y.view(4,16,1,1)
result = a.gather(3, x).gather(2, y).squeeze()
</code></pre>
<p>but I think this looks a bit clumsy and I feel like there should be a better way to accomplish this.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid there is not very elegant way to do this.<br>
Another approach would be to convert your 2D indices into a 1D index into a dimension of size height * width.<br>
Then you can use view of a and a single gather to get what you want.</p>

username: koelscha
cooked: <p>Thanks for your reply. Is there an elegant way to convert the indices to 1D?</p>

username: albanD
cooked: <p>You can do the following:</p>
<pre><code class="lang-python">import torch

batch_size = 2
num_channels = 3
height = 5
width = 5

a = torch.rand(batch_size, num_channels, height, width)
b = torch.rand(batch_size, num_channels, 2).mul(min(height, width)).long() # Make random valid indices

# Your solution
x = b[:,:,0]
x = x.view(batch_size,num_channels,1,1).expand(batch_size,num_channels,height,1)
y = b[:,:,1]
y = y.view(batch_size,num_channels,1,1)
result = a.gather(3, x).gather(2, y).squeeze()

# 1D solution
lin_indices = b.select(-1, 1) * width + b.select(-1, 0)
assert a.is_contiguous()
lin_a = a.view(batch_size, num_channels, height*width)
result2 = lin_a.gather(-1, lin_indices.unsqueeze(-1)).squeeze(-1)

print((result - result2).abs().max())


</code></pre>
Build standalone C++ program from sources

username: dnnagy
cooked: <p>I have downloaded the source code for pytorch using<br>
<code>git clone --recursive https://github.com/pytorch/pytorch</code></p>
<p>The sources are downloaded to <code>~/pytorch</code></p>
<p>Now I have a very simple C++ app:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor tensor = torch::rand({2, 3});
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}
</code></pre>
<p>Can I build this into a single standalone executable with cmake? I want to build an executable with -o3 that does not need the presence of libtorch.a. How can I do this?</p>
Univariate optimization with PyTorch

username: TinfoilHat0
cooked: <p>What’s the proper way to do univariate optimization with PyTorch?</p>
<p>Currently, what I do is something like the following which correctly finds the mean of given sample pts as 2.5.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

mean = Variable(torch.ones(1, 1), requires_grad=True)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD([mean], lr=0.01, momentum=0.9)
pts = torch.Tensor([1, 2, 3, 4])

for _ in range(10**3):
    optimizer.zero_grad()
    loss = criterion(mean, pts)
    loss.backward()
    optimizer.step()
    print(mean.data)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Pytorch is not really built to do univariate optimization.<br>
But your approach looks like the right was to do this.</p>
<p>Note that <code>Variable</code> is not needed anymore and you can simply do <code>mean = torch.ones(1, 1, requires_grad=True)</code>.<br>
And replace <code>print(mean.data)</code> by <code>print(mean)</code> or <code>print(mean.detach())</code>.</p>
Pinverse of a batch of 2d matrices

username: mfazampour
cooked: <p>Is it possible to calculate the pseudo inverse of <code>n</code> matrices in one tensor?<br>
Input tensor (<code>input</code>) has the shape of <code>[n, rows, columns]</code> and in the output should be a tensor (let’s call it <code>pinv</code>) with shape of <code>[n, columns, rows]</code> which for each <code>i</code>,  <code>pinv[i, :,:]</code> is the pseudo inverse of <code>input[i, :, :]</code>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>We do have a batch version of the <code>inverse</code> method but not the <code>pinverse</code> I’m afraid.</p>

username: mfazampour
cooked: <p>then it should be possible to calculate the batch pseudo inverse using the batch inverse like this:</p>
<pre><code class="lang-auto">def pinv(self, A):
        P1 = th.matmul(A.transpose(1,2), A)
        P2 = P1.inverse()
        P3 = th.matmul(P2, A.transpose(1,2))
        return P3
</code></pre>
<p>right?<br>
Of course it would be the <em>left inverse</em></p>

username: tom
cooked: <p>pinverse happily takes batches:</p>
<pre><code class="lang-python">a = torch.randn(5,6,3)
b = torch.pinverse(a)
c = torch.stack([torch.pinverse(a_i) for a_i in a], 0)
print((b-c).abs().max())
</code></pre>
<p>gives 5e-8 ish discrepancy.</p>
<p>As it goes with Linear Algebra, the invariably awesome <a class="mention" href="/u/vishwakftw">@vishwakftw</a> <a href="https://github.com/pytorch/pytorch/pull/26095" rel="nofollow noopener">implemented it just in time for the 1.3 release</a>.</p>
<p>As a general rule, the nicely named linear algebra functions do batches.</p>
<p>Best regards</p>
<p>Thomas</p>

username: albanD
cooked: <p>Ho nice !<br>
I was looking at the doc and this change was not reflected there. I’ll fix that quickly.</p>
Error when exporting ScriptModule to ONNX

username: gvamvou
cooked: <p>Greetings!<br>
I have an issue with the torch.onnx.export method. I have a ScriptModule object that has been tested and works well. When I try to use the torch.onnx.export method to make the model into ONNX format, I get the following message:</p>
<p>command: torch.onnx.export(scripted_model, dummy_kek, ‘model.onnx’)<br>
error message: * {AssertionError}example_outputs must be provided when exporting a ScriptModule</p>
<p>after adding an example_outputs argument in the function call I get another error message saying that this argument is unexpected:</p>
<p>command: torch.onnx.export(scripted_model, dummy_in, ‘model.onnx’, example_outputs=dummy_outp)<br>
error message: * {TypeError}export() got an unexpected keyword argument ‘example_outputs’</p>
<p>How can this happen? Has anyone ever encountered such a problem?<br>
Thank you for your time!</p>
Error in implementing custom activation function

username: astri
cooked: <p>Hello all, i am a beginner in deep learning and try to make a custom activation function from sigmoid in pytorch . I have an error when trying to implementing that custom activation function in model.</p>
<pre><code class="lang-auto">def tempsigmoid(x):
    nd=3.0
    temp=nd/torch.log(torch.tensor(9.0)) # how to write in pytorch , cant use float
    return F.sigmoid(x/(temp)) 

# CREATE MODEL
class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(1024, 100),
            nn.Sigmoid(),
            nn.Linear(100, 8)
            tempsigmoid()
            )
        
    def forward(self, x):
        x = self.layers(x)
        return x
</code></pre>
<p>Error comment :</p>
<pre><code>tempsigmoid()
          ^
</code></pre>
<p>SyntaxError: invalid syntax</p>
<p>why this is happened?</p>

username: ptrblck
cooked: <p>You’ve forgotten to add a comma after <code>nn.Linear(100, 8)</code>.</p>

username: astri
cooked: <p>thank you for your reply, after i add comma, I tried to call function with tempsigmoid() but gives error it need 1 argument. When i tried using tempsigmoid(x) i have error in my data that said</p>
<p>TypeError: mul(): argument ‘other’ (position 1) must be Tensor, not numpy.ndarray. It means my data still in the form of numpy?</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, x, y, transform=None):
        self.x = torch.from_numpy(x).float()
        self.y = torch.from_numpy(y).float()
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.x[index]
        y = self.y[index]
        
        if self.transform:
            x = self.transform(x)
        
        return x, y
    
    def __len__(self):
        return len(self.x)
</code></pre>
Anyone know well-maintained library about 3D object detection by point cloud?

username: fukatani
cooked: <p>Hi, I want to try 3D bounding box detection (bounding box) with PyTorch.<br>
I tried PointRCNN, but I failed to train with author’s best parameter.<br>
Please see <a href="https://github.com/sshaoshuai/PointRCNN/issues/123" rel="nofollow noopener">here</a><br>
Unfortunately, this repository stacked with unresolved issue.</p>
<p>Do you know any well-maintained library? What libyrary do you recommend?</p>
How to use WeightNorm with VGG architecture?

username: Jonas
cooked: <p>Hi there,</p>
<p>I’d like to use torch.nn.utils.weight_norm on a VGG net but it’s unclear to me from the docs  how exactly to add the hooks. Right now, I do this:</p>
<pre><code class="lang-auto">def make_layers(cfg, batch_norm=False,centering=False,normalize_std=False,fixed_std=False, weightnorm=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            layers += [nn.utils.weight_norm(conv2d), nn.ReLU(inplace=True)]
            in_channels = v
    return nn.Sequential(*layers)
</code></pre>
<p>But when running SGD I see absolutely no difference to the vanilla net (exact same loss values over time) and as a matter of fact the model weights in the final classification layers (no WN added) are exactly the same after any k&gt;1 iterations for both networks. How is that possible?</p>