akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to set target for YOLOv1 model?

username: Yolkandwhite
cooked: <p>I’m working on datasets for YOLOv1.</p>
<p>I have an Image with two ground truth boxes.</p>
<p>box1 = [x1, y1, w1, h1, CS1]<br>
box2 = [x2, y2, w2, h2, CS2]<br>
(CS is confidence score. I deleted Class parameter because I don’t need classification. I only need localization.)</p>
<p>below is example for my data</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/0376a0d5a74dfe6df8444bb659720630ef359ffe.png" data-download-href="https://discuss.pytorch.org/uploads/default/0376a0d5a74dfe6df8444bb659720630ef359ffe" title="스크린샷 2020-09-30 오후 2.16.31"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0376a0d5a74dfe6df8444bb659720630ef359ffe_2_250x250.png" alt="스크린샷 2020-09-30 오후 2.16.31" data-base62-sha1="uDAIKrf11p1HuV2bYrT46P5hJc" width="250" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0376a0d5a74dfe6df8444bb659720630ef359ffe_2_250x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0376a0d5a74dfe6df8444bb659720630ef359ffe_2_375x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0376a0d5a74dfe6df8444bb659720630ef359ffe_2_500x500.png 2x" data-dominant-color="FCFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2020-09-30 오후 2.16.31</span><span class="informations">1018×1016 55.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the model output is (1, 7, 7, 10)</p>
<p>how can I make targets for this?</p>
<p>if I want 2 bounding box predictions for each grid cell like yolo paper,</p>
<p>does all 7*7 grid cell should go to (x1, y1, w1, h1, CS1, x2, y2, w2, h2, CS2)?</p>
<p>or just certain 2 grid cell should go like (x1, y1, w1, h1, CS1, x1, y1, w1, h1, CS1)<br>
and (x2, y2, w2, h2, CS2, x2, y2, w2, h2, CS2) ?</p>
Torch Bicubic Upsample for 5D data

username: Mughees
cooked: <p>As per the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html" rel="noopener nofollow ugc">docs</a>,  Bicubic upsample is supported for 5D data but it’s giving an error. Also when I opened the functional file, there is no case for 5D and bicubic.<br>
For reproducing, see the code, please. Can someone explain to me that its a kind of PyTorch bug or docs needs explanation? Thanks alot.</p>
<pre><code class="lang-auto">import torch
bicubic_upsampling = torch.nn.Upsample(scale_factor=4, mode='bicubic', align_corners=True)
bu = bicubic_upsampling(torch.rand(8,7,3,100,100))
#NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 5D)
#for the modes: nearest | linear | bilinear | bicubic | trilinear (got bicubic)
</code></pre>

username: tom
cooked: <p>So in reality, bicubic is only supported for 4d inputs, i.e. spatial data. If anything you would need tricubic for 5d inputs (but that isn’t implemented). What happens is that the error messages for invalid shapes and bicubic haven’t been written, so you get a “stop-all” error message.</p>
<p>The key word in the documentation is <em>respectively</em>.</p>
<blockquote>
<p>The algorithms available for upsampling are nearest neighbor and linear, bilinear, bicubic and trilinear for 3D, 4D and 5D input Tensor, respectively.</p>
</blockquote>
<p>So this is hard to parse, but let me add paratheses: … are (nearest neighbor) and ((linear), (bilinear, bicubic) and (trilinear) for 3d, 4d, 5d, input Tensor respectively).</p>
<p>You would seem to have a point that both the error message and the documentation could use improvement.</p>
<p>The other part is that I don’t think the function you are asking for has been implemented yet.</p>
<p>Best regards</p>
<p>Thomas</p>
Adding a list of tensors together

username: jwngo
cooked: <p>I would like to add a list of tensors together.</p>
<p>I am trying test-time-augmentation (tta) with 6 images of different scales and flips.<br>
Here is the relevant code snippet. I am using ttach, a tta wrapper</p>
<pre><code class="lang-auto">for batch_idx, sample in enumerate(test_loader):
                     masks = []
                     img = sample['img'].to(self.device)
                     for transformer in tta_transforms:
                         augmented_image = transformer.augment_image(img)
                         model_output = self.tta_model(augmented_image)
                         deaug_mask = transformer.deaugment_mask(model_output)
                         masks.append(deaug_mask)
                     tta_mask = torch.zeros_like(masks[0])
                     for mask in masks:
                         tta_mask = torch.add(tta_mask, F.softmax(mask, dim=1))
                     img_name = sample['img_name']
                     #segLabel = sample['segLabel'].to(self.device) 
                     outputs, sig = self.model(img)
                     tta_mask = torch.div(tta_mask, len(masks))
</code></pre>
<p>masks contains a list of 6 tensors [ B x C x H x W ], which is [12 x 7 x 368 x 640]<br>
To add them together, I am doing torch.add(tta_mask, F.softmax(mask, dim=1)) where tta_mask is torch.zeros_like(masks[0]) and then torch.div to divide it by 6 (no. of tensors in the list).</p>
<p>I am wondering if this is the correct way to add a list of tensors together to get the mean? As I am not getting good results (Acc drops from ~95% to 89% with horizontal flip and scales [0.5,1,1.5]) and would like to ensure it is not my implementation that is causing it.</p>
<p>Thank you very much for your help!</p>

username: PistonY
cooked: <p>Use <code>torch.cat()</code> or <code>torch.stack()</code></p>

username: jwngo
cooked: <p>Oh, but i thought that<br>
<code>torch.cat()</code> or <code>torch.stack()</code><br>
would change the dimensions, from an example from Stackoverflow<br>
So if  <code>A</code>  and  <code>B</code>  are of shape (3, 4),  <code>torch.cat([A, B], dim=0)</code>  will be of shape (6, 4) and  <code>torch.stack([A, B], dim=0)</code>  will be of shape (2, 3, 4).</p>
<p>Which in my case would be<br>
<code>torch.cat([A,B], dim=1)</code> resulting in [12 x 14 x 368 x 640]?<br>
Does this still work when I want the output to be [12 x 7 x 368 x 640]?</p>

username: PistonY
cooked: <p>If you just want to collect them why not use a <code>list</code>? Or what you expect shape is?</p>

username: jwngo
cooked: <p>So what I’m actually trying to do is to get the mean of the list of 6 tensors from different augmented images to be used during testing. (test time augmentation)<br>
My expected shape would be the mean of all 6 tensors, with the same shape<br>
[12 x 7 x 368 x 640]</p>

username: PistonY
cooked: <p>So, <code>torch.stack()</code> solve your problem.</p>
Quantization on trained model weights

username: gnadaf
cooked: <p>I would like to apply quantization on trained model weights, could you please tell, how to alter the weights of the transformer model and update weights.</p>
<p>I have implemented the transformer model as an example specified in the page <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
Dirichlet distribution sometimes outputs tensor full of NaNs

username: ucals
cooked: <p>Hey guys,<br>
Found an issue with Dirichlet distribution. Sometimes, for no apparent reason, it outputs a tensor full of NaNs. The line causing the problem is as simple as:</p>
<pre><code class="lang-auto">doc_topics = Dirichlet(topic_weights).rsample()
</code></pre>
<p><code>topic_weights</code> is asserted to be a valid tensor, always.<br>
Now, I’m circumventing the problem by replacing the line above by the following code:</p>
<pre><code class="lang-auto">repeat = True
while repeat:
    doc_topics = Dirichlet(topic_weights).rsample()
    repeat = doc_topics.isnan().any()
</code></pre>
<p>It works… but it is as ugly as code can possibly be. (In Brazil we have a slang for that: <a href="https://www.urbandictionary.com/define.php?term=Gambiarra" rel="noopener nofollow ugc">“gambiarra”</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">)</p>
<p><strong>Any hints on how to solve that?</strong></p>

username: tom
cooked: <p>Thank you for the new word! I’ll be sure to use it next time I need it (and around computers, that will be soon…).</p>
<p>The obvious solution is to fix PyTorch to not do that. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> But can you give us some more idea about topic_weights (is it cuda, what shape, something unusual about the values)? The perfect reproducing example would allow us to generate the NaNs (by giving the weights and a random seed/rng state that hits the problem). A bug report with a reproducing case would be great help! (Or just paste it here and we can forward it.)<br>
There have been issues with very large/small parameters for some distributions, but a very quick search doesn’t scream the Dirichlet issue is known.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ucals
cooked: <p>Thanks <a class="mention" href="/u/tom">@tom</a>! I’m trying to create a reproducing example to work in Colab… will share soon… Thanks!</p>
Updating complex parameters and defining a real-valued MSE loss

username: alihasn
cooked: <p>I have the following fully connected architecture and parameters for a network to train on time-series data</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, input_dim, output_dim, hl_sizes):
        super(Net, self).__init__()
        current_dim = input_dim
        self.linears = nn.ModuleList()
        for hl_dim in hl_sizes:
            self.linears.append(nn.Linear(current_dim, hl_dim))
            current_dim = hl_dim
        self.L = nn.Parameter(torch.rand(output_dim,dtype=torch.cfloat))
        self.V = nn.Parameter(torch.rand(output_dim,output_dim,dtype=torch.cfloat)) 

    def forward(self, x):
        input_vecs = x
        for layer in self.linears:
            x = F.relu(layer(x))
        x = torch.cat((torch.Tensor(np.ones((x.shape[0],1))),input_vecs,x),dim=1)
        return x

net = Net(
loss_func = nn.MSELoss()

eL = torch.diag_embed(torch.exp(net.L*dt)) 
A = net.V @ eL @ torch.pinverse(net.V) 

X = net(time_series_previous) @ A
Y = net(time_series_next)

loss = loss_func(X,Y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

</code></pre>
<p>which works just fine when the parameters L and V are real valued (e.g. torch.float32). However, these parameters need to be complex valued (torch.cfloat). Making this change results in several problems arising:</p>
<ol>
<li>torch.pinverse is not defined for complex data type. Workaround:</li>
</ol>
<pre><code class="lang-auto">U,s,W = torch.svd(net.V)
pinvV = W @ torch.diag_embed(1/s) @ U.conj()
</code></pre>
<p>However, complex matrix multiplication is not defined in pytorch, so we can’t actually straightforwardly compute pinV. Okay so then maybe we can write our own complex matrix multiplication. I did that. It seems to be working. The next problems:</p>
<ol start="2">
<li>
<p>It’s likely that the parameter A will be complex, so X will likely be complex. Why does nn.MSELoss() or loss_func(X,Y) spit out a complex scalar? The norm of a complex number is real-valued! I don’t currently have a workaround for this issue.</p>
</li>
<li>
<p>Lastly, even if I do end up with a real-valued X, meaning the loss is real-valued, PyTorch gives an error like grad_is_complex == True, expected False (something like this, and the error is specifically when loss.backward() is called).</p>
</li>
</ol>
<p>I would appreciate guidance on how to properly implement a network with learnable complex parameters such as the one above. Thanks!</p>
How does backward update the weights under the hood?

username: Lei_Xing
cooked: <p>Suppose I define a new function called foo . This function has some new operations, so I define the autograd.Function myself. In the backward, I use autograd to compute the gradient automatically.</p>
<p>class foo(torch.autograd.Function):</p>
<pre><code>@staticmethod
def forward(ctx, x0, params):
      out = torch.sum( x0 * params)
      ctx.save_for_backward(x0, params)
      return out 

@staticmethod
def backward(ctx, grad_output):
      x0, params = ctx.saved_tensors
      with torch.set_grad_enabled(True):
               out = torch.sum( x0 * params)
              gradient = torch.autograd.grad(outputs=out,
                                              inputs=params,
                                              grad_outputs= grad_output,
                                              allow_unused=True,
                                              retain_graph=True,
                                              only_inputs = True)
                dfdx = torch.autograd.grad(outputs=out,
                                              inputs=x0,
                                              grad_outputs= grad_output,
                                              allow_unused=True,
                                              retain_graph=True,
                                              only_inputs = True)
              return dfdx, gradient
</code></pre>
<p>Now I want to use the foo</p>
<p>class model(nn.Module):</p>
<pre><code>     def forward(self, x):
           parameters = self.parameters()
           new_parameters = process_parameters(parameters)
           func = foo.apply 
           out  = func (x, new_parameters)
           return out
</code></pre>
<p>in the model class, I have a process_parameters function to reshape the weights to different dimension.</p>
<p>Here is my question: how does PyTorch update weights by gradient descent method? How do I know the model class has new weights? Does PyTorch realize that new_parameters is actually self.parameters() and update self.parameters() according to the new value from new_parameters ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If your <code>process_parameter</code> function is differentiable, then yes the gradient will flow back all the way to the original Parameters.</p>
<p>Note that the update of the parameters will be done by the optimizer that you have to define along with your model.</p>
Element-wise differentiation

username: amaleki
cooked: <p>I have a network with input <code>X</code> of size of <code>m x 2</code> and output <code>Y</code> of size <code>m x 1</code>. How would I be able to generate elementwise derivative of output with respect to input? i.e. a matrix of size <code>dX</code> of size <code>m x 2</code><br>
where <code>dX_i,1 = dY_i / dX_i,1</code> and <code>dX_i,2 = dY_i / dX_i,2</code> .</p>
<p>Thanks,</p>

username: tom
cooked: <p>There is <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.functional.jacobian" rel="noopener nofollow ugc">torch.autograd.jacobian</a>, but note the caveats.</p>

username: amaleki
cooked: <p>The problem with using jacobian is that it produces too big matrices, and I run into memory problems.<br>
For example if input is <code>10x2</code> and output is <code>10x1</code>, the jacobian is <code>10x20</code>, and I only need the diagonal elements of the two <code>10x10</code> matrices stack to each other.</p>

username: tom
cooked: <p>Then do a for loop over the x2.</p>

username: amaleki
cooked: <p>I can’t generate the jacobian, because it is too big. Even if I could that’ll be very slow, and inefficient.</p>

username: tom
cooked: <p>Yeah, so I don’t think any way to directly generate just the diagonal unless you have special properties you know about (like "only the <code>i</code>th input influences the <code>i</code>th output as we - except for batch norm - have in minibatches).</p>
Print tensor to string with size constraints

username: Anders_G
cooked: <p>If I print a huge tensor in libtorch to a string it will simply print it all. Is there a way to get the output in a nice way like if I print a tensor in python, where it limits what it puts out for large tensors?</p>
<p>Currently I do this to print to a string:</p>
<pre><code class="lang-auto">        std::stringstream buffer;
        buffer &lt;&lt; *t &lt;&lt; std::endl;
</code></pre>

username: glaringlee
cooked: <p><a class="mention" href="/u/anders_g">@Anders_G</a><br>
Currently, there is no way to act like python print. We do plan to improve this later.<br>
The only thing that you can control is how many chars in one line. see here:<br>
‘<a href="https://github.com/pytorch/pytorch/blob/816d37b1d88bae649db32417443d9247cc3dbb76/aten/src/ATen/core/Formatting.h#L19" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/816d37b1d88bae649db32417443d9247cc3dbb76/aten/src/ATen/core/Formatting.h#L19</a>’</p>
The missing puzzle about torch::cuda::cudnn_is_available()

username: dambo
cooked: <p>I have been battering my head about it, why isn’t the method <code>torch::cuda::cudnn_is_available()</code> viewable in my code while I can do this <code>torch::Device device(torch::kCUDA);</code> load models and run inference ?</p>
<p>Turns out, you not only have to include<br>
<code>#include &lt;torch/script.h&gt;</code><br>
But also:<br>
<code>#include &lt;torch/torch.h&gt;</code></p>
<p>Good luck,</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/dambo">@dambo</a><br>
Please always include &lt;torch/torch.h&gt; when using libtorch.</p>
Element-wise logical operation and/or

username: ghazni
cooked: <p>I have set of masks corresponding to 10 different objects from an input image (shapes and sizes below)</p>
<pre><code class="lang-auto">masks.shape:  torch.Size([10, 240, 320, 1])
masks[0].shape: torch.Size([240, 320, 1])
masks[0][:,:,0].shape:  torch.Size([240, 320])
</code></pre>
<p>How do I produce union (logical or) of the these masks using torch tensor operation? Something along the lines of combining all the masks to develop single mask.</p>
<p>Thank you.</p>

username: tom
cooked: <p>&amp; and | on bool tensors should do the trick.</p>

username: ghazni
cooked: <p>Thank you, Thomas. Are you saying</p>
<p>masks[0][:,:,0] = masks[0][:,:,0] | masks[1][:,:,0]<br>
masks[0][:,:,0] = masks[0][:,:,0] | masks[2][:,:,0]<br>
masks[0][:,:,0] = masks[0][:,:,0] | masks[3][:,:,0]</p>
<p>and so on until 9 index (or alternatively use the loop)?</p>
<p>This code is running on cuda/GPU. Is there any operation which can be applied to whole stack of tensors (i.e. 10 tensors in this case)? I mean element wise ORing of all 10 tensors</p>

username: tom
cooked: <p>Ah, sorry. In this case min and max would work.</p>
How to sinc params between 2 modules?

username: hadaev8
cooked: <p>So, i want to make self.submodule1 to refer at same weights as self.submodule0<br>
And both modules have save variable names.<br>
How to perform it?</p>
<pre><code>class SomeSubmodule(nn.Module):
	def __init__(self):
		super().__init__()
		self.layers = torch.nn.ModuleList([])
		for _ in range(5):
			self.layers.append(torch.nn.Linear(512, 512))

	def forward(self, x):
		for layer in self.layers:
			x = layer(x)
		return x

class AnotherSubmodule(nn.Module):
	def __init__(self):
		super().__init__()
		self.layers = torch.nn.ModuleList([])
		for _ in range(5):
			self.layers.append(torch.nn.Linear(512, 512))

	def forward(self, x):
		for layer in self.layers:
			x = layer(x)
		return x

class Net(nn.Module):
	def __init__(self):
		super().__init__()

		self.submodule0 = SomeSubmodule()
		self.submodule1 = AnotherSubmodule()

	def forward(self, x):
		# bla bla bla
		return x</code></pre>

username: ptrblck
cooked: <p>You could initialize the linear layers in <code>Net</code> (or outside of it) and pass them to <code>SomeSubmodule</code> and <code>AnotherSubmodule</code>.<br>
However, since these submodules only iterate these linear layers you might also just be able to reuse the linear layers directly in <code>Net</code>.</p>

username: hadaev8
cooked: <p>Well, its not what easy.<br>
My real example looks like this</p>
<pre><code>self.transformer_decoder = TransformerDecoder(
	[TransformerDecoderLayer(
		self_attention=AttentionLayer(FullAttention(), hidden, nhead),
		cross_attention=AttentionLayer(FullAttention(), hidden, nhead), d_model=hidden) for _ in range(1)])
</code></pre>
<p>and</p>
<pre><code>self.transformer_decoder = RecurrentTransformerDecoder(
	[RecurrentTransformerDecoderLayer(
		self_attention=RecurrentAttentionLayer(RecurrentFullAttention(), hidden, nhead),
		cross_attention=RecurrentAttentionLayer(RecurrentCrossFullAttention(), hidden, nhead), d_model=hidden) for _ in range(1)])
</code></pre>
<p>I don’t want to change layers’ definitions.<br>
But it has the same variables names, just different math.<br>
So I guess I can init both modules, get dict name-param of first, iterate the second module, and replace its params with params of the first one.<br>
But not sure if its the best way.</p>
How to prevent for part of the loss the create gradient

username: natank1
cooked: <p>Hi</p>
<p>I am developing a sort of creature which is similar to Auto encoder or VAE</p>
<p>I want the encoder to have a loss of his own (f) and than take its outputs (Y) and feed them the decoder with his own loss (g)</p>
<p>I want g to have derivatives up to Y (namely I dont want the gradient of g to reach the encoder)</p>
<p>However, when I run the code I can see that even if the loss is only g and the optimizer is allowed to change only the encoder, learning takes place i.e. the decoder’s loss changes the weights of the encoder.</p>
<p>What can I do?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="natank1" data-post="1" data-topic="97575">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/natank1/48/29203_2.png" class="avatar"> natank1:</div>
<blockquote>
<p>if the loss is only g and the optimizer is allowed to change only the encoder, learning takes place i.e. the decoder’s loss changes the weights of the encoder.</p>
</blockquote>
</aside>
<p>How did you make sure that the encoder doesn’t get gradients? Did you detach <code>Y</code> before feeding it into the decoder?<br>
Also, why is the optimizer allowed to update the encoder, if it shouldn’t be trained?</p>

username: natank1
cooked: <p>Thanks for the quick reply.<br>
The optimizer allows to change the encoder because it is configured to What i did was the following:<br>
loss =  encoder_f<br>
loss.backward( retain_graph=True  )<br>
optimizer_enc.step()<br>
Y= autograd.Variable(z,requires_grad=False)<br>
optimizer_dec.zero_grad()<br>
g, _ = decoderl(Y, x)<br>
loss = g<br>
loss.backward()<br>
optimizer_dec.step()</p>
<p>It works, the question if it “torch enough”?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Based on your code it doesn’t seem that the output of the encoder is fed to the decoder, so the calculated loss of the decoder won’t create any gradients in the encoder.</p>
<p>If you are worries about best practices in PyTorch:</p>
<ul>
<li>check, if you really need to use <code>retain_graph</code> since based on your code snippet it doesn’t seem as if it’s needed</li>
<li>
<code>Variable</code>s are deprecated so you can use tensors after PyTorch <code>0.4</code>
</li>
</ul>

username: natank1
cooked: <p>Sorry this line<br>
loss = encoder_f<br>
Should be written<br>
loss,Y = encoder_f<br>
." <code>Variable</code> s are deprecated so you can use tensors after PyTorch  <code>0.</code>"<br>
Not sure I understood</p>

username: natank1
cooked: <p>OK<br>
Indeed retain_graph is redundant and I replaced the auto_grad with this<br>
Y = torch.tensor(Y, requires_grad=False)</p>
YOLO v1 how to encode labels

username: Yolkandwhite
cooked: <p>Hi. I’m trying to make a model, based on YOLO v1.</p>
<p>but I’m having struggle with the dataset.</p>
<p>I already made the YOLO network and the out put is 7x7x10</p>
<p>7x7 is grid size and 10 is for 2 * (x, y, w, h, confidence score) two bounding box parameters.</p>
<p>I removed the class parameter, because my work doesn’t need classification. It only needs the bounding box for localization.</p>
<p>my current dataset looks like this (image, target)</p>
<p>target = (x1, y1, w1, h1, Confidence1, x2, y2, w2, h2, Confidence2)</p>
<p>but I need the targets to be 7x7x10.</p>
<p>how can I encode the targets to fit in the right grid cell?</p>

username: ruka
cooked: <p>The formula would be</p>
<blockquote>
<p>numpy.floor([ y/image_size * grid_size, x/image_size * grid_size ])</p>
</blockquote>
<p>Assume your input image is 256 * 256, and the center of a target is (y=10, x=123), your grid size is 7 * 7.<br>
Then the center would fall in numpy.floor([ 10/256 * 7, 123/256 * 7 ]) = [0, 3],  aka grid[0][3].</p>
Can I add hook when training and remove them when testing

username: yqian
cooked: <p>Hello, I am training a model with evaluation periodically. So I wonder if I can add hook when training and remove then when evaluating. Could I use the hook handle all the time without registering new hook again?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.register_hook"><code>register_hook</code></a> as well as <code>register_forward_hook</code> will return a <code>handle</code>, which can be used to remove the hook via <code>handle.remove()</code>. So yes, you should be able to use the hook only during training and remove them afterwards.</p>
Validation accuracy not increasing in multiple gpu but is fine when trained in single gpu

username: Nisan_Aryal
cooked: <p>I am training a Deep complex network. The Complex convolution, complex batch normalization etc are used from this repo.(<a href="https://github.com/wavefrontshaping/complexPyTorch" rel="noopener nofollow ugc">https://github.com/wavefrontshaping/complexPyTorch</a>). I am observing a unique problem which is that when I am training the model on 2 gpus(Titan RTX), The training accuracy is increasing where as the validation accuracy is stuck at around 17%. The same model is showing good validation accuracy (accuracy increasing with training ) when trained at single gpu.</p>
<p>I tried an experiment where I used a model from torchvision and tested on multiple gpu. It worked fine.<br>
For the multiple GPU I am changing only one line</p>
<p>model=nn.DataParallel(complex_model)</p>
<p>I suspect that there might be a issue with parallel computation and complex network.<br>
The tentative code is given below</p>
<pre><code class="lang-auto">  model=nn.DataParallel(complex_model)
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    for j in range(epoch):
        model.train()
        total=0
        correct=0
        for i,(real,imag, label) in enumerate(training_dataset_loader):
            real = real.cuda().float()
            imag = imag.cuda().float()
            label = label.cuda()
            label = label.view(-1).long()
            optimizer.zero_grad()
            outputs = model(real,imag)
            loss = criterion(outputs, label)
            print('iteration  == %d  epoch  == %d   loss  == %f '%(i+1,j+1,loss))
            loss.backward()
            optimizer.step()
            softmax = nn.Softmax(dim=1)
            outputs=softmax(outputs)
            _, predicted = torch.max(outputs.data, 1)
            total += label.size(0)
            correct += (predicted == label).sum().item()
        print('Accuracy of training set after %d epoch is %f ' % ((j + 1, correct / total * 100)))
        model.eval()
        num_total = 0
        total=0
        correct=0
        for i, (real,imag, label) in enumerate(validation_dataset_loader):
            real = real.cuda().float()
            imag = imag.cuda().float()
            label = label.cuda()
            label = label.view(-1).long()
            optimizer.zero_grad()
            outputs = model(real,imag)

            softmax = nn.Softmax(dim=1)
            outputs = softmax(outputs)
            _, predicted = torch.max(outputs.data, 1)
            total += label.size(0)
            correct += (predicted == label).sum().item()
        print('Accuracy of validation set after %d epoch is %f '% ((j+1,correct/total*100)))
</code></pre>
<p>for the input i am giving an spectrogram after separating real and imaginary part.<br>
Am I missing something while using multiple gpus?</p>
Inputting data with multiple variable dimensions to an LSTM

username: Remorax
cooked: <p>Hi,</p>
<p>My data is of shape <code>batch_size * num_paths * num_edges * emb_dim</code>. Here:</p>
<ul>
<li>
<code>batch_size</code>: refers to batch size (32)</li>
<li>
<code>num_paths</code>: refers to the no of paths between two given terms in a dependency tree</li>
<li>
<code>num_edges</code>: refers to the no of edges in a specific path</li>
<li>
<code>emb_dim</code>: refers to the embedding dimension (300)</li>
</ul>
<p>Both <code>num_paths</code> and <code>num_edges</code> are variable for every training sample. In other words, for every training sample, there may be variable number of paths, each with variable number of edges.</p>
<p>(Note that <code>data</code> is an n-d list of lists in native Python, since two dimensions (<code>num_paths</code> and <code>num_edges</code> are variable)</p>
<p>I want to pass each path of each training instance as input to an LSTM, since a path is a sequence of edges and I want the resultant path representation after passing it through the LSTM. After obtaining the representation for each path in a training example, I want to take the sum of all these representations.</p>
<p>I know I can pack variable number of edges through <code>pack_padded_sequence</code>. But what about variable number of paths? How do I account for that? Is there any way to do it in native Pytorch, without going for messy solutions like iterating through loops?</p>
<p>Any help would be greatly appreciated!</p>
<p>Thanks</p>

username: googlebot
cooked: <p>IIUC paths are treated as independent sequences, so you can treat their dimension as an additional batch dimension. I.e. map (batch_idx, path_idx) -&gt; seq_idx, create a compact tensor of sequences, run a RNN, map results back and aggregate by batch_idx.</p>

username: Remorax
cooked: <p>Thanks, that does sound good! Hopefully it should solve my problem.</p>
Replacing method of tensor value

username: Yoshi
cooked: <p>Hi there,</p>
<p>I am a new of Pytorch.<br>
I want to modify tensor <code>x</code> to <code>y</code> as a following rule:</p>
<pre><code class="lang-auto">y[i][j] = 1.0 ,if x[I][j] &gt; 0,
y[i][j] = 0.0 ,if x[I][j] = 0,
y[i][j] = -1.0 ,if x[I][j] &lt; 0.
</code></pre>
<p>For example:</p>
<pre><code class="lang-auto">x = tensor([
  [0.0, 1.5, 0.5, 1.0],
  [0.0, 0.0, 1.0, 1.5],
  [-0.5, -1.0, -1.5, -0.5]
])
</code></pre>
<pre><code class="lang-auto">y = tensor([
  [0.0, 1.0, 1.0, 1.0],
  [0.0, 0.0, 1.0, 1.0],
  [-1.0, -1.0, -1.0, -0.0]
])
</code></pre>
<p>(<code>x</code> should have -1.5, -1.0, -0.5, 0, 0.5, 1 or 1.5)</p>
<p>For now, I implemented them below code:</p>
<pre><code class="lang-auto">x[x&gt;0] = 1.0
x[x&lt;0] = -1.0
y = x
</code></pre>
<p>I am using GPUs to run the above code, but actually <code>x</code> and <code>y</code> are very large tensors so it takes a lot of time.<br>
I guess it is because my code isn’t parallelized properly.</p>
<p>Does anyone have a better idea to modify the tensor?</p>
Some wrong about libTorch C++ API

username: hookzeng
cooked: <p>Hi, I meet some problem about libTorch. When I load my model, I find the code can’t find torch.ops.torchvision.nms. Anybody know how to deal it? Thx~</p>
<p>The error like this:</p>
<p>terminate called after throwing an instance of ‘torch::jit::ErrorReport’<br>
what():<br>
Unknown builtin op: torchvision::nms.<br>
Could not find any similar ops to torchvision::nms. This op may not exist or may not be currently supported in TorchScript.<br>
:<br>
File “/home/hookzeng/python-project/pytorch/SSDv3_edition_v2/utils/data_process.py”, line 495<br>
“”"<br>
torch.ops<br>
return torch.ops.torchvision.nms(boxes, scores, iou_threshold)<br>
~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE</p>

username: ptrblck
cooked: <p>Your <code>torchvision</code> installation might be too old, as this operation should be supported in the JIT.<br>
Could you install the nightly binary and rerun the code (or build from source)?</p>

username: hookzeng
cooked: <p>I‘m sorry. I don’t understand how to deal it. My Torchvision Version:  0.7.0, Torchvision Version:  1.6.0.</p>

username: ptrblck
cooked: <p>I might be wrong in the assumption in a mismatch of torchvision and PyTorch versions.<br>
Could you try to install the nightly binaries and rerun it again, as it should be working based on <a href="https://discuss.pytorch.org/t/torchvision-ops-nms-in-torchscript/89286/2">this</a>.</p>

username: lysuhin
cooked: <p>Hello, <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
I can confirm facing this error too, either on 1.6/0.7 or nightly builds.<br>
I tried to re-run scripting of a model (keypoints-rcnn from torchvision.models) as well, still got no positive effect.</p>
Output of VGG16 on a classification problem

username: anotherone_one
cooked: <p>Hello everyone! I wanted to clarify a doubt I have regarding the vgg16 network. I am currently using the pre-trained vgg16 network for a classification problem with 2 labels. I already have the best weights for tthis problem, using as a criterion the nn.CrossEntropyLoss and I can get a prediction by doing:</p>
<pre><code class="lang-auto">outputs = vgg16(net_img)
_, preds = torch.max(outputs.data, 1)
</code></pre>
<p>However, my goal is not to have a binary prediction (0 or 1), but the probability and also the cross entropy metric for each class. I wanted to check if what I am doing makes sense.<br>
To get the probability of each class I am doing:<br>
<code>probabilities = torch.sigmoid(outputs)</code><br>
And to get the cross entropy os each class I am simply using the outputs I already calculated.</p>
<p>Does this make sense? If it helps, I based my work in this tutorial: <a href="https://www.kaggle.com/carloalbertobarbano/vgg16-transfer-learning-pytorch" rel="nofollow noopener">https://www.kaggle.com/carloalbertobarbano/vgg16-transfer-learning-pytorch</a></p>
<p>Thank you in advance!</p>

username: ptrblck
cooked: <p>To get the probabilities, you should probably use <code>probs = F.softmax(outputs, dim=1)</code>, since you are using <code>nn.CrossEntropyLoss</code> as the criterion which means your output should have the shape <code>[batch_size, 2]</code> (used in a multi-class classification).</p>

username: anotherone_one
cooked: <p>Thank you for your help! But without the F.softmax layer, the direct output is the cross entropy value, right? since I am using the CrossEntropyLoss as criterion</p>

username: ptrblck
cooked: <p>Without the softmax the output would contain logits. I don’t know, is that’s what “cross entropy value” would refer to.<br>
Could you explain a bit, what you mean by this?<br>
It will not contain the cross entropy (loss) between the output and target, as this will be returned by <code>nn.CrossEntropyLoss</code> itself.</p>

username: anotherone_one
cooked: <p>Sorry, I didn’t explain well. Basically I wanted to evaluate my model with something similar to this: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html" rel="noopener nofollow ugc">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html</a></p>

username: ptrblck
cooked: <p>If I understand the docs correctly, probabilities are expected as the model output, so it seems that applying a softmax should work.</p>
<p>However, this part of the docs:</p>
<blockquote>
<p>the probabilities provided are assumed to be that of the positive class</p>
</blockquote>
<p>sounds as if the loss function is dealing with a multi-label output (each output would give a probability in <code>[0, 1]</code> for a separate class)? Can you verify it?</p>