akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use TensorboardX ptorch

Aarya: <p>Hello,</p>
<p>I am new to pytorch. I have build a siamese network with a custom hand gesture video dataset with accuracy as an evaluation metric.</p>
<p>Now I have to plot the accuracy and loss function graphs. I would like to use tensorboardX for the same. I have referred the pytorch official website for the same, but did not understand how to use it in my case having a custom dataset and not MNIST dataset.</p>
<p>But, I am not aware about the structure and summary writer.</p>
<p>Can anyone please help me with this?</p>
<p>Any help is most appreciated !<br>
Thank you in advance. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I think the <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html">Tensorboard Tutorial</a> might be a good starter. I’m unsure, if the same operations would work for <code>TensorboardX</code>, so in case you need to use it you might need to adapt some calls.</p>

Aarya: <p>Thank you so much for your help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
What is behind pathwise derivative?

matheo_r: <p>Hi everyone,</p>
<p>I couldn’t find how Pytorch deals with the reparameterization trick using the rsample method (<a href="https://pytorch.org/docs/stable/distributions.html" class="inline-onebox" rel="noopener nofollow ugc">Probability distributions - torch.distributions — PyTorch 1.9.0 documentation</a>).</p>
<p>For distributions that do not belong to the location-scale family like the Gamma distribution, does it use the implicit reparametrization gradients (<a href="https://arxiv.org/pdf/1805.08498.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1805.08498.pdf</a>) or other approximations?</p>
<p>Thanks,<br>
Matheo</p>
Optimizer.step(closure)

RoitSky: <p>When shall I use ‘optimizer.step(closure)’ instead of ‘optimizer.step()’ ?<br>
I have read the PyTorch Docs, however i’m not aware of its description.</p>
<p>"<br>
Some optimization algorithms such as Conjugate Gradient and LBFGS need to reevaluate the function multiple times, so you have to pass in a closure that allows them to recompute your model. The closure should clear the gradients, compute the loss, and return it.<br>
"</p>
<p>What does ‘reevaluate the function multiple times’ mean ? What does the ‘function’ refer to ?</p>

googlebot: <p>‘function’ is a callable that returns a differentiable loss (i.e. main nn.Module with attached loss function), these algorithms have an [inner] training loop inside optimizer.step, hence this ‘closure’ is mostly a boilerplate to have a loop turned inside out (i.e. a delegate allowing nested forward+backward+change_params iterations)</p>

RoitSky: <p>But why PyTorch Docs emphasizes that ‘Some optimization algorithms such as Conjugate Gradient and LBFGS…’. When i create a SGD or an Adam optimizer, i just code in ‘optimizer.step()’.</p>

googlebot: <p>BFGS &amp; co are batch (whole dataset) optimizers, they do multiple steps on <strong>same</strong> inputs. Though docs illustrate them with an outer loop (mini-batches), that’s a bit unusual use, I think. Anyway, the inner loop enabled by ‘closure’ does parameter search with inputs fixed, it is not a stochastic gradient loop you do with SGD or Adam.</p>

RoitSky: <p>Thanks for your help, i got you!!</p>
Forcing consistency in output

Michael1: <p>I’m currently training a NN for a reconstruction task, with a structure such that its first layer is a Conv2D layer and its output as well. It takes in an image + additional parameters, where I force the additional parameters to be the same number in a tensor of the same size as the image. The parameter map is then passed through as it’s own channel. The output layer is the same as the input layer and outputs an image + additional parameters.</p>
<p>I’d like to teach the neural network / or force it that the additional parameters at the output should also be consistent, so the output channels corresponding to the parameters should be nearly or exactly the same within a channel.</p>
<p>I hope someone can help me out.</p>
Reset 'setdevice()' when creating thread

111480: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/602dc556e8c3b5b991adf5f1b85215aad8bc34d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/602dc556e8c3b5b991adf5f1b85215aad8bc34d4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/602dc556e8c3b5b991adf5f1b85215aad8bc34d4.png" alt="image" data-base62-sha1="dIPTQzngspjfssjwYuqMq2YrB4w" width="690" height="68" data-dominant-color="2D2F3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×97 4.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Even if I set it to run on GPU 1 from main() to setdevice(), if I create a thread and have the cuda function run inside the thread, it will reset back to GPU 0.</p>
<p>What’s wrong?</p>
<p>Do I always have to initialize GPU index to 1 every time I create a thread?</p>
The implementation of torch.nn.functional.kl_div()

lei_bai: <p>I am using torch.nn.functional.kl_div() to calculate the KL divergence between the outputs of two networks. However, it seems the output of F.kl_div() is not consistent with the definition.</p>
<p>For example, let assume the normalized pred = torch.Tensor([[0.2, 0.8]]), and target = torch.Tensor([[0.1, 0.9]]).</p>
<p>Then the output of F.kl_div() would be:<br>
F.kl_div(pred, target, reduction=‘sum’, log_target=False) —&gt; -1.0651<br>
or<br>
F.kl_div(pred, target, reduction=‘sum’, log_target=True) —&gt; 0.1354</p>
<p>However, if I calculate the KL divergence according to the definition:<br>
(pred * torch.log(pred/target)).sum() —&gt; 0.0444</p>
<p>Does anyone know what is the reason for the difference (the torch version is 1.8)?</p>

lei_bai: <p>Any discussions ???</p>

ptrblck: <p>I get the same outputs using the definition from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html">docs</a>:</p>
<pre><code class="lang-python">pred = torch.tensor([[0.2, 0.8]])
target = torch.tensor([[0.1, 0.9]])
F.kl_div(pred, target, reduction='sum', log_target=False)
&gt; tensor(-1.0651)
torch.sum(target * (torch.log(target) - pred))
&gt; tensor(-1.0651)

F.kl_div(pred, target, reduction='sum', log_target=True)
&gt; tensor(0.1354)
torch.sum(torch.exp(target) * (target - pred))
&gt; tensor(0.1354)
</code></pre>

lei_bai: <p>Hi, ptrblck,</p>
<p>Thanks for your reply. I think I understand:</p>
<ol>
<li>pred is already passed through log before entering F.kl_div(), that’s why there is no torch.log() for pred.</li>
<li>F.kl_div(pred, target) is KL(target | pred) instead of KL(pred | target), is it correct?</li>
</ol>
GRU Weights Lacking?

ajrheng: <p>A GRU is defined in Torch as</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4eb52ff529670bd83b0dd7fd38ecc3726eee8bc.png" alt="Screenshot from 2021-08-13 10-11-08" data-base62-sha1="yWEGjQ7T0o3NkCSQYaqKMdWdnw8" width="377" height="122"></p>
<p>As I understand, the trainable weights are W_ir, W_hr, W_iz, W_hz, W_in and W_hn along with their biases. However, if I define a GRU and print its <code>state_dict()</code>:</p>
<pre><code class="lang-auto">rnn = nn.GRU(1, 1, 1)
print(rnn.state_dict())
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">OrderedDict([('weight_ih_l0', tensor([[0.8447],
        [0.0636],
        [0.3000]])), ('weight_hh_l0', tensor([[-0.2396],
        [ 0.2593],
        [-0.7984]])), ('bias_ih_l0', tensor([-0.8617, -0.9971, -0.0588])), ('bias_hh_l0', tensor([0.9238, 0.1282, 0.9144]))])
</code></pre>
<p>According to the documentation,</p>
<pre><code class="lang-auto">~GRU.weight_ih_l[k] – the learnable input-hidden weights of the kth\text{k}^{th}kth layer (W_ir|W_iz|W_in), of shape (3*hidden_size, input_size) for k = 0. Otherwise, the shape is (3*hidden_size, num_directions * hidden_size)
</code></pre>
<p><code>weight_ih_l0</code> seems to encompass W_ir, W_iz, W_in, but there’s only one set of weights for it. Same goes with <code>weight_hh_l0</code>. Does this mean these weights are being shared?</p>

googlebot: <p>they’re concatenated because it is more efficient to do a single operation like [W1 W2 W3] @ x than follow how they appear in formulas literally, on cuda anyway</p>

ajrheng: <p>Thanks for the response. According to documentation, each weight, for example <code>W_ir</code> has dimensions <code>(3*hidden_size, hidden_size)</code> assuming unidirectional GRU which is what I have. I set <code>hidden_size=1</code>, so the tensor you see in <code>weight_ih_l0</code> which has dimensions<code>(3,1)</code>is not <code>W_ir W_iz W_in</code> concatenated. It should refer to just one tensor.</p>

googlebot: <p>why do you think that W_ir should have 3x multiplier? It appears from concatenation (r,z,n), you have to slice weight_ih_l0 to extract W_ir.</p>

ajrheng: <p>Oh, that 3x multiplier comes about <em>because</em> the three weights are being concatenated. I somehow misread the documentation. Now it makes sense. Thanks!</p>
What's the efficient way to lookup embedding from a given tensor

KylinA1: <p>Hi.<br>
I want to know is there any efficient way to get some rows from a given tensor, like lookup_embedding in tensorflow.<br>
For a given tensor with <code>requires_grad = True</code>, it will applied some functions/changes/masks, and then used as a look up table. Thus I am trying to find exactly the same way what <code>tf.nn.embedding_lookup</code> does.<br>
Directly  <code>tensor[index]</code> works, I wonder, is there more efficient way to do that.</p>
<p>Appreciate for any replies.</p>

Adherer: <p>I also have similar problems.</p>

Adherer: <p>You can use this API: nn.Embedding.from_pretrained</p>

sqlkrad: <p><a href="https://pytorch.org/docs/stable/torch.html#torch.index_select" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.index_select</a></p>

garyyyccc: <p>I found that the direct way <code>tensor[index]</code> is more efficient than <code>nn.Embedding.from_pretrained</code>.<br>
Any Comment?</p>
How to profile CUDA memory usage for each part of model

dancedpipi: <p>referece to pytorch profiler, it seem only trace cpu memory instead of gpu memory, is there any tool to trace cuda memory usage for each part of model?</p>

googlebot: <p>Try <a href="https://github.com/Stonesjtu/pytorch_memlab" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Stonesjtu/pytorch_memlab: Profiling and inspecting memory in pytorch</a>, though it may be easier to just manually wrap some code blocks and measure usage deltas (of cuda.memory_allocated).</p>

dancedpipi: <p>Thanks for your reply, I’ll try it.<br>
Is there a official pytorch profiler for gpu memory?</p>

googlebot: <p>afaik, it only has torch.profiler.profile(profile_memory=True) as an aggregator, I’m not sure if that produces useful results (there is undocumented autograd.profiler.record_function(“X”) to mark code blocks)…</p>

dancedpipi: <p>Thanks! torch.profiler.profile(profile_memory=True) seem only produce cpu memory usage, I might have to find another way</p>

googlebot: <p>there are options for cuda (version dependent, so check docs)</p>
Selecting indices from list of indices

ThunderWiring: <p>I have a 3D tensor of size (8,8,2052), all zeros. This will be the mask.<br>
Also, there is a tensor of size (29, 3), which is basically a list of indices that need to be 1 in the mask.</p>
<p><strong>Is there a pytorch way of doing this? im avoiding using python <code>for</code> loop, to iterate over the indices’ list.</strong><br>
<strong>so far i tried the following: take, index_select, “smart” indexing…</strong></p>
<p>i put a lot of time on this, tried a lot of approaches and nothing seems to give me  what im looking for.<br>
Any ideas?</p>
<p>Here is an example of the indices list:</p>
<pre><code class="lang-auto">tensor([[0, 1, 0],

        [0, 2, 0],

        [0, 7, 0],

        [1, 2, 0],

        [1, 3, 0],

        [1, 4, 0],

        [1, 6, 0],

        [1, 7, 0],

        [2, 2, 0],

        [3, 3, 0],

        [3, 4, 0],

        [4, 0, 0],

        [4, 2, 0],

        [4, 4, 0],

        [4, 6, 0],

        [5, 0, 0],

        [5, 4, 0],

        [5, 5, 0],

        [5, 6, 0],

        [6, 0, 0],

        [6, 2, 0],

        [6, 4, 0],

        [6, 5, 0],

        [6, 6, 4],

        [7, 0, 0],

        [7, 2, 1],

        [7, 4, 2],

        [7, 5, 40],

        [7, 6, 0]])
</code></pre>

ptrblck: <p>Could you post the “slow” loop approach so that we could check for a better way, please?</p>

ThunderWiring: <pre><code class="lang-auto">mask = torch.zeros((8,8,2052)) # in this case the depth is 2052, but can be much larger
idxs = tensor of size (29, 3) # the 29 can also be larger - (max is 8 * 8 * depth_of_mask)
for idx in idxs:
  # idx is a tensor of size (1, 3), 
  # using unbind to get each number on its own 
  # and fing the right 'cell' in the mast
  mask[idx.unbind()] = 1
</code></pre>
Slow inference with U-2-Net on iOS

wpmed92: <p>I’m using <a href="https://github.com/NathanUA/U-2-Net" rel="nofollow noopener">U-2-Net</a> on mobile.<br>
I converted the model to TorchScript, and used it succesfully on Android. The inference takes about 1 sec.<br>
Now, if I use the same model on iOS, the inference is about half a minute. I disable autograd, and run the model in inference mode.<br>
Any idea what could cause such a big performance difference using the same model?</p>
<p>(I use 1.6.0 nightly build on Android, and 1.5.0 production build on iOS)</p>

xta0: <p>Android 1.6.0 nightly is using XNNPACK, whereas iOS 1.5.0 is not. We’ll be enabling XNNPACK on iOS in 1.6.0 release.</p>

wpmed92: <p>Ok, thank you! Meanwhile we managed to speed up the iOS inference. The thing is that on Android we scripted to model by calling jit.script(), and we tried to use this exact same model on iOS, which was slow. I tried another approach for converting to TorchScript by tracing the model’s execution with jit.trace, and for some reason the “traced model” performance on iOS was on par with the Android inference. Can it be also related to XNNPACK somehow?</p>

xta0: <p>Cool, it’s nice to hear the issue has been resolved. Short answer, no. The XNNPACK is our new computation kernels that has nothing to do with scripting. Yea, tracing is supposed to be faster.</p>

Muhammad_Umer_Zia_Mu: <p>hello Ahmed , I am converting U2-Net for IoS app but having issue with conversion . Could you help me by your conversion code or .ml file ?</p>

wpmed92: <p>Hi, sure I can help you!</p>
<p>in u2net_test.py write this after net.eval():</p>
<pre><code class="lang-auto">scripted = torch.jit.script(net)
torch.jit.save(scripted, "fod.p")
</code></pre>
<p>This is how I build the model I use on Android. On iOS however, this model didn’t work so I used the following conversion method:</p>
<p>In u2net_test.py, in the loop which goes through the test images, write this after inputs_test is loaded:</p>
<pre><code class="lang-auto">if traced == False:
    traced_script_module = torch.jit.trace(net, inputs_test)
    traced_script_module.save("fod_traced.p")
    traced = True
</code></pre>
<p>Use <em>traced</em> boolean to only trace a single inference, not all the inferences.<br>
Hope it helps.<br>
If you need help with how to use these models on Android or iOS, let me know.</p>

Muhammad_Umer_Zia_Mu: <p>hi dear !<br>
my UNet model is displaying black image on IoS app , can I view your source code or model for help ?</p>

fahad: <p>Hi Ahmed,<br>
I successfully converted the model to pytorch script, but having some trouble using it on android platform. Can you guide me how can I use this u2net model, I mean how will we give it the input and take the output in android,<br>
Thanks in advance …</p>
Multi loss optimizer

ZidanevsRaul: <p>Hi there.</p>
<p>I have a question for the masters.<br>
Please give me hope.</p>
<p>There are many layers in my model.<br>
I get total_loss with sum of the final output and the output of a specific layer.<br>
I proceed to differentiate total_loss, and of course there is only one optimizer.</p>
<pre><code class="lang-auto">MyModel():
def __init__(self):
  ...
  many layers
  ...
  self.specific_layer()
  ...
  many layers

def forward(self, x):
  ...
  output1 = Layer1(x)
  output_s = self.specific_layer(output1.detach())
  ...
  outputN = LayerN(LayerN-1(cat(output1, LayerN-2(...), .....)))
  ...
  final_output = FinalLayer(outputN)

  return final_output, output_s


train.py
model = MyModel()
optimizer = Adam(model.parameters(), ...)
...
loop:
  optimizer.zero_grad()

  final_y_pred, specific_y_pred = model(x)

  final_loss = criterion(final_y_pred, final_target)
  specific_loss = criterion(specific_y_pred, specific_target)
  total_loss = final_loss + specific_loss

  total_loss .backward()
  optimizer.step() 
</code></pre>
<p>The total_loss converges, but the loss of a specific layer oscillates. T.T</p>
<p>I thought of two ways…</p>
<p>The first is to train by applying learning-rate differently (smaller learning-rate to the oscillated specific layer) using “per-parameters-options”.</p>
<p>The second method is to train by applying optimizer separately.</p>
<ol>
<li>Remove the specific_layer from MyModel and set the output to output1, final_output.</li>
<li>Set the separated specific_layer as MyModel2 with output1 as input and return output_s.</li>
<li>Set the optimizer for MyModel and MyModel2 , respectively.</li>
</ol>
<p>Which is a better way?<br>
And is there another good way?</p>
<p>Thank you in advance.</p>

ptrblck: <p>Based on the posted pseudo code it seems that <code>specific_loss</code> would create the gradients for <code>self.specifc_layer</code> only (<code>layer1</code> and previous layers are detached and no other layers are used to create <code>output_s</code>), while <code>final_output</code> might potentially use many more layers (thus also parameters).<br>
If so, it could be easier for the model to drive <code>final_loss</code> down as its optimizer would potentially update many more parameters, but that’s just my guess.</p>

ZidanevsRaul: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for reply.</p>
<p>You’re right!<br>
final_output use many many more layers…<br>
specific_layer has only two layer(LSTM, FC) but the other consists of multiple CNNs, LSTMs, FCs, …</p>
<p>What should I do in this case?</p>

ptrblck: <p>I’m unsure if it would work, but you could try to scale up the <code>specific_loss</code> to try to force the model to focus more on it.</p>
Correct way to do alternating updates with multiple optimizers

elkop: <p>I want to alternately update two networks, say <code>m1</code> and <code>m2</code>. Both these networks are a part of another <code>nn.Module</code>. For simplicity, we can assume that both  <code>m1</code> and <code>m2</code> are linear layers.</p>
<p>Currently, I have implemented this as shown below, however my loss remains nearly constant which indicates that gradients aren’t being updated correctly.</p>
<pre><code class="lang-auto">        optimize_m1 = True
        for x1, x2 in train_loader:
            m1_optimizer.zero_grad()
            m2_optimizer.zero_grad()

            loss = model(x1, x2, optimize_m1=optimize_m1)
            loss.backward()

            if optimize_m1:
                m1_optimizer.step()
            else:
                m2_optimizer.step()

            optimize_m1 = not optimize_m1
</code></pre>
<p>Model is defined as below:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, in_dim, out_dim
                    ) -&gt; None:
        
        super().__init__()
        self.m1 = nn.Linear(in_dim, out_dim)
        self.m2 = nn.Linear(in_dim, out_dim)
       
        self.act = Activation()

    def forward(self, x1, x2, optimize_online=True):
        out1 = self.m1(x1)
        out2 = self.m2(x2)

        out1 = self.act(out1)
        out2 = self.act(out2)
        
        loss = mse(out1, out2)
        return loss
</code></pre>
<p>For optimizers, I am using SGD defined as</p>
<pre><code class="lang-auto">m1_optimizer = torch.optim.SGD(list(model.m1.parameters()), lr=config.lr)
m2_optimizer = torch.optim.SGD(list(model.m2.parameters()), lr=config.lr)
</code></pre>
<p>Also, for loss do I need to use <code>.detach()</code> on <code>out1</code> and <code>out2</code> alternately, while optimizing the model? I am pretty sure that this simple architecture works because I replicated this with linear layers in numpy and the code works correctly. Trying to figure out where am I going wrong with PyTorch implementation.</p>

ptrblck: <p>Since you are calling the <code>step()</code> method for the corresponding optimizer only in each iteration, you wouldn’t need to <code>detach</code> tensors.</p>
<aside class="quote no-group" data-username="elkop" data-post="1" data-topic="129238">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/c57346/48.png" class="avatar"> elkop:</div>
<blockquote>
<p>which indicates that gradients aren’t being updated correctly.</p>
</blockquote>
</aside>
<p>You could verify this claim by checking the <code>.grad</code> attribute of the model, which will be updated, and check that it was indeed updated properly in the right iteration.</p>
Does Dynamic Quantization support GPU?

MattHatter: <p>Hello,<br>
I’ve tried doing dynamic quantization on the XLNet model during inference, and I got this error message:</p>
<p><code>RuntimeError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend. 'quantized::linear_dynamic' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</code></p>
<p>This leads me to believe dynamic quantization doesn’t support CUDA, and if so, do you guys plan to have CUDA support for quantization for both training and inference? I couldn’t find any issues relating to this on Github. Thanks!</p>

jerryzh168: <aside class="quote no-group" data-username="MattHatter" data-post="1" data-topic="119231">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c37758/48.png" class="avatar"> MattHatter:</div>
<blockquote>
<p><code>RuntimeError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend. 'quantized::linear_dynamic' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</code></p>
</blockquote>
</aside>
<p>yeah it is not supported on CUDA, quantized::linear_dynamic is only supported in CPU. We do not have immediate plans to support CUDA but we plan to publish a doc for custom backends which will make the extension easier.</p>

yan_kui: <p>hi, where could we get the doc?</p>

jerryzh168: <p>will think about post one in OSS, please keep an eye out for that in github issues page, we are currently working on enabling CUDA path through TensorRT as well, had a prototype here: <a href="https://github.com/pytorch/pytorch/pull/60589" class="inline-onebox" rel="noopener nofollow ugc">[not4land] Test PT Quant + TRT path by jerryzh168 · Pull Request #60589 · pytorch/pytorch · GitHub</a></p>
<p>I can share the doc early with you if you message me your email. but we may make some modifications before publishing in oss</p>

MattHatter: <p>Looking forward to this! Thanks <a class="mention" href="/u/jerryzh168">@jerryzh168</a> for your support in this matter <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Infinite value occurring in training dataset after exactly 7 epochs

Lachlan_Burne: <p>I’m having a problem where the LogSoftmax layer of my network returns NaN values at the 0th output after exactly 7 epochs, or on the beginning of the 8th to be exact. I believe this is due to the presence of an inf value somewhere in the training dataset being fed into the network. The confusion comes from the fact that the dataset in question does not originally contain an inf value. When I import the dataset from an h5py file and use <code>np.isinf(feature_data).any()</code> it returns false, then after I convert the dataset into a tensor I once again confirm that there is no inf present using <code>torch.isinf(feature_data).any()</code> and once again returns false. However, after the LogSoftmax layer returns a NaN value at the beginning of the 8th epoch,  <code>torch.isinf(feature_data).any()</code> returns true. The only possible explanation I can think of is that this occurs during the normalization process? But there are no random transformations used within the dataset, just a simple normalization utilizing the maximum and minimum values.</p>
<p>I’m also not sure if this helps, but for some reason the training loss progressively explodes over the course of 5 epochs, then suddenly lowers itself back to a reasonable value. It starts off at approximately 2-3, then exponentially increases into the millions by the 5th epoch, only to return to a value of approximately 1 on the 6th epoch. Even stranger is that the validation loss remains relatively stable at approximately 1-3 for the duration of training, and the validation accuracy still increases over time.</p>
<p>Any help would be greatly appreciated. Thanks!</p>

Lachlan_Burne: <p>Fixed my own problem. In case anyone else runs into the same issue, the problem was my custom dataset class. I mistakenly believed that python created copies of variables by default, e.g.<br>
<code>a = [5, 5, 5], b = a </code><br>
would create two distinct variables, when in actuality b is simply a reference of a. This meant that my normalization code in the dataset class was editing the values of my data every time it was passed through the dataset class which eventually lead to inf values appearing after a set number of epochs.</p>
<p>My solution was to the ndarray.copy() command to avoid the original data being edited.</p>
Decomposing a tensor

MRRP: <p>Hi Pytorch Community<br>
I want to decompose a tensor into some non_overlapping tensors. Let me show you an example of what i want to do. Assume that i have a tensor which it’s elements are only 1 like:</p>
<blockquote>
<p>a = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]</p>
</blockquote>
<p>Now i want to decompose it into a few non_overlapping tensors. here for simplicity we decompose them randomly into two tensors called b and c which their elements are only 1 and 0 as below:</p>
<blockquote>
<p>b = [0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1]<br>
c = [1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0]</p>
</blockquote>
<p>Do you have any suggestion on how to do it properly?<br>
Thanks</p>

gphilip: <p>Your question is not clear enough for me to understand. What exactly do you mean by “decomposing into a few non-overlapping tensors”? How many is “a few”? Is it always two? If not, how many?</p>
<p>Could describe what you expect the output to be like, when the input vector is:</p>
<pre><code class="lang-auto">a = [10, -5, 7, 8, 12, -15, 65, -1001, 78, 4]
</code></pre>

ptrblck: <p>Given your example, you might be looking for <code>scatter_</code>:</p>
<pre><code class="lang-python">a = torch.tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float32).unsqueeze(0)
b = torch.zeros(2, a.size(1))
idx = torch.randint(0, 2, (1, a.size(1)))
b.scatter_(0, idx, a)
print(b)
&gt; tensor([[0., 0., 1., 1., 0., 0., 1., 1., 0., 0., 1.],
          [1., 1., 0., 0., 1., 1., 0., 0., 1., 1., 0.]])
</code></pre>

MRRP: <p>Thank you so much for replying.<br>
That is exactly what i was looking for.</p>
Load several rows of a large embedding matrix from disk

supervan: <p>Hi, I have a large embedding matrix saved on disk, I wonder if Pytorch supports loading a submatrix from the large embedding matrix? Thanks.</p>

pascal_notsawo: <p>How is it stored? With what (as a library)?<br>
Pytorch? If yes in some cases some store directly the instance of the model, others the parameters …</p>
<p>More precision please.</p>
Download model .pth files saved in Google Drive to a different environment

JohnPolo: <p>I used torch.save() with the model and the model weights to save a .pth for each object. I have those .pth files saved in a Google Drive. I want to read those files in for another environment. How should I do this?</p>
<p>I’m trying to use PyDrive2 to download the files to another environment. This is my code for downloading from the Google Drive where the files are saved:</p>
<pre><code class="lang-auto">from pydrive2.auth import GoogleAuth
from pydrive2.drive import GoogleDrive

gauth = GoogleAuth()
gauth.LocalWebserverAuth()

drive = GoogleDrive(gauth)

model = drive.CreateFile({'id': '1MbDv8hwtmWS_oV90El_4C9IKypG1yGLS'})
model.GetContentFile('MaskInstanceModel.pth') # Download file as 'MaskInstanceModel.pth'.

modelweights = drive.CreateFile({'id': '1EKaPhcaMmCQVhzCaH0jlaqWscf-JbSKO'})
model.GetContentFile('MaskModelParams.pth') 
</code></pre>
<p>The files seem to download fine. They’re the right size and there is no message or warning about an error.</p>
<p>In the script that uses the model and weights, this is the snippet of code:</p>
<pre><code class="lang-auto">import torch
import torch.utils.data
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
from torchvision.io import read_image
from torchvision.transforms.functional import convert_image_dtype
import torchvision.transforms.functional as F

...
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

# Load the model from the saved file:
model = torch.load('MaskInstanceModel.pth', map_location=device)

model.load_state_dict(torch.load('MaskModelParams.pth', map_location=device))
</code></pre>
<p>But I get this error:&gt;</p>
<blockquote>
<p>bash-4.3$ python3 model.py<br>
Traceback (most recent call last):<br>
File “/home/john/imseg/src/model.py”, line 92, in <br>
model.load_state_dict(torch.load(‘MaskModelParams.pth’, map_location=device))<br>
File “/home/john/.local/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1379, in load_state_dict<br>
state_dict = state_dict.copy()<br>
File “/home/john/.local/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1130, in <strong>getattr</strong><br>
raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(<br>
AttributeError: ‘MaskRCNN’ object has no attribute ‘copy’</p>
</blockquote>
<p>I guess this has something to do with how PyDrive downloaded the file from Google Drive. I’m not married to PyDrive, it just seemed like the easiest way to download files from a Google Drive. Is there a better way to move the model files around?</p>

ptrblck: <p>Based on the error message it seems that</p>
<pre><code class="lang-python">torch.load('MaskModelParams.pth', map_location=device)
</code></pre>
<p>returns a <code>MaskRCNN</code> object, not the <code>state_dict</code> and would thus fail, so you could double check that <code>MaskModelParams.pth</code> is indeed containing the <code>state_dict</code>.</p>
Nearest Neighbor Imputation

mlizhardy: <p>I have images which have empty values, I want to fill the empty values with nearest neighbor.  What is an efficient method for this sort of imputation?</p>
<p>The images will already be pytorch tensors on GPU, and sending to cpu and back to gpu is not an option, so I cannot use numpy nor scikit options.</p>

JosueCom: <p>How do you define a pixel that is empty? Do those pixels have an specific value?</p>

mlizhardy: <p>yes they have a specific value, could be any value you want (including nan). without loss of generality consider it to be [0,0,0]</p>

googlebot: <p>efficient method to fix isolated holes would be to prepare a substitute image, and do conditional blending with torch.where(hole, sub, original)</p>
<p>how to prepare such a substitute is another matter, using a smoothed image would be the simplest approach perhaps.</p>

mlizhardy: <p>creating that substitute image is essentially the problem of imputation in and of itself. I know how to place the pixel values once I have them, but generating them is the problem of imputation. I’m not sure what you mean by “using a smoothed image” because when there are empty pixels, there are no values to “smooth”</p>

googlebot: <p>if you’re talking about bigger regions missing, the problem is not cheaply solveable with generic math operators (built-in kernels), look for noise reduction algorithm implementations. for small holes, using something like gaussian blur may suffice</p>
How to load this data in dataoader, which is in pickle file?

hema_rathore: <p>I have a pickle file having my training data in format as given below:</p>
<pre><code class="lang-auto">'Ses01M_script03_1_F011': {'audio_data': array([ 97, 173, 139, ...,  74,  22,  56], dtype=int16),
  'emo_label': 0,
  'gen_label': 0,
  'transcript': 'mmm',
  'features': array([[-0.72736262,  0.43802561,  2.14846694, ..., -2.69805505,
          -2.6575799 , -2.62012389],
         [ 1.64373051,  1.45195623,  2.5100556 , ..., -3.49289518,
          -3.33669469, -3.24162892],
         [ 0.06487407,  2.2071517 ,  2.77620356, ..., -4.97157829,
          -5.32930986, -9.44662813],
         ...,
         [ 1.9662066 ,  2.45063788,  2.64235023, ..., -5.53699691,
          -5.63285812, -8.2494293 ],
         [ 2.46171323,  2.83857374,  2.91717236, ..., -5.23233626,
          -6.12257485, -6.33155109],
         [ 2.27569999,  2.81201242,  2.78668791, ..., -3.50327683,
          -3.70423857, -3.75825296]])},
 ...}
</code></pre>
<p>so basically its a dictionary. I want to load this information using dataloader , For that i want to create a speechdatagenrator class similar to given below:</p>
<pre><code class="lang-auto">

import numpy as np
import torch
from utils import utils_wav

class SpeechDataGenerator():
    """Speech dataset."""

    def __init__(self, manifest, mode):
        """
        Read the textfile and get the paths
        """
        self.mode=mode
        self.audio_links = [line.rstrip('\n').split(' ')[0] for line in open(manifest)] # here manifest is the location where training data is saved.
        self.emo_labels = [int(line.rstrip('\n').split(' ')[1]) for line in open(manifest)]
        self.gen_labels = [int(line.rstrip('\n').split(' ')[2]) for line in open(manifest)]
        

    def __len__(self):
        return len(self.audio_links)

    def __getitem__(self, idx):
        audio_link =self.audio_links[idx]
        class_id = self.emo_labels[idx]
        #lang_label=lang_id[self.audio_links[idx].split('/')[-2]]
        audio_data = utils_wav.load_data_wav(audio_link,min_dur_sec=10)
        sample = {'raw_speech': torch.from_numpy(np.ascontiguousarray(audio_data)), 'labels': torch.from_numpy(np.ascontiguousarray(class_id))}
        return sample
        
    

</code></pre>
<p>Can any on please tell me how to modify this class to read this pickle file, inside which there is a dictionary to load features , emo_labels and gen_labels.</p>
<p>seeking guidance.<br>
good day all.</p>
Torch.nn missing

madbats: <p>I’ve been trying to compile a programme using the pytorch libraries for Android. However when i compile from the source code, only the JIT module’s methods are compiled.<br>
How to i get the torch::nn Module’s methods ?</p>

guangy10: <p>Hey, could you provide the log or the error message so that we know where to take a look?<br>
cc: <a class="mention" href="/u/linbin">@Linbin</a> <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a></p>

madbats: <p>When I compile my programme i point to the folder containing the libraries and include torch, torch_cpu and c10.<br>
But i always get the following errors</p>
<pre><code class="lang-auto">ld: error: undefined symbol: torch::cuda::is_available()
ld: error: undefined symbol: torch::nn::Module::named_buffers(bool) const
ld: error: undefined symbol: torch::nn::Module::named_parameters(bool) const
ld: error: undefined symbol: torch::nn::Module::eval()
ld: error: undefined symbol: torch::nn::EmbeddingImpl::forward(at::Tensor const&amp;)
ld: error: undefined symbol: torch::nn::detail::RNNImplBase&lt;torch::nn::LSTMImpl&gt;::flatten_parameters()
ld: error: undefined symbol: torch::nn::LSTMImpl::forward(at::Tensor const&amp;,c10::optional&lt;std::__ndk1::tuple&lt;at::Tensor, at::Tensor&gt; &gt;)
ld: error: undefined symbol: typeinfo for torch::nn::Module
</code></pre>
<p>I’ve been told it’s because i’ve only compiled the JIT version of pytorch. But looking thought the build_android.sh file on the git repo, I can’t see any options that would not build the JIT version.</p>

madbats: <p>It appears that by turning <code>NO_API=OFF</code> and  <code>BUILD_MOBILE_AUTOGRAD=ON</code> in the root CMake file,builds the api.</p>
Multidim Indexing

Ilia_Karmanov: <p>I’m having a hard time adapting my code to work across a batch example. Previously I had</p>
<pre><code class="lang-auto">idx.shape == torch.Size([50, 4])
points.shape == torch.Size([400, 3])
# Index
points[idx].shape == torch.Size([50, 4, 3])
</code></pre>
<p>So I would sample four, 3D points, from a set of 400 points and have this repeated 50 times. This was just for one observation. Now I would like to run this for each observation in a batch of 256, thus:</p>
<pre><code class="lang-auto">idx.shape == torch.Size([256, 50, 4])
points.shape == torch.Size([256, 400, 3])

# Doesn't work:
points[idc].shape
# Doesn't work:
torch.gather(points, 1, idx).shape

# Works but I want to avoid the loop
torch.stack([points[i][idx[i]] for i in range(len(points))]).shape == torch.Size([256, 50, 4, 3])
</code></pre>
<p>Edit:</p>
<p>With a bit of messing around I have a solution but it doesn’t look pretty:</p>
<pre><code class="lang-auto">idx.shape == torch.Size([256, 50, 4])
points.shape == torch.Size([256, 400, 3])

points = points.unsqueeze(1).repeat(1, idx.size(1),,1,1)
idx= idx.unsqueeze(-1).repeat(1,1,1, points.size(3))

# torch.Size([256, 50, 400, 3])
print(m.shape)

# torch.Size([256, 50, 4, 3])
print(ids.shape)

torch.Size([256, 50, 4, 3]
print(m.gather(2, ids).shape)
</code></pre>
<p>It also seems to run slower:</p>
<pre><code class="lang-auto"># 3.66 ms ± 6.41 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
A = torch.stack([pt_coords[i][idc[i]] for i in range(len(pt_coords))])

# 8.05 ms ± 10.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
coords_rep = pt_coords.unsqueeze(1).repeat(1, idc.size(1), 1, 1)
B = coords_rep.gather(2, idc.unsqueeze(-1).repeat(1,1,1, coords_rep.size(3)))
</code></pre>
Why can `torch.tensor` be an index value for slicing?

marsggbo: <pre><code class="lang-python">a = [4,5,6,3,5]
b = torch.tensor([2])
print(a[:b])
&gt;&gt;&gt;[4,5]
</code></pre>
<p><code>b</code> is <code>torch.tensor</code>, I wonder why it can be used as an index value for slicing, performing like an <code>int</code> type?</p>

albanD: <p>Hi,</p>
<p>Yes integer type Tensors can be used this way.</p>
<p>I see that you added the autograd label. It is not a problem for autograd because integer type Tensor can never require gradients.</p>
Loss.backward() mismatch

PyNoob: <p>Hello there!</p>
<p>I am really new in PyTorch. When training the discriminator Network for a GAN I am running with an error in the shapes of the gradient generated for the backpropagation.</p>
<p>“<strong>RuntimeError: Function AddmmBackward returned an invalid gradient at index 1 - got [1, 16] but expected shape compatible with [1, 5520]</strong>”</p>
<p>My inputs are images of the following <em>(batch_size, channels, height, width)</em>: [1, 3, 270, 387]</p>
<p>This is the classification model defined for the discrimination task:</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
  def __init__(self, im_chan = 3, hidden_dim = 32):
    super().__init__()

    self.model = nn.Sequential(

        nn.Conv2d(3, 32, kernel_size = 3, stride = 2),
        nn.LeakyReLU(0.2),
        nn.BatchNorm2d(32),

        nn.Conv2d(32, 64, kernel_size = 3, stride = 2),
        nn.LeakyReLU(0.2),
        nn.BatchNorm2d(64),

        nn.Conv2d(64, 32, kernel_size = 3, stride = 2),
        nn.LeakyReLU(0.2),

        nn.Conv2d(32, 16, kernel_size = 3, stride = 2),
        nn.LeakyReLU(0.02),

        nn.Flatten(),
        nn.Linear(16, 1),
        nn.Sigmoid()

    )
    self.loss_function = nn.BCELoss()

    self.optimiser = torch.optim.Adam(self.parameters(), lr = 0.0001)

    self.counter = 0

    self.progress = []

def forward(self, inputs):

    return self.model(inputs)

def train(self, inputs, targets):

    outputs = self.forward(inputs)

    print(outputs.shape)

    targets = targets.unsqueeze(0)

    print(targets.shape)

    loss = self.loss_function(outputs, targets)
    print('Loss calculated!')
    self.counter += 1

    if (self.counter % 10 == 0):
      self.progress.append(loss.item())

    if (self.counter % 1000 == 0):
      print('counter = ', self.counter)

    self.optimiser.zero_grad()
    loss.backward()
    self.optimiser.step()
</code></pre>
<p>When training the discriminator, the target is defined as a torch.cuda.FloatTensor([1.0]) for real images and torch.cuda.FloatTensor([0.0]) for the fake ones.</p>
<p>I am stuck here. Any help would be much appreciated!</p>

albanD: <p>Hi,</p>
<p>Can you share a full script that we can run to reproduce the issue?</p>
Is there a way to include model.training control flow in traced model?

Alexander_Soare: <p>When tracing a model where some of the logic depends on <code>self.training</code>, we can only have one of the branches. Here’s a minimal example:</p>
<pre><code class="lang-auto">from torch import nn
import torch.fx

tracer = torch.fx.Tracer()

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.path = False

    def forward(self, x):
        if self.training:
            return x + 1
        else:
            return x


module = MyModule()
module.eval()
graph = tracer.trace(module)
print(graph)
graph_module = torch.fx.GraphModule(module, graph)

graph_module.train()  # This has no effect on the outcome
print(graph_module(torch.zeros(1)))
</code></pre>
<p>As you can see, <code>graph_module.train()</code> does nothing, because the tracing operation completely ignored the <code>self.training</code> branch.</p>
<p>A more realistic example is when we have some sort of dropout we only want to switch on during training.</p>
<p>Any way to somehow build both branches of the control flow into the resulting graph module?</p>
How to implement this calculation without large memory usage?

Wang_Zhuo: <p>I have two tensors, X and W, and their shapes are (n, m) and (m, l), respectively. I want to get a new tensor R that<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22f0d7eaf4999377bbb98a34db9940a76954b25a.png" data-download-href="https://discuss.pytorch.org/uploads/default/22f0d7eaf4999377bbb98a34db9940a76954b25a" title="WX20210812-211514@2x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22f0d7eaf4999377bbb98a34db9940a76954b25a_2_345x95.png" alt="WX20210812-211514@2x" data-base62-sha1="4Z6eNpNWWS5t1qSuJjhgnQGueWC" width="345" height="95" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22f0d7eaf4999377bbb98a34db9940a76954b25a_2_345x95.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22f0d7eaf4999377bbb98a34db9940a76954b25a_2_517x142.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22f0d7eaf4999377bbb98a34db9940a76954b25a_2_690x190.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20210812-211514@2x</span><span class="informations">3252×902 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
. My current solution is using <code>torch.prod(1 - X.unsqueeze(-1) * W, dim=1)</code>. However, it will generate a tensor in the shape of (n, m, l) to store intermediate results. This (n, m, l) tensor could be too large to store in my GPU. Is there any way to avoid this large intermediate result tensor and can still keep the speed? For example, the calculation to get R is very similar to matrix multiplication, but the matrix multiplication costs much less memory.</p>
Autocast Slows Down the Code

Osman_Yasar: <p>Hi,<br>
I have RTX 3070. Somehow using autocast slows down my code.</p>
<p>torch.version.cuda prints 11.1, torch.backends.cudnn.version() prints 8005 and my PyTorch version is 1.9.0. I’m using Ubuntu 20.04 with Kernel 5.11.0-25-generic.</p>
<p>That’s the code I’ve been using:</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for epoch in range(10):

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):

        inputs, labels = data

      
        optimizer.zero_grad()
        
        
        with torch.cuda.amp.autocast(): 
             outputs = net(inputs)
             oss = criterion(outputs, labels) 

         scaler.scale(loss).backward() 
         scaler.step(optimizer)
         scaler.update() 

end.record()

torch.cuda.synchronize()

print(start.elapsed_time(end))
</code></pre>

ptrblck: <p>I don’t know which model you are using, but you could try to either use the nightly binaries with cuDNN8.2.2 or build PyTorch from source with the latest cudnn version to check the latest performance.<br>
Also, <code>torch.backends.cudnn.benchmark = True</code> could help in case you are working with static input shapes (or a low variance in the shape).</p>

Osman_Yasar: <p>Well I solved it by increasing the out channels of the conv2d. And it worked that way. Thanks for help.</p>
How to increase loss

hix: <p>Hello</p>
<p>I am trying to reconstruct Sinogram.</p>
<p>However, the loss decreases so quickly that the psnr converges at 31 and no longer increases.</p>
<p>So I want to improve psnr by slowing down the loss.</p>
<p>How can I increase loss and is my method correct?</p>

ptrblck: <p>Based on your description it seems you are calculating a specific loss while caring about the PSNR, which is not represented in your loss.<br>
I don’t know if slowing down the convergence would increase the PSNR, so could you explain this assumption a bit more and why this would be the case?<br>
Without knowing details about your use case I would guess the better approach would be to track some kind of information of the PSNR in the loss.</p>

hix: <p>Thank you for reply I concentrate my psnr code</p>
<p>Thank you ptrblck</p>
How can i convert model's parameter data type to Double?

yunjey: <p>Default data type of parameters is Float.<br>
I want to convert it to Double.<br>
How can i do this?</p>

albanD: <p>You can convert your model to double by doing <code>model.double()</code>.<br>
Note that after this, you will need your input to be <code>DoubleTensor</code>.</p>

yunjey: <p>Thanks very much! It’s very simple.</p>

Marwan_Elghitany: <p>Convert Your Data to float instead … as it’s <a href="https://discuss.pytorch.org/t/problems-with-target-arrays-of-int-int32-types-in-loss-functions/140/3">very dangerous to convert model type to Double …</a> as GPUs are not good at double precision computation<br>
Try:</p>
<pre><code class="lang-python">X_train = torch.FloatTensor(X_train)
</code></pre>
<p>instead</p>