akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch Vanilla and Multi Layer Autoencoder Network

Uma: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66b386bedb895a366eec0068e7e69e07bd584c1a.png" data-download-href="https://discuss.pytorch.org/uploads/default/66b386bedb895a366eec0068e7e69e07bd584c1a" title="vanillaLoss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66b386bedb895a366eec0068e7e69e07bd584c1a_2_690x482.png" alt="vanillaLoss" data-base62-sha1="eExkNl57lANWBjNiLZfumCvEFBg" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66b386bedb895a366eec0068e7e69e07bd584c1a_2_690x482.png, https://discuss.pytorch.org/uploads/default/original/3X/6/6/66b386bedb895a366eec0068e7e69e07bd584c1a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/66b386bedb895a366eec0068e7e69e07bd584c1a.png 2x" data-dominant-color="FEFDFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">vanillaLoss</span><span class="informations">1000×700 30.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I got the above figure for the reconstruction loss of vanilla autoencoder.  I used the learning rate of IE-2 and Adam optimization. Can we expect a graph like this for the reconstruction loss? But when I change the learning rate I got a normal loss function which decreases the loss at each iteration.</p>
AIQC 2.0.0: preprocessing, tuning, metrics for PyTorch!

HashRocketSyntax: <p>AIQC is a framework for rapid &amp; reproducible deep learning that empowers open science.</p>
<p>Support for PyTorch was added in v2.0.0! ReadTheDocs includes examples of classification (binary, multi) and regression in PyTorch.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/aiqc/AIQC">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/aiqc/AIQC" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/17c6c52dd74a3ad52f9ff60ca25787affa96de1260ea0dd581c95392f0400eb0/aiqc/AIQC" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/aiqc/AIQC" target="_blank" rel="noopener nofollow ugc">GitHub - aiqc/AIQC: End-to-end deep learning on your desktop or server.</a></h3>

  <p>End-to-end deep learning on your desktop or server. - GitHub - aiqc/AIQC: End-to-end deep learning on your desktop or server.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff" title="Screen Shot 2021-04-29 at 7.24.38 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff_2_690x388.png" alt="Screen Shot 2021-04-29 at 7.24.38 AM" data-base62-sha1="memTjpKeIdmRCVKLa0CaB2cGehV" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bcfa5f0c780c7714ac55e771ea44fb39fec65ff.png 2x" data-dominant-color="242422"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-04-29 at 7.24.38 AM</span><span class="informations">1036×583 95.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<div class="youtube-onebox lazy-video-container" data-video-id="cN7d8c-3Vxc" data-video-title=" - YouTube" data-video-start-time="" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=cN7d8c-3Vxc" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="" title=" - YouTube" width="" height="">
  </a>
</div>

<p><code>pip install aiqc</code></p>
F1 sample for multi-label classification

ireneb612: <p>def train(net,data_loader,optimizer,cost_function, device=device):</p>
<p>samples = 0.</p>
<p>cumulative_loss = 0.</p>
<p><span class="hashtag">#cumulative_accuracy</span> = 0.</p>
<p>net.train() # Strictly needed if network contains layers which has different behaviours between train and test</p>
<p>for batch_idx, (inputs, targets) in enumerate(data_loader):</p>
<pre><code># Load data into GPU

inputs = inputs.to(device)

targets = targets.to(device)

  

# Forward pass

outputs = net(inputs)

# Apply the loss

targets = targets                              #.unsqueeze(1)           # attenzione

loss = cost_function(outputs,targets)

  

# Backward pass

loss.backward()


# Update parameters

optimizer.step()

# Resets the gradients

optimizer.zero_grad()

f1 = f1_score(targets, outputs &gt; 0.5, average="samples")

samples+=inputs.shape[0]         # batches

cumulative_loss += loss.item()


return cumulative_loss/samples,  f1                   
</code></pre>
<p>Does this make sense?</p>
Loss exploding after resume training SAC

deepfailure: <p>Hello,</p>
<p>I am training a SAC agent and want to resume training. However, the loss explodes after resuming training.</p>
<p>In SAC we have as trainable parameters: an actor (also called policy), a critic, and a parameter alpha.</p>
<p>Moreover, if I want to resume training I have to reload the experience replay.</p>
<p>I think the issue is in the alpha parameter as the loss for alpha goes from near 0 to 40 K, do I load it correctly?</p>
<pre><code class="lang-auto">def load_model(actor_path, critic_path, optimizer_actor_path, optimizer_critic_path, optimizer_alpha_path):

  policy = torch.load(actor_path)
  self.alpha = policy['alpha'].detach().item()
  self.log_alpha = torch.tensor([policy['log_alpha'].detach().item()], requires_grad=True, device=self.device)
  self.alpha_optim = Adam([self.log_alpha], lr=self.lr)

  self.policy.load_state_dict(policy['model_state_dict'])
  self.policy.train()
  self.critic.load_state_dict(torch.load(critic_path))
  self.critic.train()

  self.policy_optim.load_state_dict(torch.load(optimizer_actor_path))
  self.critic_optim.load_state_dict(torch.load(optimizer_critic_path))
  self.alpha_optim.load_state_dict(torch.load(optimizer_alpha_path))
</code></pre>
<p>Additional note: when testing the policy and critic it seems to output the same values as on the previous training, the problem has to be in the alpha or optimizers.</p>
Batch inference of softmax does not sum to 1

Rasoul: <p>I am working with REINFORCE algorithm with PyTorch. I noticed that the batch inference/predictions of my simple network with Softmax doesn’t sum to 1 (not even close to 1). I am attaching a minimum working code so that you can reproduce it. What am I missing here?</p>
<pre><code class="lang-auto">import numpy as np
import torch

obs_size = 9
HIDDEN_SIZE = 9
n_actions = 2

np.random.seed(0)

model = torch.nn.Sequential(
            torch.nn.Linear(obs_size, HIDDEN_SIZE),
            torch.nn.ReLU(),
            torch.nn.Linear(HIDDEN_SIZE, n_actions),
            torch.nn.Softmax(dim=0)
        )

state_transitions = np.random.rand(3, obs_size)

state_batch = torch.Tensor([s for s in state_transitions])
pred_batch = model(state_batch)  # WRONG PREDICTIONS!
print('wrong predictions:\n', *pred_batch.detach().numpy())
# [0.34072137 0.34721774] [0.30972624 0.30191955] [0.3495524 0.3508627]
# DOES NOT SUM TO 1 !!!

pred_batch = [model(s).detach().numpy() for s in state_batch]  # CORRECT PREDICTIONS
print('correct predictions:\n', *pred_batch)
# [0.5955179  0.40448207] [0.6574412  0.34255883] [0.624833   0.37516695]
# DOES SUM TO 1 AS EXPECTED
</code></pre>

rahulvigneswaran: <p><strong>In case 1:</strong> <code>model(state_batch)</code></p>
<ul>
<li>The output dimension of <code>torch.nn.Linear</code> is <code>(3,2)</code> - 2 Dimensional</li>
</ul>
<p><strong>In case 2:</strong> <code>[model(s).detach().numpy() for s in state_batch]</code></p>
<ul>
<li>The output dimension of <code>torch.nn.Linear</code> is just <code>(2)</code> - 1 Dimensional</li>
</ul>
<p>The <code>torch.nn.Softmax(dim=0)</code> does softmax at <code>dim=0</code> which will work properly for Case 2 as there is only 1 dimension but in case of Case 1 there are 2 dimensions and that is the root cause for the issue.</p>
<blockquote>
<p>Changing your code to <code>torch.nn.Softmax(dim=1)</code> should fix this issue for Case 1.</p>
</blockquote>
<p>And if you still want the new changed code to work for Case 2, then do the following,<br>
<code>pred_batch = [model(s.unsqueeze(dim=0).detach().numpy() for s in state_batch]</code></p>
Build C++ only cpp libtorch with version lower than 1.2.0a0

thymbahutymba: <p>Hi guys, I’m trying to build libtorch&lt;=1.2.0a0, the version requirement is due to the <code>torch::jit::load</code> that has to return the <code>shared_ptr&lt;Model&gt;</code> instead of just the <code>Model</code>.<br>
However I’m not able to build it due to the following error:</p>
<pre><code class="lang-auto">/tmp/pytorch/third_party/foxi/foxi/onnxifi_dummy.c:99:60: error: conflicting types for 'onnxInitGraph'
   99 | ONNXIFI_PUBLIC ONNXIFI_CHECK_RESULT onnxStatus ONNXIFI_ABI onnxInitGraph(
</code></pre>
<p>I’m building with the following command:</p>
<pre><code class="lang-auto">cmake .. -DUSE_CUDA=OFF -DBUILD_TEST=OFF -DPYTHON_EXECUTABLE=$(which python2) -DBUILD_PYTHON=OFF -DBUILD_SHARED_LIBS=ON
make -j8
</code></pre>
<p>I tried the following version: 1.2.0a0, 1.2.0, 1.1.0, 1.0.1, 1.0.0.<br>
At the end of the day I would like to have libtorch.so libc10.so libtorch_cpu.so. Any help?</p>
Difference between val loss and train loss

887574002: <p>Hi, I am working on a Segmentation task with Unet. I am using a cucle step learning schedule with cycle length =50 and CrossEntropy Loss.<br>
I am using IOU as a metric to evaluate performnace.</p>
<p>I obtained the following learning curve that is not a sign of overfitting in my opinion.   But I can not understand the reason of difference between Train and Val loss. and if there is a problem in training how can I fix it.</p>
<p>Another question is that the leaning curve shows training is not finished yet and I should leave it to be trained for longer epochs, right?</p>
<p>My training set has 740 images and valset has 267. It is cityscapes dataset. Since the dataset is really large I chose a smaller subset of training and validation set for training.</p>
<p>thank you in advance.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88fd373ea0fe048b2cf7df62aff7e350134a030b.png" data-download-href="https://discuss.pytorch.org/uploads/default/88fd373ea0fe048b2cf7df62aff7e350134a030b" title="sgld_exp_107_cost"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fd373ea0fe048b2cf7df62aff7e350134a030b_2_690x325.png" alt="sgld_exp_107_cost" data-base62-sha1="jxRrNnpho95O87pjAbMOUkH4frR" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fd373ea0fe048b2cf7df62aff7e350134a030b_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fd373ea0fe048b2cf7df62aff7e350134a030b_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88fd373ea0fe048b2cf7df62aff7e350134a030b.png 2x" data-dominant-color="F6F2EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sgld_exp_107_cost</span><span class="informations">1136×536 50 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
CuDNN execution fails on moving parameters to GPU

sambbhavgarg: <p>Hey guys<br>
Im facing the following error on moving my model parameters and state to the GPU -</p>
<blockquote>
<p>torch.backends.cudnn.CuDNNError: 8: b’CUDNN_STATUS_EXECUTION_FAILED’</p>
</blockquote>
<p>(can produce stacktrace if req.) with the following configuration -</p>
<pre><code class="lang-auto">cuda90                    1.0                  h4c72538_0    peterjc123
cudatoolkit               9.0                           1
pytorch                   0.3.1           py36_cuda90_cudnn7he774522_2  [cuda90]  peterjc123
</code></pre>
<p>However, with -</p>
<pre><code class="lang-auto">cudatoolkit               8.0                           4
pytorch                   0.3.1           py36_cuda80_cudnn6he774522_2    peterjc123
</code></pre>
<p>the execution runs with just a minor bump in the road. (incompatible compilation warnings)</p>
<p>Following are my hardware specs -</p>
<blockquote>
<p>Card: RTX 2080S<br>
CUDA: 10.1, CuDNN 7.6.5</p>
</blockquote>
<p>Any help would be appreciated. Thanks!</p>
<p>PS This is my first post on the Torch forum. It would be fine if minor formatting suggestions are commented too.</p>

ptrblck: <p>PyTorch <code>0.3.1</code> is quite old by now, so could you update to the latest stable release (<code>1.8.1</code>) and rerun your script, please?</p>

sambbhavgarg: <p>Thanks for your response, <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
The reason I’m holding back on upgrading the Pytorch versions is that the same piece of code works marvellously on my Nvidia GT 730 (which I know is an old fellow). Could you help me out with what this error means? or where I need to look?</p>
<p>Also, other developers who updated to a later version of Pytorch complained of the results going haywire on the same code. I wanted to create an initial version of the thing im trying to build before resolving versioning issues.</p>
<p>Thanks again for your response!</p>

ptrblck: <p>Since you are using a PyTorch, CUDA, and cudnn release from early 2018, you might be facing already fixed issues.<br>
In any case, if you want to use this old setup, you could disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and see, if your script would be working.</p>
Using LBFGS, CUDA and "custom" parameters

ChrisHaddi: <p>Hi,<br>
I ran into the following issue: I have a simple neural network class with one custom variable ‘a’:</p>
<pre><code class="lang-auto">class NeuralNetwork(torch.nn.Module):
    def __init__(self, a):
        super().__init__()
        
        self.a = a
        self.fc1 = torch.nn.Linear(3,5)

    
    def forward(self, x):
        x = self.a * self.fc1(x)
        return x
</code></pre>
<p>I am trying to do the following: create an instance of NeuralNetwork, put it to cuda and train it with the LBFGS optimizer like follows:</p>
<pre><code class="lang-auto">a = torch.tensor(1.0, requires_grad = True)

device = torch.device("cuda:0")

net = NeuralNetwork(a)
net.to(device)

optimizer = torch.optim.LBFGS([net.a] + list(net.parameters()))
# optimizer = torch.optim.Adam([net.a] + list(net.parameters()))

xIn = torch.tensor((1.0,2.0,3.0), device = device)

def closure():
    optimizer.zero_grad()
    xOut = net(xIn)
    loss = xOut.sum() + a
    loss.backward()
    return loss

optimizer.step(closure)
</code></pre>
<p>The variable ‘a’ is also a parameter that should be changed by my optimizer, hence it is included in the parameters in the initialization of ‘optimizer’. When running the whole code, I run into the following error:</p>
<blockquote>
<p>RuntimeError: All input tensors must be on the same device. Received cpu and cuda:0</p>
</blockquote>
<p>in the last line.<br>
I thought that this might happen because ‘a’ is for some reason not on the GPU, as when leaving ‘a’ out of the optimizer parameters, the error is gone. But then again, when swapping LBFGS for Adam (in the code above comment the LBFGS-line and uncomment the Adam-line), but keeping ‘a’ in the optimizer parameters, the error is also gone.<br>
Does anyone have an idea what might be the problem here and how to solve it?</p>
<p>Thank you!<br>
Chris</p>

googlebot: <p>yes, it is not auto-moved, as it is not considered a part of the model, just because you’ve assigned it as an object attribute (without nn.Parameter wrapper). adam probably mitigates this error by having more robust code.</p>

ChrisHaddi: <p>Hi, I now changed the definition of a to</p>
<pre><code class="lang-auto">a = torch.nn.Parameter(torch.tensor(1.0), requires_grad = True)
</code></pre>
<p>and now it seems to work. Thank you!</p>
What Data Should i Validate my Model Against while Training?

newbieee: <p>Hello Guys,</p>
<p>i want to research the impact of diffrent Data-Augmentation-Techniques for STT-Models. For this im Using a Pretrained (Wav2Vec2) Model trained on CommonVoice. If i want to test the impact of augmenting the Data by 1) adding a completely new dataset 2) adverserial attack 3) maybe trying TTS-synthesized data</p>
<p>What should i validate the Modell against while Training? For my final comparison i want to use the Test-Split of CommonVoice, but can i just reuse that Dataset for my Validation Step (while Training) or should i use a split what ever im training on?</p>
<p>Since English is not my native language, I’ll try to explain my problem via a graphic.:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c380259da8c97c49bd5e2c319dfc4301f4a926ab.png" alt="c380259da8c97c49bd5e2c319dfc4301f4a926ab" data-base62-sha1="rTtEW86hXnAzn4Zhz5y1aSySRBN" width="431" height="412"><br>
I am using a pre-trained Wav2Vec2 model and would like to train it further with “Augmented Data”. So for example by taking either a complete new dataset, by TTS syntesized data or by Adverserial Attack based on CommonVoice training data.</p>
<p>I will compare the augmentation methods later using that of the Common Voice -test split. Now, when I train my models, what should I validate them on during training? Should I just split the new (augmented) training data into 80% training 20% validation, or should I take 100% of the augmented data and validate on Common Voice - test split? Is it problematic during training and final comparison to validate the systems with the same dataset-split (as seen in the graph (see dataset above right)</p>
<p>ty in advcaned</p>
Pytorch Quantization Inference data type

fwz-opt: <p>I am new to pytorch quantization, confused with two points <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><br>
1、pytorch two quantization ways: qat and ptq. Are they use the same operations in framework during inference phase when deploy on mobile device? will they have the same performance?<br>
2、Why activations are quantized to uint8 meanwhile weights quantized to int8. Because int8 weights doesn’t need to subtract the zero-point? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

supriyar: <ol>
<li>yes the operators used during inference for both QAT and PTQ flows will remain the same. On mobile some of the kernels use QNNPACK for inference so the performance may differ on mobile compared to server.</li>
<li>This is due to the requirement of underlying kernels that perform the GEMM operation, i.e. FBGEMM and QNNPACK.</li>
</ol>

fwz-opt: <p>Thanks!!<br>
About the point 2, what’s the data type for acivations and weights in QNNPACK. The code shows that activations are uint8 but weights are void*. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/QNNPACK/blob/master/src/q8dwconv/up8x9-neon.c#L18" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/QNNPACK/blob/master/src/q8dwconv/up8x9-neon.c#L18" target="_blank" rel="noopener nofollow ugc">pytorch/QNNPACK/blob/master/src/q8dwconv/up8x9-neon.c#L18</a></h4>
<pre class="onebox"><code class="lang-c"><ol class="start lines" start="8" style="counter-reset: li-counter 7 ;">
<li>
</li>
<li>#include &lt;arm_neon.h&gt;</li>
<li>
</li>
<li>#include &lt;qnnpack/q8dwconv.h&gt;</li>
<li>
</li>
<li>
</li>
<li>void q8dwconv_ukernel_up8x9__neon(</li>
<li>    size_t channels,</li>
<li>    size_t output_width,</li>
<li>    const uint8_t** input,</li>
<li class="selected">    const void* weights,</li>
<li>    uint8_t* output,</li>
<li>    size_t input_stride,</li>
<li>    size_t output_increment,</li>
<li>    const union qnnp_conv_quantization_params quantization_params[restrict static 1])</li>
<li>{</li>
<li>  const uint8x8_t vkernel_zero_point = vld1_dup_u8((const uint8_t*) &amp;quantization_params-&gt;neon.kernel_zero_point);</li>
<li>  const int32x4_t vmultiplier = vld1q_dup_s32(&amp;quantization_params-&gt;neon.multiplier);</li>
<li>  const int32x4_t vright_shift = vld1q_dup_s32(&amp;quantization_params-&gt;neon.right_shift);</li>
<li>  const int16x8_t voutput_zero_point = vld1q_dup_s16(&amp;quantization_params-&gt;neon.output_zero_point);</li>
<li>  const uint8x8_t voutput_min = vld1_dup_u8(&amp;quantization_params-&gt;neon.output_min);</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Using MaskRCNN for only detection (not instance segmentation)

Lakshay_Sharma: <p>Is there a way to use MaskRCNN for only detection labels and for detection bounding-box prediction? All the tutorials I’ve seen so far seem to require segmentation mask labels as well. My problem only has bounding box labels and only needs bounding box predictions.</p>

KFrank: <p>Hi Lakshay!</p>
<aside class="quote no-group" data-username="Lakshay_Sharma" data-post="1" data-topic="119745" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lakshay_sharma/48/22347_2.png" class="avatar"> Lakshay_Sharma:</div>
<blockquote>
<p>Is there a way to use MaskRCNN for only detection labels and for detection bounding-box prediction?</p>
</blockquote>
</aside>
<p>Not that I’m aware of (without modifying it significantly).</p>
<p>Consider using <a href="https://pytorch.org/vision/0.8/models.html#torchvision.models.detection.fasterrcnn_resnet50_fpn" rel="noopener nofollow ugc">Faster R-CNN</a> which is designed for detection rather<br>
than for instance segmentation.  (And it is pretty close to being the<br>
bounding-box piece of Mask R-CNN.)</p>
<p>Best.</p>
<p>K. Frank</p>
Custom node parsing

Arkady: <p>Hi,<br>
We currently develop a Glow backend for CEVA target DSP that supports AOT (offline) compilation mode.<br>
We would like to be able to compile a model containing layers which glow does not currently support, for example, the Loop layer, so that we could offload them to the DSP.<br>
As far as I understand, the model parsing is done in the frontend.<br>
Is there a way to add custom model parsing for a specific layer by the backend, such that it’d be automatically called during the parsing stage, without modifying the frontend code?<br>
Thanks.</p>

jfix: <p>Hi <a class="mention" href="/u/arkady">@Arkady</a> – we have currently have an <code>ExternalFunctionCall</code> node which is capable of representing a node that only a specific backend understands how to handle/execute. There is also a more extensive/feature rich version of this same idea in <a href="https://github.com/pytorch/glow/pull/5208" rel="noopener nofollow ugc">this PR</a> but not sure when it will land.</p>

Arkady: <p>Thanks for the reply.<br>
However, it looks like both the ExternalFunctionCall node and the enhancement of this feature, as described in the link you mentioned, add the possibility to use custom code during execution.<br>
But how can I trigger custom layer parsing during the parsing of the model?</p>
<p>For example:<br>
I’m trying to compile a bundle from a tensorflow-lite model which contains the “SpaceToBatch” layer, which isn’t supported by glow, and as a result I receive the following error:<br>
“Error message: TensorFlowLite: Operator type ‘SPACE_TO_BATCH_ND’ is not supported!”</p>
<p>But the backend knows how to parse this layer - When this layer is encountered, I’d like to create a new MySpace2BatchNode (which is defined in the backend), and add it to the graph.<br>
Is there a way to instruct glow (without frontend code changes), that if the model parser encountered a layer it cannot parse, ask the backend if it can parse it, instead of failing immediately?<br>
In other words, can the control pass to the backend before the parsing of the model is complete, or is the only way to change the behavior of the parser is to modify the parser code itself?<br>
Thanks.</p>

jfix: <p>Got it.</p>
<blockquote>
<p>Is there a way to instruct glow (without frontend code changes), that if the model parser encountered a layer it cannot parse, ask the backend if it can parse it, instead of failing immediately?</p>
</blockquote>
<p>Depends by what you mean without frontend code changes. The current <code>master</code> cannot do this. We would need to make changes to do something like, if the node isn’t supported in Glow IR, then check the backend if it supports it as an <code>ExternalFunctionCall</code> or something similar, and if so load it as such. I.e. this would require a single code change to the frontend so that such layers could be loaded as something like an <code>ExternalFunctionCallNode</code>. It would then be possible to load them and then the backend would be responsible for its execution.</p>
Backward takes a long time while GPU usage is zero. Train one LSTM model with another LSTM model

UCDuan: <p>Hi all,</p>
<p>I’m training an LSTM model with the following code. First I will generate a time series from my LSTM model <code>inverse_lstm</code>. Then the generated series is fed to another pre-trained LSTM model <code>test_lstm</code>, which is in evaluation mode. The idea is to train <code>inverse_lstm</code> with the help of <code>test_lstm</code>. The output from <code>inverse_lstm</code> is the input to <code>test_lstm</code> and the output of <code>test_lstm</code> is used in the loss function.</p>
<pre><code class="lang-auto"># generate input from inverse_lstm. 
# Here the data loader is not shuffled to follow chronological order. 

for i, data in enumerate(loader):
    x_d, x_s, x_one_hot, y = data
    x_d, x_s = x_d.to(device), x_s.to(device)
    x_one_hot, y = x_one_hot.to(device), y.to(device)
    x_ = inverse_lstm(x_d=x_d, x_s=x_s, x_one_hot=x_one_hot, y_true=y)[0] 
    x_ = x_[:, 0, :]  
    x_ = x_.cpu().data
    x_generated.append(x_)  

# The generated data will be fed into a fixed LSTM model. 

x_generated = torch.cat(x_generated, dim=0)  # concat along batch. 
x_generated = x_generated.requires_grad_(True)
ds_gen = Dataset_Precip(ds=ds, x_precip=x_generated)
gen_loader = DataLoader(ds_gen, batch_size=128, shuffle=True)

# Train inverse_lstm with test_lstm. 
# The gradient needs to trace back to inverse_lstm. 

for data in tqdm(gen_loader):
    x_d, x_s, x_one_hot, y = data
    x_d, x_s = x_d.to(device), x_s.to(device)
    x_one_hot, y = x_one_hot.to(device), y.to(device)
    y_hat = test_lstm(x_d=x_d, x_s=x_s, x_one_hot=x_one_hot)[0]
    y_hat_sub = y_hat[:, -1:, :]
    y_sub = y[:, -1:, :]
    optimizer.zero_grad()
    loss = mse_loss(y_hat_sub, y_sub)
    loss.backward(retain_graph=True)
    optimizer.step()
</code></pre>
<p>The training process takes a long time, especially <code>loss.backward()</code>. It takes around 2 minutes for one batch. Also by monitoring CPU and GPU, the CPU usage is pretty high and GPU usage is zero during the backward process. Is it because that the gradient is hard to calculate from the loss to the first lstm model? Or is there anything wrong with my code?</p>
<p>Thank you for any suggestion and help!</p>
Custom optimizer class

Samue1: <p>I would like to implement a custom optimizer class so that I have more control during the gradient descent step. Here is my approach:</p>
<pre><code class="lang-auto">class SGD:

    def __init__(self, model, learning_rate, momentum):
        self.model = model
        self.learning_rate = learning_rate
        self.momentum = momentum

        self.v_old = [torch.zeros_like(p.data) for p in model.parameters()]

    def step(self):
        with torch.no_grad():
            update = lambda v_old, grad: self.momentum * v_old + self.learning_rate * grad
            self.v_old = [update(v_old, params.grad) for v_old, params in zip(self.v_old, self.model.parameters())]
            for parameters, v_old in zip(self.model.parameters(), self.v_old):
                parameters -= v_old
</code></pre>
<p>I use this class as follows:</p>
<pre><code class="lang-auto">train_loader, test_loader = get_data_loader(batch_size)
model = Model()
cross_entropy_loss = nn.CrossEntropyLoss()
optimizer = SGD(model=model, learning_rate=0.01, momentum=0.9)

for epoch in range(10):

    running_loss = 0.0
    running_counter = 0

    for i, data in enumerate(train_loader):
        inputs, labels = data[0], data[1]
        model.zero_grad()
        outputs = model(inputs)
        loss = cross_entropy_loss(outputs, labels)
        loss.backward()

        optimizer.step()

        running_loss += loss.item()
        running_counter += labels.size(0)

    running_loss = running_loss / running_counter
    print(f"epoch {epoch} loss {running_loss}")
</code></pre>
<p>I would like to know if I can improve the SGD class and if the approach is considered good style? Or are there more elegant ways to create custom optimizer classes?</p>

albanD: <p>Hi,</p>
<p>That looks good to me.</p>
<p>One thing we usually do in core optimizer is to lazily create the buffers (like v_old). Namely only create it during the first step.<br>
That way, you can do things like:</p>
<pre><code class="lang-auto">optimizer = SGD(model=model, learning_rate=0.01, momentum=0.9)
mode.cuda()
# Your training loop
</code></pre>
<p>With your current code, this would fail because the buffer will be on cpu and the weights on gpu.</p>

Samue1: <p>I’m not completely sure if I understood you correctly. It is better to move the buffer to the GPU, right? But how can I do that?</p>
<p>Can I just do something like</p>
<pre><code class="lang-auto">self.v_old = [torch.zeros_like(p.data).to(device) for p in model.parameters()]
</code></pre>
<p>or is this not possible?</p>

albanD: <p>Ho what I meant is to do:</p>
<pre><code class="lang-auto"># inside the __init__
self.v_old = None

# inside the step
if self.v_old is None:
    self.v_old = [torch.zeros_like(p) for p in model.parameters()]
</code></pre>
Label Smoothing CrossEntropy in the Semantic Segmentation

choco9966: <p>Hello, I’m having trouble creating a Label Smoothing function for CrossEntropy.</p>
<p>input : (1, 3, 512, 512)<br>
output : (512, 512) # n_class = 12</p>
<pre><code class="lang-auto">class LabelSmoothingLoss(nn.Module): 
    def __init__(self, classes, smoothing=0.0, dim=-1): 
        super(LabelSmoothingLoss, self).__init__() 
        self.confidence = 1.0 - smoothing 
        self.smoothing = smoothing 
        self.cls = classes 
        self.dim = dim 
        
    def forward(self, pred, target): 
        pred = pred.log_softmax(dim=self.dim) 
        with torch.no_grad():
            true_dist = torch.zeros_like(pred) 
            true_dist.fill_(self.smoothing / (self.cls - 1)) 
            true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence) 
        return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))
</code></pre>
<p>In this situation, which part of the code above should I modify?</p>
Pytorch vs. Torchscript about GIL policy

111480: <p>PyTorch should use multi-processing instead of multi-threading due to GIL policy, but can torchscript have multi-threading effect because it is JIT?<br>
Is it correct that JIT does not follow GIL policy?<br>
So, is it correct to use Python interpreter when running Python and JIT interpreter when using Torch script?</p>
Masking the gradients before the update

Emanuele_Mercanti: <p>Hello</p>
<p>I was reading this paper <a href="https://arxiv.org/abs/2009.00329" rel="noopener nofollow ugc">Learning explanations that are hard to vary</a> and found the relative <a href="https://github.com/gibipara92/learning-explanations-hard-to-vary/blob/main/notebooks/linear_regression_ilc.ipynb" rel="noopener nofollow ugc">github repo</a>. To keep it short, before updating the parameters <code>theta = theta - lr * final_grads</code> pytorch (cuda) computes by default the arithmetic mean of the gradients, whereas I want to compute the geometric mean or to apply a mask as shown in the code.</p>
<p>Is there a way to do this leveraging pytorch autograd + cuda without the need to write a custom training loop?</p>
<p>Code taken from the linked notebook</p>
<pre><code class="lang-auto">def opt(x, y, method, lr, weight_decay, n_iters, verbose=False):
    thetas, iters, losses = [], [0], []
    theta = torch.randn(5, requires_grad=True) * 0.1
    thetas.append(theta.data.numpy())
    
    with torch.no_grad():
        loss = loss_fn(x, theta, y)
        losses.append(loss.item())
    
    for i in range(n_iters + 1):
        lr *= 0.9995
        
        grads = []
        loss_e = 0.
        for e in range(x.shape[0]):
            loss_e = loss_fn(x[e], theta, y[e])
            grad_e = torch.autograd.grad(loss_e, theta)[0]
            grads.append(grad_e)

        grad = torch.stack(grads, dim=-1)
        
        if method == 'geom_mean':
            n_agreement_domains = len(grads)
            signs = torch.sign(grad) 
            mask = torch.abs(signs.mean(dim=-1)) == 1
            avg_grad = grad.mean(dim=-1) * mask
            prod_grad = torch.sign(avg_grad) * \
                        torch.exp(torch.sum(torch.log(torch.abs(grad) + 1e-10), dim=1)) \
                        ** (1. / n_agreement_domains)
            final_grads = prod_grad
        elif method == 'and_mask':
            signs = torch.sign(grad) 
            mask = torch.abs(signs.mean(dim=-1)) == 1
            avg_grad = grad.mean(dim=-1) * mask
            final_grads = avg_grad
        elif method == 'arithm_mean':
            avg_grad = grad.mean(dim=-1)
            final_grads = avg_grad
        else:
            raise ValueError()
            
        theta = theta - lr * final_grads
        
        # weight decay
        theta -= weight_decay * lr * theta
        
        if not i % (n_iters // 200):
            thetas.append(theta.data.numpy())
            iters.append(i)
            with torch.no_grad():
                loss = loss_fn(x, theta, y)
                losses.append(loss.item())
        
        if not i % (n_iters // 5):
            print(".", end="")
            with torch.no_grad():
                loss = loss_fn(x, theta, y)
                if verbose:
                    print(f"loss: {loss.item():.6f}, theta: {theta.data.numpy()}, it: {i}")
              
    with torch.no_grad():
        loss = loss_fn(x, theta, y)
        print(f"loss: {loss.item():.6f}, theta: {theta.data.numpy()}, it: {i}")
    return np.stack(thetas), iters, losses
</code></pre>
Libdnnl and libmlkdnn in LibTorch

Kablys: <p>Why are there two identical libraries inside LibTorch: <code>libdnnl</code> and <code>libmlkdnn</code>?</p>

ptrblck: <p>I would assume one of them is a symbolic link, but you are right. It seems both are shipped and should be identical, if I’m not mistaken:</p>
<pre><code class="lang-auto">ll libtorch/lib
[...]
185727360 Mar 23 21:02 libdnnl.a
185727360 Mar 23 21:02 libmkldnn.a
[...]
</code></pre>
<p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and link to this thread, please?</p>

Kablys: <p>Thank you, I did: <a href="https://github.com/pytorch/pytorch/issues/57123" class="inline-onebox" rel="noopener nofollow ugc">LibTorch ships with two identical library files: libdnnl and libmlkdnn · Issue #57123 · pytorch/pytorch · GitHub</a></p>
My flask app just stops when it starts loading my torch model

vishak_bharadwaj: <p>Hello,<br>
I have saved my Pytorch model as a state_dict, and am now trying to load it into flask for inference.<br>
However when I run my flask API, it just stopped. And using my debugger got me here.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e56ae3a87d68d9d28344828face2ec8f84357de5.png" data-download-href="https://discuss.pytorch.org/uploads/default/e56ae3a87d68d9d28344828face2ec8f84357de5" title="Screenshot from 2021-04-27 15-53-22"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e56ae3a87d68d9d28344828face2ec8f84357de5.png" alt="Screenshot from 2021-04-27 15-53-22" data-base62-sha1="wJwlHjZHoanvfrSvuJXXFTgUAtL" width="690" height="218" data-dominant-color="292A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-27 15-53-22</span><span class="informations">727×230 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The same model loads in my notebook. Why is this happening? Any help would be great!</p>

Yuanchao_Du: <p>you can check this <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html?highlight=flask" class="inline-onebox" rel="noopener nofollow ugc">Deploying PyTorch in Python via a REST API with Flask — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>

vishak_bharadwaj: <p>Hey, It doesn’t seem to be a code problem. The flask app used to work on my old system, but doesn’t on my new one. Same code in both places.</p>

Yuanchao_Du: <p>please provide more code in you <code>utils.py</code> and <code>pytorch_model.py</code></p>

vishak_bharadwaj: <p>Sure<br>
this is from pytorch_model.py</p>
<pre><code class="lang-auto">from torch import nn 
import torch.nn.functional as F



class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linearA = nn.Linear(13,100)
        self.linearB = nn.Linear(100, 50)
        self.linearC = nn.Linear(50, 1)

    def forward(self, x):
        yA = F.relu(self.linearA(x))
        yB = F.relu(self.linearB(yA))
        return self.linearC(yB)
</code></pre>
<p>and this is the part from utils.py</p>
<pre><code class="lang-auto">data.model = Model()
data.model.load_state_dict(torch.load(data.model_path))
</code></pre>
Image classification Input Features

Tornike: <p>When classifying images with cnn what the input feature should be? As i know, it should be 1 if the images are grayscale and 3 if the images are RGB. Is it necessary to input feature to be<br>
image pixels * image pixels?</p>

JuanFMontesinos: <p>In general convolutions works on a grid. Therefore the input can be an image of any size.<br>
Some networks (usually old ones) have fully connected layers in the edn which work with fixed size inputs. In short networks should work with any size but may not.</p>
<p>Another thing is the subjective size of the object in the image. If a network is trained with images of size X and you feed a image of size 10X then it would be too big. Imagine someone brings an object really really close to your face, you may not recognize it.</p>
How to separate between one trayectory from another in trainset

Laura_Montalvo: <p>HI there! I’ve created a dataset in which 5 trajectories taken at the same scenario are concatenated in this way:</p>
<ul>
<li>1 trajectory has N of positions measured (this is the number of rows) &amp; a fixed number of columns (shared between trajectories) that represent the data of each position of the trajectory. So each trajectory has shape (N, n_columns)<br>
So the dataset has shape: (N*number_trajectories, n_columns). My problem is that I’m trying to train a LSTM with different sequence length, being the sequence length how many positions does the LSTM see for predicting the output.<br>
However I have a problem separating the trayectories for training: some of the final positions of one trajectory are fed with the first positions of the new trayectory because of how I feed the network with the sequence length.<br>
A comparison just in case I’m not explaining: it’s like if my dataset is compose by 7 reviews expressed as one large string, and I mix the last words of my first review with the first words of my second review that have nothing to do one with another. So I know that I can separate this by spliting the reviews by new line, creating a vector in which each row is a review.</li>
</ul>
<p>However I dont know how to do this with my vector data. Any help?</p>

googlebot: <p>You’ll have to either write code for hierarchical sampling (timeseries, then random slice) or cut some tails (with nn.utils.rnn.pack_padded_sequence perhaps). Sounds like you’re using a simple 2d data loader, it is probabably best to fix things at that level, than attempting to implement mid-sequence reset schemes.</p>

Laura_Montalvo: <p>Hi there! Thanks I get what you’re telling me. That’s why I’m trying to change how i get the item within the custom dataset. However my sequences are of different sizes so I don’t really know how can I obtain when I call the dataloader different sequence for each idx. I know this wont work for batch_size=! 1 but I wanna try first if it works for batch_size=1, any help?</p>

googlebot: <p>You can use torch’s DataLoader to iterate over timeseries, and in addition manually sample random time offsets. This can be done inside collate_fn (see collate_batch in cell 20 <a href="https://github.com/pytorch/text/blob/50bd1b3f65b4b5ffb07267c34ac54ca13e08d406/examples/legacy_tutorial/migration_tutorial.ipynb" rel="noopener nofollow ugc">here</a> as an example of collate_fn, though it is a bit different - full sequences are padded). Or inside <code>__iter__</code> (IterableDataset), that’s more conventional way.</p>

Laura_Montalvo: <p>Thanks, I’ve seen that I can create a custom sampler for my custom dataset so thats what I’m trying. However I’ll take a look at your proposal.</p>
Saving and reloading model: save and reload vocab as well?

st-vincent1: <p>When I train a text generation model and then reload it in a separate session, I get abysmal performance. I figured out that the problem might be the vocabulary changing. Since everytime I start a session I build a new vocabulary from the training data, which is generated using random split from a Dataframe, the “stoi” mapping will be different every time.</p>
<p>Is there a way to save the vocabulary along with the model parameters to make sure that inference will be successful?</p>

pascal_notsawo: <p>If you are using random, I suggest you look here:</p>
<ul>
<li>pytorch : <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.8.1 documentation</a>
</li>
<li>numpy : <a href="https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.seed.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.random.seed — NumPy v1.15 Manual</a>
</li>
<li>python : <a href="https://docs.python.org/3/library/random.html#random.seed" class="inline-onebox" rel="noopener nofollow ugc">random — Generate pseudo-random numbers — Python 3.9.4 documentation</a>
</li>
</ul>
<p>I think it will allow you to have the same random split at all times.</p>

st-vincent1: <p>Thanks for replying. I am using the following function to set the random seed at the beginning of every session:</p>
<pre><code class="lang-auto">    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>Nevertheless, the issue remains.</p>
Building torchaudio on ppc64le

nullpunkt: <p>I am trying to build the torchaudio extension on my university’s HPC, which provides PowerPC computing nodes for machine learning applications. So I have to build from source since no wheels for ppc64le are available, AFAIK.</p>
<p>I am also limited to PyTorch 1.6.0, because there is an optimized module available on the HPC and I would like to use it. I therefore am trying to build torchaudio v0.6.0 (according to the compatibility table in the torchaudio repo).</p>
<p>Here is the command I am running:</p>
<pre><code class="lang-auto">CC=gcc CXX=g++ BUILD_SOX=1 python setup.py install --user
</code></pre>
<p>When I do so, I first get an error about the third-party libs MAD and LAME and their config.guess, which cannot guess my platform. I was able to get around that by overwriting the respective config.guess files using an updated version taken from <a href="http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD" rel="noopener nofollow ugc">here.</a></p>
<p>After that, everything seems to build just fine, but when I import torchaudio, I get the warning:</p>
<blockquote>
<p>UserWarning: torchaudio C++ extension is not available.</p>
</blockquote>
<p>Can anyone tell me what this means exactly and how I can make the C++ extension available?</p>

ptrblck: <p>The error message is raised, if the backend cannot be found as described <a href="https://pytorch.org/audio/stable/backend.html#availability">here</a>. Did the log indicate that SOX will be installed after you’ve fixed the config or was it showing any warnings/errors?</p>

nullpunkt: <p>I did not notice anything. Are the logs saved anywhere?</p>
<p>By the way, the warning has disappeared as of today. Since it is an HPC, I have no idea what changed, though. Maybe they installed the required libraries out of sheer coincidence.</p>