akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Style loss stays zero

username: yamand16
cooked: <p>Hello all,</p>
<p>I am trying to use feature reconstruction and style reconstruction losses on my model. For this, I followed the example code on PyTorch website for “Neural Style Transfer”.</p>
<p><a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/neural_style_tutorial.html</a></p>
<p>Although the feature loss is calculated without problem, the style loss is always zero. And I could not find the reason since everything looks okay in the implementation. The calculation formulas are the same with the proposed mathematical formulas. Besides, as you know, the style and feature losses are almost same in terms of calculation except Gram matrix step in style loss and no problem in feature loss.</p>
<p>Could anyone help me about this situation?</p>
<p>Best.</p>
How to run inference on a trained model

username: Meriem
cooked: <p>After building and training a regression model, I saved then loaded the model and I am now trying to run inferences on the loaded model in order to get the loss value and calculate other metrics.<br>
Is there anything similar to Keras’:</p>
<pre><code class="lang-auto">model.evaluate(x_val, y_val,verbose=1)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">model.predict(x_val)
</code></pre>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You should call <code>model.eval()</code> to set the model to evaluation mode, which will change the behavior of some modules (e.g. batchnorm layers will use their running stats and dropout will be disabled) and can then directly pass your test data:</p>
<pre><code class="lang-python">model.eval()
out = model(test_tensor)
...
</code></pre>

username: Meriem
cooked: <p>Thanks for the quick reply!</p>
<p>I already tried that, but surprisingly, didn’t get the same output as the one I got when I first ran the training and testing loop… I thought there must be others ways to do it.<br>
I’ll take a closer look at the code… there should be something missing.</p>
Using torch.load on a torch hub model

username: Luca_Pamparana
cooked: <p>I am currently playing around with the DETR object detection toolkit and I am downlloading the pretrained models as:</p>
<pre><code class="lang-auto"> model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)
</code></pre>
<p>Now it downloads files locally in the pytorch cache directtory. My question is would it be possible to load this model using the <code>torch.load</code> method subsequently? My use-case is that I would like to sort of us these models offline and once they are downloaded what would be the best way to use them without replying on torch.hub?</p>
<p>Is it possible to generate some torch jiit traced model from this, which I can simply use with <code>torch.jit.load</code> or something similar?</p>

username: diskontinuum
cooked: <p>Have you tried downloading the DETR model directly from the <a href="https://github.com/facebookresearch/detr" rel="nofollow noopener">facebookresearch repo on github</a> and loading it?</p>
<pre><code class="lang-auto">state_dict = torch.load("detr-r50-e632da11.pth")
</code></pre>
<p>To load the dict to the model this <a href="https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb#scrollTo=GJhCp8y0-b-H" rel="nofollow noopener">detr-demo-notebook</a> suggests building a class and instantiating it minimally (with just the number of classes) and then calling the method <code>load_state_dict()</code>, which is not defined in the notebook.</p>
<pre><code class="lang-auto">state_dict = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)
detr.load_state_dict(state_dict) 
</code></pre>
<p>I’m still figuring out how to avoid a dict mismatch (‘missing keys’, etc.).</p>
First iteration is 10x faster than subsequent iterations in for loop

username: WillThomson
cooked: <p>I have a for loop in which the first iteration is substantially faster than subsequent iterations. This discrepancy only appears when I use a GPU. On a CPU every iteration takes more or less the same time. I’m wondering what causes this and whether there’s anything I can do to prevent the slow down in later iterations.</p>
<p>The code is here. <a href="https://colab.research.google.com/drive/1TC6khI8T44KcYCrdAkHpldw_ejRG_eO2?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1TC6khI8T44KcYCrdAkHpldw_ejRG_eO2?usp=sharing</a></p>
<p>Apologies if this is a common question or there’s a simple explanation, but I haven’t been able to find any answers.</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you would have to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>. Otherwise you’ll only profile the Python overhead as well as the kernel launch in the first iterations until your script encounters a blocking operation.</p>

username: WillThomson
cooked: <p>Ah! Thanks, things make sense after this.</p>
Conv1d internal weight calculations

username: Keith72
cooked: <p>Hi, I am trying to figure out how a nn.conv1d processes an input for a specific example related to audio processing in a WaveNet model. I have input data of shape (1,1,8820), which passes through an input layer (1,16,1), to output a shape of (1,16,8820). That part I understand, because you can just multiply the two matrices. The next layer is a conv1d, kernel size=3, input channels=16, output channels=16, so the state dict shows a matrix with shape (16,16,3) for the weights. When the input of (1,16,8820) goes through that layer, the result is another (1,16,8820).  What multiplication steps occur within the layer to apply the weights to the audio data? In other words, if I wanted to apply the layer(forward calculations only) using only numpy for this example, how would I do that?</p>

username: yoyololicon
cooked: <p>Hi <a class="mention" href="/u/keith72">@Keith72</a>, this is how pytorch conv1d actually do in your case:</p>
<pre><code class="lang-python">x = torch.rand(1, 16, 8820)
weight = torch.rand(16, 16, 3)

# first pad zeros along the time dimension
x = torch.pad(x, [1, 1])    #shape = (1, 16, 8822)

#unfolded, so you have 8820 moving windows with size = (16, 3)
x = x.unfold(2, 3, 1)    #shape = (1, 16, 8820, 3)

# matrix multiplication, I use tensordot for simplicity
y = torch.tensordot(x,weight, dims=([1, 3], [1, 2]))     #shape = (1, 16, 8820)
</code></pre>
<p>In numpy you can simply replace <code>pad</code> and <code>tensordot</code> with corresponding numpy function; for <code>unfold</code> you can use <code>numpy.lib.stride_tricks.as_strided</code>.</p>

username: Keith72
cooked: <p>Thanks for the quick response! My initial implementation seems to fit using your steps, except the last step gave me a shape (16,1,8820), so I just swapped the first two dimensions. Now if I wanted to account for layer dilation, how would that work?</p>

username: yoyololicon
cooked: <p>That can be achieved easily by using indexing:</p>
<pre><code class="lang-python">x = torch.pad(x, [1 * dilation] * 2)
x = x.unfold(2, 2 * dilation + 1, 1)[..., ::dilation]    #shape = (1, 16, 8820, 3)
...
</code></pre>
<p>In numpy you can alternate the stride size of the ndarray to do dilated convolution. Here’s an example <a href="https://github.com/yoyololicon/building-nn-with-numpy/blob/fffb87126e6a9fd4f580654403d2cf35693829ee/nn.py#L85" rel="nofollow noopener">implemention</a>, you can check it for details.</p>

username: Keith72
cooked: <p>The example helps a lot, and thank you for taking the time to explain all that.</p>