akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model.load_state_dict(torch.load(args.model_loc)) after using Inception V3 pretrained model

username: Mona_Jalal
cooked: <p>I am reading a <a href="https://github.com/arjunvekariyagithub/camelyon16-grand-challenge/blob/660000a79775fbc5cfa8c5b44a591e62ce714089/camelyon16/postprocess/build_heatmap_multi_thread.py" rel="noopener nofollow ugc">GitHub code</a> that uses a pretrained version of Inception V3. I am confused by this line <code>    model.load_state_dict(torch.load(args.model_loc))</code>  in the code:</p>
<pre><code class="lang-auto">    # define the model 
    model = models.inception_v3(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, 2)

    # load the model
    model.load_state_dict(torch.load(args.model_loc))
    model.eval()
    model.cuda()
</code></pre>
<p>As we see, <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/inception.py" rel="noopener nofollow ugc">in the original Inception code</a>, it is already loading the state dict. I am wondering why the authors are also doing same after setting their model to the pretrained Inception V3?</p>
<p>At the same time, I see <a href="https://www.kaggle.com/beastlyprime/pytorch-beginner-transfer-learning" rel="noopener nofollow ugc">this piece of Kaggle code</a> for using a pretrained Inception V3 model. So, which method do you comment as correct?</p>
<pre><code class="lang-auto">import torchvision.models as models
import torch.optim as optim

model_ft = models.inception_v3(pretrained=True)
for param in model_ft.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
# Handle the auxilary net
num_ftrs = model_ft.AuxLogits.fc.in_features
model_ft.AuxLogits.fc = nn.Linear(num_ftrs, 2)
# Handle the primary net
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)
</code></pre>

username: ptrblck
cooked: <p>I guess the author of the first code snippet is loading a pretrained <code>state_dict</code> for the manipulated model (note that <code>model.fc</code> is changed)?</p>
Can't install deepspeed

username: Klink
cooked: <p>× python setup.py egg_info did not run successfully.<br>
│ exit code: 1<br>
╰─&gt; [11 lines of output]<br>
Traceback (most recent call last):<br>
File “”, line 2, in <br>
File “”, line 34, in <br>
File “C:\Users\USER\DeepSpeed\setup.py”, line 138, in <br>
abort(f"Unable to pre-compile {op_name}")<br>
File “C:\Users\USER\DeepSpeed\setup.py”, line 43, in abort<br>
assert False, msg<br>
AssertionError: Unable to pre-compile cpu_adam<br>
DS_BUILD_OPS=1<br>
[WARNING]  One can disable cpu_adam with DS_BUILD_CPU_ADAM=0<br>
[ERROR]  Unable to pre-compile cpu_adam<br>
[end of output]</p>
<p>I use windows10 and i installed requirements programs : cuda, cpp build tools, pytorch… etc</p>
<p>I was looking for many sites to find this error, but there is no data about this error.</p>
<p>How i can solve this problem?</p>

username: ptrblck
cooked: <p>The error seems to be raised by <code>cpu_adam</code>. Did you try to disable it as suggested in the warning?</p>
<pre><code class="lang-python">[WARNING] One can disable cpu_adam with DS_BUILD_CPU_ADAM=0
[ERROR] Unable to pre-compile cpu_adam
</code></pre>

username: Klink
cooked: <p>I tried to disable CPU_ADAM.<br>
But it called next compile error, CPUAdagrad/FusedAdam/FusedLamb that imported in op_builder folder.</p>
Custom parameter not updating weights

username: Honnang
cooked: <p>I am trying to create a custom parameter as a scale factor to resize the tensor via bilinear interpolation</p>
<p>self.scl = nn.Parameter(torch.tensor(0.5, requires_grad=True))</p>
<pre><code>def forward(self, inx):
    x = self.lrelu(self.part1_PW(inx))
    x0 = x
    #==========
    b, c, h, w = x.shape

    scl = self.scl.item()
    scl = round(scl, 1)
    #print(scl)
    if scl == 0:
        scl = 1.0
    elif scl &lt;0:
        if scl &gt; -1:
            scl = scl*(-1.0)
        else:
            scl = 1.0
    else:
        if scl &gt; 1:
            scl = 1.0

    downscale = nn.Upsample(size=(int(h * scl), int(w * scl)), mode='bilinear', align_corners=False)
</code></pre>
<p>But when I checked, I realized the weights aren’t updating<br>
is the problem from the fact that I didn’t use the weights on the input tensors?</p>

username: ptrblck
cooked: <p>Based on your code it seems you are using <code>self.scl.item()</code> which will return a plain Python <code>float</code> and will thus not update the parameter.<br>
To get valid gradients in the parameter you would have to use it directly.<br>
However, based on your code you are also assigning new values to it, so I’m unsure how the actual usage and update should work.</p>
Undefine opencv symbol when compiling a CPP Extension with pytorch

username: ethanquix
cooked: <p>Hi !</p>
<p>I am trying to make my custom C++ extension for pytorch.</p>
<p><strong><em>Intro:</em></strong></p>
<p>C++ file look like this:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;opencv2/cudaarithm.hpp&gt;
#include &lt;opencv2/cudaimgproc.hpp&gt;

//CODE ...

cv::pyrDown(in, out, cv::Size(X, Y));

//CODE ...
</code></pre>
<p>setup.py look like this:</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import CppExtension, BuildExtension

setup(name='lltm',
      ext_modules=[
          CppExtension(
              'gaussianpyramid',
              ['src/pyramid.cpp'],
              extra_compile_args=['-lopencv_calib3d', '-lopencv_contrib', '-lopencv_core', '-lopencv_features2d',
                                  '-lopencv_flann', '-lopencv_highgui', '-lopencv_imgproc', '-lopencv_legacy',
                                  '-lopencv_ml', '-lopencv_objdetect', '-lopencv_photo', '-lopencv_stitching',
                                  '-lopencv_superres', '-lopencv_ts', '-lopencv_video', '-lopencv_videostab'],

          )],
      cmdclass={'build_ext': BuildExtension},
      )
</code></pre>
<p><strong><em>The problem:</em></strong></p>
<p>After compilation, if in a python script I have:<br>
<code>import MY_LIB</code></p>
<p>It stop by an error: <code>undefined symbol: _ZN2cv7pyrDownERKNS_11_InputArrayERKNS_12_OutputArrayERKNS_5Size_IiEEi</code></p>
<p><strong><em>Info:</em></strong></p>
<p>OpenCV library are in my <code>LD_LIBRARY_PATH</code><br>
<code>ldconfig -N -v $(sed 's/:/ /g' &lt;&lt;&lt; $LD_LIBRARY_PATH) | grep opencv</code> find them</p>
<p>This symbol is inside the opencv library (I have check this with a grep on the library)<br>
Another error (and I think it come from there, is than if I do a <code>ldd my_lib.so</code> (generated by the setup.py), I don’t have the opencv lib, only:</p>
<pre><code class="lang-auto">	linux-vdso.so.1 =&gt;  (0x00007ffc46dc4000)
	libstdc++.so.6 =&gt; XXX (0x00007fbda60e6000)
	libm.so.6 =&gt; XXX (0x00007fbda5db3000)
	libgcc_s.so.1 =&gt; XXX (0x00007fbda5ba1000)
	libpthread.so.0 =&gt; XXX (0x00007fbda5985000)
	libc.so.6 =&gt; XXX (0x00007fbda55c1000)
	/lib64/ld-linux-x86-64.so.2 (0x00005573b895d000)
</code></pre>
<p>And no opencv library…</p>

username: lshiwjx
cooked: <p>Hi, I meet the same problem, have you solved it?</p>

username: Tengxu-Sun
cooked: <p>The same problem, any update?</p>
How to stack over for loop?

username: smani
cooked: <p>I have a loop, and I am getting a 10x10 tensor for each iteration of that loop. Lets assume that I am running that loop five times, and the output after the loop completes should be the concatenation of these tensors, i.e., a size of 10x10x5. How to concatenate this?</p>
<pre><code class="lang-auto">outx = []
for i in range(5):
    tmp = net(x) # this will return a 10x10 tensor
    outx = # need to cat tmp with outx in dim=2

outx - should have a dimension of 10x10x5
</code></pre>

username: tom
cooked: <p>Building the list and then using <code>stack</code> at the end is reasonable:</p>
<pre><code class="lang-python">outx = []
for i in range(5):
    tmp = net(x) # this will return a 10x10 tensor
    outx.append(tmp)

outx = torch.stack(outx, 2)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: abhibha1807
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> ,<br>
I had a question, if the outputs that I want to append to a list are my model outputs, Will appending them to a list and then applying torch.stack break the computation graph?<br>
Thank you.</p>

username: tom
cooked: <p>No. If any of the inputs in the list (which <code>torch.stack</code> will iterate over) requires gradient, so does the output and the graph is connected accordingly.</p>
<p>Best regards</p>
<p>Thomas</p>
nn.ModuleDict vs OrderedDict vs dict

username: denexo
cooked: <p>What is the advantage of using <code>nn.ModuleDict</code> versus <code>OrderedDict</code> or <code>dict</code> (in case the order doesn’t matter)?</p>
<pre><code class="lang-auto">self.activations = nn.ModuleDict([
        ['lrelu', nn.LeakyReLU()],
        ['prelu', nn.PReLU()]
])
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">self.activations = OrderedDict([
        ['lrelu', nn.LeakyReLU()],
        ['prelu', nn.PReLU()]
])
</code></pre>

username: ptrblck
cooked: <p><code>nn.ModuleDict</code> will properly register the modules to the parent as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.module_activations = nn.ModuleDict([
            ['m_lrelu', nn.LeakyReLU()],
            ['m_prelu', nn.PReLU()]
        ])



        self.activations = OrderedDict([
            ['lrelu', nn.LeakyReLU()],
            ['prelu', nn.PReLU()]
        ])
    
model = MyModel()
print(dict(model.named_modules()))
</code></pre>
<p>This makes sure that the parameters are pushed to the desired device in e.g. <code>to()</code> calls and the <code>model.parameters()</code> call (used to pass the parameters to e.g. an optimizer) will return all parameters of the registered modules.</p>

username: Eta_C
cooked: <p>The parameters may not be updated while training if you use <code>OrderedDict</code>.</p>
Unable to install Pytorch on Centos

username: Takumi3
cooked: <p>am trying to install pytorch on Centos 6.7 using the below code</p>
<p><em>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</em></p>
<p>But I am getting the below shared error</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
</code></pre>
<p>I am Nvidia GeForce RTX A5000 on my Desktop PC.<br>
Please let me know where am I going wrong.</p>
<p>and I get GPU information like below.</p>
<pre><code class="lang-auto"> NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6
</code></pre>
Lazily loading inputs of different sizes without padding

username: MHSA
cooked: <p>Hello,</p>
<p>I have a dataset consisting of large images. I only need to look at a specific area in each image, so I have converted the entire dataset into an hdf5 file and I plan to <strong>lazily load</strong> that specific area per image.</p>
<p>However, since these areas are different between images, I end up with <strong>inputs of different sizes</strong>. The suggested way to handle that is to write a <strong>custom collate function</strong> to pad all the inputs into the same size. The catch is, <strong>instead of increasing the size by padding</strong> (which adds useless information to my input), I prefer to increase the selected area and <strong>just load a larger area</strong> from my hdf5 file.</p>
<p>In order to do this, I need to calculate the input size of that batch before I load anything; and for that I need to know which images will be in each batch. However, I don’t have access to this information inside the Dataset.</p>
<p>One solution I can think of is to load the data inside the collate function instead of in the Dataset, but I am worried that this will interfere with the parallelization that is done in DataLoader and slow the training.</p>
<p>Is there a better way to solve this problem? Or is there a way to access all the indices of a batch inside the Dataset?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MHSA" data-post="1" data-topic="145402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/3be4f8/48.png" class="avatar"> MHSA:</div>
<blockquote>
<p>However, I don’t have access to this information inside the Dataset.</p>
</blockquote>
</aside>
<p>That’s correct in the default use case. However, you could use a <code>BatchSampler</code>, which would allow you to get <em>all</em> indices of the current batch in the <code>__getitem__</code> and apply your resizing logic as described e.g. <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">here</a>.</p>
JIT inference time issue with variable batch-size

username: ZdsAlpha
cooked: <p>I have noticed when I perform inference with a unique batch size for the first time it takes around &gt;1s to do inference. After that it only takes &gt;0.01s. I think it recompiles the graph for different batch-size. Is it possible to compile graph for all batch sizes before performing inference (without passing tensors of every possible batch) so it doesn’t take too much time for initial inference. I can pad batch to fill it to fixed batch size but other models are larger and adding useless samples to batch will further increase time for cached batch.</p>
Pass correct image shape to model that uses a conv1d and con2d in the same network

username: Allaye
cooked: <p>so i am trying to implement the VGG network, everything in the paper, but i have when i am using the architecture that has a conv1-255 as part of it network. below is my code</p>
<pre><code class="lang-auto"> def _make_convo_layers(architecture) -&gt; torch.nn.Sequential:
        """
        Create convolutional layers from the vgg architecture type passed in.
        :param architecture:
        """
        layers = []
        in_channels = 3
        for layer in architecture:
            if type(layer) == int:
                out_channels = layer
                layers += [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
                # layers.append([nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=1) + nn.ReLU()])
                in_channels = layer
            elif (layer == 'Conv1-256'):
                out_channels = 256
                layers += [nn.Conv1d(256, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
            elif (layer == 'LRN'):
                layers += [nn.LocalResponseNorm(5, alpha=0.0001, beta=0.75, k=1)]
            elif (layer == 'M'):
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        return nn.Sequential(*layers)
</code></pre>
<p>below is me passing some random data to the model</p>
<pre><code class="lang-auto">
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
vgg = VGGNet(config['vgg16-C1']).to(device)
x = torch.randn(1, 3, 224, 224).to(device)
model = vgg(x).to(device)
print(model.shape)
</code></pre>
<p>below is the error i received when i passed the x variable to the model</p>
<p><code>RuntimeError: Expected 3-dimensional input for 3-dimensional weight [256, 256, 3], but got 4-dimensional input of size [1, 256, 56, 56] instead</code></p>
<p>any help, will do please</p>

username: ptrblck
cooked: <p>In your code you are using <code>nn.Conv1d</code> which seems to be a typo:</p>
<pre><code class="lang-python">elif (layer == 'Conv1-256'):
    out_channels = 256
    layers += [nn.Conv1d(256, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
</code></pre>
<p>Replace it with <code>nn.Conv2d</code> and it should work.</p>
Wrong training result with changing forward function in VAE

username: Khadijeh_Arabi
cooked: <p>I have a VAE model with the below forward function and loss function:</p>
<pre><code class="lang-auto">def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 2))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def loss_function(recon_x, x, mu, logvar):
       L2 = torch.mean((recon_x-x)**2)
       KLD = -0.5 * torch.mean(1 + logvar - mu.pow(2) - logvar.exp())
    return L2 + KLD
</code></pre>
<p>the result of training for the multivariate gaussian dataset is similar to the photo below:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa27cf477dcbeb4c5708970720e26b9215c4d24d.jpeg" alt="12" data-base62-sha1="zGYESZJLxe6VjCsaUqzV0bsvhxz" width="361" height="317"></p>
<p>To do something, I changed the forward and loss function as follows</p>
<pre><code class="lang-auto">def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 2))
        z = self.reparameterize(mu, logvar)
        q0 = torch.distributions.normal.Normal(mu, (0.5 * logvar).exp())
        prior = torch.distributions.normal.Normal(0., 1.)
        log_prior_z = prior.log_prob(z).sum(-1)
        log_q_z = q0.log_prob(z).sum(-1)
        return self.decode(z), log_q_z - log_prior_z

def loss_function(recon_x, x, KL):
    L2 = torch.mean((recon_x-x)**2)
    return L2 + KL
</code></pre>
<p>but the training result is as follows<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1960a7a022ef4949e2ce6207db56cb9a44f177c9.jpeg" alt="32" data-base62-sha1="3Cv19PPHBxHwOsH29x11GDy4r0Z" width="395" height="321"></p>
<p>How do I make these changes to get the same result?</p>

username: Khadijeh_Arabi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please help me? Thanks in advance</p>
Calculate MseLoss over 2 tensors - errors

username: adv010
cooked: <p>Hi, I’m trying to calculate MseLoss between 2 tensors of size torch.Size([64]),torch.Size([64]) as follows:</p>
<pre><code class="lang-auto">criterion = nn.MSELoss(reduction = 'sum')
  
    for epoch in range(5):
        total_loss = 0
        n_batches =0
    for batch_idx, data in enumerate(train_dataloader):
        
        s1, s2, labels, s1_length, s2_length = data
        optimizer.zero_grad()
        outputs, att1, att2 = model(s1,s2,s1_length,s2_length)
        print(outputs.shape) #torch.Size([64])
        print(labels.shape) #torch.Size([64])
        loss = criterion(outputs,labels) + pen_s1 + pen_s2
        print(loss)
        loss.backward()
</code></pre>
<p>I keep getting this error : <em><strong>grad can be implicitly created only for scalar outputs</strong></em> , despite giving the <code>reduction = sum</code> argument inside nn.MSELoss. Could someone tell me where am I going wrong with this?</p>

username: aiGeekProgrammer
cooked: <p>This line makes me wonder:</p>
<pre><code class="lang-auto">loss = criterion(outputs,labels) + pen_s1 + pen_s2
</code></pre>
<p>Could you elaborate on <em>pen_s1</em> and <em>pen_s2</em>?</p>
<p>I’m asking about this as <em>loss.backward()</em> works if loss contains single element and here something happens <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Does loss.sum().backward() work?</p>

username: adv010
cooked: <p>Hey, turns out my pen_s1 implementation was returning a matrix instead of the scalar output. It worked when I used pen_s1.mean()</p>
<p>Thanks for the reply, made me look again!</p>
Nn.functional.conv2d add more dimentions to convolve 3 color channels

username: mreister
cooked: <p>0</p>
<p>I am getting the following error message:</p>
<p><strong>RuntimeError: w groups=3, expected weight to be at least 3 at dimension 0, but got weight of size [1, 1, 2, 2] instead</strong></p>
<p>when i try to convolve a image with a filter using the “functional version of conv2d”</p>
<p>i know why i am receiving this error message… it is because i need to have 3 dimensions in channel 0. But i have now i idea how to add two more dimensions.</p>
<p>I have flailed around for quite some time trying to add two more, but i cant figure it out. I want to the kernal applied one all color channels… so i just want to replicate it 2 more times.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import torch.nn.functional as nnf
from PIL import Image
from torchvision import transforms

img = Image.open("GOPR0305.jpg")

preprocess = transforms.Compose([transforms.ToTensor()])
img_t = preprocess(img)
img_t = torch.unsqueeze(img_t, 0)

hci = [1, -1]
hri = [-1, 1]

hc = [1.0, -1.0]
hr = [-1.0, 1.0]

lc = [0.5, 0.5]
lr = [0.5, 0.5]



hh_k = torch.tensor([hc ,hr])[None, None, ...]
hl_k = torch.tensor([hc ,lr])[None, None, ...]
lh_k = torch.tensor([lc ,hr])[None, None, ...]
ll_k = torch.tensor([lc ,lr])[None, None, ...]
in_t = torch.tensor([ [14, 7, 6, 2,] , [4 ,8 ,11 ,1], [3, 5, 9 ,10], [12, 15, 16, 13] ])[None, None, ...]
in_t = torch.tensor([ [14.0, 7.0, 6.0, 2.0,] , [4.0 ,8.0 ,11.0 ,1.0], [3.0, 5.0, 9.0 ,10.0], [12.0, 15.0, 16.0, 13.0] ])[None, None, ...]

def wave_haar(in_t):
    hh = nnf.conv2d(in_t, hh_k,stride=2,groups=3)
    ll = nnf.conv2d(in_t, ll_k,stride=2)
    hl = nnf.conv2d(in_t, hl_k,stride=2)
    lh = nnf.conv2d(in_t, lh_k,stride=2)
    return [ll,hl,lh,hh]
    
[ll,hl,lh,hh] = wave_haar(img_t[:,2:])

print(img_t.shape)
print(img_t.size())
print(img_t)
print(img_t.shape)
print(ll.shape)
</code></pre>

username: mreister
cooked: <p>I was able to solve this by changing my kernal to  this:<br>
from this:</p>
<pre><code class="lang-python">hh_k = torch.tensor([hc ,hr])[None, None, ...]
</code></pre>
<p>to this:</p>
<pre><code class="lang-python">hh_k3 = torch.tensor([[[hc, hr]],[[hc, hr]],[[hc, hr]]])
</code></pre>
Text classification using ensemble of transformers

username: the_coder
cooked: <p>I am trying to do text classification using ensemble of transformers. My dataset contains about 64000 tweets. For each tweet i got prediction as follows, So for each tweet each transformer model is giving probabilities whether a tweet a sarcastic or not. As you can see below there are 10 probabilities for each tweet.</p>
<p>Tweet 1 : tensor([0.4219, 0.5781, 0.5237, 0.4763, 0.4977, 0.5023, 0.4618, 0.5382, 0.5324,0.4676])<br>
Tweet 2 : tensor([0.4295, 0.5705, 0.4761, 0.5239, 0.4859, 0.5141, 0.4979, 0.5021, 0.5025,0.4975])<br>
Tweet 3 : tensor([0.4000, 0.6000, 0.4832, 0.5168, 0.4932, 0.5068, 0.5023, 0.4977, 0.4939,0.5061])<br>
.<br>
.<br>
.<br>
Tweet 64000: tensor([0.4213, 0.5787, 0.4904, 0.5096, 0.4870, 0.5130, 0.5084, 0.4916, 0.4900,0.5100])</p>
<p>I need to feed in these probabilities to another neural network or apply logistic regression so that i can get the final prediction for each row. I am not sure how i can achieve these or whether it is actually possible to do. Can you please look into it and provide some insights?</p>
<p>Your answer will be much appreciated.</p>

username: sandeep1
cooked: <p>Just treat the values as feature as training set and train the model against the golden value.</p>

username: the_coder
cooked: <p>Thank you so much for your reply. Can you please elaborate/pseudocode how i can deal with 64000 rows with 10 probabilites /10 columns? Let say i want to feed these probabilities to a neural network, how do i write a class for my expected output? Please note that i am trying to ensemble 5 transformers, so thats why i got 10 probabilities.</p>
How we use GRU encoder state as input to decoder?

username: sandeep1
cooked: <p>The following code is taken from <a href="https://d2l.ai/chapter_recurrent-modern/seq2seq.html" class="inline-onebox" rel="noopener nofollow ugc">9.7. Sequence to Sequence Learning — Dive into Deep Learning 0.17.2 documentation</a><br>
The state shape from the encoder(X)[1] is torch.Size([2, 4, 16])), where 2-would be the number of layers, 4- batch size, 16- embedding size.</p>
<pre data-code-wrap="class"><code class="lang-nohighlight">    """The RNN decoder for sequence to sequence learning."""
    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,
                 dropout=0, **kwargs):
        super(Seq2SeqDecoder, self).__init__(**kwargs)
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.rnn = nn.GRU(embed_size + num_hiddens, num_hiddens, num_layers,
                          dropout=dropout)
        self.dense = nn.Linear(num_hiddens, vocab_size)

    def init_state(self, enc_outputs, *args):
        return enc_outputs[1]

    def forward(self, X, state):
        # The output `X` shape: (`num_steps`, `batch_size`, `embed_size`)
        X = self.embedding(X).permute(1, 0, 2)
        # Broadcast `context` so it has the same `num_steps` as `X`
        context = state[-1].repeat(X.shape[0], 1, 1)
        X_and_context = torch.cat((X, context), 2)
        output, state = self.rnn(X_and_context, state)
        output = self.dense(output).permute(1, 0, 2)
        # `output` shape: (`batch_size`, `num_steps`, `vocab_size`)
        # `state` shape: (`num_layers`, `batch_size`, `num_hiddens`)
        return output, state

decoder = Seq2SeqDecoder(vocab_size=10, embed_size=8, num_hiddens=16,
                         num_layers=2)
decoder.eval()
state = decoder.init_state(encoder(X))
output, state = decoder(X, state)
output.shape, state.shape
</code></pre>
<p>My doubt is In line  context = state[-1].repeat(X.shape[0], 1, 1)<br>
Why  state[-1] is taken ? Do we take the last layer’s output from GRU and give it to every input embedding of X by concatenating it?</p>
"Minimum" loss across several predictions

username: Andrei_C
cooked: <p>Greetings! Looking to get some help with the following task.</p>
<p>I’m training a model that can reconstruct occluded parts of an original image. The correct reconstruction can be multimodal in pixel color space, and the different correct representations can even be very very far apart (e.g. suppose the thing to be reconstructed is a ying-yang sign in black and white, and in the data half the time the sign has white on the left and black on the right, and half the time it has black on the left and white on the right). I’d like to allow my model to make N different predictions and for the loss function to be the minimum MSE distance (minimum across the N predictions) between the prediction and the target. That way the model can say something like “I think the missing piece is a ying-yang symbol either with black on the left and white on the right, or vice-versa” and be correctly rewarded for the most accurate of the predictions.</p>
<p>Is this kind of minimum loss function possible in Pytorch? If so, has anyone used it before and can share best practices? I can imagine that there are issues related to the fact that it’s kinked around the minimum threshold and the first derivative is not easily defined for autograd.</p>
<p>Thank you!</p>
Adding a classifier on the ouputs of the intermediate layers

username: pymon
cooked: <p>I have defined a simple resnet18 model with a two-node output layer. I extracted features from the last convolutional layer that might be useful to get image embeddings, now I would like to use a classifier with this vector.</p>
<p>How can I implement it? is it possible?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I’m not exactly sure what “classifier” means in this context, but your resnet18 should already contain the feature extractor (the conv, pool, relu layers) and a classifier (linear, relu, etc.).<br>
In the common setup the features (output of the last conv layer) would be flattened and passed to the classifier afterwards.</p>

username: Mona_Jalal
cooked: <p>If I gather correctly, the person means he/she/they wants to use the frozen features from the last layer of resnet18 (dimension 512) and then fine-tune it for their own problem (e.g., feeding these 512D embeddings to an MLP layer for classification) rather than training the ResNet18 from scratch, maybe because they have limited data compared to that of ImageNet on which original ResNet18 is trained on.</p>

username: pymon
cooked: <p>I did something similar to this example: <a href="https://kozodoi.me/python/deep%20learning/pytorch/tutorial/2021/05/27/extracting-features.html" rel="noopener nofollow ugc">Extracting Intermediate Layer Outputs in PyTorch | Nikita Kozodoi</a></p>
<p>I have obtained output from one of the intermediate layers of the resnet18 neural network and now I would like to try to add a linear classifier as a decision tree, KNN…<br>
I hypothesise that from the information obtained from intermediate layers I have features that I can then use in a classifier.</p>
Averaging some model parameters

username: Evan_Vogelbaum
cooked: <p>I have k classes, for each of which I have trained a model. I would like to create a new model whose architecture is identical and whose parameters are the <strong>weighted</strong> average of the corresponding parameters of each of my k models according to some specified distribution p (e.g. [0.1, 0.2, 0.7] for k=3).</p>
<p>My goal is to construct such a model, evaluate it on data, and then have gradient propagate back to the original models (the intermediate averaged model does not matter to me). Would someone be able to help me understand the best way to do this?</p>

username: ptrblck
cooked: <p>To simply average the parameters of multiple models you could use <a href="https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/2">this approach</a>. However, based on your description you would like to calculate the gradients for the parameters in the original models, so using the averaged <code>state_dict</code> wouldn’t work.<br>
Instead I think the right approach would be to create the new averaged tensors from all models and use the functional API for these new tensors.<br>
Something like this would work:</p>
<pre><code class="lang-python">model1 = nn.Linear(10, 10)
model2 = nn.Linear(10, 10)

new_weight = (model1.weight + model2.weight) / 2.
new_bias = (model1.bias + model2.bias) / 2.

x = torch.randn(1, 10)
out = F.linear(x, new_weight, new_bias)

out.mean().backward()

for name, param in model1.named_parameters():
    print(name, param.grad)
    
for name, param in model2.named_parameters():
    print(name, param.grad)
</code></pre>
<p>Although it seems to be quite cumbersome, so maybe someone else would have a better and cleaner approach.</p>

username: Evan_Vogelbaum
cooked: <p>Thanks so much! This certainly works for very simple models but I agree is extremely cumbersome for larger ones. Do you know why it is that <code>nn.Parameter</code>s don’t retain the gradient of the tensors which are used to create then?</p>

username: ptrblck
cooked: <p><code>nn.Parameter</code> would recreate a new parameter without any gradient history. While this new parameter would then get the gradients the two tensors used to create it wouldn’t.</p>

username: Evan_Vogelbaum
cooked: <p>Yeah; a bit of searching around showed me that <a href="https://discuss.pytorch.org/t/nn-parameter-doesnt-retain-grad-fn/29214">this is still an open issue</a>. Hopefully this is a feature added soon. For now the thread is closed. Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>

username: borgr
cooked: <p>If there are “real” models, this approach is infeasible (e.g. there is no “attention” functional and you need to reimplement everything anew). Is there another way someone figured for that?</p>
Structural similarity index with a non-rectangular window

username: witl
cooked: <p>Usually, Structural Similarity Index Metric (SSIM) is used for images, through a rectangular window. (<a href="https://piq.readthedocs.io/en/latest/modules.html#structural-similarity" class="inline-onebox" rel="noopener nofollow ugc">Class Interface — PyTorch Image Quality (PIQ) 0.6.0 documentation</a>)</p>
<p>One could use SSIM for sequences just by reshaping the sequence to <code>L x L</code> where <code>L = sqrt(length)</code>, such that <code>length</code> is the length of the sequence, and it is a perfect square number, i.e., a sequence of length <code>100</code> can be seen as an image <code>10 x 10</code>.</p>
<p>But is it possible to use SSIM for sequences without reshaping them? For example, through a window <code>1 x L</code>? Does it even make sense?</p>
<p>The reason I’m asking is that I’m doing self-supervised learning on sequences using 1D CNNs. Surprisingly, when I use SSIM instead of L2, even though I have to reshape the sequences to a rectangle, it gives me better results. I wonder if it would be possible to apply SSIM without the requirement of reshaping the sequences.</p>
<p><span class="mention">@Edit</span>: <a href="https://github.com/photosynthesis-team/piq/issues/306" class="inline-onebox" rel="noopener nofollow ugc">SSIM non-rectangular kernel size · Issue #306 · photosynthesis-team/piq · GitHub</a></p>
Implementation of RNN with an input which is a func of output at previous time-step

username: ahq1993
cooked: <p>Hi, I got a question regarding the implementation of RNNs. In my implementation, the input to RNN at next time-step is basically the output of the RNN at previous timestep augmented with some external input. However, I was going through pytorch examples on RNNs (e.g., image captioning) and it seems we need to input the whole sequence of inputs at once into RNN as the input tensor should be of shape (seq_len, batch, input_size)  which means my implementation wouldn’t be possible to implement. Is it correct or is there anyway to implement the architecture shown below?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e5dcfec3068b112cdb750babf2809ff23dbf27b0.png" alt="image" data-base62-sha1="wNsOSIBJibc70d4RTfBhGF16hb2" width="220" height="251"></p>

username: alexis-jacq
cooked: <p>Unfortunately, it seems that we have to make a for loop, giving at each step the last output as a new input (one example here: <a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html">http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a>).<br>
Maybe there is already a way to make this without a hand-made loop, but I don’t know how.</p>
<p>One of my projects is to implement such a “self-feeding LSTM”, going into the underlying C code. I need to find the time.</p>

username: ahq1993
cooked: <p>Hi,</p>
<p>Thank you for your reply. I am already trying to do through for loop, as follow:<br>
for epoch in range(args.num_epochs):<br>
print “epoch” + str(epoch)<br>
for i in range (0,int(dataset_train),args.batch_size):<br>
# Forward, Backward and Optimize<br>
RNN.zero_grad()<br>
for b in range(i,i+args.batch_size):<br>
inp,sg = get_input(dataset[i]) # inp=torch.cat((s1,sg))<br>
for l in range(0,len(targets)):<br>
inp=to_var(inp)<br>
outputs = RNN(inp)<br>
inp=torch.cat((sg,outputs))<br>
loss = criterion(outputs,targets[l])<br>
loss.backward()<br>
optimizer.step()</p>
<p>But it tells to give the input in the form (seq_length, batch, input_size).</p>

username: alexis-jacq
cooked: <p>You should make something like:</p>
<pre><code class="lang-python">input = Variable(torch.Tensor([[sos]*batch_size])
hidden, cell = RNN.init_hidden_cell(batch_size)
for di in range(MAX_LENGTH):
    output, (hidden,cell) = RNN(input, (hidden,cell))
    loss += criterion(output, targets[di,:])
    # for ex of a NLP output:
    # contains the probabilities for each word IDs.
    # The index of the probability is the word ID.
    _, topi = torch.topk(output,1)
    # topi = (batch*1)
    input = torch.transpose(topi,0,1)
</code></pre>
<p>And if you don’t use batchs, you can break the loop at the first “eos”.</p>

username: dizcza
cooked: <p>Any updates on this topic?</p>
<p>Is there a way to feed the output of a model (in my case, it’s RNN + Linear on top) as the input for the next timestamp without the explicit for-loop?</p>
How to freeze a specific layer in a custom network?

username: ckato
cooked: <p>when i try to freeze  a specific layer in a custom network,it report errors how to solve this problem?</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
def conv_bn(inp, oup, stride):<br>
return nn.Sequential(<br>
nn.Conv2d(inp, oup, 3, stride, 1, bias=False),<br>
nn.BatchNorm2d(oup),<br>
nn.ReLU(inplace=True)<br>
)<br>
def conv_dw(inp, oup, stride):<br>
return nn.Sequential(<br>
nn.Conv2d(inp, inp, 3, stride, 1, groups=inp, bias=False),<br>
nn.BatchNorm2d(inp),<br>
nn.ReLU(inplace=True),</p>
<pre><code>            nn.Conv2d(inp, oup, 1, 1, 0, bias=False),
            nn.BatchNorm2d(oup),
            nn.ReLU(inplace=True),
        )
    self.model = nn.Sequential(
        conv_bn(3, 32, 2),
        conv_dw(32, 64, 1),
        conv_dw(64, 128, 2),
        conv_dw(128, 128, 1),
        conv_dw(128, 256, 2),
        conv_dw(256, 256, 1),
        conv_dw(256, 512, 2),
        conv_dw(512, 512, 1),
        conv_dw(512, 512, 1),
        conv_dw(512, 512, 1),
        conv_dw(512, 512, 1),
        conv_dw(512, 512, 1),
        conv_dw(512, 1024, 2),
        conv_dw(1024, 1024, 1),
        nn.AvgPool2d(7),
    )
    self.fc = nn.Linear(1024, 1000)
def forward(self, x):
    x = self.model(x)
    x = x.view(-1, 1024)
    x = self.fc(x)
    return x
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c8b1c02835e3d038eaea889b44a8571ebff9d45.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c8b1c02835e3d038eaea889b44a8571ebff9d45" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c8b1c02835e3d038eaea889b44a8571ebff9d45_2_690x370.png" alt="image" data-base62-sha1="aV8klYwMPIG1Yv7Pjg8z0QCyknj" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c8b1c02835e3d038eaea889b44a8571ebff9d45_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c8b1c02835e3d038eaea889b44a8571ebff9d45_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c8b1c02835e3d038eaea889b44a8571ebff9d45.png 2x" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1328×714 84.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The error is raised by:</p>
<pre><code class="lang-python">mobilenet.weight.requires_grad = False
</code></pre>
<p>since <code>mobilenet</code> (and your <code>Net</code> implementation) doesn’t have a <code>.weight</code> attribute.<br>
If you want to freeze parameters, you would need to access the desired layer e.g.  via <code>mobilenet.model[0][0].weight</code>.</p>
Num_works cause insufficient shared memory

username: ran_wu
cooked: <p>I try to training imagenet on four 3090 graph cards with 4096 batchsize. However, when I set num_works bigger than 8, this erorr shut down my training program.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/345b7070abe44a18067c0e6339c2f2fa03029d14.png" data-download-href="https://discuss.pytorch.org/uploads/default/345b7070abe44a18067c0e6339c2f2fa03029d14" title="74ac40b68fb606a3667e532b34d2d1f"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/345b7070abe44a18067c0e6339c2f2fa03029d14.png" alt="74ac40b68fb606a3667e532b34d2d1f" data-base62-sha1="7taJDw9lBbyGr9nRwSzkgFJxG7i" width="690" height="394" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">74ac40b68fb606a3667e532b34d2d1f</span><span class="informations">1488×850 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ran_wu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9077845fa53e87f1e1022975ef47855440dfc783.png" data-download-href="https://discuss.pytorch.org/uploads/default/9077845fa53e87f1e1022975ef47855440dfc783" title="e964a23c84231bdeb2bcfafec533f4f"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9077845fa53e87f1e1022975ef47855440dfc783_2_690x127.png" alt="e964a23c84231bdeb2bcfafec533f4f" data-base62-sha1="kC0OPLcGS77yOV8AZdG9BhmyoWD" width="690" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9077845fa53e87f1e1022975ef47855440dfc783_2_690x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9077845fa53e87f1e1022975ef47855440dfc783_2_1035x190.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9077845fa53e87f1e1022975ef47855440dfc783_2_1380x254.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">e964a23c84231bdeb2bcfafec533f4f</span><span class="informations">1439×265 17.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Did you try to increase your shared memory limit or checked which limit is currently set?</p>

username: ran_wu
cooked: <p>I am working on a leased server. The shared memory of virtual environment is 20G. I can’t change this setting. Is there any method to avoid this issue?</p>

username: ptrblck
cooked: <p>Assuming you are indeed using all 20GB of it, I think the only workaround would be to reduce the shared memory usage by reducing the number of workers.</p>
Dataloader returns first batch but throws error on the second batch

username: Afaq_Saeed
cooked: <p>I am returning three tensors from the getitem function and it only returns the first batch and then start throwing error</p>
<p>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512])</p>
<pre><code>def __getitem__(self,idx):  


    semihardtriplet = possible_triplets[idx]#Choice the at about 75 % position      

    anchor_t    = self.path_to_tensor(semihardtriplet.anchor,self.device,False) 

    positive_t  = self.path_to_tensor(semihardtriplet.positive,self.device,False)

    negative_t  = self.path_to_tensor(semihardtriplet.negative,self.device,False)

      

    return anchor_t,positive_t,negative_t

 def path_to_tensor(self,img_path,device,check):

    # img       = Image.open(img_path)

    try:

      tensor       = read_image(img_path,ImageReadMode.RGB)

    except:

      print("Cant read Image",img_path)

  

    t = transforms.Compose([transforms.Resize((160,160))])

    tensor    = t(tensor)

    if check:       

      tensor    = tensor.unsqueeze(0)

    tensor    = tensor.float().to(device)

    # tensor    = tensor.permute(0,3,1,2)

    return tensor
</code></pre>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1-3e5pnGAz00hDAzDLbAcmGV7QC0HFlQq#scrollTo=H_6L8F7qzkzl&amp;uniqifier=1">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220228-060055-RC00_431397495" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1-3e5pnGAz00hDAzDLbAcmGV7QC0HFlQq#scrollTo=H_6L8F7qzkzl&amp;uniqifier=1" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1-3e5pnGAz00hDAzDLbAcmGV7QC0HFlQq#scrollTo=H_6L8F7qzkzl&amp;uniqifier=1" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Will tensor.to float32 to float16 lose precision or values?

username: karangoswamikenz
cooked: <p>Sorry if this is an obvious question. I know it will lose precision and accuracy when converting from float32 to float16 but is there a better way than to use tensor.to() function?</p>
<p>I have a 1x3x576x960 tensor that I want to convert from float32 to float16</p>
<p>I used</p>
<p>l<code>rs_prev_fp16 = lrs_prev.to(torch.float16)</code></p>

username: ptrblck
cooked: <p>I’m not sure I understand the question completely, but you are right that transforming a <code>float32</code> tensor to <code>float16</code> would lose precision.<br>
The <code>to()</code> operation is the standard way of converting a <code>dtype</code>. Alternatively, you could also use <code>a = b.half()</code> which would have the same effect.</p>

username: Eta_C
cooked: <p>About the accuracy, if your network architecture is <code>backbone + classification</code>, there may be a better choice that apply f16 only in <code>backbone</code> while training.</p>
<p>PS: sometimes, f16 may lead to <code>NaN</code> loss if there are BN layers in your network.</p>
How to calculate the percentage of correct answers for the top 5 individuals?

username: LEELANGU
cooked: <p>How to calculate the percentage of correct answers for the top 5 individuals?<br>
I made object recognition model using CNN and it worked.<br>
Already succeeded print top-5 answer but want more thing.<br>
I want to print their percentage individually.<br>
Please teach me how to calculate the percentage of correct answers for the top 5 individuals.</p>
<p>from PIL import Image</p>
<p>import numpy as np</p>
<p>import time</p>
<p>def imshow(input):</p>
<pre><code>input = input.numpy().transpose((1, 2, 0))

mean = np.array([0.485, 0.456, 0.406])

std = np.array([0.229, 0.224, 0.225])

input = std * input + mean

input = np.clip(input, 0, 1)

plt.imshow(input)

plt.show()
</code></pre>
<p>image = Image.open(‘0008.jpg’)</p>
<p>image = transform_resnet(image).unsqueeze(0).to(DEVICE)</p>
<p>with torch.no_grad():</p>
<pre><code>outputs = resnet34_3(image)

indices = torch.topk(outputs, 5).indices

#for x in indices[0]: print(x.item())

#_, preds = torch.topk(outputs, 5)

print('prediction: ')

for x in indices[0]:

    print(class_names[x.item()])

imshow(image.cpu().data[0])</code></pre>