akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GPU Memory usage, Windows

Samuel_Bachorik: <p>Hello, i am using two computers for training, one have linux and second windows. These PCs have same HW. When i try to run same training on windows witch batch 4 iam getting CUDA out of memory. On Linux i can have batch 16 and it works fine. To me it seems windows is using too much of GPU memory, is there any way to fix this ? Decrease GPU memory usage by windows ?</p>

ptrblck: <p>Are the GPUs completely empty in both setups or are you using e.g. a display or other applications in the Windows machine?<br>
If they are empty (check it via <code>nvidia-smi</code>), are you seeing the larger memory usage in all models on the Windows system?</p>
How to generate random pairs at each epoch?

kyy: <p>Hi, I am trying to generate a series of random pairs <code>{x_1, x_2}</code> from a specific dataset X, and I have done similar things before in TensorFlow by implementing a pair shuffling function in <code>on_epoch_end()</code> function of the TensorFlow generator, I am just wondering if there are similar functions for dataloader in PyTorch?</p>
<p>Note: setting <code>shuffle=True</code> in the dataloader would not work for me, since it only shuffle data in X, not generates random pairs.</p>
<p>Thank you!</p>

ptrblck: <p>How would these pairs be generated? Would samples be redrawn from the dataset or would all combinations be used?<br>
I think the proper way would be to write a custom <code>sampler</code> and implement your sampling logic there.<br>
The <code>sampler</code> passes its indices to <code>Dataset.__getitem__</code> to load each sample and could pass indices for each pair to it.</p>

kyy: <p>Thanks for the reply! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="112065">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Would samples be redrawn from the dataset or would all combinations be used?</p>
</blockquote>
</aside>
<p>Sample pairs are randomly drawn from the dataset.  I am not sure if I understand it correctly, would <code>Sampler</code> only output list of indices, rather than index pairs?</p>

conrad: <p>Could you just double the batch size of the dataloader?</p>
<p>E.g.:</p>
<pre><code class="lang-python"># assuming your data are images: batch of (128, 3, H, W)
batch_size = 128
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True, drop_last=True)

for batch in dataloader:
    # pairs is a tuple of (64, 3, H, W) and (64, 3, H, W)
    pairs = torch.split(batch, batch_size//2, dim=0) 
</code></pre>

kyy: <p>Yeah this should work, might also duplicate the dataset as well (so each point has chance to pair to itself), thank you so much!</p>
Using torchvision models in Lightning

BaruchG: <p>I’m just getting started with pytorch lightning.  Is there a way to use a precreated torchvision model in lightning?  All of the examples that I see build a new model which inherits from <code>LightningModule</code> such as at <a href="https://pytorch-lightning.readthedocs.io/en/stable/converting.html" rel="noopener nofollow ugc">How to organize PyTorch into Lightning — PyTorch Lightning 1.1.8 documentation (pytorch-lightning.readthedocs.io)</a> but I have a model, <code>net = models.segmentation.fcn_resnet50(pretrained=False, num_classes=91)</code> which I would like to make a lightning module.</p>
Pytorch Convolutions with Large Batch Sizes

George_Stepaniants: <p>Hi, I am currently working on a project where I have 10,000 images of size 100x100 and I have to convolve them with a filter of size mxm where m can range from 5 to 100.</p>
<p>I have found that when the size of my kernel is small m&lt;=10, convolving with 10,000 images takes tens of seconds and when my kernel is large m=100 then convolving with 10,000 images take several minutes.</p>
<p>Is there any way to speed up these computations for m small or large? I have tried using fft convolutions but they also don’t give me the desired speed up due to how large my batch size is. My goal is to perform this operation of convolution with 10,000 images in milliseconds if possible.</p>
<p>P.S. If this is not possible, I also know that each one of my 10,000 images that I convolve my kernel with is a rank one matrix if this is of any use.</p>
ValueError: emissions must have dimension of 3, got 2

Roberto: <p>I would like to classify 1d binary array (340 ) to 8 classes (each is list hot vector of length 8), I want to  add conditional random field at the top my model. I have 256 batch size . the output should be list of  length 8, each is probability for each class. I apply it as the following code . But I got an error from mthat line <code>loss = - self.crf(out, labels)</code>. The error is “emission must have dimension of 3 and got 2”. PS: out and labels are both [256,8]. Is there something wrong with my code because it is my first time to use conditional random field layer.</p>
<pre><code class="lang-auto">class MM(nn.Module):
    def __init__(self, M=1):
        super(MM, self).__init__()
        self.numb_label = 8
        #input layer
        self.layer1 = nn.Sequential(
            nn.Conv1d(340, 256, kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.layer2 = nn.Sequential(
           nn.Conv1d(256, 128, kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.layer3 = nn.Sequential(
            nn.Conv1d(128, 64, kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.drop1 = nn.Sequential(nn.Dropout())
        self.batch1 = nn.BatchNorm1d(64)
        # LSTM
        #self.multihead_attn1 = nn.MultiheadAttention(680, num_heads=2)
        self.lstm1=nn.Sequential(nn.LSTM(
        input_size=64,
        hidden_size=32,
        num_layers=1,
        bidirectional=True,
        batch_first= True))
        self.multihead_attn2 = nn.MultiheadAttention(32 * 2, num_heads=2)
        self.gru = nn.Sequential(nn.GRU(
            input_size=32*2,
            hidden_size=16,
            num_layers=2,
            bidirectional=True,
            batch_first=True))
        self.multihead_attn3 = nn.MultiheadAttention(16*2, num_heads=2)
        self.fc1 = nn.Linear(16 * 2, 8)
        self.sot = nn.LogSoftmax(dims=-1)
        self.crf = CRF(self.numb_label, batch_first=True)
     
    def forward(self, x,labels= None):
        #print(x.shape)
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.drop1(out)
        out = self.batch1(out)
        out = out.squeeze()
        out = out.unsqueeze(0)
        out,_ = self.lstm1(out)
        out, _ = self.multihead_attn2(out, out, out)
        out,_ = self.gru(out)
        out = out.squeeze()
        out = self.fc1(out)
        out = self.soft(out)
        loss = -self.crf(out, labels)
        output = loss
        print(out)
        pred_list = self.crf.decode(out)
        if output is None:
            return out
        else:
            return (output,pred_list)
</code></pre>

nitaifingerhut: <p>labels should be of size=batch_size with the actual class numbers and not one hot vectors</p>
Add convolutional layer to match other image size

anotherone_one: <p>Hello everyone!<br>
I have a network whose first layer is:</p>
<p>nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1), that expects a 3x96x96 image.</p>
<p>I want to add a previous layer in order to make the neural network accept images of 3x384x384 .</p>
<p>I tried to use the formula in the pytorch documentation, but I am not sure if I understood how to use them: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" class="inline-onebox" rel="noopener nofollow ugc">Conv2d — PyTorch 1.7.1 documentation</a></p>
<p>Thank you in advance</p>

nitaifingerhut: <p>You need:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html</a><br>
or:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html</a></p>
Dual-number dtype

Andrea_Cortis: <p>I was reading the paper</p>
<p><a href="https://arxiv.org/pdf/1912.01703.pdf" rel="noopener nofollow ugc">PyTorch: An Imperative Style, High-Performance Deep Learning Library</a></p>
<p>and found this quote:</p>
<p><em>PyTorch can be easily extended to perform forward-mode differentiation using array-level dual numbers</em></p>
<p>Can anybody point me to an example of how to define a tensor with a <code>dual number</code> dtype?</p>
<p>Thanks</p>
<p>Andrea</p>
How to use lr_scheduler in opacus?

hkz: <p>Hi,<br>
Thanks for this great project! I want to adjust the learning rate in opacus. When I create <code>scheduler</code> before attaching <code>optimizer</code> just like the following code.</p>
<pre><code class="lang-python">model = Net()
optimizer = SGD(model.parameters(), lr=0.05)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[30,80], gamma=0.1)
privacy_engine = PrivacyEngine(
    model,
    batch_size,
    sample_size,
    alphas=[10, 100],
    noise_multiplier=1.3,
    max_grad_norm=1.0,
)
privacy_engine.attach(optimizer)
</code></pre>
<p>It raises UserWarning: Seems like <code>optimizer.step()</code> has been overridden after learning rate scheduler initialization. <strong>What’s the potential risk in the above code?</strong></p>
<p>I also try to attach <code>optimizer</code> before learning rate scheduler initialization, thus this warning goes away. <strong>Is it necessary?</strong></p>
<p>Any help is appreciated!</p>

karthikprasad: <p>Hello <a class="mention" href="/u/hkz">@hkz</a>,<br>
Thank you for surfacing this quirk.</p>
<p>Opacus does indeed override the <code>optimizer.step()</code> when privacy engine is attached to it. In the current implementation, we perform the per-sample gradient clipping and subsequently call the original <code>step()</code> function of the optimizer (<a href="https://github.com/pytorch/opacus/blob/master/opacus/privacy_engine.py#L186-L195" class="inline-onebox" rel="noopener nofollow ugc">opacus/privacy_engine.py at master · pytorch/opacus · GitHub</a>). Therefore, your code should work correctly regardless of when you initialize the scheduler.</p>
<p>However, this is risky as it might not work correctly should the implementation of <code>optimizer.step()</code> change in a manner inconsistent with the scheduler’s expectations; although, I don’t see this happening.</p>
<p>Bottom line: heeding the warning is ideal and recommended. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Build Libtorch with AMD support

yngtodd: <p>Hey everyone,</p>
<p>Does anyone have advice for building <code>libtorch</code> with AMD support? I am currently poking around the <a href="https://github.com/pytorch/pytorch/blob/master/tools/amd_build/build_amd.py" rel="nofollow noopener">build_amd.py</a> and <a href="https://github.com/pytorch/pytorch/blob/master/tools/build_libtorch.py" rel="nofollow noopener">build_libtorch.py</a> scripts.</p>

rchavezj: <p>I am not giving advice but commenting to resurface this post for I am also curious. I have an AMD support GPU and keep getting the following error. I’m working on a Rust backend program using tch-rs  (PyTorch C++ Rust Bindings). <br></p>
<p>note: “cc” “-Wl,–as-needed” “-Wl,-z,noexecstack” “-m64” “-Wl,–eh-frame-hdr” “-L” "<br>
<br></p>
PyTorch support on Apple's M1 chip

SharanSMenon: <p>I was wondering if PyTorch will support Apple’s M1 chip and its 16 core Neural Engine. Tensorflow already supports the M1 GPU. I was hoping PyTorch would do the same.</p>

SharanSMenon: <p>Maybe you could use the <em>mlcompute</em> api like TensorFlow.</p>

SharanSMenon: <p>Are they even considering M1?</p>
Pining multiple cpu cores to each gpu processes

moh_sen: <p>I am using <code>torch.multiprocessing</code> to create processes for each gpu. The processes are mixture of cpu and gpu. Data X is pass to all processes. Other arguments passe to each process to tell to act in which portion of data X.</p>
<pre><code class="lang-auto">def foo(X, args, gpu):
  with torch.no_grad():
    t1 = time.time()
    classes_to_process = list(range(1,int(number_of_classes)+1))
    list_of_tuples = [(0,0)] * number_of_classes
    Q = mp.Queue()
    done_event = [ mp.Event()  for k in range(number_of_classes)]

    NG = min(  len(gpu) , number_of_classes  )
    assert NG &gt; 0
    processes = []
  
    for k in range(number_of_classes):
      r = k % NG
      p = mp.Process(target==my_process, args_my_process=([classes_to_process[k]], X, args, gpu[r], Q, done_event[k]))
      p.start()
      processes.append(p)
      if r == ( NG  - 1 ):
        for n in range(NG):
          L , D = Q.get()
          list_of_tuples[L-1] = (L, D)
          done_event[L-1].set()
        for p in processes:
          p.join()
        processes.clear()
    if r !=  ( NG  - 1 ):
      for n in range(r+1):
        L , D = Q.get()
        list_of_tuples[L-1] = (L, D)
        done_event[L-1].set()
      for p in processes:
        p.join()
    Y = OrderedDict(list_of_tuples)
    t2 = time.time()
    print("foo time = ", t2 - t1)
    del Q, done_event, L, D, p, list_of_tuples, processes
    return Y
</code></pre>
<p>It work without error. However, It assign a gpu and a cpu core to each processes. The single cpu core for each process slow done the whole program.</p>
<p>How can I assign multiple cpu cores and a single gpu to each process?</p>
Need help passing in 2D Matrix to Conv1D layer and outputting a softmax probability

torch_user_92: <p>Hi how’s it going? I’m trying to build a model that takes in a 2D Matrix as a single sample and outputs the row index that’s the best action by using softmax.</p>
<p>This is what I have so far:</p>
<pre data-code-wrap="python"><code class="lang-python">
names = ['Bob','Henry','Mike','Phil']

max_squat = [300,400,200,100]

max_bench = [200,100,225,100]

max_deadlift = [600,400,300,225]

strongest_worker_df = pd.DataFrame({'Name':names,'Max_Squat':max_squat,'Max_Bench':max_bench,'Max_Deadlift':max_deadlift})

</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab937a4bde6f4ae179e0e4a1705cbe8dcbd300b.png" alt="Screenshot (33)" data-base62-sha1="hvF4OSK2bzfZuimH5WGcgk1jL1p" width="458" height="222"></p>
<pre data-code-wrap="python"><code class="lang-python">
class Policy(nn.Module):  

   def __init__(self): 
       super(Policy, self).__init__() 
       self.layer1 = torch.nn.Conv1d(in_channels=4, out_channels=4, kernel_size=3, stride=1) 

   def forward(self, x): 

       x = self.layer1(x) 
       x = F.softmax(x,dim=1)

       return x 

   def act(self, state):

       state = state.float()
       value = self.forward(state)

       return value

policy = Policy()
result = policy.act(input_torch)

</code></pre>
<p>The result shape is torch.Size([1,4,1]). How do I get this to output a column vector instead?</p>
<p>Also, is Conv1D with kernel size equal to number of features per row the most logical approach to this input state representation?</p>
How to modify intermediate layer parameter gradients in DataParallel?

jerrybai1995: <p>I was hoping to print and manually verify the gradient of intermediate layer parameters when using DataParallel. An example is below:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.func = nn.Linear(3,3, bias=False)
        self.func2 = nn.Linear(3,3,bias=False)
        
    def forward(self, x):
        z = self.func(x)

        # I want the gradient of self.func2.weight here. I can 
        # get it when using a single GPU, but not in multi GPU setting...

        z = self.func2(z)
        return z 

net = Model()
para_net = nn.DataParallel(net)
xx = torch.randn(2,3).requires_grad_()
yy = para_net(xx)
loss = yy.mean()        # Just to produce a scalar
loss.backward()
</code></pre>
<p>Everything works fine when I’m using a single GPU (e.g., I can intercept with a <code>torch.autograd.Function</code> and manually modify the content of <code>self.func2.weight.grad</code>). However, once I use multiple GPUs by setting <code>CUDA_VISIBLE_DEVICES=0,1</code>, I can no longer access or modify it (e.g., if I intercept it, the printed <code>self.func2.weight.grad</code> will be <code>None</code>).</p>
<p>It’d be great if someone can help me resolve this issue, or point me to a solution!</p>

kikic: <p>Got the same issue here. Took me two days to realize it is with the DataParallel but not autograd.grad.</p>

kikic: <p>I wonder what kind of error did you get??</p>

jerrybai1995: <p>Well, the error was simply that my “modified gradient” didn’t go through in the backward pass of DataParallel; but I eventually found that having a backward hook solves this issue.</p>
Is it possible to do backward in this case?

chatterboy: <p>Let’s see an example with a simple architecture.</p>
<pre><code class="lang-auto">... -- conv1 -- conv2 -- ...
</code></pre>
<p>If the weights of the conv2 layer can be learned then it is possible to learn the weights of the conv1.<br>
Now, let’s see another example with a similar architecture except adding tile function.</p>
<pre><code class="lang-auto">... -- conv1 -- tile -- conv2 -- ...
</code></pre>
<p>In this case, is it feasible to apply backpropagation algorithm to conv1 layer?<br>
Thank you.</p>

albanD: <p>Hi,</p>
<p>If you’re <code>tile</code> function is differentiable, then yes. Could you give more details about that function?</p>

chatterboy: <p>An example for the function is <a href="https://pytorch.org/docs/master/generated/torch.tile.html" class="inline-onebox" rel="noopener nofollow ugc">torch.tile — PyTorch master documentation</a>. Btw, I found this link <a href="https://discuss.pytorch.org/t/are-all-operations-defined-in-torch-tensor-differentiable/4318" class="inline-onebox">Are all operations defined in torch.Tensor differentiable?</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Getting an error while built a bundle for ONNX model

mr2020: <p>Hi,<br>
I am getting an error while built a bundle for the ONNX model. I have downloaded the pre-trained model from here <a href="https://github.com/onnx/models/tree/master/vision/classification/mnist" rel="nofollow noopener">https://github.com/onnx/models/tree/master/vision/classification/mnist</a><br>
I have followed the instruction from here <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" rel="nofollow noopener">https://github.com/pytorch/glow/blob/master/docs/AOT.md</a><br>
My system OS is Windows 10.</p>
<p>I have used below command<br>
model-compiler -model=E:\Glow\onnx_mnist_model\mnist-8.onnx -emit-bundle=E:\Glow\converted_onnxModel -backend=CPU</p>
<p>It’s through the below error<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
F0813 17:15:02.143579  3536 Error.cpp:119] exitOnError(Error) got an unexpected ErrorValue:<br>
Error code: MODEL_LOADER_INVALID_PROTOBUF<br>
Error message: Failed to parse ModelProto<br>
Error return stack:</p>

jfix: <p>Have you been able to load the protobuf elsewhere? That error points to some issue due to the format of the protobuf itself, i.e. it isn’t being parsed correctly into C++ before Glow can even start taking a look at it.</p>
Dealing with tiling artifacts in boundaries (GAN)

Li-Yun: <p>I am dealing with some tiling artifacts which appear after the first epoch in an image-to-image translation task. I use a patch gan (70x70) as proposed in CycleGAN. Furthermore, as I read in the paper: <a href="https://arxiv.org/pdf/1611.07004.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1611.07004.pdf</a>, these artifacts come up when a vanilla discriminator or a one with a smaller receptive field is used. As epochs grow, this phenomenon is getting gradually worse.</p>
<p>Any ideas would be appreciated.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/650fafd6f4eb2b10c33f1acefbdf82ab60598484.png" alt="image" data-base62-sha1="eq1PFVmDlXDkw6AUgQRmOnFmozG" width="214" height="199"></p>
Torch assign independent tasks to different gpus

nikolaos_karaliolios: <p>i have a script in which train two models in an independent way, on the same dataset and using the same initialization</p>
<p>pseudo code would read something like</p>
<ol>
<li>initialize warm-up model</li>
<li>train and store warm-up model</li>
<li>load warm-up model and train using 1st strategy</li>
<li>load warm-up model and train using 2nd strategy</li>
<li>compare performances</li>
</ol>
<p>is there a way to assign in the script the training to different GPUs so that steps 3&amp;4 happen in parallel without calling a different .py file for each step?</p>

janhenr: <p>Hi,<br>
I am not aware of any way to do this in PyTorch. However, it seems like your usecase is very easily fixed writing a bash script. You can use the same .py script using different arguments  and call it in  a bash script as such:</p>
<pre><code class="lang-auto">python train.py --warmup --model_out "weights.ckpt" # warmup
python train.py --strategy_1 --model_in "weights.ckpt" --model_out "strategy_1.ckpt"
python train.py --strategy_2 --model_in "weights.ckpt" --model_out "strategy_2.ckpt"
python train.py --compare --models_to_compare  "strategy_1.ckpt" "strategy_2.ckpt"
</code></pre>
<p>Then you just run the above script with <code>bash run.sh</code>.<br>
Hope this helps!</p>

nikolaos_karaliolios: <p>that seems to be the only way to do it, but the problem is that i want to call the script many times and with different values of hyperparameters, so it gets way too complicated…</p>
<p>thanks!</p>
Do we have coding tutorial about how to train video dataset?

JonathanSum: <p>I have heard nvidia has their own library. I have also heard <a href="http://torch.vision.io" rel="noopener nofollow ugc">torch.vision.io</a> does this. But I don’t know what I should do.</p>

mathematics: <p><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji only-emoji" alt=":grinning:"><br>
Hope to make it soon ?</p>

raivokoot: <aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="15" height="15">
      <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/29647087?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="59" height="59">

<h3><a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">RaivoKoot/Video-Dataset-Loading-Pytorch</a></h3>


  <p><span class="label1">Generic PyTorch Dataset Implementation for Loading, Preprocessing and Augmenting Video Datasets</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Initialize weights of this model

Samuel_Bachorik: <p>Hello please, iam using this model for segmentation and i need to initialize weights by <code>torch.nn.init.</code> <code>xavier_uniform_</code> can someone show me please how can i do it on this model ?</p>
<pre><code class="lang-auto">def double_conv(in_channels, out_channels):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, 3, padding=1),
        torch.nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(out_channels, out_channels, 3, padding=1),
        torch.nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True)
    )

class Model(nn.Module):
    def __init__(self, n_class=5):
        super(Model).__init__()

        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        self.dconv_down1 = double_conv(3, 64)
        self.dconv_down2 = double_conv(64, 128)
        self.dconv_down3 = double_conv(128, 256)
        self.dconv_down4 = double_conv(256, 512)

        self.maxool = nn.MaxPool2d(2)
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

        self.dconv_up3 = double_conv(256 + 512, 256)
        self.dconv_up2 = double_conv(128 + 256, 128)
        self.dconv_up1 = double_conv(128 + 64, 64)

        self.conv_last = nn.Conv2d(64, n_class, 1)

    def forward(self, x):
        conv1 = self.dconv_down1(x)
        x = self.maxpool(conv1)

        conv2 = self.dconv_down2(x)
        x = self.maxpool(conv2)

        conv3 = self.dconv_down3(x)
        x = self.maxpool(conv3)

        x = self.dconv_down4(x)

        x = self.upsample(x)
        x = torch.cat([x, conv3], dim=1)

        x = self.dconv_up3(x)
        x = self.upsample(x)
        x = torch.cat([x, conv2], dim=1)

        x = self.dconv_up2(x)
        x = self.upsample(x)
        x = torch.cat([x, conv1], dim=1)

        x = self.dconv_up1(x)

        out = self.conv_last(x)

        return out
</code></pre>

ptrblck: <p>You could write a <code>weight_init</code> method and <code>apply</code> it on the model:</p>
<pre><code class="lang-python">def weight_init(m):
    if isinstance(m, nn.Conv2d):
        print('initializing conv2d weight')
        torch.nn.init.xavier_uniform_(m.weight)

model.apply(weight_init)
</code></pre>
Create a new layer with transpose matrix

Alfred: <p>Hi, I have to define a new layer, which does the following:</p>
<p>Given an input x (vector with N elements), and a NxM  matrix  W, I want as an output W^T(ReLU(Wx)), where W^T is the transpose of W. What I have is the following:</p>
<pre><code class="lang-auto">class AML(nn.Module):
    def __init__(self, in_features: int, out_features: int,n: int) -&gt; None:
        super(AML, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        param = torch.Tensor(out_features, in_features)
        param_t = torch.transpose(param,0,1)
        self.weight = Parameter(param)
        self.weight_t = Parameter(param_t)
        self.relu = ReLu(n)

    def forward(self, input: Tensor) -&gt; Tensor:
        out = F.linear(input, self.weight)
        out = self.relu(out)
        out = F.linear(out,self.weight_t)
        return out
</code></pre>
<p>But I keep getting random guessing results even on very simple tasks. Is there anything wrong is my code?</p>

ptrblck: <p>The code looks generally alright, but I would not use <code>torch.Tensor</code> to create the initial parameter, as it will use uninitialized memory and will contain whatever was stored in the memory you are using.<br>
This would also mean that it might contain NaN values etc.<br>
Use the factory methods, such as <code>torch.randn</code> etc. instead.</p>
How can I get the Hidden Embeddings from the 2nd last fully connected layer for t-SNE visualization?

shakeel608: <p>how can we do this in pytorch?<br>
After training the model,   get the learned representations from the 2nd last FC layer for t-SNE visualization. The Model is defined below</p>
<pre><code class="lang-auto">class VisualNet(nn.Module):
    def __init__(self):
        super(VisualNet, self).__init__()
        self.conv1 = nn.Conv2d()  #Ignore the function values
        slef.conv2 = nn.Conv1d()
        slef.conv3 = nn.Conv1d()
        slef.conv4 = nn.Conv1d(512,512)

       self.fc1   = nn.Linear(512,256)
        self.fc2   = nn.Linear(256, 8)

    def forward(self, x):
        x = self.conv1(self.conv2(self.conv3(self.conv4(x)))
        x_fc1 = self.fc1(x)
        x = self.fc2(x_fc1)        

        return x

</code></pre>
<pre><code class="lang-auto">model = VisualNet()

</code></pre>
<p>After finishing Training, How can I get <strong><code>x_fc1</code> embeddings</strong><br>
for t-SNE ??</p>

ptrblck: <p>Assuming you are looking for the output activations of <code>self.fc1</code> (unsure what embeddings would mean in this context), you could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
AMP initialization with fp16

JIALI_MA: <p>I’d like to know how should I initialize the model if the model is separated into several modules.<br>
For example:</p>
<pre><code class="lang-auto">model = def_model()  # backbone layers
model_loss = def_loss()  # FC classifier 
params =  list(model.parameters()) + list(model_loss.parameters())  # all the parameters
optimizer = torch.optim.SGD(params, lr)
</code></pre>
<p>Then if I want to train the model using apex fp16, which operation is correct?</p>
<ol>
<li>Init all the sub-modules</li>
</ol>
<pre><code class="lang-auto">[model, model_loss], optimizer = amp.initialize([model, model_loss], optimizer, opt_level="O1")
</code></pre>
<ol start="2">
<li>Only init the main module</li>
</ol>
<pre><code class="lang-auto">model, optimizer = amp.initialize(model, optimizer, opt_level="O1")
</code></pre>
<p>Since the params already contains all the parameter, it is able to train using both methods. But what will happen if I only include partial modules in the init operation?</p>

ptrblck: <p>We recommend to switch to the native mixed-precision training utility used via <code>torch.cuda.amp</code> as described <a href="https://pytorch.org/docs/stable/amp.html">in this doc</a>.</p>

JIALI_MA: <p>Thanks for the reply.<br>
Actually we switched from torch.cuda.amp to apex due to some internal reasons.<br>
I studied the doc for apex, but found no instruction on this aspect. Could you please provide some suggestions regarding the above issue? Thanks!</p>

ptrblck: <p>The passed modules to <code>amp.initialize</code> will get patched based on the used <code>opt_level</code>. If you are not passing all modules to it, autocasting might not work and/or the code might break in another unexpected way.<br>
Note that <code>apex.amp</code> won’t get any new features anymore, as the current development is focused on the native implementation.</p>
Use different dataloaders in different epochs?

kyy: <p>Hi, I am trying to train a model using different dataloaders in different epochs.  The main purpose of doing this (instead of simply concat different datasets) is that I am interested in applying different <code>collate_fn</code> for different epochs, which cannot be done with simple data concatenation.</p>
<p>One possible dirty approach I think of is to use if else condition to manually switch between different dataloaders, something like:</p>
<pre><code class="lang-auto">if idx_epoch % 4 in (0, 1):
    # use dataloader A for training
elif idx_epoch % 4 in (2,3):
    # use dataloader B for training
...
</code></pre>
<p>I am wondering if anyone know better ways to do this?</p>
<p>Thank you!</p>
Best Way to Reweight Faster R-CNN Loss?

ai_padawan: <p>I’m trying to figure out what’s the best way to reweight the component losses in Faster R-CNN.</p>
<p>As far as I can tell this part of the code returns the component loss (<a href="https://github.com/pytorch/vision/blob/af97ec2f4c9daac091b9a87355c4f22d37488004/torchvision/models/detection/roi_heads.py#L16" class="inline-onebox" rel="noopener nofollow ugc">vision/roi_heads.py at af97ec2f4c9daac091b9a87355c4f22d37488004 · pytorch/vision · GitHub</a>):</p>
<pre><code class="lang-auto">def fastrcnn_loss(class_logits, box_regression, labels, regression_targets):
    # type: (Tensor, Tensor, List[Tensor], List[Tensor]) -&gt; Tuple[Tensor, Tensor]
    """
    Computes the loss for Faster R-CNN.
    Args:
        class_logits (Tensor)
        box_regression (Tensor)
        labels (list[BoxList])
        regression_targets (Tensor)
    Returns:
        classification_loss (Tensor)
        box_loss (Tensor)
    """

    labels = torch.cat(labels, dim=0)
    regression_targets = torch.cat(regression_targets, dim=0)

    classification_loss = F.cross_entropy(class_logits, labels)

    # get indices that correspond to the regression targets for
    # the corresponding ground truth labels, to be used with
    # advanced indexing
    sampled_pos_inds_subset = torch.where(labels &gt; 0)[0]
    labels_pos = labels[sampled_pos_inds_subset]
    N, num_classes = class_logits.shape
    box_regression = box_regression.reshape(N, box_regression.size(-1) // 4, 4)

    box_loss = det_utils.smooth_l1_loss(
        box_regression[sampled_pos_inds_subset, labels_pos],
        regression_targets[sampled_pos_inds_subset],
        beta=1 / 9,
        size_average=False,
    )
    box_loss = box_loss / labels.numel()

    return classification_loss, box_loss
</code></pre>
<p>I can add some custom weights to change the importance of classification vs box loss.</p>
<p>What’s the best way of modify this function, but without needing to create a separate ecosystem of custom FasterRCNN classes and objects?</p>
Deep copying PyTorch modules

juliohm: <p>Hi, I am new to PyTorch, and was wondering if there is an API defined for deep copying Modules? I have a function that accepts a Module as input, and trains it. Because I don’t want to keep track of object’s state, I am planning to just deep copy the trained model and save it somewhere else in a database.</p>
<p>Is it safe to just do <code>copy.deepcopy(mymodel)</code>?</p>

jpeg729: <p>If I had to deep copy a model, I would do something like this…</p>
<pre><code class="lang-auto">model_copy = type(mymodel)() # get a new instance
model_copy.load_state_dict(mymodel.state_dict()) # copy weights and stuff
</code></pre>
<p>I can’t see how that would help you “save it somewhere else in a database”. I assume you have read the <a href="http://pytorch.org/docs/0.3.0/notes/serialization.html" rel="nofollow noopener">docs on serialising models</a> and the recommendations and warnings on that page.</p>

Jayant_Parashar: <p>In type(mymodel)(), are arguments also copied by default from mymodel instance? Or do we need to make sure the arguments match the mymodel instance?</p>

asalimih: <aside class="quote no-group" data-username="jpeg729" data-post="2" data-topic="13514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<pre><code class="lang-auto">model_copy = type(mymodel)() # get a new instance
model_copy.load_state_dict(mymodel.state_dict()) # copy weights and stuff
</code></pre>
</blockquote>
</aside>
<p>if there are arguments in the <code>__init__</code> function of <code>mymodel</code> you must pass them too. <code>type(mymodel)(args)</code></p>

Gayal_Shamane: <p>what if I do</p>
<p><strong>model_copy.load_state_dict(mymodel.state_dict()).to(device=‘cpu’)</strong> ?</p>
<p>All I want is to use an instance of a model as an input to another function while training. Here I do not need any gradient flow for this opperation.</p>
Shapenet v1 and shapenet v2 transformation

lycaenidae: <p>hi, I would like to find the pose that converts shapenet v1 model to shapenet v2 model.<br>
The difference is front is -z in the v2, but x in the v1.<br>
Somehow I could not figure out the pose or flipping axis</p>
<p>How can I do that?</p>
Best way to backprop() with respect to the input variable

Nreon: <p>Hello everyone, I come to you in time of great need,</p>
<p><strong>I need only the input for my model to be affected by backpropagation.</strong></p>
<p>It is an experiment. I know it might sound silly, but I was wondering how to achieve this.</p>
<p>I tried setting <code>x.requires_grad = True</code> for my input variable and then <code>layer.requires_grad = False</code> for all the layers, but it seems that the computational graph isn’t created because of this.</p>
<p>Should I let requires_grad be true for everything and then manually set gradients to zero for all the layers?</p>
<p>How can I do that? Or is there a better way?</p>
<p>Thank you very much in advance for your help and patience. I’m a beginner.</p>

albanD: <p>Hi,</p>
<p>If you only want the gradient for the input, the simplest thing I can think of is:</p>
<pre><code class="lang-python">model = # Your model
input = # Your input

# Make sure input requires grad
input.requires_grad_(True)

# Do the forward as usual
output = model(input)
loss = crit(output, label)

# Ask only for the grad wrt the input:
grad_input, = autograd.grad(loss, input)
</code></pre>

Nreon: <p>This solved it! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Getting different results with code, but i think they should be the same

fred: <p>I’m getting different results when using these two pieces of code interchangeably, but i don’t know why. They should give the same results. It’s the code for my optimizer configuration.</p>
<h2> Code 1 </h2>
<code>
return torch.optim.Adam(model.parameters(), weight_decay=1e-6, lr=0.001)
</code>
<h2> Code 2</h2>
<code>
<pre><code class="lang-auto">    updown_weights = []
    updown_bias = []
    for m in model.modules():
        if isinstance(m, mcnnsae_parts.DownConv) or isinstance(m, mcnnsae_parts.UpConv):
            for name, p in m.named_parameters():
                if 'bias' in name:
                    updown_bias.append(p)
                if 'weight' in name:
                    updown_weights.append(p)

    return torch.optim.Adam([
        {'params': updown_bias, 'weight_decay': 1e-6},
        {'params': updown_weights, 'weight_decay': 1e-6}],
        lr=0.001, weight_decay=1e-6)
</code></pre>
</code>

ptrblck: <p>How are you comparing these different approaches? Are you seeding the code or reusing the same <code>state_dict</code>s? Is each approach deterministic but yields different results when compared to each other?<br>
In the latter case how large are the differences?</p>

fred: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="111973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>ow are you comparing these different approaches? Are you seeding the code or reusing the same <code>state_dict</code>s? Is each approach deterministic but yields different results when compared to each other?<br>
In the latter case how large are the differences?</p>
</blockquote>
</aside>
<p>I’m seeding the code using the following:</p>
<pre><code class="lang-auto">    seed=1024
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
</code></pre>

ptrblck: <p>Additionally to these flags you could also use <code>torch.set_deterministic(True)</code>, which should raise errors for known non-deterministic methods as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>.</p>
Is it possible to pass two samplers in a dataloader?

kyy: <p>Hi all,</p>
<p>This is related to <a href="https://discuss.pytorch.org/t/how-to-generate-random-pairs-at-each-epoch/112065" class="inline-onebox">How to generate random pairs at each epoch?</a>, I am thinking that one possible solution to that problem is to pass two samplers (each generate a random index, so we have a random index pair) to the dataloader.  I am just wondering if this is possible?  Thank you!</p>
Modifing nn.Module._load_from_state_dict

tshmak: <p>Hi, I’m trying to modify <code>nn.Module._load_from_state_dict()</code> (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module.load_state_dict" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.module — PyTorch 1.7.1 documentation</a>).</p>
<p>My class inherits <code>nn.Module</code>:</p>
<pre><code class="lang-auto">class foo(nn.Module):

    def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict,
                          missing_keys, unexpected_keys, error_msgs):


        # The following lines are copied from the first few lines in the original _load_from_state_dict()
        for hook in self._load_state_dict_pre_hooks.values():
            hook(state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs)

        persistent_buffers = {k: v for k, v in self._buffers.items() if k not in self._non_persistent_buffers_set}
        local_name_params = itertools.chain(self._parameters.items(), persistent_buffers.items())
        local_state = {k: v for k, v in local_name_params if v is not None}
        
        pdb.set_trace()

</code></pre>
<p>However, I found that it doesn’t seem so simple. I found that <code>self._load_state_dict_pre_hooks</code>, <code>persistent_buffers</code>, <code>local_name_params</code> are now empty dictionaries when I stepped into it in <code>pdb.set_trace()</code>.</p>
<p>Can someone explain why this might be and how I may modify <code>_load_from_state_dict()</code>?</p>

albanD: <p>Hi,</p>
<p>All functions in python that start with <code>._foo</code> are considered private and not part of the user API (you can see it is not documented).<br>
Why do you want to modify that function?</p>

tshmak: <p>Currently, if the dimensions of the tensor don’t match, then <code>_load_from_state_dict()</code> will ignore that tensor. However, I want it to try and copy the parameters anyway. For example, I may change the dimension of a tensor from (10, 100) to (10, 110). I would still like to copy over the first 100 columns of the tensor as starting values.</p>

albanD: <p>I think the simplest way to do that is to pre-process your state_dict before giving it to the <code>Module.load_state_dict</code> function. That will make sure that your code keeps working with new versions of PyTorch as internal API might change without notice.</p>
Select batches according to label in data column

Bruno_Morgado: <p>I have a dataset like such:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>index</th>
<th>tag</th>
<th>feature1</th>
<th>feature2</th>
<th>target</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>tag1</td>
<td>1.4342</td>
<td>88.4554</td>
<td>0.5365</td>
</tr>
<tr>
<td>2</td>
<td>tag1</td>
<td>2.5656</td>
<td>54.5466</td>
<td>0.1263</td>
</tr>
<tr>
<td>3</td>
<td>tag2</td>
<td>5.4561</td>
<td>845.556</td>
<td>0.8613</td>
</tr>
<tr>
<td>4</td>
<td>tag3</td>
<td>6.5546</td>
<td>8.52545</td>
<td>0.7864</td>
</tr>
<tr>
<td>5</td>
<td>tag3</td>
<td>8.4566</td>
<td>945.456</td>
<td>0.4646</td>
</tr>
</tbody>
</table>
</div><p>The number of entries in each tag is not always the same.</p>
<p>And my objective is to load only the data with a specific tag or tags, so that I get only the entries in <code>tag1</code> for one mini-batch and then <code>tag2</code> for another mini-batch if I set <code>batch_size=1</code>. Or for instance <code>tag1</code> and <code>tag2</code> if I set <code>batch_size=2</code></p>
<p>The code I have so far disregards completely the <code>tag</code> label and just chooses the batches randomly.</p>
<p>I built the datasets like such:</p>
<pre><code># features is a matrix with all the features columns through all rows
# target is a vector with the target column through all rows
featuresTrain, targetTrain = projutils.get_data(train=True, config=config)
train = torch.utils.data.TensorDataset(featuresTrain, targetTrain)
train_loader = make_loader(train, batch_size=config.batch_size)
</code></pre>
<p>And my loader (generically) looks like this:</p>
<pre><code>def make_loader(dataset, batch_size):
loader = torch.utils.data.DataLoader(dataset=dataset,
                                     batch_size=batch_size, 
                                     shuffle=True,
                                     pin_memory=True,
                                     num_workers=8)
return loader
</code></pre>
<p>Which I then train like this:</p>
<pre><code>for epoch in range(config.epochs):
    for _, (features, target) in enumerate(loader):
        loss = train_batch(features, target, model, optimizer, criterion)
</code></pre>
<p>And the <code>train_batch</code>:</p>
<pre><code>def train_batch(features, target, model, optimizer, criterion):
features, target = features.to(device), target.to(device)

# Forward pass ➡
outputs = model(features)
loss = criterion(outputs, target
return loss</code></pre>
Why do I get the error "TypeError: object of type 'NoneType' has no len()"?

xsquared: <p>hello,</p>
<p>I am getting the error “TypeError: object of type ‘NoneType’ has no len()” and I do not know why it is happening.</p>
<p>This error is triggered by the training loop which is:</p>
<pre><code class="lang-auto">def training_loop(
        n_epochs,
        optimizer,
        model,
        loss_fn,
        train_loader):
    for epoch in range(1, n_epochs + 1):
        loss_train = 0.0
        for i, (imgs, labels) in enumerate(train_loader):

            outputs = model(imgs)

            loss = loss_fn(outputs, labels)

            optimizer.zero_grad()

            loss.backward()

            optimizer.step()

            loss_train += loss.item()

        if epoch == 1 or epoch % 10 == 0:
            print('{} Epoch {}, Training loss {}'.format(
                datetime.datetime.now(),
                epoch,
                loss_train / len(train_loader)))
</code></pre>
<p>here is the rest of the relevant code:</p>
<pre><code class="lang-auto">
def instantiate_data(data_path, isTraining):
    """A higher-order procedure that instantiates training or val datasets.

      It transforms each PIL image to a Pytorch Tensor.

      Moreover, it normalizes the data.

      DATA_PATH -&gt; DATASET"""
    return datasets.MNIST(data_path,
                          train=isTraining,
                          download=True,
                          transform=transforms.Compose([
                              transforms.ToTensor(),
                              transforms.Normalize((0.0839, 0.2039, 0.1042),
                                                   (0.2537, 0.3659, 0.2798))]))

def instantiate_training_data(data_path):
    """This function instantiates the training dataset."""
    instantiate_data(data_path, True)
model = Net()

data_path = "../Mnist/"

print("instatiate data")
mnist = instantiate_training_data(data_path)
print("training loader")
train_loader = torch.utils.data.DataLoader(mnist, batch_size=64)
print("optimizer")
optimizer = optim.SGD(model.parameters(), lr=1e-2)
print("loss")
loss_fn = nn.CrossEntropyLoss()
print("training loop")

training_loop(
    n_epochs = 100,
    optimizer = optimizer,
    model = model,
    loss_fn = loss_fn,
    train_loader = train_loader,
    )

</code></pre>
<p>here is the error message:<br>
<code>File "/home/sphere3/classify_mnist/src/train.py", line 11, in training_loop     for i, (imgs, labels) in enumerate(train_loader):</code></p>
<p>thanks!</p>

Erhan: <p>You must add return term beginning of the instantiate_data(data_path, True) in the ‘instantiate_training_data’ function.</p>