akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Empy parameters in optimizer.param_groups

kuzand: <p>I have encountered a strange behavior. The following piece of code works as expected</p>
<pre><code class="lang-auto">model = models.resnet152(pretrained=True)
</code></pre>
<pre><code class="lang-auto">params_to_update = [{'params': model.fc.parameters(), 'lr': 0.001}]

optimizer = optim.Adam(params_to_update) 

print(optimizer.param_groups)
</code></pre>
<p>However if I do something like this:</p>
<pre><code class="lang-auto">params_to_update = [{'params': model.fc.parameters(), 'lr': 0.001}]

for d in params_to_update:
    for param in d['params']:
        print(param.requires_grad)

optimizer = optim.Adam(params_to_update) 

print(optimizer.param_groups)
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">True
True
[{'params': [], 'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}]
</code></pre>
<p>where the <code>'params': []</code> is empty now (before inserting the for-loops it was not empty). I find it very strange since I haven’t altered the <code>params_to_update</code> dictionary and just made some print statements in the for-loop.</p>

tom: <aside class="quote no-group quote-modified" data-username="kuzand" data-post="1" data-topic="34579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kuzand/48/8268_2.png" class="avatar"> kuzand:</div>
<blockquote>
<p>params_to_update = [{‘params’: model.fc.parameters(), ‘lr’: 0.001}]</p>
</blockquote>
</aside>
<p>Try</p>
<pre><code class="lang-auto">params_to_update = [{'params': list(model.fc.parameters()), 'lr': 0.001}]
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

kuzand: <p>Thank you, that works! However I was wondering why is that happening and why using the list helps?</p>

tom: <p>Without the list cast you have an iterator that you can only consume once. If you do that in the for loop, there won’t be anything left when you pass it to the optimizer.</p>
<p>Best regards</p>
<p>Thomas</p>

Upgrade_Yourself: <p>Hi, is this a general property of all iterators or is it something that’s special for model.parameters() ?</p>

tom: <p>No that’s iterators in general. I believe that if one would want to avoid that, you’d need something like generators (e.g. what datasets do), where you can get multiple (“rewound to start”) iterators from a generator, but then you’d probably have other drawbacks.</p>
Anything like transformer for TensorDataset?

siberiamark: <p>Up to now I’ve figure out two approaches to use a custom dataset(with the standard DataLodar).<br>
The first is to use ImageFolder, somehow  similar to creating a filelist and get access to data. Transformations such as randomcrop and scaling is convenient with that scheme,but  I guess it  is I/O-consuming.</p>
<blockquote>
<p>dataset = trochvision.datasets.ImageFolder(root=‘main_dir’,transformer=torchvision.transform.Compose([…])</p>
</blockquote>
<p>So the second scheme is to read in a bunch of data into the memory at a time into tensors:</p>
<blockquote>
<p>dataset = torch.utils.data.TensorDataset(data_tensor,label_tensor)</p>
</blockquote>
<p>Anything like the transformer in the first approach for TensorDataset？</p>

apaszke: <p>No, we don’t support that at the moment but it would make sense to add that. We’re going to be rewriting some transforms soon and we’ll do that then. Thanks for a suggestion!</p>

adelaide: <p>Is transform available for TensorDataset now？</p>

zym1010: <p>this looks like something you can integrate into.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/ncullen93/torchsample" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/13004360?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/ncullen93/torchsample" target="_blank" rel="nofollow noopener">ncullen93/torchsample</a></h3>

<p>High-Level Training, Data Augmentation, and Utilities for Pytorch - ncullen93/torchsample</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


RylanSchaeffer: <p><a class="mention" href="/u/apaszke">@apaszke</a> any updates on adding transforms for TensorDataset?</p>
nn.Sequential(..) vs without nn.Sequential(..) gives totally different things

telecom: <pre><code class="lang-auto">def draft_number_1(self, x):
    layers = [nn.Conv2d(1, 16, kernel_size=5, padding=1), 
    nn.ReLU(), 
    nn.MaxPool2d(kernel_size=2, stride=1), 
    nn.Conv2d(16, 32, kernel_size=5, padding=1), 
    nn.ReLU(),  
    nn.MaxPool2d(kernel_size=2, stride=1)]
    return nn.Sequential(*layers)(x)
</code></pre>
<pre><code class="lang-auto">def draft_number_2(self, x):
    x = nn.Conv2d(1, 16, kernel_size=5, padding=1)(x)
    x = nn.ReLU()(x)
    x = nn.MaxPool2d(kernel_size=2, stride=1)(x)
    x = nn.Conv2d(16, 32, kernel_size=5, padding=1)(x)
    x = nn.ReLU()(x)
    x = nn.MaxPool2d(kernel_size=2, stride=1)(x)
    return x
</code></pre>
<p>Hello, everytime I run a training with the draft number 1, I will get an accuracy of ~0.8 after 1000 iterations, but if I try with draft number 2, I will get only an accuracy of ~0.4 after same number of iterations.</p>
<p>I repeated again and again and I get always these metrics (0.8 and 0.4). But they are supposed to be (modulo some epislon due to the random initialization of weights) “nearly” equals. What could be the problem? oO</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>If you are using both methods as the forward pass, I’m wondering, how it is working in any approach at all.<br>
In both methods you are reinitializing all layers from scratch, so that no real training will happen.</p>
Even after freezing parameters, accuracy is changing

Kim_KA: <p>if epoch &gt;=10:<br>
model.train(False)<br>
for param in model.parameters():<br>
print(param)<br>
param.requires_grad = False</p>
<p>I have freeze the network after some epoch but forward pass results after that epoch are still changing (training and validation accuracy is changing).</p>
<p>Please help me how to overcome.</p>
<p>Thanks in advance.</p>

ptrblck: <p>Could you post the model architecture, so that we can have a look?</p>

Kim_KA: <p>It is a simple binary classifier:</p>
<p>(classifier): SimpleClassifier(<br>
(main): Sequential(<br>
(0): Linear(in_features=1024, out_features=512, bias=True)<br>
(1): ReLU()<br>
(2): Dropout(p=0.5, inplace=True)<br>
(3): Linear(in_features=512, out_features=2, bias=True)<br>
)<br>
)</p>

Kim_KA: <p>Fully connected 1024 --&gt; 512<br>
ReLU<br>
Dropout<br>
512 --&gt; 2</p>
<p>Thanks for response.</p>

ptrblck: <p>Thanks for the code snippet.<br>
The model correctly reproduces the same results in <code>eval</code> mode using a static output:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(1024, 512),
    nn.ReLU(),
    nn.Dropout(0.5, inplace=True),
    nn.Linear(512, 2)
)

model.train(False)
for param in model.parameters():
    param.requires_grad = False
    
    
x = torch.randn(1, 1024)
out_reference = model(x)

for _ in range(100):
    out = model(x)
    print((out-out_reference).abs().max())
</code></pre>
<p>Are you using any other (random) operations?</p>

Kim_KA: <p>No, other function I am using.<br>
This is exactly the same architecture of my model.</p>
<p>Thanks for reply.</p>

ptrblck: <p>Does my code snippet reproduce different results using your setup?</p>

Kim_KA: <p>It worked.<br>
Thanks for your help.</p>

Upgrade_Yourself: <p>Hi, I don’t understand how the problem is solved. Based on the code he gave at first, he wanted the training to stop after a certain epoch. But in the code you gave him, you were initializing the model and setting train to false immediately without training it first.<br>
Thanks for your answer.</p>

ptrblck: <p>Based on the initial question, <a class="mention" href="/u/kim_ka">@Kim_KA</a> was wondering, why his outputs change even after freezing all parameters and setting the model to <code>eval()</code>.<br>
Since his model architecture was not responsible for these changes, something else might have created the randomness, but that’s on <a class="mention" href="/u/kim_ka">@Kim_KA</a> to report. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>From the perspective of his code snippet and model architecture, the outputs are deterministic and static.</p>

Upgrade_Yourself: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
LSTM in PyTorch (many to one)

Sherzod_Bek: <p>Hello,<br>
I want to implement many-to-one LSTM model class in PyTorch (as shown in image).  When found some code from internet I couldn’t understand(manage) inputs and outputs of the model…<br>
In short: How can I implement LSTM class model like shown in image…?   (Could you write some simple example code, or …)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1c233eeac934a1241fca027969f0d651c9b5b70b.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c233eeac934a1241fca027969f0d651c9b5b70b" title="lstm"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1c233eeac934a1241fca027969f0d651c9b5b70b_2_690x265.png" alt="lstm" data-base62-sha1="40USJOBCNEzUCkhgkKx58ncVBfZ" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1c233eeac934a1241fca027969f0d651c9b5b70b_2_690x265.png, https://discuss.pytorch.org/uploads/default/original/2X/1/1c233eeac934a1241fca027969f0d651c9b5b70b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/1c233eeac934a1241fca027969f0d651c9b5b70b.png 2x" data-dominant-color="EDECEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm</span><span class="informations">966×372 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Before LSTM I want to use Encoder and after LSTM I want to use Decoder, that’s why I need exact size of input and outputs like shown above image…</p>

Prerna_Dhareshwar: <p>Hi,</p>
<p>The input to a pytorch LSTM layer (<code>nn.LSTM</code>) has to be an input with shape (sequence length, batch, input_size). So you will likely have to reshape your input sequence to be of the shape <code>(10, 1, 512*7*7)</code> which you can do with - <code>x = x.view(10,1,512*7*7)</code>.</p>
<p>You can do the following after that-</p>
<pre><code class="lang-auto">model = nn.LSTM(input_size = 512*7*7, hidden_size = 512*7*7)
out, (h,c) = model(x)
</code></pre>
<p>Where <code>h</code> is the many-to-one output you need. It will have the shape <code>(1,1,512*7*7)</code>.</p>
<p>Hope this helps.</p>

Sherzod_Bek: <p>Thank you very much Prerna_Dhareshwar!</p>

anandaltekar: <p>What should be the input size if I want to feed in 18 images each of size (3,128,128) into an LSTM?<br>
when I draw the batch using data loader my batch size is 18, so my data tensor is (18,3,128,128)</p>
Exception is Expected object of scalar type Float but got scalar type Long for argument #2 'other'

Zeev: <p>Hi,<br>
Got this error but I don’t have the full log.</p>
<pre><code class="lang-auto">def encode(words):
    max_l = max([len(i) for i in words])
    result = torch.empty((1,max_l, 26)).int()
    for word in words:
        ints = (np.fromstring(word,dtype=np.uint8)-ord('a'))
        addition = np.zeros((max_l - ints.shape[0],)) -1
        tr = torch.Tensor(np.expand_dims(np.hstack((ints,addition)),-1))
        tr = (tr[:] == torch.arange(26)).int()
        tr = torch.unsqueeze(tr, 0)
        result = torch.cat((result,tr))
    result = result[1::]
    return result
</code></pre>
<p>I have no Idea why this error occurs. could anybody help?</p>
Question on ImageFolder?

shwetank: <p>Hello, Actually, there are two folder on kaggle on path “’…/input/aptos2019-blindness-detection/”, train_images and test_images, and imageFolder is loading the all images from both folder but i want dataset from train folder only, please help how can i achieve this?<br>
if i give path to tain_images, it gives error i.e:</p>
<p>RuntimeError: Found 0 files in subfolders of: …/input/aptos2019-blindness-detection/train_images<br>
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp</p>
<p>but if i give path: “…/input/aptos2019-blindness-detection/”, it reads images from both folder.</p>
<p>i can’t even create a new folder because data is read only in kaggle</p>

beaupreda: <p>Hello,</p>
<p>If you look at the documentation of <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder" rel="nofollow noopener">ImageFolder</a>, it specifies that it expects more than one folder in the specified location (<code>root</code> parameter), and not a folder containing only images.</p>
<p>What you could do, in your case, is load “…/input/aptos2019-blindness-detection/” which I presume contains both the training and testing folder. You can then split manually to keep only the training data. I wrote this piece of code to show how you could do that, but you will have to change the names of certain folders.</p>
<pre><code class="lang-auto">    root = '/store/dabeaq/datasets/middlebury/'
    folder = ImageFolder(root)
    print(folder.imgs)
    # [('/store/dabeaq/datasets/middlebury/cones/disp2.png', 0), ('/store/dabeaq/datasets/middlebury/cones/disp6.png', 0), ('/store/dabeaq/datasets/middlebury/cones/im2.png', 0), ('/store/dabeaq/datasets/middlebury/cones/im6.png', 0), ('/store/dabeaq/datasets/middlebury/other_cones/disp2_t.png', 1), ('/store/dabeaq/datasets/middlebury/other_cones/disp6_t.png', 1), ('/store/dabeaq/datasets/middlebury/other_cones/im2_t.png', 1), ('/store/dabeaq/datasets/middlebury/other_cones/im6_t.png', 1)]
    train_folder_id = folder.class_to_idx['cones']
    test_folder_id = folder.class_to_idx['other_cones']
    train_images = [img for img, i in folder.imgs if i == train_folder_id]
    test_images = [img for img, i in folder.imgs if i == test_folder_id]
    print(train_images)
    # ['/store/dabeaq/datasets/middlebury/cones/disp2.png', '/store/dabeaq/datasets/middlebury/cones/disp6.png', '/store/dabeaq/datasets/middlebury/cones/im2.png', '/store/dabeaq/datasets/middlebury/cones/im6.png']
    print(test_images)
    # ['/store/dabeaq/datasets/middlebury/other_cones/disp2_t.png', '/store/dabeaq/datasets/middlebury/other_cones/disp6_t.png', '/store/dabeaq/datasets/middlebury/other_cones/im2_t.png', '/store/dabeaq/datasets/middlebury/other_cones/im6_t.png']
</code></pre>
<p>Hope this is clear!</p>

shwetank: <p>Thanks for the solution,i got it</p>
LSTM for RL with batching

Sheepsody: <p>Hi everyone,</p>
<p>I hope this subject was not already created.</p>
<p>I’m currently implementing IMPALA (<a href="https://arxiv.org/abs/1802.01561" rel="nofollow noopener">https://arxiv.org/abs/1802.01561</a>), a deep reinforcement learning method that aims to be distributed without suffering from the same bottlenecks as A3C (no GPU) or GA3C (instability). It basically collects trajectories and applies the v-trace algo to perform importance sampling.<br>
It also uses an LSTM, and that’s where i’m struggling! I chose to have same length trajectories, but with a mask that tells when the LSTMs states should be reset, and that’s the implementation I decided to use :</p>
<pre><code class="lang-python">x_out = []

for i in range(seq_len):
    result, lstm_hxs = self.model.lstm(x[i], lstm_hxs)
    lstm_hxs = [(done_mask[:, :, i]*state) for state in lstm_hxs]
    x_out.append(result)

x = torch.stack(tensors=x_results, dim=0)
</code></pre>
<p>I though of other methods, such as having different lengths episodes (the LSTM state is reset to zero once an episode is finished) and do padding, but I don’t know which approach would be the best…</p>
<p>Thanks a lot !</p>
Efficient way of selectively replacing vectors from a tensor in pytorch

Passepartout: <p>Given a batch of text sequences, the same is converted into a tensor with each word represented using word embeddings or vectors (of 300 dimensions). I need to selectively replace vectors for certain specific words with a new set of embeddings. Further, this replacement will occur only for not all occurrences of the specific word, but only randomly. Currently, I have the following code to achieve this. It traverses through every word using 2 for loops, check if the word is in a specified list, <code>splIndices</code> . Then it checks if the word needs to be replaced or not, based on T or F value in <code>selected_</code> .</p>
<p>But could this be done in a more efficient manner?</p>
<p>The below code may not be an MWE, but I have tried to simplify the code by removing the specifics, so as to focus on the problem. Please ignore the semantics or purpose of the code as it may not have been appropriately represented in this snippet. The question is about improving performance.</p>
<pre><code class="lang-auto">splIndices = [45, 62, 2983, 456, 762] # vocabulary indices which needs to be replaced
splFreqs = 2000 # assuming the words in splIndices occurs 2000 times
selected_ = Torch.Tensor(2000).uniform_(0,1)&gt;(0.2) # Tensor with 20% of the entries True
replIndexCtr = 0 # counter for selected_
diffVector = {45: Torch.Tensor(300).uniform_(0,1), ...... 762: Torch.Tensor(300).uniform_(0,1) }
 # Dictionary with vectors to be replaced. This is a dummy function. Original function depends on some property of the word


embeding = nn.Embedding.from_pretrained(embedding_matrix, freeze=False)
tempVals = x  # shape [32, 41] - batch of 32 sequences with 41 words each
x = embeding(x) # [32, 41, 300] - the sequence now has replaced vocab indices with embeddings

for i,item in enumerate(x):        # iterate through batch for sequences    
    for j,stuff in enumerate(item):   # iterate sequences for words          
        if tempVals[i][j].item() in splIndices: 
                        if self.selected_[self.replIndexCtr] == True:                   
                            x[i,j] = diffVector[tempVals[i][j].item()]
                            replIndexCtr += 1
</code></pre>
Using multiple models and optimizer in training

yuyaya: <p>Hello,</p>
<p>I’m now trying to use multiple models and optimizers in training.<br>
During the training, it seems like the loss is not decreasing.</p>
<p>Can anyone know what the problem is?<br>
The code is below</p>
<p>I have multiple models and optimizers, and in the <code>for c in range(classnum):</code><br>
loop, I am trying to compute the loss from each model and optimizing the model with optimizers corresponds to the model.</p>
<p>Note that each  optimizer is different alghorithm.</p>
<pre><code class="lang-auto">models = list[mode1, model2,...., model10]
optimizers = list[optimizer1, optimizer2,...., optimizer10]

for model in models:
            model.train()
            
for step, inputs in enumerate(tqdm(train_loader)):
     inputs = inputs.to(device)
            
     for c in range(classnum):
                
          optimizers[c].zero_grad()
                
                             :
                             :
                             :
           outputs, mean, logvar = models[c].forward(each_input)
           kld, recon = loss_function(outputs, each_input, mean, logvar, each_weights)
           loss = kld + recon
        
           loss.backward()
           optimizers[c].step()

           running_loss += loss.item()
</code></pre>
<p>Thank you.</p>

albanD: <p>Hi,</p>
<p>This approach looks good to me. You should not have any problem with it.<br>
Does training a single model with your loss works?</p>

yuyaya: <p>Thank you for you’re reply.<br>
<a class="mention" href="/u/alband">@albanD</a></p>
<p>Are you asking whether training loss decreased with single model and singe optimizer?</p>
<p>Yes it did</p>

yuyaya: <p><a class="mention" href="/u/alband">@albanD</a><br>
When I look at the training loss graph, looked like the loss was decreasing.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f0ef6acbd2f659c1872d2152087c41abb33c768.png" alt="00" data-base62-sha1="kpykVFgXacHoTq1wOTgtDeTk1p6" width="632" height="418"></p>
<p>Although the difference of the loss between each loss is very small…</p>
<pre><code class="lang-auto">1st epoch
 72170060.70782048
2st epoch
 72124209.65666279
3st epoch
 72124202.77555539
4st epoch
 72124197.6106462
5st epoch
 72124195.73002665
     :
     :</code></pre>

albanD: <p>Ho.</p>
<p>So it is training just very very slowly? Have you tried different learning rate? What about only 2 models?<br>
I don’t know what your loss is, but is it properly averaged over the samples?</p>

yuyaya: <p>I’m using Adam as an optimizer and tried several learning rate.<br>
But nothing actually changes. And haven’t tried training with 2 models yet.</p>
<p>The loss function is <code>torch.nn.BCELoss</code> and the reason that loss is large is that because reduction is sum.</p>
<p>The original data’s shapes are (batch size, C, H, W) = (32, 10, 100, 100) and during the training, I’m training each channel with an independent model and optimizer. Which means the shape of input data to model is (32, 1, 100, 100).</p>
<p>And also original data is the output of the segmentation model, so each channel is the probability.</p>
How do I store tensors in a database？

LinGeLin: <p>I want to store the calculated eigenvectors in a database, such as Mysql, and then take them out of the database during validation, and then calculate the similarity. The method I used to calculate the similarity is as follows:</p>
<p><code> </code> `</p>
<p>/ / cvec1. Sizes () = (1, 512], cvec1. Sizes () = [1, 512]</p>
<p>Tensor score = at::cosine_similarity(cvec1, cvec2);</p>
<p>Return score. The item &lt; float &gt; ();</p>
<p>}</p>
<p><code> </code> `</p>
<p>Is there a good solution?</p>
<p>Thanks</p>
Is this a bug in batch norm?

Jianjin_Xu: <p>I went through the code of batchnorm.py and found this statement in batchnorm.py:</p>
<p>line 78 in pytorch 1.3.1:</p>
<pre><code class="lang-auto">        return F.batch_norm(
            input, self.running_mean, self.running_var, self.weight, self.bias,
            self.training or not self.track_running_stats,
            exponential_average_factor, self.eps)
</code></pre>
<p>Maybe this statement should be  <code>self.training and self.track_running_stats</code>  according to the documentation  <code>If track_running_stats is set to False, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.</code></p>
<p>F.batch_norm should only receive training=True if self.training=True and self.track_running_stats=True</p>

albanD: <p>Hi,</p>
<ul>
<li>During training, you should always use the computed stats.</li>
<li>During evaluation, you should use computed stats if <code>track_running_stats</code> is False. Because you were not tracking stats, you cannot use them at evaluation time.</li>
</ul>
<p>So <code>self.training or not self.track_running_stats</code> looks correct to me.</p>

Jianjin_Xu: <ol>
<li>There is some cases that in training we don’t want to track the statistics.</li>
<li>In evaluation, if track_running_stats=False, then this statement will make training=True and the batch_norm will compute statistic as I understand. This is not correct because in evaluation we want to use the pre-computed statistic rather than compute this in evaluation.</li>
</ol>

albanD: <p>Hi,</p>
<ol>
<li>No there is not. That’s the definition of training mode of batchnorm (at least the one we use and to the best of my knowledge, the one from the original paper): use the statistics computed on the current batch.</li>
<li>But if <code>track_running_stats</code> was <code>False</code>, there are no pre-computed statistics to use as we never tracked them. We have to use the running one like during training.</li>
</ol>
Reduce GPU memory when loading the model for inference

netpcvnn: <p>Hi all,</p>
<p>When I load the model for inference. The model always consumes a lot of memory even the model size is small. For example: model size is 70MB (Encoder + Decoder + attention with Resnet 50 as backbone for encoder) but it occupies approximately 1GB GPU memory.<br>
Can we have any ways to reduce the GPU memory when loading the model for inference?</p>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>I’m afraid there is not much we can do.<br>
You can check this other topic on the same subject: <a href="https://discuss.pytorch.org/t/moving-a-tiny-model-to-cuda-causes-a-2gb-host-memory-allocation/61282" class="inline-onebox">Moving a tiny model to cuda causes a 2Gb host memory allocation</a></p>

111137: <p>Hi,</p>
<p>How about use of float16 instead of float32?, this conversion makes room twice.</p>
<p>Best,<br>
S.Takano</p>

netpcvnn: <p>Thanks all for supports.<br>
with Float16, I have used Apex (mixed precision) for training, and the model size is reduced to 50 MB, but when I load the model to GPU, it still consumes 950 MB.<br>
Look likes I need to see more for CUDA context for this issue.</p>
Multi-gpu inference with torchscript model is slow

Arvind_Vepa: <p>I traced some classification models from torchvision and then wanted to apply  <code>DataParallel</code> for multi-gpu inference. While no error is thrown, for more than 1 gpu passed as <code>device_ids</code> to <code>DataParallel</code>, inference on the torchscript model is much slower than the regular torch model (5x-10x slower), even if batch_size=1, which results in effectively only 1 gpu being used.</p>

ptrblck: <p>If the batch size is set to 1, only a single GPU will be used, as the data cannot be chunked and send to each device.</p>

Arvind_Vepa: <p>so i know that. I was just giving an example of the issues I was encountering with <code>DataParallel</code>. The issue is the torchscript model inferences are significantly slower than the standard torch model inferences. I tested a few different parameters, like <code>batch_size</code> and different number of gpu device ids with <code>DataParallel</code> (as well as a few different architectures). I found with 1 device_id or when not using <code>DataParallel</code> at all, the model inference speed was similar between the torchscript and torch model. However, when there was more than 1 device_id passed to <code>DataParallel</code>, the inference speed for the torchscript model reduced dramatically in comparison to the standard torch model with the same parameters (5x-10x slower). I brought up the example because it was curious that even when only 1 gpu was effectively being used when batch_size=1 with 2 device_ids passed, the torchscript model was still significantly slower than the regular torch model. However, when 1 device_id was passed, the torchscript model was similar in speed to the torch model.</p>
Scalar operation observers

pshashk: <p>Scalar op observers doesn’t seem to record statistics.</p>
<pre><code class="lang-auto">class HardSigmoid(nn.Module):
    def __init__(self):
        super().__init__()
        self.act = nn.ReLU6()
        self.add = nn.quantized.FloatFunctional()
        self.mul = nn.quantized.FloatFunctional()
    
    def forward(self, input):
        output = self.add.add_scalar(input, 3)
        output = self.act(output)
        output = self.mul.mul_scalar(output, 1/6)
        return output

class Residual(nn.Module):
    def __init__(self, *layers):
        super().__init__()
        self.layers = nn.Sequential(*layers)
        self.add = nn.quantized.FloatFunctional()

    def forward(self, input):
        return self.add.add(input, self.layers(input))

model = Residual(HardSigmoid())
model.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(model, inplace=True)
model(torch.rand(4,3,2,1))

</code></pre>
<p><code>model.add.activation_post_process.min_val</code> has a value, but <code>model.layers[0].mul.activation_post_process.min_val</code> and <code>model.layers[0].add.activation_post_process.min_val</code> is None</p>

lly-zero-one: <aside class="quote no-group" data-username="pshashk" data-post="1" data-topic="60840">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/9d8465/48.png" class="avatar"> pshashk:</div>
<blockquote>
<p>act</p>
</blockquote>
</aside>
<p>add_scalar and mul_scalar actually do not need observer. It could get the quantized parameter from the input qtensor and also compute the quantized parameter for the output qtensor.</p>

pshashk: <p>I’m glad there is no issue and it works as designed. But still it’s a bit strange for me to use <code>FloatFunctional</code> if it doesn’t record any state. Why not simply overload scalar operations at convert time?</p>

lly-zero-one: <p>Eager mode quantization is based on module level replacement. Current flow didn’t do the function level swapping, though it should be feasible to replace it with quantized kernel.</p>
How deal with ' raise NotImplementedError'?

Foolish: <p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net,self).__init__()
    self.fc1=nn.Linear(28*28,256)
    self.fc2=nn.Linear(256,64)
    self.fc3=nn.Linear(64,10)

def forword(self,x):
    x=F.relu(self.fc1(x))
    x=F.relu(self.fc2(x))
    x=self.fc3(x)
    return x
</code></pre>
<p>I heard that it is an indentation problem, how can I not find the above two pieces of code?</p>

Eta_C: <p><code>forword</code> → <code>forward</code></p>

Foolish: <p>oh,I am so sloppy.Thank you very much for your answer.</p>
Why does limiting the output layer makes accuracy drop?

alx: <p>I have 5 classes and I’m using a Densenet121.</p>
<p><strong>In scenario A I’m setting the ouput layer as follow:</strong></p>
<pre><code class="lang-auto">def get_trainable(model_params):
    return (p for p in model_params if p.requires_grad)

model =models.densenet121(pretrained=True)
num_ftrs = model.classifier.in_features
model.classifier = nn.Linear(num_ftrs, 5)
  
optimizer = torch.optim.Adam(
get_trainable(model.parameters()),
    lr=0.001,
)
</code></pre>
<p><strong>In scenario B, I’m leaving the default output layer:</strong></p>
<pre><code class="lang-auto">def get_trainable(model_params):
    return (p for p in model_params if p.requires_grad)

model = models.densenet121(pretrained=True)
  
optimizer = torch.optim.Adam(
get_trainable(model.parameters()),
    lr=0.001,
)
</code></pre>
<p>With a fixed output, the model plateaus at 76% accuracy at 20 epochs.</p>
<p>With the default output, I’m a 76% acc at the 2nd epoch! and it goes up to 82%.</p>
<p>Can someone explain why the last layer has such an influence when everything else is the same?</p>
Per Tensor/Channel quantization equivalents in PyTorch/Caffe2

Dorozhko-Anton: <p>The weights obtained from PyTorch per tensor quantization of Conv2d can be used in Int8Conv from caffe2. But from Int8Conv definitions, I understood that it only accepts <code>scale</code> as a float and not an array.</p>
<p>Is it possible to use the PerChannel quantization in Caffe2?</p>

Jongsoo_Park: <p>Unfortunately, Caffe2 Int8Conv doesn’t support per-channel quantization. The DNNLOWP engine that uses FBGEMM backend does support group-wise quantization if that helps you. Please see <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/quantization/server/conv_groupwise_dnnlowp_op_test.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/caffe2/quantization/server/conv_groupwise_dnnlowp_op_test.py</a> for example of using group-wise quantization.</p>
How to add Dense Layer after the last Conv layer when you don't know the shape of the last Conv layer activation map?

macab: <pre><code>    layer1 = nn.Sequential(
        nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2))
    layer2 = nn.Sequential(
        nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2))
    # How to add Dense Layer here ?</code></pre>

ptrblck: <p>One simple method would be to add a print statement with the shape information after <code>layer2</code> and define the number of input features based on this.<br>
Otherwise, you could also calculate the shape manually using the convolution / pooling arguments, but the first approach might be faster and simpler.</p>
Matrix quantization while training

Toshi: <p>I am having a bit trouble navigating the new quantization lib. I’m trying to train while quantizing a tensor matrix which is multiplied with a float32 convolutional layer weight. What would be the best way going about it?</p>

raghuramank100: <p>Hi <a class="mention" href="/u/toshi">@Toshi</a>, can you provide more details? Ideally a small code snipped would help. Have you seen the tutorials under <a href="https://pytorch.org/tutorials/#quantization-experimental" rel="nofollow noopener">https://pytorch.org/tutorials/#quantization-experimental</a>?</p>
When should I use sparse embedding instead of dense embedding

CMS_Flash: <p>When should I choose to set <code>sparse=True</code> for an <code>Embedding</code> layer? What are the pros and cons of the sparse and dense versions of the module?</p>

vainaijr: <p>When most of the embeddings are not learnt during training, that is representation of only few words is updated, rest of the representations stay as they were, for example,</p>
<pre><code class="lang-auto">class A(nn.Module):
  def __init__(self):
    super().__init__()
    self.embedding = nn.Embedding(10, 10, sparse=True)
  def forward(self, x):
    return 2*self.embedding(x)
</code></pre>
<pre><code class="lang-auto">net = A()
loss = net(torch.LongTensor([8, 7])).sum()
for param in net.parameters():
    print(param.grad)
</code></pre>
<p>None</p>
<pre><code class="lang-auto">loss.backward()
for param in net.parameters():
  print(param.grad)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor(indices=tensor([[8, 7]]),
       values=tensor([[2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
                      [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]),
       size=(10, 10), nnz=2, layout=torch.sparse_coo)
</code></pre>
<p>not all word embeddings that we have represented are updated during training, only a few are updated, if we do not use sparse=True, then,</p>
<pre><code class="lang-auto">loss.backward()
for param in net.parameters():
  print(param.grad)
</code></pre>
<p>would give something like,</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>most of the embeddings are not being updated during training, so probably it is better to use sparse=True, if we were passing all of our inputs to our neural network, and all of the embeddings were getting updated, then we would have set sparse=False.</p>
Why do we do subtraction in optimizer?

vainaijr: <p>Why do we do</p>
<pre><code class="lang-auto">w -= lr* w.grad
</code></pre>
<p>and not something like</p>
<pre><code class="lang-auto">w *= lr*w.grad
</code></pre>
<p>or</p>
<pre><code class="lang-auto">w /= lr*w.grad
</code></pre>
<p>does only subtraction lead to best weights?</p>

albanD: <p>You can check the wikipedia article on <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a> for example.<br>
But the substraction is interpreted as moving in the space of weights along the descent direction given by the negative of the gradients.</p>
<p><code>w /= lr*w.grad</code> can have a fun interpretation where it is also doing a gradient step in the space of exponentiated values: if you take exponentials, you will recover the gradient descent update with <code>-</code>.</p>
<p>Multiplication, would be like <code>w += lr * w.grad</code> in the exponentiated space. Which sounds bad <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Shared weights in convolution filters

jbo: <p>Hi,</p>
<p>I want to create a new module that implements a custom convolution layer that, given one input channel, produces two output channels using e.g. the following two filters:</p>
<p><img src="https://latex.codecogs.com/gif.latex?%5Cbegin%7Bbmatrix%7D&amp;space;w0&amp;space;&amp;&amp;space;w1&amp;space;&amp;&amp;space;w2&amp;space;%5C%5C&amp;space;w7&amp;space;&amp;&amp;space;0&amp;space;&amp;&amp;space;w3&amp;space;%5C%5C&amp;space;w6&amp;space;&amp;&amp;space;w5&amp;space;&amp;&amp;space;w4&amp;space;%5Cend%7Bbmatrix%7D&amp;space;%5Ctext%7Band%7D&amp;space;%5Cbegin%7Bbmatrix%7D&amp;space;w7&amp;space;&amp;&amp;space;w0&amp;space;&amp;&amp;space;w1&amp;space;%5C%5C&amp;space;w6&amp;space;&amp;&amp;space;0&amp;space;&amp;&amp;space;w2&amp;space;%5C%5C&amp;space;w5&amp;space;&amp;&amp;space;w4&amp;space;&amp;&amp;space;w3&amp;space;%5Cend%7Bbmatrix%7D" title="\begin{bmatrix} w0 &amp; w1 &amp; w2 \\ w7 &amp; 0 &amp; w3 \\ w6 &amp; w5 &amp; w4 \end{bmatrix} \text{and} \begin{bmatrix} w7 &amp; w0 &amp; w1 \\ w6 &amp; 0 &amp; w2 \\ w5 &amp; w4 &amp; w3 \end{bmatrix}" width="268" height="67"></p>
<p>Those two filters share the weights w0, …, w7 and the center weight is fixed to zero. Optimally, I want to use the filters together in one F.conv2d(inputs, filters) call.</p>
<p>Is this possible? If yes, how do I get the shared weights and the weights fixed to zero?</p>
<p>Thanks in advance!</p>

albanD: <p>Hi,</p>
<p>The simplest solution to get these is for you only to store a 1D Tensor that contains [w0, w1…, w7] and then reconstruct the filters on every forward before giving them to <code>F.conv2d</code>.</p>

jbo: <p>Ok, I understand.</p>
<p>But is there a way to construct this in the constructor and not the forward pass?  Because if there are many input- and output-channels, this might be a lot of work for every forward pass.</p>

albanD: <p>Well not really. Because this needs to be recomputed at every forward like any other operation you perform in your forward pass.</p>
Create a network where the number/type of layers are given as parameter input

DeepLearner17: <p>Hi,</p>
<p>Is there any possibility to build a network with variable number of layers ?</p>
<p>for L number of inputs we have L first layers which are run in parallel and don’t share parameters. L  is given as parameter.</p>
<p>To make it simple let’s suppose that all the layers to create are linear.</p>
<p>At a given intermediate layer the results are concatenated.</p>
<p>Here is my pseudo code.</p>
<pre><code class="lang-auto">class  My_net(nn.Module):

    def __init__(self, net_parameters,args):

	d,n,m,c,n_classes,number_of_layers=net_parameters

	self.designed_layers=[]

	for  layer in number_of_layers :
             self.layer=nn.Sequential(torch.nn.Linear(m,m*c),torch.nn.ReLU(True))
	     self.designed_layers.append[self.layer]


	self.fc1 = nn.Linear(n*m, n_classes)
    

    def forward(self,x,args.number_layers):

	
	new_x=[]

	for layer in number_layers:
		x_layer=self.designed_layers[layer](x[layer])
		new_x.append(x_layer)
	x=torch.cat(new_x)
	x=self.fc1(x)
	


     return x
     
Thank you
</code></pre>
Please help me debug

sbzcyx123: <p>x = torch.tensor([np.array(df.iloc[:,15]),np.array(df.iloc[:,14]),np.array(df.iloc[:,13]),<br>
np.array(df.iloc[:,12]),np.array(df.iloc[:,11]),np.array(df.iloc[:,10]),<br>
np.array(df.iloc[:,9]),np.array(df.iloc[:,8]),np.array(df.iloc[:,7]),<br>
np.array(df.iloc[:,6]),np.array(df.iloc[:,5]),np.array(df.iloc[:,4]),<br>
np.array(df.iloc[:,3]),np.array(df.iloc[:,2]),np.array(df.iloc[:,1]),</p>
<pre><code>                      np.array(df.iloc[:,30]),np.array(df.iloc[:,29]),np.array(df.iloc[:,28]),
                      np.array(df.iloc[:,27]),np.array(df.iloc[:,26]),np.array(df.iloc[:,25]),
                      np.array(df.iloc[:,24]),np.array(df.iloc[:,23]),np.array(df.iloc[:,22]),
                      np.array(df.iloc[:,21]),np.array(df.iloc[:,20]),np.array(df.iloc[:,19]),
                      np.array(df.iloc[:,18]),np.array(df.iloc[:,17]),np.array(df.iloc[:,16])]
                     , dtype=torch.float32)
</code></pre>
<p>y = torch.tensor(np_y_buy, dtype=torch.float32)</p>
<h1>============================================================</h1>
<p>D_in, H1, H2, D_out = 30, 10, 5, 3</p>
<p>class TwoLayerNet(torch.nn.Module):<br>
def <strong>init</strong>(self, D_in, H1, H2, D_out):<br>
# define the model architechture<br>
super(TwoLayerNet, self).<strong>init</strong>()<br>
self.linear1 = torch.nn.Linear(D_in, H1, bias = True)<br>
self.linear2 = torch.nn.Linear(H1, H2, bias = True)<br>
self.linear3 = torch.nn.Linear(H2, D_out, bias = True)<br>
def forward(self, x):<br>
y_pred = self.linear3(self.linear2(self.linear1(x).sigmoid()).sigmoid()).sigmoid()<br>
return y_pred</p>
<p>model = TwoLayerNet(D_in, H1, H2, D_out)<br>
loss_fn = nn.BCELoss()</p>
<p>learning_rate = 1e-4</p>
<p>optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)</p>
<p>for it in range(100):<br>
# forward propagation<br>
y_pred = model(x) # 等价于用 y_pred = model.forward(x)</p>
<pre><code># compute loss
loss = loss_fn(y_pred, y)
print(it, loss)

optimizer.zero_grad()

# backward propagation
loss.backward()

# update the parameters
optimizer.step()
</code></pre>
<h1>It said I got a size mismatch error. I do not know why is that</h1>

InnovArul: <p>Can you print the shape of <code>x</code>and see if it is two dimensional?</p>
Error in calculating similarity，get -nan

LinGeLin: <p>my code as follows：</p>
<p>at::Tensor output1 = model_.forward({inputs}).toTensor();</p>
<p>float ttt[128];</p>
<p>// Some assignment operations</p>
<p>auto output2 = torch::from_blob(ccc, {1,128});</p>
<p>at::Tensor score = at::cosine_similarity(output1, output2);</p>
<p>return score.item();</p>
<p>got -nan return</p>
<p>How to fix it ?</p>
[C++] missing:TORCH_LIBRARY on initial setup

Hamhub7: <p>Hey guys, I’m trying to get my first PyTorch project up and running on C++. I’ve used the python platform in the past but I would like to program something in a language I’m more comfortable with, so I’m checking out libtorch for C++. I have gotten to the point in the installation where I use the cmake command to generate the build files. However, my command prompt says that configuring is incomplete, errors occurred! the error it points out is – Could NOT find torch (missing: TORCH_LIBRARY). It sounds like its an error with where my DCMAKE_PREFIX_PATH is pointing, but I am pointing directly to the unzipped libtorch folder. I am on Windows 10 Pro, if that helps any. If needed, I can attach log files, etc. Thanks for the help!</p>
<p>Edit: upon further inspection, pthread.h is also unable to be found. If that is of any help</p>

Hamhub7: <p>I also tried installing the debug version with CUDA, and that one gives me an error with being unable to find the cuDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)</p>

Haydnspass: <p>have you maybe missed to copy the relevant dll’s?<br>
This is needed on windows.<br>
Following the examples, I include those lines at the bottom of the CMakeLists.txt</p>
<pre><code class="lang-auto">if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET deepsmlm_interface
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:deepsmlm_interface&gt;)
endif (MSVC)
</code></pre>
Pytorch tensor to opencv GPUMat

Msabih: <p>Hi,</p>
<p>I compiled OpenCV 4 with cuda and python bindings which makes it possible to use certain opencv algorithms on GPU. I am doing some post processing on output of a neural network. I was wondering if there is a way to directly convert a pytorch tensor on the GPU to opencv GPUMat ?</p>
<p>I could first bring tensor to cpu and then convert it into numpy and then to GPUMat but that will bring put extra penalty due to GPU-&gt;CPU-&gt;GPU transfer.</p>
<p>So is there any direct way to utilize pytorch tensor’s pointer or something ?</p>
<p>Best Regards,</p>

Shoaib_Ahmed_Siddiqu: <p>I am looking for something similar. This would definitely be useful for directly importing CV algorithms as CPP extensions.</p>

Msabih: <p>Hi,</p>
<p>Did you find a reasonable solution ? I did convert the GPUMat but the algorithm still did’nt work. As it was not a crucial thing so I just left it out.</p>
Master weights meaning

Upgrade_Yourself: <p>In the context of mixed precision training, I’ve found the following:</p>
<blockquote>
<p>Using mixed precision training requires three steps:<br>
1/Converting the model to use the float16 data type where possible.<br>
2/Keeping float32 master weights to accumulate per-iteration weight updates.<br>
3/Using loss scaling to preserve small gradient values.</p>
</blockquote>
<p>What does <strong>master weights</strong> mean exactly?</p>
<p>Thanks!</p>

mailcorahul: <p>Hi!<br>
I am not aware of half-precision training, but it looks like the <code>original/float32-type weights</code> is what being referred to as <code>master weights</code> here.</p>
Pytorch specific operation for the finding dimension wise mean for a list of tensors

Passepartout: <p>I have a list of tensors, each of <code>300x1</code> dimensions and I need to find the mean and variance for each dimension based on the list, how exactly can I achieve this. I could make this work in numpy with numpy arrays, with the following code snippet:</p>
<pre><code class="lang-auto">for item in embeddingLists:    # embeddingLists is a list of lists
                               # item contains list of numpy arrays
    tempVal =  np.mean(np.array(item),axis=0)
    meanVects.append(tempVal)

    temVar = np.var(np.array(item),axis=0,ddof=1) 
    varVects.append(temVar)
</code></pre>

Eta_C: <p>maybe</p>
<pre><code class="lang-auto">for item in embeddingLists:                           
    tempVal =  torch.mean(torch.tensor(item).float(),dim=0)
    meanVects.append(tempVal.item())

    temVar = torch.var(torch.tensor(item).float(),dim=0) * len(item) / ddof 
    varVects.append(temVar.item())
</code></pre>

Passepartout: <p>Thanks for the reply. I figured out that the following code can help here</p>
<pre><code class="lang-auto">for item in embeddingLists:       
    tempItem = [stuff.unsqueeze(0)  for stuff in item]
    coomn = torch.cat(tempItem)             
    temMean = torch.mean(coomn,dim=0)
    meanVects.append(temMean)
    temVar = torch.var(coomn,dim=0)
    varVects.append(temVar)
</code></pre>
Median filtering for 2D Tenso

Thiyagu: <p>Hi,<br>
Is there any way to do median filtering for 2D Tensor(1X130) for post-processing of classification problem in pytorch?</p>
<p>Thanks</p>

diggerdu: <p><a href="https://torchgeometry.readthedocs.io/en/latest/_modules/kornia/filters/median.html" class="onebox" target="_blank" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/_modules/kornia/filters/median.html</a></p>
Train multiple models on single gpu in parallel?

stevethesteve: <p>I am doing leave-1-out cross-validation and I only have 1 GPU at my disposal (I use Colab). Currently the different model instances are trained sequentially. This takes a lot of time and also the “RAM” bar in Colab is nearly empty, which leads me to believe that one can do better by somehow loading multiple models into the GPU and training them simultaneously. Is this possible? Paid services / frameworks are not an option for me.</p>