akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loading pre-batched data from directory

username: xandrovich
cooked: <p>If I have a directory of .h5 files, each containing 500 images (of size (300,300,1)), how do I proceed to load each .h5 file, and then pass a batch_size quantity of images i.e. if batch_size = 2, I’ll pass 2 images of those 500, to have the neural network to train on? Currently I can’t think of how to do this without including a loop of some sort within the <code>__getitem__</code> attribute of the Dataset class - though I think this will cause issues with it being called by the data loader.</p>
<p>Much appreciated,<br>
xandrovich</p>
How to have two optimizers such that one optimizer trains the whole parameter and the other trains partial of the parameter?

username: bendangnuksung
cooked: <p>Hi! I am new to PyTorch.<br>
I have a model:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(128, 128, (3,3))
        self.conv2 = nn.Conv2d(128, 256, (3,3))
        self.conv3 = nn.Conv2d(256, 256, (3,3))
                
    def forward(self,):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        return x

model = MyModel()
</code></pre>
<p>I want to train model in such a way that in every training step <code>DATA_X1</code> should train<br>
<code>['conv1', 'conv2', 'conv3']</code> layers and <code>DATA_X2</code> should train only <code>['conv3']</code> layers.<br>
Is there a way I can do this? Any help will be appreciated.</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>The following topic (<a href="https://discuss.pytorch.org/t/two-optimizers-for-one-model/11085">Two optimizers for one model</a>) discussed what you can do with two optimizers. You only have to specify the correct parameters to each optimizers. For example:</p>
<pre><code class="lang-auto">optim1 = torch.optim.SGD(model.parameters(), lr=0.001)
optim2 = torch.optim.Adam(model.conv3.parameters(), lr=0.05)
</code></pre>
<p>Good luck!</p>
Apply a function to a tensor columns

username: guitarman
cooked: <p>I’m trying to find an efficient way of applying a function to the axis of a tensor. The function could be a simple sum() and return a scalar.</p>
<p>So for an example for (32,3,1,16) I would get (32,3,1,1)</p>
<p>Right now I’m looping through a tensor and applying function. I guess this is not the most efficient way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thank you</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">x.sum(dim=-1, keepdim=True)
</code></pre>

username: guitarman
cooked: <p>I only need to define my own function instead of sum(x)</p>
LSTMCell batch size confusion

username: rustagiadi95
cooked: <p>Hi all,</p>
<p>I have a dataset which is fed into my model with batch_size 32, now being an nlp problem, the input samples are sentences.<br>
I have padded the sentence to 60 words so that while batching the dimensions are consistent, taking the embeddings of each word = 300.<br>
The input size is (32, 60, 300)</p>
<p>I am trying to feed this whole thing in an LSTMCell, who according to pytorch documentation takes the tensor of dimension of (batch, input_size). Now I have a confusion that how the input tensor should be reshaped or sliced off((32, 300) or (60, 300)) in order to feed it to LSTMCell.</p>
<p>If I iterate over the tensor feeding sentences one by one, still the input will be of shape (60, 300) ; but 60 is not the batch size(as it should be according pytorch docs for LSTMCell) here. This is the exact confusion.</p>
How to visualize activation in neural network

username: OBouldjedri
cooked: <p>Hi I was wondering if there is a good tool that helps to visualize the Result of activations</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>If you just want to plot the activation maps, you could use <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/2">this approach</a>.</p>

username: Eta_C
cooked: <p>Maybe this repository  <a href="https://github.com/utkuozbulak/pytorch-cnn-visualizations" rel="nofollow noopener">PyTorch CNN Visualization</a> can help you.</p>
Libtorch move Variable from CPU to GPU Excessive CPU usage

username: zihua_xiong
cooked: <pre data-code-wrap="cpp"><code class="lang-cpp">for (int k = 0; k &lt; img_vars.size(); ++k) {
		
		img_var = img_vars[k]; //**vector of variable on CPU**
		if (use_gpu)                                
			img_var = img_var.cuda();//to(at::kCUDA);// **I find this code will Excessive CPU usage**
}
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8620d70a340a69000e293169f4dab6cc7d30907d.png" alt="image" data-base62-sha1="j8yl6yiIwwDZVzQqlyjLBEaiU7r" width="407" height="165"></p>
<p><strong>Is such a high CPU usage normal? and any way to reduce CPU usage?</strong></p>

username: yf225
cooked: <p>Could you try setting <code>OMP_NUM_THREADS=1</code>, and see if there is still a high CPU usage?</p>

username: dancedpipi
cooked: <p>still too high, any other suggestions?</p>
Libtorch C++ beginners

username: duola
cooked: <p>Hello , guys<br>
i am learning the libtorch C++ front, but it is rather slow because i don’t know where to beging. Do you have any good idea for the beginners? Such as some website to learn it?<br>
Thanks for sharing!</p>

username: 21fa417e3fb06e56040c
cooked: <p>See here.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/pytorch/examples/tree/master/cpp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/pytorch/examples/tree/master/cpp" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>my repo<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/kerry-Cho/transfer-learning-Libtorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/20694053?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/kerry-Cho/transfer-learning-Libtorch" target="_blank" rel="nofollow noopener">kerry-Cho/transfer-learning-Libtorch</a></h3>

<p>Libtorch Examples. Contribute to kerry-Cho/transfer-learning-Libtorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: duola
cooked: <p>Thanks a lot , i am not sure whether the transfer-learning is difficult for me right now <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: 21fa417e3fb06e56040c
cooked: <p>Tell me  if you need help.</p>
List.pop() can't work in Dataloader

username: feiyuhuahuo
cooked: <p>I’m training videos. Each video frames are in one folder, and I have ten videos. I let ‘indice’ which comes from Dataloader to decide which folder to use. So I need another random index to decide which frame in a specified folder to use. And I want to make full use of every frame in one folder without repetition. So I set a random sequence. Every time I pop a index from the sequence as the index of the frame, but list.pop() doesn’t work well as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e744c9670283cc9d9cd6862172e6f20148d77650.png" data-download-href="https://discuss.pytorch.org/uploads/default/e744c9670283cc9d9cd6862172e6f20148d77650" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e744c9670283cc9d9cd6862172e6f20148d77650_2_690x408.png" alt="image" data-base62-sha1="wZTFzCXFv7z18EeMgBujXsJhNew" width="690" height="408" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e744c9670283cc9d9cd6862172e6f20148d77650_2_690x408.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e744c9670283cc9d9cd6862172e6f20148d77650_2_1035x612.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/7/e744c9670283cc9d9cd6862172e6f20148d77650.png 2x" data-dominant-color="303132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1171×693 89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’ve pop() many times, but the sequence remains unchanged. How deal with this, please?</p>

username: alex.veuthey
cooked: <p>Could you try with <code>num_workers=0</code> instead? I think multiprocessing breaks your intended behaviour.</p>
<p>The correct approach for this kind of problem is using a custom sampler, that will give you indices for all batches. This way, the dataloader can have multiple processes and be faster, it will ask the sampler (only one instance) for indices, and transfer those indices to the dataset, therefore loading the correct data.</p>
<p>More detailed explanations <a href="https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler" rel="nofollow noopener">in the doc</a>.</p>
Quantized Squeeze block MobilenetV3

username: thancaocuong
cooked: <p>Hi, I’m trying to quantize mobilenet V3 but get stuck in quantizing Squeeze Block. When I train it in subset dataset and use convert to quantized model, it works. but when I evaluate quantized model in eval set, It throws an error of Mul operation. Can you tell me how can I implement Squeeze block (SElayer). thank you so much.</p>
<pre><code class="lang-auto">class SqueezeBlock(nn.Module):
    def __init__(self, exp_size, divide=4):
        super(SqueezeBlock, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.dense = nn.Sequential(
            nn.Linear(exp_size, exp_size // divide),
            nn.ReLU(inplace=False),
            nn.Linear(exp_size // divide, exp_size),
            h_sigmoid()
        )
        self.mul = torch.nn.quantized.FloatFunctional()

    def forward(self, x):
        batch, channels, height, width = x.size()
        out = self.avg_pool(x).view(batch, channels)
        out = self.dense(out)
        out = out.view(batch, channels, 1, 1)
        return self.mul.mul(x,out)
</code></pre>
<pre><code class="lang-auto">File "/home/X/Documents/thancuong/mb3_quantized/quantized_mb3.py", line 121, in forward
    return self.mul.mul(x,out)
  File "/home/X/.virtualenvs/torch_0.4/lib/python3.5/site-packages/torch/nn/quantized/modules/functional_modules.py", line 146, in mul
    zero_point=self.zero_point)
RuntimeError: Mul operands must be the same size! (check_inputs at /pytorch/aten/src/ATen/native/quantized/cpu/qmul.cpp:20)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x33 (0x7f077b622813 in /home/X/.virtualenvs/torch_0.4/lib/python3.5/site-packages/torch/lib/libc10.so)
</code></pre>

username: lly-zero-one
cooked: <p>You can actually try to comment out the two lines as <a href="https://github.com/pytorch/pytorch/pull/30442" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/30442</a>, since the tensor iterator supports broadcast.</p>

username: thancaocuong
cooked: <p>thank you, I will try it</p>
How to free GPU memory (Nothing works)

username: KnHuq
cooked: <p>Hi <a class="mention" href="/u/smth">@smth</a> , I tried all the discussion and everywhere but can’t find the correct solution with pytorch. I am seeking your help.</p>
<p>How can I free up the memory of my GPU ?</p>
<p>[time 1] used_gpu_memory = 10 MB<br>
[time 2] model = ResNet(Bottleneck, [3, 3, 3, 3],100).cuda()<br>
[time 2] used_gpu_memory = 889 MB<br>
[time 3] del model<br>
[time 4] torch.cuda.empty_cache()<br>
[time 4]  used_gpu_memory = 627 MB</p>
<p>I tried gc.collect(). It is also not helping. I am having huge problem during training due to this un-referenced memories.</p>

username: PTA
cooked: <p>Have you tried to terminate your script and remount the GPUs?</p>

username: KnHuq
cooked: <aside class="quote no-group" data-username="PTA" data-post="2" data-topic="23158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c5a1d2/48.png" class="avatar"> PTA:</div>
<blockquote>
<p>rminate your script and remount the GPUs?</p>
</blockquote>
</aside>
<p>Yeah. When I terminate the script it frees the memory but the same thing again when I run it. <a class="mention" href="/u/smth">@smth</a> any thought on that? I want to free that redundant memory when I am running the script so <a class="mention" href="/u/pta">@PTA</a> remounting the gpu during the run does not make sense at all.</p>

username: PTA
cooked: <p>I think after <code>torch.cuda.empty_cache()</code> there is a reduced number of remaining memory does make sense. From my experience that should not affect your model training, unless there is a memory leak.</p>

username: KnHuq
cooked: <p>That memory is occupied by something. I want to remove that memory as i want to use that memory. May be trying to put some new tensor in the gpu but that space is still occupied.</p>

username: KnHuq
cooked: <p><a class="mention" href="/u/smth">@smth</a> Still waiting for your response.</p>

username: smth
cooked: <p><a class="mention" href="/u/knhuq">@KnHuq</a> we do occupy some base memory of ~400MB per GPU for cuda context, CUDA RNG context, streams etc. and ~200MB per GPU for CuDNN handles etc.<br>
It depends on GPU model as well. The numbers I quoted are for Volta V100 GPU.</p>

username: 111197
cooked: <p>Hi:<br>
if I do ops like:</p>
<pre><code class="lang-auto">x = conv(x)
x = conv(x)
</code></pre>
<p>I found it would save the very first x even if it has been replaced by conv(x). And it would consume 3 times memory as much as x. Could I free the useless very first x like that?</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/how-to-save-memory-by-doing-op-like-x-conv-x-while-inferencing/62906">here</a>. Lets continue the discussion in the other topic. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
What is the different between torch.max(1)[1] and torch.max(1)[1].data

username: wulongjian
cooked: <p>It seems no different between these .</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a=torch.randn(4,4)
&gt;&gt;&gt; a
tensor([[ 2.0155, -1.1138, -1.7522,  0.7299],
        [ 1.0620,  0.1840,  0.2790,  1.1942],
        [ 1.7519,  1.8871,  1.4988, -0.1911],
        [-1.6222, -0.2044,  1.6316, -1.0949]])
&gt;&gt;&gt; a.max(1,keepdim=True)[1]
tensor([[0],
        [3],
        [1],
        [2]])
&gt;&gt;&gt; a.max(1,keepdim=True)[1].data
tensor([[0],
        [3],
        [1],
        [2]])
</code></pre>

username: ptrblck
cooked: <p>The usage of <code>.data</code> is not recommended, as you’ll get access to the “real” underlying data and Autograd cannot track the changes applied on it, which might yield wrong results and thus silent errors.<br>
If you don’t want to track some changes on your tensors, you should wrap these operations in a <code>with torch.no_grad()</code> block.</p>
Requires_grad = False on Specific Filters Weights

username: capoling
cooked: <p>Hello!</p>
<p>I am working on pruning filters of a fully convolutional neural network, and am starting off by zeroing out the weights of the filters I deem least important, for all layers. I’d like to achieve something similar to setting a requires_grad = False flag for specific filters within a layer. I know that setting requires_grad = False on a layer basically freezes a certain layer’s weights from being updated with the backprop steps, however I’d like that to be the case for only some weights of that layers, not all. Are you aware / do you have any suggestions on how to do so?</p>
<p>Thanks so much!!</p>
<p>Giulio</p>
Human body detector

username: Surekha_Gaikwad
cooked: <p>I would like to know does there exists any pre-trained detector which can detect  human body excluding face?</p>

username: osousa
cooked: <p>I really don’t know anything about pytorch, but if this helps, it seems it can detect various objects, including human forms. <a href="https://gilberttanner.com/blog/detectron-2-object-detection-with-pytorch" rel="nofollow noopener">Here</a></p>
Fail to convert PyTorch module to TorchScript

username: herbertguoqi
cooked: <p>I’ve created a model with a forward function like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
...
  def forward(self, num_nodes, num_feats, nodes):
          features = nn.Embedding(num_nodes, num_feats)
          features.weight = nn.Parameter(torch.FloatTensor(feat_data), requires_grad=False)
</code></pre>
<p>then save that model using</p>
<pre><code class="lang-auto">traced_script_module = torch.jit.script(net)
traced_script_module.save(model_path1)
</code></pre>
<p>I have train model successfully, but get this error when save the model.</p>
<pre><code class="lang-auto">NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:...
'Embedding' is being compiled since it was called from '__torch__.___torch_mangle_0.Net.forward'
at &lt;ipython-input-5-501dbaacc7a5&gt;:42:8
    def forward(self, num_nodes, num_feats, nodes):
        
        features = nn.Embedding(num_nodes, num_feats)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        features.weight = nn.Parameter(torch.FloatTensor(feat_data), requires_grad=False)

</code></pre>
<p>And pytorch’s version is 1.3<br>
What is the best way to handle this?<br>
Any help appreciated!!!</p>

username: driazati
cooked: <p>You should move the initialization of submodules out into <code>__init__</code> rather than in your model’s <code>forward</code>. This will also likely improve performance since costly parameter allocations only need to happen once instead of on each run of your model’s <code>forward</code>.</p>
<pre><code class="lang-python">class M(nn.Module):
    def __init__(self, num_nodes, num_feats):
        self.features = nn.Embedding(num_nodes, num_feats)
        self.features.weight = nn.Parameter(torch.FloatTensor(feat_data), requires_grad=False)
    
    def forward(self, nodes):
        result = self.features(...)
        ...

model = M()
script_model = torch.jit.script(model)
script_model.save("script_model.pt")
</code></pre>
<p>When classes are instantiated in TorchScript, the entire class must be compatible with the TorchScript compiler (<a href="https://pytorch.org/docs/master/jit.html#id3">details</a>), which is not the case for most <code>nn.Module</code>s. However, if <code>nn.Module</code>s are saved on <code>self</code> in <code>__init__</code>, only the methods that are actually used in the <code>forward</code> of your model <code>M</code> need to be compatible with the compiler (which should work for any module in <code>nn</code> except for <a href="https://pytorch.org/docs/master/jit.html#builtin-functions">these 3</a>).</p>

username: herbertguoqi
cooked: <p>Got it！<br>
Thank you for the answers.</p>
Does detach make a copy of the tensor while still mainintng the tensor in the graph?

username: Lewis
cooked: <p>Hi. I’d like to ask about tensor.detach()<br>
I’ve read <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861">this</a> post already. But i’d like to ask. If performing<br>
<code>tensor_copy = tensor.detach()</code>,<br>
This will create a copy of <code>tensor</code> detached from the graph and store the detached copy in <code>tensor_copy</code>. My question is, when performing this operation, does <code>tensor</code> gets removed from the computation graph? Or will it still remain? (I know that <code>tensor_copy</code> will be removed from the graph, but I’m not sure about <code>tensor</code>).</p>
<p>This is because I’d like to modify one of my variables which requires grad, perform and operation and then use it again along with the original <code>tensor</code>. But when I do <code>.detach()</code>, I get an error that the tensor that required grad is modified when performing <code>loss.backward()</code></p>
<p>Thanks!</p>

username: Lewis
cooked: <p><a class="mention" href="/u/alband">@albanD</a> may you please help? Since you’ve answered previous posts regarding this question.</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>.detach()</code> returns a new Tensor without history but they share content. To get an actual copy, you can add a <code>.clone()</code> after the detach().</p>

username: Lewis
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>.<br>
Thanks for your reply. My question is not to get an actual copy. It is wether the actual copy (<code>tensor</code> in my example) will be removed from the graph.</p>

username: albanD
cooked: <p><code>tensor.detach()</code> is an out of place operation. So it does not modify <code>tensor</code> iteself.</p>
Save/reload torchvision model after I changed its inner structure

username: evgeniititov
cooked: <p>Hello folks,</p>
<p>I am pretty sure this question might have been asked multiple times, but I failed to find answer to my question, so here we go.</p>
<ol>
<li>Long story short, I use pretrained ResNet18 to classify some custom objects of mine. I changed its classifier from the one that predicts 1k classes to just 2. Trained the mode with transfer learning (didnt freeze any layers), saved its weights correctly, no dramas.</li>
</ol>
<p>My question is, how do I load this model? I mean I do know how to load it, but during training I change default resnet18’s classifier, so I cannot just import the default model from torchvision in another script I’ve got, load my weights for 2 classes and hope it will work, right? Does it mean that I need to somehow save the model as well so that I could open it in another script and start using?</p>
<ol start="2">
<li>
<p>Also, I suppose i can relatively easy find it online but still I am here asking questions, I want to ask one more question as well. I want to train a model to classify defected (with cracks etc) and not defected concrete poles. I’ve got relatively big dataset of concrete and pavement with cracks that should get the job done, but I still want to perform data augmentation. The question is can it backfire if you include too many image modifications? Like all those flips, angle changes, colour changes etc. Should I be careful and do not include too many? For this problem I am going to test the most popular model like VGG, inception etc. If you could also give any advice what you think might work for this task I’d be very grateful.</p>
</li>
<li>
<p>What transfer learning approach would you suggest? Since the problem is quite straightforward, I was thinking to unfreeze last conv layers during training so that a model better adapts to the shapes it is to work with. Do you think it is correct I should just training a new classifier without unfreezing the layers?</p>
</li>
</ol>
<p>Thank you very much in advance guys. I do hope I managed to properly word my questions.</p>
<p>Regards,<br>
Eugene</p>

username: ptrblck
cooked: <ol>
<li>
<p>Use the same workflow as was done before training. I.e. create the model, change the last layer, and load the <code>state_dict</code>. The most important part is to save the source code somehow to be able to get the same architecture, since the <code>state_dict</code> only stores the parameters and buffers without any information about e.g. the forward pass.</p>
</li>
<li>
<p>Use the validation data to check how much augmentation is necessary and when it hurts the model performance.</p>
</li>
<li>
<p>Same as above. As long as you don’t touch the test set for these experiments, you should be fine.</p>
</li>
</ol>

username: evgeniititov
cooked: <p>Hi ptrblck,</p>
<p>Thanks a lot for your answer. Very helpful.</p>
<p>All the best</p>
<p>Eugene</p>
How come I am getting different results when doing mean using list of dimensions vs step by step?

username: MLSlayer
cooked: <p>For example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
tensor([[[[ 1.,  3.],
          [ 4.,  6.]],

         [[-1., -3.],
          [-4., -7.]]]])
&gt;&gt;&gt; x.mean([3, 0, 1])
tensor([ 0.0000, -0.2500])
&gt;&gt;&gt; x.mean([3, 0])
tensor([[ 2.0000,  5.0000],
        [-2.0000, -5.5000]])
&gt;&gt;&gt; x.mean([3, 0]).mean([1])
tensor([ 3.5000, -3.7500])
</code></pre>
<p>I had believed that, doing mean([3, 0, 1]) is equivalent of doing x.mean([3, 0]).mean([1])</p>

username: vainaijr
cooked: <p>if we do,</p>
<pre><code class="lang-auto">x.mean(3).mean(0).mean(0)
</code></pre>
<p>then it would give same result<br>
basically,<br>
if we have tensor</p>
<pre><code class="lang-auto">x = torch.randn(a, b, c, d)
</code></pre>
<p>and we apply</p>
<pre><code class="lang-auto">torch.mean(x, dim=0)
</code></pre>
<p>then its shape would be</p>
<pre><code class="lang-auto">torch.Size([b, c, d])
</code></pre>
<p>so here, when we do,</p>
<pre><code class="lang-auto">x = torch.tensor([[[[ 1.,  3.],
          [ 4.,  6.]],

         [[-1., -3.],
          [-4., -7.]]]])
x.shape
</code></pre>
<p>torch.Size([1, 2, 2, 2])<br>
let us assume a=1, b=2, c=2, d=2</p>
<pre><code class="lang-auto">x.mean([3, 0, 1])
</code></pre>
<p>the resulting shape would be,</p>
<pre><code class="lang-auto">torch.Size([c])
</code></pre>
<p>while if we do,</p>
<pre><code class="lang-auto">x.mean([3, 0]).mean([1])
</code></pre>
<p>then after</p>
<pre><code class="lang-auto">x.mean([3, 0])
</code></pre>
<p>shape would be</p>
<pre><code class="lang-auto">torch.Size([b, c]) # [2, 2]
</code></pre>
<p>and when we do</p>
<pre><code class="lang-auto">x.mean([3, 0]).mean([1])
</code></pre>
<p>means shape would be</p>
<pre><code class="lang-auto">torch.Size([b])
</code></pre>
<p>so, we get different results, if we did</p>
<pre><code class="lang-auto">x.mean([3, 0]).mean(0)
</code></pre>
<p>then results would be same.</p>

username: MLSlayer
cooked: <p>This is exactly what I was looking for!</p>
Using autograd to compute Jacobian of partial derivatives

username: arijitthegame
cooked: <p>I apologize if this question is obvious or trivial. I am very new to pytorch and I am trying to understand the autograd.grad function in pytorch. I have a neural network G that takes in inputs (x,t) and outputs (u,v). Here is the code for G:</p>
<pre><code class="lang-auto">class GeneratorNet(torch.nn.Module):

"""
A three hidden-layer generative neural network
"""

def __init__(self):
    super(GeneratorNet, self).__init__()
    self.hidden0 = nn.Sequential(
        nn.Linear(2, 100),
        nn.LeakyReLU(0.2)
        )

    self.hidden1 = nn.Sequential(
        nn.Linear(100, 100),
        nn.LeakyReLU(0.2)
        )

    self.hidden2 = nn.Sequential(
        nn.Linear(100, 100),
        nn.LeakyReLU(0.2)
        )

    self.out = nn.Sequential(
        nn.Linear(100, 2),
        nn.Tanh()
        )

def forward(self, x):
    x = self.hidden0(x)
    x = self.hidden1(x)
    x = self.hidden2(x)
    x = self.out(x)
    return x
</code></pre>
<p>Or simply G(x,t) = (u(x,t), v(x,t)) where u(x,t) and v(x,t) are scalar valued. Goal: Compute $\frac{\partial u(x,t)}{\partial x}$ and $\frac{\partial u(x,t)}{\partial t}$. At every training step, I have a minibatch of size $100$ so u(x,t) is a [100,1] tensor. Here is my attempt to compute the partial derivatives, where coords is the input (x,t) and just like below I added the requires_grad_(True) flag to the coords as well:</p>
<pre><code class="lang-auto">tensor = GeneratorNet(coords)
tensor.requires_grad_(True)
u , v = torch.split(tensor, 1, dim=1)
du = autograd.grad(u, coords, grad_outputs = torch.ones_like(u), create_graph=True, 
        retain_graph=True, only_inputs=True, allow_unused=True)[0]
</code></pre>
<p>du is now a [100,2] tensor.  <strong>Question :</strong> Is this the tensor of the partials for the 100 input points of the minibatch?</p>
<p>There are similar questions like computing derivatives of the output with respect to inputs but I could not really figure out what’s going on. I apologize once again if this is already answered or trivial. Thank you very much.</p>
Can I load whole 3D MRI image into my memory without performing patching or subsampling?

username: Aliktk
cooked: <p>I have 17 GB of data collected from ADNI where I am going to classify them without any preprocessing and augmentation steps. I load a single image using <code>nibable</code> library. Now I am trying to load all the <code>.nii</code> format files from my directory.<br>
For this purpose I already studied the basic dataloading method shown in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">WRITING CUSTOM DATASETS</a> and also studied custom dataloader for 3D files .<br>
Please help me someone with some sort of code which help me to understand that I am on right path or I have to change the direction of implementing it?</p>

username: Balamurali_M
cooked: <p>Are you planning to train the model with 2d or 3d ?</p>

username: Aliktk
cooked: <p>I will train my CNN modal using 3D data</p>

username: Aliktk
cooked: <p><strong>Please Help!!!</strong><br>
I am stuck here to load the data from the directory for that please specify some direction.</p>

username: Balamurali_M
cooked: <aside class="quote" data-post="7" data-topic="48531">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/balamurali_m/48/12574_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-load-niftii-images-nii-using-pytorch-dataloader/48531/7">What is the best way to load niftii images (.nii) using pytorch dataloader?</a> 
  </div>
  <blockquote>
    Save the nib file to h5 file. Create a list with [(h5_file_name,slice_no),(h5_file_name,slice_no)]. Iterate this using dataloader, during runtime read the h5 file again and take out the slice. Let me know if you understand this, else i will share the pseudocode.
  </blockquote>
</aside>

<p>Will this help ?</p>

username: Aliktk
cooked: <p>I can’t get this I am not too much expert in pseudocode<br>
I can load a single image but I can save the file in h5 file using <strong>HDF5</strong> software which convert and save into h5 file.? Does It will be ok ?<br>
If can share the pseudocode than It will help more to understand your point.<br>
Thanks for fast reply</p>
How does one deal with different categories in pytorch train, test, and holdout set

username: Jordan_Howell
cooked: <p>I have a tabular pytorch model that takes in cities and zipcodes as a categorical embeddings. However, my hold out set has zipcodes and cities that were not in my train/test set.  This is also a real possibility if this model goes into deployment.</p>
<p>How do I set up a model to run even if the holdout set, or new data to infer has categorical values that are new?</p>
<p>For example, I have 2819 unique zip codes in my training and test set.  That leaves an embedding size of <code>(2918, 50)</code>.  I have new zip codes in my hold out sample that yields an embedding size of <code>(8684, 50)</code>.</p>
<p>As I get new values in, I may have a zip code never seen before.  Is there a way to account for this?</p>
Yolo/SSD , how are features localized?

username: MC95
cooked: <p>Hello, Iwant to fully understand how Yolo or SSD does work for object detection.</p>
<p>Here is what I understand:</p>
<ul>
<li>in a simple CNN for image recognition, features are extracted, flattend and used for a classification</li>
<li>yolo devides the image into a grid. For each grid, some values like class probabilities and the bounding box parameters are calculated.</li>
<li>SSD not only uses one grid , but a combination of different sizes to better detect objects at any size.</li>
</ul>
<p>What I don’t understand:</p>
<ul>
<li>in yolo and ssd, there is a classification per grid cell? How do we know, which features are in that specific cell?</li>
<li>are features extracted per grid cell or per bounding box or non of both?</li>
<li>how does ssd combine results from different grids?</li>
</ul>
<p>Thank you!</p>
Help! Recently installed conda and now pytorch is unstable

username: testin
cooked: <p>Hello! I’ve recently installed conda and got pytorch, but I’ve been getting a bunch of errors, such as:<br>
AssertionError: Torch not compiled with CUDA enabled<br>
cnn = cnn.cuda()<br>
File “c:\apps\Miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 260, in cuda<br>
return self._apply(lambda t: t.cuda(device))<br>
File “c:\apps\Miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 187, in _apply<br>
module._apply(fn)<br>
File “c:\apps\Miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 187, in _apply<br>
module._apply(fn)<br>
File “c:\apps\Miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 193, in _apply<br>
param.data = fn(param.data)<br>
File “c:\apps\Miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 260, in <br>
return self.<em>apply(lambda t: t.cuda(device))<br>
File "c:\apps\Miniconda3\lib\site-packages\torch\cuda_<em>init</em></em>.py", line 161, in _lazy_init<br>
<em>check_driver()<br>
File "c:\apps\Miniconda3\lib\site-packages\torch\cuda_<em>init</em></em>.py", line 75, in _check_driver<br>
raise AssertionError(“Torch not compiled with CUDA enabled”)</p>
<p>I’m not sure how to fix this, i’ve been at it for 3 hours now.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This happens because you try to use cuda. By doing <code>cnn.cuda()</code> but you installed a cpu-only version of pytorch.<br>
You should either remove the calls to <code>.cuda()</code> or install a cuda enabled version of pytorch.</p>
Share Cuda Context among multiple Processes on same GPU

username: maaft
cooked: <p>For hyperparameter search I start multiple training-processes on the same GPU.<br>
Unfortunately each process creates its own cuda context, consuming an excessive 700MB of memory.</p>
<p>Is there anything I can do to make the cuda context shared among multiple processes?</p>
<p>I’d love to use threads instead of processes, but according to the API this does bring even more issues.</p>
Increase dimension when multiply ([a x b], [a x c] ==> [a x (b*c)])

username: curie-ahn
cooked: <p>m1 : [a x b]<br>
m2 : [a x c]<br>
and I want to make a new matrix with shape</p>
<p>m3 : [a x d]<br>
where d  = b*c.<br>
i.e, the new matrix column-wisely multiply m1 with m2.<br>
Which can I use?</p>

username: ptrblck
cooked: <p>I’m not sure, if I understand the use case correctly, but could you check, if this code snippet would work for you:</p>
<pre><code class="lang-python">a, b, c = 2, 3, 4

x = torch.randn(a, b)
y = torch.randn(a, c)

z = torch.matmul(x.unsqueeze(2), y.unsqueeze(1))
z = z.view(a, -1)
</code></pre>
<p>If not, could you provide dummy tensors and the expected result?</p>

username: curie-ahn
cooked: <p>This command works for me. Thanks!</p>
Modelling prior in a gaussian process using Gpytorch

username: prvnkmr
cooked: <p>I am new to Pytorch and I am using a data of size 1600 to create a Gaussian process model.  Since data is big I could not create the model and I get out of memory error. I found that the prior can be modeled rather than feeding the data directly so that I can use it to create a Gaussian process model. Is there a way to do this in Pytorch. Does pytorch have any prior predict/model check methods ?</p>
DataParallel freezes

username: eric01300
cooked: <p>I just got a new machine with 2 gtx1080ti, so I wanted to try using nn.DataParallel for faster training. I have created a test code to make sure nn.DataParallel works, but it seems to get stuck in the forward().</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
class RandomDataset(Dataset):
	def __init__(self):
		self.a = torch.randn(123, 15)
		pass
	def __getitem__(self, index):
		return self.a[index]
	
	def __len__(self):
		return 123

class Model(nn.Module):
	def __init__(self, i_s, o_s):
		super(Model, self).__init__()
		self.fc = nn.Linear(i_s, o_s)
	def forward(self, x):
		output = self.fc(x)
		return output

dl = DataLoader(dataset=RandomDataset(), batch_size=16, shuffle=True)
model = Model(15, 3)
print(torch.cuda.device_count())
model = nn.DataParallel(model)


model.to(device)
print('it works!!')
for data in dl:
	x = data.to(device)
	y = model(x)
	print('!!it works')
</code></pre>
<p>Training works perfectly if I don’t use nn.DataParallel.</p>
<p>I did not know anything about setting up cuda because I was using cloud service previously, so I just followed the steps <a href="https://askubuntu.com/questions/1077061/how-do-i-install-nvidia-and-cuda-drivers-into-ubuntu" rel="nofollow noopener">here</a>.</p>

username: JuanFMontesinos
cooked: <p>Does it throw any error? or just get frozen</p>

username: JuanFMontesinos
cooked: <p>Such a strange thing.  Can you train in both gpus independently? Have you tried both gpus? Nvidia-smi works?</p>

username: sh0416
cooked: <p>I encountered the same issue above, and I resolve this issue by chaning the following way.</p>
<pre><code class="lang-auto">model = ResNet().to(device)
model = nn.DataParallel(model)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">model = ResNet()
model = nn.DataParallel(model).to(device)
</code></pre>
<p>In my case, it works.</p>
<p>BTW, the model should be in the <code>output_device</code> as the official document mentioned.</p>
<p>Thanks,</p>
Is Saving/Restoring Adam State strictly required?

username: maaft
cooked: <p>Until now I didn’t bother to restore the previous state of my Adam Optimizer to continue training.<br>
In fact, I observed that by doing this, a new best performing model is often found after this “hard” continuation.</p>
<p>Is there anything I am missing that makes this practice absolutely wrong?</p>

username: tom
cooked: <p>Well, so restoring the state and continuing is the equivalent of doing a single larger training run.<br>
What happens is that during the first few steps, the statistics gathered by the optimizer are still “rubbish”, and so you will take steps of more or less not terribly controlled size (the could be more precise, I guess). This has - at the beginning of the training - bothered some people enough to run a few steps with a learning rate of 0 in the beginning or re-initialize after a few steps (i.e. just update statistics), even if I cannot find the reference at the moment.<br>
So in a way, your method of re-starting Adam is “take a step in a lucky direction”. You might try to get a similar effect more systematically by varying the learning rate upwards or somesuch.</p>
<p>Best regards</p>
<p>Thomas</p>
Tutorial and End to End Example for Pytorch Mobile on Android

username: johnolafenwa
cooked: <p>Hi everyone, i recently published a tutorial on <strong>Image Classification on Android with Pytorch Mobile</strong></p>
<p>Since documentation and tutorials are still few on this, i included an easy to comprehend open source android application for image recognition.</p>
<p>The post is here: <a href="https://heartbeat.fritz.ai/pytorch-mobile-image-classification-on-android-5c0cfb774c5b" rel="nofollow noopener">https://heartbeat.fritz.ai/pytorch-mobile-image-classification-on-android-5c0cfb774c5b</a></p>
<p>And the repository for the android application is here: <a href="https://github.com/johnolafenwa/PytorchMobile" rel="nofollow noopener">https://github.com/johnolafenwa/PytorchMobile</a></p>
<p>This is intended as a reference to help pytorch developers easily deploy models with Pytorch Mobile.</p>

username: Kiki_Rizki_Arpiandi
cooked: <p>Great tutorial, do you manage to deploy object detection on android?. I need to multiply detection output tensor with priorbox tensor somehow</p>
torch::Tensor from gstreamer

username: Kiki_Rizki_Arpiandi
cooked: <p>I wonder how to create tensor from gstreamer data?</p>

username: tom
cooked: <p>Probably <code>torch::from_blob</code> if you/gstreamer keep owning the memory for as long as the Tensor exists or allocating a tensor and passing the memory pointer (<code>t.data_ptr</code>) to gstreamer. There also is a variant where PyTorch calls a deleter function when it doesn’t need the memory anymore (this is at the heart of the DLPack mechanism).</p>
<p>I usually recommend the latter if you have the option to avoid having to track memory.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Kiki_Rizki_Arpiandi
cooked: <p>Thankyou tom its works like charm</p>
Transfer learned model from binary to multiclass

username: alx
cooked: <p>Hi, I’m working on a disease detection project. I found 10+ datasets with binary labels (disease/no disease) and 4 dataset with labels indicating the gravity of the disease (0-3, 0 being no disease).</p>
<p>I was wondering if it was possible to train on the 10 datasets binary datasets and then transfer what I learned to a multiclass training.</p>

username: alex.veuthey
cooked: <p>Sure, this is equivalent to finetuning. The low-level features (closer to your input) will be pretrained by the large datasets and you only learn the high-level features (the ones helping decision / classification) when finetuning on the smaller datasets.</p>

username: alx
cooked: <p>Great! Do I simply have to load the weightss of the binary pretrained model on the multiclass one, and then train some more on the multiclass?</p>

username: alex.veuthey
cooked: <p>(Sorry for the delay, I was away)<br>
That is one way of doing it, but the classical approach is generally to freeze the low-level features (disable gradient computing &amp; updating of weights) and only finetune the high-level ones with a lower learning-rate.</p>
When to use which norm?

username: vainaijr
cooked: <p>Is there a technique to decide which norm should one use, as I find different kinds of norms being used in different modules.</p>
<p>For example, multi head attention uses LayerNorm<br>
In pytorch/examples, after conv2d, they use BatchNorm<br>
In GANs they use spectral_norm<br>
Then there is weight norm also, which internally uses frobenius norm<br>
I also found about nuclear norm, which is sum of singular values of matrix<br>
And some place, they consider maximum singular value of the matrix as the norm.</p>
<p>Is the eventual goal to bring all weights to a smaller range, so that we get convex curve between weight and loss?<br>
If so, how does one see this plot, for example if I do dog cat classification, then how to plot this convex curve?<br>
And does making all weight matrices symmetric and positive definite contribute to convexity?<br>
Is a higher norm of matrix bad for convexity, and norm should be less than one and (&gt;0)?</p>
Autograd on custom loss function

username: ROODAY
cooked: <p>Would autograd still work if I used non torch functions on my tensors? My model outputs a series of human poses in the form of 3D coordinates (each frame/pose is composed of 17 such coordinates), so the shape of output and target being fed to my loss function would be (seq_len, batch_len, 17, 3). I want my loss function to be the euclidean distance between the output and target summed (i.e. distance between output[0][0][0] and target[0][0][0], then output[0][0][1] and target[0][0][1], and so on, and all those distances summed). My question is, if I were to compute this via a loop that iterated through all the frames/poses, would autograd still know what to do? Or do I have to do this calculation using built in torch functions?</p>

username: Balamurali_M
cooked: <p>You have to use built-in torch functions.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ROODAY" data-post="1" data-topic="62822">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rooday/48/18299_2.png" class="avatar"> ROODAY:</div>
<blockquote>
<p>if I were to compute this via a loop that iterated through all the frames/poses, would autograd still know what to do?</p>
</blockquote>
</aside>
<p>Yes. A loop does not detach the computation graph and as long as you are using PyTorch methods, Autograd will create the backward pass for you.<br>
If you need to leave PyTorch and e.g. use numpy in the forward pass, you would need to implement the backward function manually.</p>