akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Scalar product of tensors with different dimensions

username: S_Krishna
cooked: <p>Hi,</p>
<p>I have a 2 stream conv network, one of which is a vision subnet. The output of this subnet is a tensor of dimension [b, 14,14,128] where b is the batch size. The output of the send subnet is a tensor of dim [b, 128].  I need to perform a scalar product to compute the similarity between each of the 14x14 128-dim vector from the visual subnet and the corresponding 128-dim vector from the 2nd subenet, so that the result is a [b,14,14,1] similarity map. I tried using torch.dot() and torch.bmm() but I am getting an error and I am not sure how I should transform the dimensions to get the 14x14x1 similarity map. I would appreciate any guidance in this regard. Here is a portion of my code:</p>
<pre><code>def forward(self, x_v, x_a):
  v_out = self.vfeatures(x_v)  #[b,14,14,128] feature map
  a_out = self.afeatures(x_a) #[b,128] 

  # sim map between embeddings needs to be a [bx14x14x1] map.
  #how can I define the pairwise_sim function to compute the scalar product?
  sim_map  = pairwise_sim(v_out, a_out)
</code></pre>
<p>thank you.</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">import torch
x = torch.randn(2, 14, 14, 128)
y = torch.randn(2, 128)
torch.einsum('abcd, ad -&gt; abc', x, y).unsqueeze(-1).shape
</code></pre>
<p>torch.Size([2, 14, 14, 1])</p>

username: S_Krishna
cooked: <p>It does produce the appropriate shape. I am not very familiar with einsum though, so I need to verify with some examples if the result  matches the  scalar product. Thank you very much for your help.</p>
How to use torch fft for color images

username: Deeply
cooked: <p>How to use <a href="https://pytorch.org/docs/0.4.0/torch.html?highlight=fft#torch.fft" rel="nofollow noopener">torch.fft</a> for a batch containing a number (52 here) of 2D RGB images.</p>
<pre><code class="lang-auto">imgs.shape
torch.Size([52, 3, 128, 128])
</code></pre>
<p>Thanks</p>

username: ksanjeevan
cooked: <p>So the Fourier transform works on intensities and an RGB image won’t have a defined intensity given a pixel. You could try and splitting the image in the rgb channels and then running <code>torch.fft</code> on them?</p>
<pre><code class="lang-auto">image = torch.randn([52,3,128,128])
r =torch.rfft(image[:,0:1].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])
g =torch.rfft(image[:,1:2].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])
b =torch.rfft(image[:,2:3].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])

image_fft = torch.cat([r,g,b], dim=1)
image_fft.shape
&gt;&gt;torch.Size([52, 3, 128, 128, 2])

</code></pre>

username: Deeply
cooked: <p>Apologies, what I wanted is <code>fft2</code>.</p>

username: Rui_Wang
cooked: <p>Hi why is the <code>signal_ndim=1</code>? Instead of 2?</p>

username: Rui_Wang
cooked: <p>For anyone using, I believe this is a more elegant implementation</p>
<pre><code class="lang-auto">torch.rfft(imgs, signal_ndim=2, normalized=True)
</code></pre>
<p>As <code>torch.rfft()</code> should be running from the last dimension, meaning that anything before imgs[2] will be considered as a batch size. Hence <code>[52, 3]</code> is treated as a whole and the <code>fft</code> is performed only at <code>[128, 128]</code>.</p>
Normalize embeddings using nn.BatchNorm1d(embedding_size)

username: vainaijr
cooked: <p>is this a correct way to normalize embeddings with learnable parameters?</p>
<pre><code class="lang-auto">x = nn.Embedding(10, 100)
y = nn.BatchNorm1d(100)
a = torch.LongTensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
y(x(a))
</code></pre>

username: fly2mars
cooked: <p>Looks like your input batch size is 10 (from <code>a</code>), which happens to equal your vocabulary size in <code>x</code>.</p>
<p>Your code looks fine if the above assumption is right.</p>
<p>Just to remind that if your input contains multiple input, say<br>
<code>a = torch.LongTensor([[0,1,2],[0,4,6]])</code><br>
this means you’ve got a batch of size 2 and each sample has 3 features.</p>
<p>Then after embedding, you’ll get tensor of size <code>(2, 3, 100)</code>,<br>
From my point of view, a <code>reshape(-1, 3*100)</code> or <code>view(-1, 3*100)</code> is needed in order to apply <code>BatchNorm1d</code></p>
<p>Pls correct me if Im wrong.</p>

username: vainaijr
cooked: <p>I think better way is to set max_norm in nn.Embedding, as compared to using nn.BatchNorm1d, it re normalizes each embedding vector norm to be less than or equal to max_norm.</p>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 128 and 256 in dimension 3 at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/generic/THCTensorMath.cu:71

username: Shrutika994
cooked: <p>/home/shru/abdominal-multi-organ-segmentation-master/net/ResUnet_CE.py:229: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.<br>
nn.init.kaiming_normal(module.weight.data, 0.25)<br>
/home/shru/abdominal-multi-organ-segmentation-master/net/ResUnet_CE.py:230: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.<br>
nn.init.constant(module.bias.data, 0)<br>
/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2457: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.<br>
warnings.warn(“nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.”)<br>
/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2539: UserWarning: Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.<br>
“See the documentation of nn.Upsample for details.”.format(mode))<br>
Traceback (most recent call last):<br>
File “val.py”, line 144, in <br>
outputs = net(ct_tensor)<br>
File “/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 150, in forward<br>
return self.module(*inputs[0], **kwargs[0])<br>
File “/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/shru/abdominal-multi-organ-segmentation-master/net/ResUnet_CE.py”, line 215, in forward<br>
inputs_stage2 = torch.cat((temp, inputs), dim=1)<br>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 128 and 256 in dimension 3 at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/generic/THCTensorMath.cu:71<br>
Hello, I am very new to pytorch. I am trying run a project available on github for abdominal multi organ segmentation and I keep getting this error. Any help would be great.</p>
Efficient Training for Neural Nets with Evolving Architecture

username: quantumjoel
cooked: <p>I’ve been experimenting with neural nets where the architecture can change, and as a result I am trying to make my code support any directed acyclic graph where each node is a neuron.</p>
<p>Right now to evaluate this net on a data point, the output neurons are evaluated recursively, working backwards through the graph (and caching values of nodes visited to avoid repeated work).</p>
<p>The basic optimization code I’m using looks standard</p>
<pre><code class="lang-auto">        criterion = nn.MSELoss()
        optimizer = torch.optim.SGD(self.parameters(), lr=0.1)
        for step in range(num_epochs):
            for target, vector in data:
                vector = vector[0]
                optimizer.zero_grad()
                pred = self(vector)
                loss = criterion(pred, target)
                loss.backward()
                optimizer.step()
                print(f'loss after {step} step optimization: ', loss.item())
</code></pre>
<p>except evaluating the line pred=self(vector) uses my custom-built forward evaluation function.</p>
<p>Does having something close to fully connected feedforward layers allow torch to better optimize?  Or should I just have a bunch of “layers” which only have one neuron?  I wonder if I am somehow making it repeat a lot of unnecessary work (such as if it’s rebuilding a computational graph each time but wouldn’t normally do that).</p>
<p>I’m somewhat of a beginner to how Torch works under the hood (before this experiment was usually training more standard kinds of neural nets and basing my code on templates).</p>
<p>Thanks so much in advance!<br>
Joel</p>
Copied Neural Nets Not Training Properly

username: quantumjoel
cooked: <p>I want to experiment with using genetic algorithms to evolve neural net architectures, and this requires creating duplicate copies of a neural net, mutating them in different ways, and training them separately to compare.</p>
<p>Calling deepcopy doesn’t work, but I built my own copy function which appears to copy everything over correctly (essentially by serializing the net and then building a new net from that data).  However, when I then attempt to train the duplicate neural net, the loss stays the same at each iteration.  Training the first net works as it should on the other hand.</p>
<p>I’m wondering if it has something to do with the optimizer looking at the computational graph for the first net instead of its copy, but I’m not sure - I don’t know a lot about how torch works under the hood.</p>
<p>Thanks in advance!  I’m new around here.</p>

username: ptrblck
cooked: <p>If you are creating a new model, you would also have to create a new optimizer and pass the newly created parameters to it.<br>
Using the old optimizer might work, if you just change the parameters somehow inplace, but we would need to see some code, which you are using, to debug it properly. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: quantumjoel
cooked: <p>Thank you!  It ended up being something simple I was overlooking, and I felt silly in hindsight.</p>
Turning 1d pixel image into labeled coordinates

username: kashdog
cooked: <p>Hi,</p>
<p>I’m trying to label the coordinates of 2 objects in 1D space using a neural network.</p>
<p>My input would look like 28 time steps of something like</p>
<p>[0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]</p>
<p>and the output should output the coordinates of both objects(e.g. [ 9. 21.]) for each time step</p>
<p>Could anyone provide some initial points to look at at and what loss I should use to train the network?</p>
Pytorch nvidia gtx460 version

username: keyser1
cooked: <p>Graphic card: nvidia GeForce GTX 460<br>
Operating system: Ubuntu 18.04<br>
Cuda version: ???</p>
<p>Hi! I need a little help, I am trying to get pytorch for my computer but don’t know what version i need. <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow noopener">https://pytorch.org/get-started/previous-versions/</a>  My graphics card is a little old, however I feel a little overwhelmed by the number of options. My software is up to date.</p>

username: ptrblck
cooked: <p>Unfortunately, your GPU uses the Fermi architecture with compute capability 2.1, if I’m not mistaken. The minimal compute capability is 3.5 and the binaries ship with 3.7, so you would either need a newer GPU or could use e.g. Google Colab, which provides free GPU runtimes for a specific time.</p>
The concept of rotating a tensor

username: Lame
cooked: <p>Hi all,</p>
<p>Could anyone please explain (theoretically) the meaning of the rotation along specific axis?</p>
<p>Let’s have an example of tensor of shape <code>x=[3,16]</code> and how if we want to rotate it along axis <code>Y</code> or <code>Z</code>.</p>
<p>Thanks for your help in advance.</p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://en.wikipedia.org/wiki/Rotation_matrix">Rotation Matrix Wikipedia article</a> which gives a good overview using these matrices.<br>
There are other approaches, e.g. quaternions, which might be a bit more complicated to get started.</p>
help！The number of model parameters decreases, but the memory consumption increases

username: Q_lee
cooked: <p>I replaced Conv2D layer with depthwise separable convolution. like this:</p>
<pre><code class="lang-auto">def conv_dw(in_channels, out_channels, kernel_size, stride, padding, dilation=1, with_relu=True):
    order_dict = collections.OrderedDict()
    order_dict['conv1'] = nn.Conv2d(in_channels, in_channels, kernel_size, stride, padding,
                                    dilation=dilation, groups=in_channels)
    if with_relu:
        order_dict['relu1'] = nn.ReLU(inplace=True)
    order_dict['conv2'] = nn.Conv2d(in_channels, out_channels, 1, 1, 0)
    if with_relu:
        order_dict['relu2'] = nn.ReLU(inplace=True)
    return nn.Sequential(order_dict)
</code></pre>
<p>The number of model parameters decreases to 1/4, but the memory consumption increases while training. And I got OOM,What should I do?</p>

username: aauker
cooked: <p>The memory consumption of a model during training is actually mostly dominated by the size or number of feature maps, not the network weights (parameters). Replacing one convolution with two may reduce the parameters but roughly doubles the number feature maps. This is normal and doesn’t require a fix.</p>
Efficient way to convert multiple PIL images to 5D batch tensor

username: glefundes
cooked: <p>I’m training a network that takes three RGB images as input. I have them stored as bytes in a LMDB file and I’m using tensorpack’s Dataflow to read them.</p>
<p>For each batch, I need to convert my images and ground truth to PyTorch tensors, yielding the images as a [N, 3, 3, 64, 64] Tensor.</p>
<p>Currently, I’m doing it the “easy” way with a nested loop to convert each byte array to a PIL Image and subsequentialy a tensor, and then stacking each set of three images into the final 5D batch tensor.</p>
<blockquote>
<pre><code>crops_tensor = torch.stack([torch.stack([transform_fn(Image.open(io.BytesIO(crop))) for crop in crops]) for crops in crops_batch])
</code></pre>
</blockquote>
<p>where <code>crops_batch</code> is yielded from the Dataflow at each iteration.</p>
<p>As you can imagine, this is a huge bottleneck for my script (as seen in the snakeviz profile below).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97d13b6882529548ac041f9a96b3200a5bb6439b.png" data-download-href="https://discuss.pytorch.org/uploads/default/97d13b6882529548ac041f9a96b3200a5bb6439b" title="snakeviz"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97d13b6882529548ac041f9a96b3200a5bb6439b_2_517x170.png" alt="snakeviz" data-base62-sha1="lF2ni3N6VU6P4Sr8aJHYOn6TWyL" width="517" height="170" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97d13b6882529548ac041f9a96b3200a5bb6439b_2_517x170.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97d13b6882529548ac041f9a96b3200a5bb6439b_2_775x255.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97d13b6882529548ac041f9a96b3200a5bb6439b_2_1034x340.png 2x" data-dominant-color="B5AEBF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">snakeviz</span><span class="informations">1067×352 33.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
What is a more efficient way for me to go from N 3D byte arrays to the 5D tensor used in my training routine?</p>
Mask only diagonal factor

username: izuna385
cooked: <p>I have an tensor, say, 128 * 128, like</p>
<pre><code class="lang-auto">tensor([[3851, 3971,  535,  ..., 3803, 3963,  485],
        [1437, 3264, 3824,  ..., 1611, 3128,  649],
        [3947, 3634,  511,  ..., 1114, 3478, 1490],
        ...,
        [3174, 1381,   -1,  ..., 3436,   -1, 1213],
        [2325, 2251,   -1,  ...,   70, 3354, 3038],
        [2031, 3100, 2623,  ..., 3309, 1328,  785]])

</code></pre>
<p>and filter this tensor.<br>
What I want to get is</p>
<pre><code class="lang-auto">tensor([[-1, 3971,  535,  ..., 3803, 3963,  485],
        [1437, -1, 3824,  ..., 1611, 3128,  649],
        [3947, 3634,  -1,  ..., 1114, 3478, 1490],
        ...,
        [3174, 1381,   -1,  ..., -1,   -1, 1213],
        [2325, 2251,   -1,  ...,   70, -1, 3038],
        [2031, 3100, 2623,  ..., 3309, 1328,  -1]])
</code></pre>
<p>I’m trying to find permutation by which I can get this, but can’t.<br>
If you’d know how to do this, I’d appreciate it much.<br>
Thanks.</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">x[x == x.diag()] = -1
</code></pre>

username: izuna385
cooked: <p>Thanks, I’ll try that.</p>
Size of training set

username: Upgrade_Yourself
cooked: <p>In the context of multilabel image classification, I am aware that the complexity of the model must match the size of the training set to not overfit. However, I don’t know what is the criteria.<br>
Can someone give me an approximate number of  samples depending on the standard models say resnetxxx, densenetxxx, xceptionxxx, or others. Also, it will surely depend on the number of the layers trained in the case of transfer learning.<br>
I could use all the help I can get.<br>
Thanks in advance</p>

username: mailcorahul
cooked: <p>There is no real limit for the size of training data. It does depend on the number of layers, but as long as you have more samples(say in the range 10k-100k, I have trained with even 500-1k samples) with diverse augmentations, it should be okay.</p>
<p>Also it depends on your usecase. For some applications, you may not find enough samples in which case you have to go with more augmentations.</p>
How to know which image is at what index in torchvision.datasets.ImageFolder?

username: Abhijeet_Ganapule
cooked: <p>Hi everyone,<br>
i have used   “trainset=datasets.ImageFolder(root=’/home/jessiepinkman/Traindata’,transform=apply_transform)” in my code to load images and convert them to tensors. Now after processing those, I get some label corresponding to each tensor. Now my problem is I need to know which index of the trainset[i], is related to which image in my dataset.</p>

username: ptrblck
cooked: <p>You can see the image paths and their corresponding class index using</p>
<pre><code>print(dataset.samples)</code></pre>

username: lalithnag
cooked: <p>Using the class_to_idx attribute gives the labels corresponding to each class. You can use:</p>
<pre><code class="lang-auto">print(trainset.class_to_idx)
</code></pre>
How to best use large images for training neural networks

username: Upgrade_Yourself
cooked: <p>I would like to ask you about how I should deal with the images I have. They are really large. They have this shape:  <strong>(3000, 4000, 3)</strong> .</p>
<p>I’m working on a  <em><strong>multilabel classification model</strong></em> .</p>
<p>And I want to know if it’s wise to  <strong>slice the images into equal tiles</strong>  (using  <strong>image_slicer</strong> ) and to feed them to the model  <strong>as well as resize the original</strong>  images to the requisite input size of the model and feed those too.</p>
<p><em>So the training set would be: resized_original_images + tiles_of_original_image</em></p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I’m not sure, if slices would work and I assume it depends a bit on your use case.<br>
E.g. if you are dealing with a classification use case, I would assume that smaller patches of the original image could not contain the valid class at all.<br>
So passing a patch with just “sky”, while the prediction could be “dog” could confuse the model.<br>
However, probably the best way is just to try it out and see, if the model is able to catch enough information from the smaller patches.</p>

username: Upgrade_Yourself
cooked: <p>Thanks for your answer.I’m actually going to slice said images before labeling each slice.Same goes for the original image. That way there will be no problem with the labels. I was just wondering if using these patches aas well as the original image would be a bit redundant for the neural network. I want to ask what is an appriximate good  number of samples. I don’t wish for the model to overfit.</p>
Pybind11 and pytorch dataloader

username: santisy
cooked: <p>Hello everyone,</p>
<p>In my experiment setting, I am trying to do some data pre-processing using C++ code. I have tried two approaches:</p>
<ol>
<li>Using subprocess to run C++ built executable and save the results to the hard disk. Then on the Python side, it reads the processed result.</li>
<li>Using pybind11 to warp the C++ code, and return a buffer to Python directly, which can be converted to numpy array.</li>
</ol>
<p>Both methods have been tested to be correct and coherent solely. However, when being integrated inside a dataloader (worker number &gt; 1), the second method met some problem. The situation was that the training would get some NaN value very soon. I do not know how to look into this. Is there any suggestion and tip for me to use pybind11 warpped C++ code along with pytorch dataloader?</p>
<p>Best</p>
<p>santis</p>

username: albanD
cooked: <p>Do you use the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">builtin cpp_extension</a> to bind your cpp code? That would allow you to work directly with Tensors and avoid potential errors where your buffer memory might get freed/reused before your python code is done with it.</p>

username: santisy
cooked: <p>No I do not use that. But I do return a buffer cv::Mat buffer, which is accessed by py::bufer_info and is converted to numpy array then. This numpy array is then converted to numpy array by <code>torch.from_numpy</code>.</p>

username: santisy
cooked: <p>Other information is that I was building the python module using pybind11 2.4 with C++ standard 11. I have seen some related question on pybind11 thread that talked about potential memory leak using C++ standard 11 building pybind11 2.4.</p>
Implementation of backward

username: ChenMinQi
cooked: <p>suppose I have:</p>
<blockquote>
<p>a = torch.tensor([3.0,], requires_grad=True)<br>
b = a * 3<br>
c = b * b * 3<br>
loss = c.mean()<br>
loss.backward()</p>
</blockquote>
<p>when the backward is executed:<br>
step1. calculate ∂loss/∂c<br>
step2. calculate ∂loss/∂b</p>
<p>actually step2 has 2 possibility<br>
case1:  ∂loss/∂b =  ∂loss/∂c *  ∂c/∂b<br>
case2:  ∂loss/∂b =  ∂(loss/∂c*c)/∂b</p>
<p>sow which case is happened?</p>
Size dismatch when use multi-cuda(torch.nn.DataParallel)

username: t6am3
cooked: <p>Hi~ I’m using a custom model like this</p>
<pre><code class="lang-auto">class SimpleNN(nn.Module):
    def __init__(self, vectors_size, features_size, hidden_size=15, dropout_rate=0.1):
        super(SimpleNN, self).__init__()

        self.vectors_size = vectors_size
        self.features_size = features_size
        self.hidden_size = hidden_size
        self.dropout_rate = dropout_rate

        self.vectors_hidden = nn.Sequential(
            nn.Dropout(self.dropout_rate),
            nn.Linear(vectors_size, vectors_size//2),
            nn.Tanh(),
            nn.Linear(vectors_size//2, features_size),
            nn.Tanh()
        )
        self.hidden = nn.Sequential(
            nn.Linear(features_size*2, hidden_size),
            nn.ReLU(),
        )

        self.output = nn.Linear(hidden_size, 2)

    def forward(self, pairs, features):
        """
        features: (n_samples, features_size)
        """
        vectors = pairs2vectors(train_pub, pairs).to(device)
        embedding_features = self.vectors_hidden(vectors)
        combined_features = torch.cat([features, embedding_features], dim=1)
        return self.output(self.hidden(combined_features))
</code></pre>
<p>This model works well when i use only one cuda, but after ‘DataParallel’ used like below, it always tell me the size of  <code>features</code>  and  <code>embedding_features</code>  are not match, i find that the n_samples shape of features doesn’t follow my expectation just like another batch data, i dont know why and how to solve this problem.</p>
<pre><code class="lang-auto">if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
        model = nn.DataParallel(model)
</code></pre>
<p>BTW, here is the pic of error message is <a href="https://i.stack.imgur.com/ZNb8x.png" rel="nofollow noopener">Error message</a></p>
Squeeze on first dimension when export onnx and convert tensorrt

username: jinfagang
cooked: <p>There is very common usage:</p>
<p>a = a.squeeze(dim=0)</p>
<p>this op export to onnx, can not convert to tensorrt since tensorrt doesn’t support operation on the first dimension.</p>
<p>Does there anyway to make it done?</p>

username: ptrblck
cooked: <p>Do you have this operation inside the model?<br>
If so, would it be possible to move it outside of it and add the batch dimension to your tensors beforehand?</p>
Ruby library, powered by LibTorch

username: ankane
cooked: <p>Hi, I wanted to let the community know that there are now <a href="https://github.com/ankane/torch-rb" rel="nofollow noopener">Ruby bindings</a> for LibTorch. Here’s a short example:</p>
<pre><code class="lang-auto">x = Torch.ones(2, 2, requires_grad: true)
y = x + 2
z = y * y * 3
out = z.mean
out.backward
p x.grad
</code></pre>
<p>If you have any feedback, feel free to create an issue in the project for it. Thanks!</p>

username: ptrblck
cooked: <p>Thanks for sharing!<br>
May I ask you, what you are using it for and which use cases are now enabled by Ruby? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ankane
cooked: <p>Good questions. I created it as part of a larger effort to improve machine learning in Ruby and am planning to try it for collaborative filtering. It currently supports most of the features of these two tutorials (<a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" rel="nofollow noopener">PyTorch intro</a> and <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" rel="nofollow noopener">torch.nn intro</a>).</p>
Detectron under Android

username: Chame_call
cooked: <p>Is that possible to embed Detectron into Android application?<br>
I can use object detection model in my Android app like any other model, can’t I??</p>

username: viswanatha_reddy
cooked: <p>I think it is not a good idea to embed models like Faster-RCNN and Mask-RCNN into Android. In order to do that you can use quantization techniques to reduce the memory.</p>

username: Chame_call
cooked: <p>what about SSD, Yolo?</p>

username: viswanatha_reddy
cooked: <p>For SSD and Yolo you can find many optimized versions easily. If you are aiming to run the model on gpu you can use the optimized model that is available in TensorRT for yolo. If your device contains cpu then try to do optimizations in openvino.</p>
LSTM change batch_first=True，loss does not decrease

username: ZhangJ
cooked: <p>I use CRNN in <a href="https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py" rel="nofollow noopener">https://github.com/meijieru/crnn.pytorch/blob/master/models/crnn.py</a>. The training process is normal.<br>
But for tensorrt deployment, when I change the batch_first=True in LSTM，the loss does not decrease.<br>
WHY？</p>
<pre><code class="lang-auto">class CRNN(nn.Module):

    def __init__(self, imgH, nc, nclass, nh, n_rnn=2, leakyRelu=False):
        super(CRNN, self).__init__()
        assert imgH % 16 == 0, 'imgH has to be a multiple of 16'

        # ks = [3, 3, 3, 3, 3, 3, 2]
        # ps = [1, 1, 1, 1, 1, 1, 0]
        # ss = [1, 1, 1, 1, 1, 1, 1]
        ks = [3, 3, 3, 3, 3]
        ps = [1, 1, 1, 1, 0]
        ss = [1, 1, 1, 1, 1]

        # nm = [64, 128, 256, 256, 512, 512, 512]
        nm = [64, 128, 128, 256, 256]

        cnn = nn.Sequential()

        def convRelu(i, batchNormalization=False):
            nIn = nc if i == 0 else nm[i - 1]
            nOut = nm[i]
            cnn.add_module('conv{0}'.format(i),
                           nn.Conv2d(nIn, nOut, ks[i], ss[i], ps[i]))
            if batchNormalization:
                cnn.add_module('batchnorm{0}'.format(i), nn.BatchNorm2d(nOut))
            if leakyRelu:
                cnn.add_module('relu{0}'.format(i),
                               nn.LeakyReLU(0.2, inplace=True))
            else:
                cnn.add_module('relu{0}'.format(i), nn.ReLU(True))

        convRelu(0)
        cnn.add_module('pooling{0}'.format(0), nn.MaxPool2d(2, 2))  # 64x16x64
        # convRelu(1)
        # cnn.add_module('pooling{0}'.format(1), nn.MaxPool2d(2, 2))  # 128x8x32
        convRelu(1, True)
        convRelu(2)
        cnn.add_module('pooling{0}'.format(2),
                       nn.MaxPool2d((3, 2), (2, 1), (0, 1)))  # 256x4x16
        convRelu(3, True)
        # convRelu(5)
        cnn.add_module('pooling{0}'.format(3),
                       nn.MaxPool2d((3, 2), (2, 1), (0, 1)))  # 512x2x16
        convRelu(4, True)  # 512x1x16

        self.cnn = cnn
        # self.rnn = BidirectionalLSTM(256, nh, nclass)
        self.rnn = nn.LSTM(256, nh,2,bidirectional=True, batch_first=True)
        self.linear = nn.Linear(nh * 2, nclass, bias=True)
        '''
        self.rnn = nn.Sequential(
            BidirectionalLSTM(512, nh, nh),
            BidirectionalLSTM(nh, nh, nclass))
        '''

    def forward(self, input):
        # conv features
        cnn = self.cnn(input)
        # print(cnn.size(),cnn._trt.shape)
        b, c, h, w = cnn.size()
        # assert h == 1, "the height of conv must be 1"
        # conv = conv.squeeze(2)
        conv = cnn.view(b,c,w)
        conv = conv.permute(0, 2, 1) # [w, b, c]
        self.rnn.flatten_parameters()
        rnn, (h0, c0) = self.rnn(conv)

        b, T, h = rnn.size()
        rec = rnn.reshape(b*T,h)

        e = self.linear(rec)  # [T * b, nOut]
        output = e.view(T, b,-1)
        return output

</code></pre>

username: ZhangJ
cooked: <p>Could Someone solve my problem?</p>
Dataloader does not return batch-size cut iterator. each data in iterator has full size as total data

username: 111192
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28da351a38703bb4fe1e6bdc8bd9934e6d023d0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/28da351a38703bb4fe1e6bdc8bd9934e6d023d0b" title="torch_forum_1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28da351a38703bb4fe1e6bdc8bd9934e6d023d0b.png" alt="torch_forum_1" data-base62-sha1="5PoBzMFkVJn1ROon5ugRfHZML4D" width="303" height="500" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch_forum_1</span><span class="informations">987×1628 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want features from every iteration have the size of (32, 390).<br>
but my custom dataloader gave me full size of data</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>The problem that you have is a MemoryError since you continue to use your whole dataset in your <code>__getitem__</code> method. In <code>__getitem__</code>, what you need to do is return 1 example from your dataset (so 1 feature tensor and 1 output tensor in your case). The PyTorch <code>DataLoader</code> will take care of giving you a batch tensor (it will concatenate <code>batch_size</code> tensors from your <code>__getitem__</code> function). With that in mind, here is a small example of how to adapt your code so you don’t get the error:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, data):
        self.data = data

    def __getitem__(self, idx):
        # do whatever you do with your columns.
        output_col = 4
        # only take the index of the data that you are interested in.
        output = self.data[idx, output_col]

        # arbitrary number on my part, use your columns!
        normalized_data = self.data[idx, 10:12]
        one_hot_encoded_data = self.data[idx, 90:100]

        features = np.concatenate([normalized_data, one_hot_encoded_data])
        features = torch.from_numpy(features)

        return features, output

    def __len__(self):
        return len(self.data)

data = np.random.rand(838682, 390)
dataset = CustomDataset(data)
loader = torch.utils.data.DataLoader(dataset, batch_size=32)

for feature, label in loader:
    print(feature.shape)
    # torch.Size([32, 12])
</code></pre>
<p>Hope it helps!</p>

username: 111192
cooked: <p>Thank you! It solves my problem:)</p>
Diffence of depthwise convolution in PyTorch and ONNX

username: pu_heng
cooked: <p>Hi, I found a strange problem when I tried to speedup model inference using depthwise convolution. I constructed a mobilenet model with depthwise conv(Model A) and then replaced dw conv with standard conv(Model B). I expected Model A would be faster than Model B because of less params and less computation cost, but what I oberserved was that Model B ran faster than Model A. Then I exported these two models to onnx format and made a test. A miracle happened!!! Model B performed much slower than Model A(about 3 times). What a confused problem it was. Moreover, I tested these two models in MXNet. Model A was still slower than Model B as same as in PyTorch. So I guess whether there are some differences between PyTorch and onnxruntime in internal mechanism. HELP ME, PLEASE…</p>
About memory reuse in pytorch

username: arsen
cooked: <p>The test code is:</p>
<p>import torch<br>
import torch .nn as nn</p>
<p>seq0 = nn.Sequential(nn.Conv2d(3, 3, 1), nn.Conv2d(3, 3, 1), nn.Conv2d(3, 3, 1), nn.Conv2d(3, 3, 1))</p>
<p>inp = torch.randn(1, 3, 224, 224)</p>
<p>def register_hook(module):<br>
def hook_func(module, input, output):<br>
print(type(input), id(input[0]), type(output), id(output))<br>
if (isinstance(module, hooked_modules)):<br>
module.register_forward_hook(hook_func)</p>
<p>seq0.train()<br>
seq0.apply(register_hook)</p>
<hr>
<p>It prints as:<br>
&lt;class ‘tuple’&gt; 139634598790880 &lt;class ‘torch.Tensor’&gt; 139634598874800<br>
&lt;class ‘tuple’&gt; 139634598874800 &lt;class ‘torch.Tensor’&gt; 139634598874880<br>
&lt;class ‘tuple’&gt; 139634598874880 &lt;class ‘torch.Tensor’&gt; 139634598874800<br>
&lt;class ‘tuple’&gt; 139634598874800 &lt;class ‘torch.Tensor’&gt; 139634598874880</p>
<p>Obviously, memory reuse comes up in the network forward procedure. When pytorch reuses memory, how to calculate gradients because the intermediate Tensors may have been covered？</p>

username: albanD
cooked: <p>Hi,</p>
<p>Do you mean how does pytorch computes gradients? It will keep all the values it needs but not necessarily as python objects as they are quite expensible to work with.</p>

username: arsen
cooked: <p>Thank you very much.<br>
I have realized that when I saw the code:<br>
<a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function</a></p>
<p>In the forward procedure,</p>
<blockquote>
<blockquote>
<blockquote>
<pre><code>def forward(ctx, i):
    result = i.exp()
    ctx.save_for_backward(result)
    return result
</code></pre>
</blockquote>
</blockquote>
</blockquote>
<p>it saves the results for backward and the memory footprint is reused.</p>
Module design: optimizer and loss inside module

username: Valiox
cooked: <p>We’re working on a fairly big model that has a lot of “plug-in” components (that I’ll call submodules), i.e. smaller models that are trained on intermediate results of the big model during the forward pass. Each of these submodules is trained at the same time as the big model, through an individual optimizer and loss function for each. Submodules can be individually enabled or disabled.</p>
<p>The problem we have is that the individual optimizers and loss functions are starting to clutter the main training loop code. For each submodule there needs to be a check whether that module is activated or not, a call to the backward pass, a call to the optimizer step function and a call to zero grad. Only the forward pass is made in a single call, as calling the forward pass on the big model “cascades” into calling the forward pass of each submodule at some point.</p>
<p>Furthermore, to compute the loss for each submodule the ground truth tensors and the predicted tensors of the submodules must make their way to the loss function somehow. As we only have a single loss function that computes it for each submodule, all these tensors end up in the main training loop so as to be used by that loss function. This is clearly not scalable design.</p>
<p><strong>Does torch have an inbuilt mechanism to deal with this situation?</strong> I was thinking of refactoring the code in a way that these submodules would have their own individual optimizer and loss function. To avoid having to cache the result of each submodule’s forward function, I was thinking of having a <code>step</code> function that would make the forward-loss-backward pass operations in one go, and an <code>optim_step</code> function that calls the optimizer step and then zero grad. As the step function is called on the big model, it is also called on every submodule in a cascading fashion. The same goes for the optim step.</p>

username: ptrblck
cooked: <p>Your idea makes generally sense.<br>
One thing I would worry about is, if you are planning on using <code>(Distributed)DataParallel</code> at some point.<br>
I’ve seen a lot of errors trying to just wrap the model into <code>DDP</code>, if there were a lot of these “convenience functions”.</p>

username: Valiox
cooked: <blockquote>
<p>One thing I would worry about is, if you are planning on using  <code>(Distributed)DataParallel</code>  at some point.<br>
I’ve seen a lot of errors trying to just wrap the model into  <code>DDP</code> , if there were a lot of these “convenience functions”.</p>
</blockquote>
<p>That is precisely what I’m dealing with at the moment indeed: the main module is wrapped in DP and I’ve had to disable that for now. Hope I’ll find a nice way to come around it</p>
Failed Compile source code in develop model in Mac Os Catalina 10.15.1

username: Wei_Ma
cooked: <p>In file included from …/caffe2/python/pybind_state.cc:1:<br>
In file included from …/caffe2/python/pybind_state.h:16:<br>
In file included from …/caffe2/python/pybind_state_dlpack.h:9:<br>
In file included from …/torch/include/pybind11/pybind11.h:44:<br>
In file included from …/torch/include/pybind11/attr.h:13:<br>
In file included from …/torch/include/pybind11/cast.h:13:<br>
In file included from …/torch/include/pybind11/pytypes.h:12:<br>
…/torch/include/pybind11/detail/common.h:112:10: fatal error: ‘Python.h’ file not found<br>
<span class="hashtag">#include</span> &lt;Python.h&gt;<br>
^~~~~~~~~~<br>
1 error generated.<br>
[6/680] Building CXX object caffe2/CMakeFiles/math_test.dir/utils/math_test.cc.o<br>
…/caffe2/utils/math_test.cc:459:23: warning: comparison of integers of different signs: ‘int’ and ‘std::__1::vector&lt;float, std::__1::allocator &gt;::size_type’ (aka ‘unsigned long’) [-Wsign-compare]<br>
for (int i = 0; i &lt; Y_data.size(); ++i) {<br>
~ ^ ~~~~~~~~~~~~~<br>
…/caffe2/utils/math_test.cc:167:17: warning: unused variable ‘kEps’ [-Wunused-const-variable]<br>
constexpr float kEps = 1e-5;<br>
^<br>
2 warnings generated.<br>
ninja: build stopped: subcommand failed.<br>
Traceback (most recent call last):<br>
File “setup.py”, line 755, in <br>
build_deps()<br>
File “setup.py”, line 316, in build_deps<br>
cmake=cmake)<br>
File “/Users/mawei/ml_mutation/pytorch/tools/build_pytorch_libs.py”, line 62, in build_caffe2<br>
cmake.build(my_env)<br>
File “/Users/mawei/ml_mutation/pytorch/tools/setup_helpers/cmake.py”, line 337, in build<br>
self.run(build_args, my_env)<br>
File “/Users/mawei/ml_mutation/pytorch/tools/setup_helpers/cmake.py”, line 141, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/Users/mawei/anaconda3/lib/python3.7/subprocess.py”, line 347, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command '[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’</p>
Brew install libtorch

username: ankane
cooked: <p>Hi, I wanted to let the community know that LibTorch is <a href="https://github.com/Homebrew/homebrew-core/pull/47222" rel="nofollow noopener">now available</a> on Homebrew.</p>
<pre><code class="lang-bash">brew install libtorch
</code></pre>
<p>If improvements can be made to the formula, please submit a pull request directly to <a href="https://github.com/Homebrew/homebrew-core" rel="nofollow noopener">Homebrew</a>. Thanks!</p>
Enforcing parameter constraints in a Module

username: Sam_Pr
cooked: <p>I have a custom <code>Module</code> that is a reusable layer, and it needs to constrain its weight parameters to between 0 and 1. The simplest and most obvious way to do this is to clip them after every optimiser step. I’m not interested in techniques that try to enforce them by guiding them with a modified loss, as it’s mathematically invalid for them to ever go out of that range.</p>
<p>I can make this work by calling a constraint function after <code>optimiser.step()</code> in my training loop. However, this is an implementation detail of the module, and I don’t want every training loop for any model that internally uses this module to have to remember to call this function in order to work. The responsibility for this constraint should be contained within the module itself.</p>
<p>I’ve not been able to find any hooks that fire after an update occurs, so I’m not sure how to go about enforcing this. Is there a nice way to do this?</p>
<p>The best I’ve come up with is enforcing it in <code>forward</code>, but that’s wasteful if it hasn’t changed since the last run.</p>

username: Sam_Pr
cooked: <p>Another approach would be to apply some kind of normalisation in the forward pass, but that feels like a hacky approach when all I need to do is constrain the values in a tensor. I don’t want to add extra complexity to the graph.</p>
Is there any way to parallelize code over modules that don’t depend on each other?

username: amatsukawa
cooked: <p>In TF, independent sections of the graph are automatically executed in parallel if called together in one session.run(…) call. A simple case is this:</p>
<pre><code class="lang-auto">xs = [ ... ] # list of torch.Tensors
models = [ ... ]  # list of nn.Modules
out = [m(x) for m, x in zip(models, xu)]
</code></pre>
<p>It seems to me if this code is dynamically executed, it is not possible to parallelize the calls of m1(x1), m2(x2), … mn(xn) for some obvious performance benefits.</p>
<p>What is the best way in PyTorch to achieve this effect? Is it to use the jit?</p>

username: driazati
cooked: <p>The JIT execution won’t run anything in parallel, right now it mostly does code transformations and fusions to optimize performance.</p>
<p>If you have multiple GPUs, <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">DataParallel</a> might be helpful.</p>
Torch.max() - Get two highest values

username: Maria_Fernanda_De_La
cooked: <p>I want to get the two highest values from a number. I am thinking of getting the max, removing it, and then get the max again. Does anyone have a link to the code implementation for this function?</p>
<p>Thank you in advance</p>

username: KFrank
cooked: <p>Hi Maria!</p>
<aside class="quote no-group" data-username="Maria_Fernanda_De_La" data-post="1" data-topic="62341" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria_fernanda_de_la/48/11812_2.png" class="avatar"> Maria_Fernanda_De_La:</div>
<blockquote>
<p>I want to get the two highest values from a number.</p>
</blockquote>
</aside>
<p>Does <a href="https://pytorch.org/docs/stable/torch.html#torch.topk" rel="noopener nofollow ugc">torch.topk()</a> do what you want?  I would expect it to be the most<br>
efficient approach.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Maria_Fernanda_De_La
cooked: <p>Amazing! Thank you so much!</p>
Use linear.weight or linear.weight.data

username: Allan_Jie
cooked: <p>Here is my problem, I have a linear layer.</p>
<ol>
<li>Everytime I forward, I want to make the linear weight multiply a mask.</li>
</ol>
<pre><code class="lang-python">linear = nn.Linear(10, 10) ## for example
</code></pre>
<p>Now during forward</p>
<pre><code class="lang-python">   def forward(inputs):
       ## which way below is the correct way to use? mask is with all 0 and 1 values/
        self.linear.weight = self.linear.weight * torch.from_numpy(self.mask)
        self.linear.weight.data= self.linear.weight.data * torch.from_numpy(self.mask)

     ### In the following code, I will use the updated linear layer to do something and 
     ### want the model to check gradient
</code></pre>

username: albanD
cooked: <p>You want to do:</p>
<pre><code class="lang-auto">with torch.no_grad():
   self.linear.weight.mul_(torch.from_numpy(self.mask))
</code></pre>

username: SimonW
cooked: <p>actually you want</p>
<pre><code class="lang-auto">weight = self.linear.weight * torch.from_numpy(self.mask)
output = F.linear(input, weight, ...)
</code></pre>
CUDA error while passing hidden states to convLSTM

username: anandaltekar
cooked: <p>Help need regarding the following question:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/59075481/cuda-error-while-passing-hidden-states-to-convlstm" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4060920/qbit" target="_blank" rel="nofollow noopener">
    <img alt="qbit" src="https://i.stack.imgur.com/asxWD.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/59075481/cuda-error-while-passing-hidden-states-to-convlstm" target="_blank" rel="nofollow noopener">CUDA error while passing hidden states to convLSTM</a>
</h4>

<div class="tags">
  <strong>cuda, conv-neural-network, pytorch, cpu</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4060920/qbit" target="_blank" rel="nofollow noopener">
    qbit
  </a>
  on <a href="https://stackoverflow.com/questions/59075481/cuda-error-while-passing-hidden-states-to-convlstm" target="_blank" rel="nofollow noopener">05:23PM - 27 Nov 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Average Stacked Bidirectional LSTM hidden states, based on directions

username: gionanide
cooked: <p>Is that right if I want to average the hidden states h, c based on the directions. So the resulting tensors will be (num_layers, batch, hidden_units), and the hidden units are averaged based on directions?</p>
<pre><code class="lang-auto">                product_lstm_output, (product_h, product_c) = self.product_lstm(product_inputs)                   
                product_h = product_h.view(self.number_of_layers, self.directions, self.batch, self.hidden_units)#break them down
                product_c = product_c.view(self.number_of_layers, self.directions, self.batch, self.hidden_units)
                
                product_h = torch.mean(product_h,dim=1)# average with respect to directions
                product_c = torch.mean(product_c,dim=1)
</code></pre>