akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Depthwise convolution for gradient filters

markorei94: <p>I am trying to make a Sobel-type filter by adapting the follwoing Tensorflow code into Pytorch.</p>
<pre><code class="lang-auto">def get_gradient_filters():
    np_grad_x = np.asarray([[-3,0,3], [-10,0,10], [-3,0,3]], dtype=np.float32).reshape((3, 3, 1, 1))
    np_grad_x /= np.sum(np.abs(np_grad_x), keepdims=True)
    tf_grad = tf.constant((np.concatenate([np_grad_x, np_grad_x.transpose((1, 0, 2, 3))], axis=-1)))
    return tf_grad

tf_grad_filter = get_gradient_filters()
tf_grad_filter = tf.tile(tf_grad_filter, (1, 1, 3, 1))

image=skimage.data.astronaut().astype(np.float32)/255.

tf_f=tf.expand_dims(tf.constant(image),0)
tf_f_grad = tf.nn.depthwise_conv2d(tf_f, tf_grad_filter, [1, 1, 1, 1], "SAME")/(1./32)
tf_f_grad=tf.reshape(tf_f_grad,(1,512,512,3,2))
</code></pre>
<p>Here is my attempt which does not give the same result.</p>
<pre><code class="lang-auto">def gradient_filter(C):
    np_grad_x = np.asarray([[-3, 0, 3], [-10, 0, 10], [-3, 0, 3]], dtype=np.float32).reshape((3, 3, 1, 1))
    np_grad_x /= np.sum(np.abs(np_grad_x), keepdims=True)
    np_grad = np.concatenate([np_grad_x, np_grad_x.transpose((1, 0, 2, 3))], axis=-1)
    np_grad = np.tile(np_grad, (1, 1, C, 1))
    torch_grad = torch.Tensor(np_grad).reshape(1, 3, 3, 2 * C).permute(3, 0, 1, 2)
    filter = torch.nn.Conv2d(in_channels=C, out_channels=2 * C, kernel_size=(3, 3), padding=(1, 1), padding_mode='zeros',
                       groups=C, bias=False)
    filter.weight.data = torch_grad
    return filter

filter = gradient_filter(3)
def img_gradient(torch_f, filter):
    M, N, _ = torch_f.shape
    return filter(torch_f.unsqueeze(dim=0).permute(0, 3, 1, 2)) /(1./max(M,N))

torch_f=torch.Tensor(image)
torch_f_grad=img_gradient(torch_f,filter)
torch_f_grad=torch_f_grad.permute(0,2,3,1).reshape(1,512,512,3,2)
</code></pre>
<p>I also tried some minor variations (first permute then reshape) but it is still not the same. Any ideas?</p>

ptrblck: <p>How large is the max. absolute difference? If it’s in the range of approx. <code>1e-6</code> you might be running into the floating point precision for <code>float32</code>.</p>

markorei94: <p>The differences are not that small (some values in outputs are even of size 1e2) but for some examples of images it did seem as if every term of one output is corresponding the term in other output with some precision error. What do you recommend in that case?</p>

markorei94: <p>Edit: I tried with float64 and the difference is indeed now at most 1e-5. Is it possible to even decrease this?</p>

ptrblck: <p>Did you see a difference of <code>1e2</code> for some output values and others were much smaller?<br>
If so, this sounds like an overflow issue, which I wouldn’t expect to happen in <code>float32</code>.</p>

markorei94: <p>I can not reproduce the example with difference <code>1e2</code> right now and I think it was caused by something else at that moment. It seems that my code might be correct after all and that difference is due to floating point precision.</p>
Linear layer with local connections

haalim: <p>I want to implement linear regression in Pytorch with sparse connections. To build such a network I cannot use nn.Linear because it is a densely connected layer. I have first tried to make a binary matrix with 0’s and 1’s to indicate the presence and absence of connections. But it won’t work in my case because it is not computationally efficient. I am looking for a sparse weight matrix or sparse linear layer type solutions. I would be really grateful if you would help me in this regard. I have attached a screenshot also to explain what im trying to do.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/297371eb273f355caa653754fd097917034d82ea.png" alt="local_connections" data-base62-sha1="5UGUNhW3TEFB9BcPjBUerYxE1Au" width="191" height="277"></p>

googlebot: <p>How is binary mask matrix not efficient? That should only be the case with 1)no gpu 2)huge weight matrix (megabytes) 3)big sparsity, like 90%. If your scenario is similar to that, you can try torch.sparse.mm.</p>

haalim: <p>Im working with images. Consider 128x128 pixels color image, A linear transformation for these size images can be represented by a matrix of size 49152 x 49152.</p>
<p>So, in this case dense layer would not work for me. Thats why im looking look for sparse solution. Can you please explain how can i use torch.sparse.mm instead of nn.Linear?</p>
<p>I didn’t get your point.</p>

pfloat: <p>are you looking for matrix operations over matrices that are sparse by block ? That would surely reduce you dimensionality problem.</p>
<p>In that case, what about using several <code>Linear</code> layers that are not sparse instead of a large sparse<code>Linear</code> layer?</p>

googlebot: <aside class="quote no-group" data-username="haalim" data-post="3" data-topic="96201">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haalim/48/11990_2.png" class="avatar"> haalim:</div>
<blockquote>
<p>images</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="haalim" data-post="3" data-topic="96201">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haalim/48/11990_2.png" class="avatar"> haalim:</div>
<blockquote>
<p>49152 x 49152</p>
</blockquote>
</aside>
<p>Can’t you express the needed connectivity with convolutions?</p>
<p>Anyway, about sparse, I just checked if it works:</p>
<pre><code class="lang-auto">i = torch.LongTensor([[0, 1, 1], [2, 0, 2]])
v = nn.Parameter(torch.FloatTensor([3, 4, 5]))
w = torch.sparse.FloatTensor(i,v, torch.Size([2,3]))
w.to_dense()
  tensor(cpu,(2, 3)[[0., 0., 3.],
        [4., 0., 5.]], grad_fn=&lt;ToDenseBackward&gt;)
x = torch.randn(5,3)
y=torch.sparse.mm(w, x.t()).t()
y.shape
  torch.Size([5, 2])
y.backward(torch.ones_like(y))
v.grad
  tensor(cpu,(3,)[-1.6546, -3.2455, -3.2455])
</code></pre>
Unable to understand generating the data process in parallel

bing: <p>Hi Guys,<br>
I am trying to generate data in parallel using <a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" rel="nofollow noopener">this</a> tutorial. I am using it for image classification problem so I have image data in jpeg format.<br>
I have a very basic doubt, in the below code , the data is being accessed using <code>X = torch.load('data/' + ID + '.pt')</code>, is this some sort of pickling the data or is the general notation for data of every kind.<br>
How to convert my image data in this format(Also is this a better approach or not)?</p>
<pre><code>def __getitem__(self, index):
'Generates one sample of data'
# Select sample
    ID = self.list_IDs[index]
# Load data and get label
    X = torch.load('data/' + ID + '.pt')
    y = self.labels[ID]

    return X, y</code></pre>

ptrblck: <aside class="quote no-group" data-username="bing" data-post="1" data-topic="95918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<p>Also is this a better approach or not</p>
</blockquote>
</aside>
<p>It depends a bit on your use case.<br>
You could save some data loading time, if you preload the data (e.g. images), process it all, and store it as binary data. However, since e.g. JPEG images are compressed, while binary data is usually not, you could potentially use more space on your device.<br>
Also, in order to augment the data on the fly during training you would have to transform the samples back to images, if you want to use <code>PIL.Image</code> transforms, and then back again to tensors, so that your initial benefit might not be visible anymore.</p>
Error with "ImbalancedDatasetSampler"

swap: <p>Hi,<br>
I am dealing with imbalanced data (mere 2% minority samples). I tried “WeightedRandomSampler” approach which only works OK for my validation set, but it fails in case of independent test set. I came across <a href="https://github.com/ufoym/imbalanced-dataset-sampler" rel="nofollow noopener">https://github.com/ufoym/imbalanced-dataset-sampler</a> and I wanted to try this approach on my data. The problem is - this ‘ImbalancedDatasetSampler’ module can’t figure out labels in my TensorDataset object.</p>
<p><code>train_loader = data_utils.DataLoader(train_dataset,                                       batch_size = BATCH_SIZE,                                     sampler=ImbalancedDatasetSampler(train_dataset))</code></p>
<p>it returns error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
&lt;ipython-input-13-b7cc711025fa&gt; in &lt;module&gt;
      2 train_loader = data_utils.DataLoader(train_dataset, 
      3                                      batch_size = BATCH_SIZE,
----&gt; 4                                     sampler=ImbalancedDatasetSampler(train_dataset))

~/py_torch_sampler.py in __init__(self, dataset, indices, num_samples, callback_get_label)
     30         label_to_count = {}
     31         for idx in self.indices:
---&gt; 32             label = self._get_label(dataset, idx)
     33             if label in label_to_count:
     34                 label_to_count[label] += 1

~/py_torch_sampler.py in _get_label(self, dataset, idx)
     51             return self.callback_get_label(dataset, idx)
     52         else:
---&gt; 53             raise NotImplementedError
     54 
     55     def __iter__(self):

NotImplementedError: 

</code></pre>
<p>Could someone tell me how can I solve this problem?</p>

ptrblck: <p>Based on the provided stack trace it seems the <code>NotImplementedError</code> is raised by the <code>ImbalancedDatasetSampler</code>, so I would recommend to create an issue in the project’s GitHub repository for better visibility. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Pytorch boradcasting

pytorch_lzwhard: <p>I have a weight tensor(64,3, 3, 3) and a scale tensor(64), how to multiply a scale to per filter, a slow solution is showed below:</p>
<pre><code class="lang-python">for i in range(weight.shape[0]):
    weight[i] = weight[i] * scale[i]
</code></pre>
<p>can boradcasting apply to this? please help me out, tks!</p>

pfloat: <p>This should work:</p>
<pre><code class="lang-auto">q = torch.randn(64,3,3,3)
t = torch.randn(64)
qq = t.unsqueeze(1).unsqueeze(1).unsqueeze(1)  * q
for i in range(64):
    torch.allclose(t[i] * q[i], qq[i]) # should be true all the time
</code></pre>

Sunshine352: <p>This should work:<br>
weight = weight* scale.unsqueeze(dim=1).unsqueeze(dim=1).unsqueeze(dim=1)<br>
or<br>
weight = weight* scale.view(-1, 1, 1, 1)<br>
etc.</p>

pytorch_lzwhard: <aside class="quote no-group" data-username="Sunshine352" data-post="3" data-topic="96202">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshine352/48/7667_2.png" class="avatar"> Sunshine352:</div>
<blockquote>
<p>gh</p>
</blockquote>
</aside>
<p>It works! thanks a lot.</p>
Sound event detection model not learning

fcola: <p>Hello everyone,</p>
<p>I am fine tuning a model for sound event detection, taken from <a href="https://github.com/qiuqiangkong/audioset_tagging_cnn" rel="nofollow noopener">https://github.com/qiuqiangkong/audioset_tagging_cnn</a>, on the Urbansed dataset. In this task, the model should predict a [batch, n_classes, time_steps] matrix, with a value of one indicating the presence of an event at a certain time step.<br>
However, my network does not seem to train. Specifically, after about the first 10 epochs, my loss stops decreasing. If i check the predictions of the model, the output is composed entirely of 0.5s.<br>
I have tried:</p>
<ul>
<li>Changing the amount of l2 regularization, even turning it off completely</li>
<li>Changing the learning rate</li>
<li>Doing a mock training with only 2 samples to see if the network could learn the simple problem. The result was the same matrix of 0.5.</li>
<li>Different optimizers (Adam and SGD so far)</li>
<li>BCEWithLogitsLoss with reduction = mean and sum (using this loss as in theory multiple classes can be active at a time)</li>
</ul>
<p>My loss and optimizer:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss(reduction='sum')
optimizer = optim.SGD(model.parameters(), lr=0.001)
</code></pre>
<p>My training loop:</p>
<pre><code class="lang-auto">    for i, data in enumerate(dataloader_train):
        inputs, labels = data
        inputs = inputs.type(torch.FloatTensor)
        optimizer.zero_grad()
        outputs = model(inputs).cpu()
        loss = 0
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
</code></pre>
<p>I can’t figure out what’s wrong. Any thoughts?<br>
Thanks,<br>
Federico</p>

Henry_Chibueze: <p>Try viewing ur data to see what it’s actually training with</p>

fcola: <p>Thank you for your suggestion. Tried to visualize the data and nothing seems out of place, the spectrograms appear correct.</p>

Henry_Chibueze: <p>I think u should manually initalize ur networks weights and try again<br>
Btw what is ur loss func, learning rate and optimizer?</p>

fcola: <p>Thanks, I will try that, although that would mean I cannot do transfer learning. As for loss function, optimizer and lr I’m using:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss(reduction='sum')
optimizer = optim.SGD(model.parameters(), lr=0.001)
</code></pre>
<p>Although I also tried Adam and different learning rates and l2 regularization values.</p>

Henry_Chibueze: <p>Well try using the Adams optimization included with the weights initialization (weight values should be very close to zero by not too small and uniformly distributed)<br>
If it still continues not to learn then try changing network architecture</p>
What affects quality/accuracy of image classification results?

Forelli: <p>Hi</p>
<p>For a further education I have analyzed PyTorch for image classification (with Kaggle Dogs vs. Cats). In particular I investigated what influences the quality/accuracy of the results.</p>
<p>I claim that the following points are most important (sorted by importance):</p>
<ol>
<li>amount of data (I recommend at least 20’000 images for categorizations with two types, DogsVsCats example have 25’000)</li>
<li>data quality</li>
<li>division Train/Test-Data (ideal results at 90%/10%)</li>
<li>model used</li>
<li>script modifications (e.g. optimizer, transforms)</li>
</ol>
<p>The following properties have a small influence on the accuracy:</p>
<ul>
<li>Size of the images</li>
<li>Alignment of the images</li>
</ul>
<p>Furthermore I claim that the hardware used (CPU/CUDA) does not influence the quality of the results, but clearly the duration.</p>
<p>One more question for long-time users:Have new versions of PyTorch significantly improved the quality/accuracy or is that just model related? Unfortunately I cannot test this.</p>
<p>Would you accept this as a professor or have I forgotten important points?</p>
<p>Cheers<br>
Forelli</p>
Can we concatenate two GPUs to run single big model?

Yupjun: <p>Let’s say I have two GPUs with 4GB VRAM and I have a 6GB model.</p>
<p>Can I run this model by utilizing the two GPUs?</p>

ptrblck: <p>You could use model sharding, i.e. executing different parts of the model of different devices, as given in <a href="https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239/2">this example</a>.</p>
Compiling Python Code

kempj: <p>I’m new to glow and am trying to understand exactly what it does and how it can be used.</p>
<p>From reading the paper I get the impression that normal pytorch code can be compiled an optimized, but from what I see, the closest thing to this is the torch_glow library. Is this the case or have I missed something?</p>
<p>Thanks.</p>

jfix: <p>Hi <a class="mention" href="/u/kempj">@kempj</a>, Glow has multiple front ends. That is, we can load models from ONNX or Caffe2 protobufs, or through TorchScript from PyTorch (<code>torch_glow</code>). Or you could even write your model directly in C++ with our C++ APIs if you wanted <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">.</p>
<p><code>torch_glow</code> is under active development/is less mature than our ONNX and Caffe2 protobuf loaders. If you have problems with <code>torch_glow</code>, another path from PyTorch to Glow is to export your model to ONNX and then load into Glow.</p>

tkg.rahul: <p>Hi <a class="mention" href="/u/jfix">@jfix</a>, as per your Apr 8 comment torch_glow is under active dev. Requesting you to comment on current state of troch_glow as per current date?</p>

jfix: <p>It’s definitely still under active development. It’s gotten more mature, but not sure if it’s easily buildable still. You can see the latest build instructions from a couple months ago <a href="https://github.com/pytorch/glow/blob/master/docs/pytorch.md" rel="nofollow noopener">here</a>.</p>
Get image from webcam and load into pytorch

Andre_Almeida: <p>Hi.</p>
<p>I’m doing a project with pytorch and I need to get a single image and pass to my CNN.<br>
I already have the code working to get the image from my webcam. My CNN is trained and now I need to pass my image to the CNN.</p>
<p>Is that possible with some custom Dataset? Anyone has some tip to give me about that?</p>

tom: <p>Usually, you just take the image )say as numpy array from OpenCV), convert to a tensor and unsqueeze the batch dimension. You need to apply the same transformations, for OpenCV you might also have to shuffle the channels BGR-&gt;RGB.</p>
<p>Best regards</p>
<p>Thomas</p>

Andre_Almeida: <p>Thank you!<br>
I will try that!</p>

edgarriba: <p>in <a href="http://www.kornia.org" rel="nofollow noopener">kornia</a> we have this handy function for that:</p>
<pre><code class="lang-python">import cv2
import kornia
import torch
import numpy as np

def load_opencv_img(img_path: str) -&gt; torch.Tensor:
    img: np.ndarray = cv2.imread(img_path, cv2.IMREAD_COLOR)  # HxWx3
    img_t: torch.Tensor = kornia.image_to_tensor(img)  # 1xCxHxW
    img_t = kornia.bgr_to_grayscale(img_t)  # Bx1xHxW
    return img_t
</code></pre>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:84

wupatty: <p>Hi,<br>
I currently tried to run yolov5 + deepsort.<br>
When I trained yolov5 by using MOT16 datasets, I got this error.<br>
Looks like something happened when evaluating validation data.<br>
I can’t figured out what happened. (I’ve checked that the bboxes are normalized to 0,1)<br>
Could you give me some suggestions?<br>
Thanks a lot!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f60eba96a4db63564a3fe7d7ebf98abfe5b7c11.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f60eba96a4db63564a3fe7d7ebf98abfe5b7c11" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f60eba96a4db63564a3fe7d7ebf98abfe5b7c11.png" alt="image" data-base62-sha1="ksnVDorKeaF7YtMW0GQ8jVstzBn" width="690" height="95" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1911×265 19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b61e392a0bdb55615ae3e70963ff3e9aaca219f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/b61e392a0bdb55615ae3e70963ff3e9aaca219f4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b61e392a0bdb55615ae3e70963ff3e9aaca219f4_2_690x182.png" alt="image" data-base62-sha1="pZ5Et7U4HFGkOoi4gq8iuOYUiTq" width="690" height="182" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b61e392a0bdb55615ae3e70963ff3e9aaca219f4_2_690x182.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b61e392a0bdb55615ae3e70963ff3e9aaca219f4_2_1035x273.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b61e392a0bdb55615ae3e70963ff3e9aaca219f4_2_1380x364.png 2x" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1902×503 36.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems an indexing operation is creating an out of bounds error.<br>
You could try to run the code on the CPU, which would error directly in the offending line of code or rerun your script with <code>CUDA_LAUNCH_BLOCKING=1 python script.pt args</code>, which should make sure the stack trace points to the right line of code where the indexing is failing.</p>
Increse the dimensions of tensor in java

SungmanHong: <p>I have (pytorch) tensor object in java.</p>
<p>The shape of tensor is (3 x H x W) where 3 represent (R,G,B) each.</p>
<p>The tensor is actually gained from bitmap image (shape = H x W)<br>
by performing the following command, outputting (3 x H x W) which is in the TensorImageUtils.<br>
(<a href="https://pytorch.org/docs/1.3.1/org/pytorch/TensorImageUtils.html" class="inline-onebox" rel="noopener nofollow ugc">TensorImageUtils — PyTorch master documentation</a>)</p>
<pre><code class="lang-auto">final Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);
</code></pre>
<p>but I need (1 x 3 x H x W) tensor, not (3 x H x W) in java.<br>
In python, that can be performed by</p>
<blockquote>
<p>torch.unsqueeze(x, 0) where x is (3 x H x W) tensor<br>
(<a href="https://pytorch.org/docs/stable/generated/torch.unsqueeze.html" class="inline-onebox" rel="noopener nofollow ugc">torch.unsqueeze — PyTorch 2.1 documentation</a>)</p>
</blockquote>
<p>in java, I think there is no unsqueeze command, how can i expand the dimension of tensor in java?</p>
Extract raw data or dataset from shuffled dataloader

Jona: <p>Hi,</p>
<p>Do you know how to extract the raw data or Dataset (in the original order when the Dataloader was created) from a shuffled Dataloader?</p>
<p>Best regards</p>

mathematics: <p>Did you mean, Extracting inputs and targets from DataLoader?</p>

Jona: <p>Yes, that’s what I mean … I want to extract the raw features/target in the same order of the original data, from which the dataset was constructed … and from which the shuffled dataloader was loaded</p>
<p>I’m looking for something like</p>
<p><code>dl = DataLoader(....shuffled=True)</code><br>
<code>dl.data or dl.dataset # ordered and original data, not shuffled</code></p>
<p>Best regards</p>

mathematics: <p>apologies but i dont know to revert the shuffling process of <code>DataLoader</code>,</p>
<p>I think to extract raw files we can get using like this,</p>
<pre><code class="lang-auto">
for input, target in dl:
     ...
</code></pre>

Jona: <p>But it that, the raw files/data would be shuffled (unordered) …</p>
<p>Thanks anyway <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <a class="mention" href="/u/mathematics">@mathematics</a></p>

mathematics: <p>While building <code>DataLoader</code>  , if <code>shuffle</code> argument is not passed,<br>
I think It will be ordered like raw files because <code>shuffle=False</code> is default arg in Dataloader  <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader</a> , unless you specified to be <code>True</code> which will shuffle raw dataset.</p>

Jona: <p>Yes, I know that. That’s why I posted:</p>
<blockquote>
<p>I’m looking for something like</p>
<p><code>dl = DataLoader(....shuffled=True)</code><br>
<code>dl.data or dl.dataset  # or something like this, to get ordered and original data, not shuffled</code></p>
</blockquote>

ptrblck: <p>Your proposed approach using <code>dl.dataset</code> should work:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(10)
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(dataset, shuffle=True, batch_size=2)

for data in loader:
    print(data)

for data in loader.dataset:
    print(data)
</code></pre>

ethanbaruh: <p>If you want to just get the raw data from the dataset you don’t need to use DataLoader. You can just index the dataset. For example if your dataset is called DS and <strong>getitem</strong>() returns an image and target you could use a for loop to iterate over the dataset in the order that the raw data is in.</p>
<pre><code class="lang-auto">for i in range(DS.__len__()):
    img, target = DS[i]
</code></pre>
Updating specific values in tensor (and ignoring rest in cost function)

batlike: <p>I’m trying to build a Hidden Markov Model in PyTorch where I specify a transition matrix as a set of weights I’d like to update. The tensor looks something like:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9e62e1bf919f2b294af0b5a14b2e92087177033.png" alt="image" data-base62-sha1="xnapn6cm3crEu2HH795HSkj8UEP" width="297" height="111"></p>
<p>I’m trying to solve for the <em>w</em>’s in this tensor while holding the <em>c</em>’s fixed (that is, they cannot be updated during optimization). The constraint I am solving under is that the row sums of the tensor is 1 (<code>X.sum(axis=1) = [1, 1, 1, 1]</code>).</p>
<p>I guess two things that aren’t obvious to me are (1) how to fix specific values of a Tensor (prevent them from updating during optimization or factoring into the error function?) (2) how to symbolically represent certain weights (notice that w_{b} is repeated at multiple entries!). At first I confused myself into thinking this is a system of linear equations that I can solve, but now I am thinking maybe I need to use a deep learning framework to approximate these values. Does PyTorch have this flexibility?</p>

ptrblck: <p>To freeze certain parameters, you could either zero out their gradients (e.g. using hooks) or avoid passing them to the optimizer. Note that parameters could still be updated, if your optimizer uses internal running estimates or a momentum even if the gradients are zero.</p>
<aside class="quote no-group" data-username="batlike" data-post="1" data-topic="95829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/batlike/48/21562_2.png" class="avatar"> batlike:</div>
<blockquote>
<p>Does PyTorch have this flexibility?</p>
</blockquote>
</aside>
<p>It should have the flexibility, as there are some HMM implementations in PyTorch, e.g. <a href="https://colab.research.google.com/drive/1IUe9lfoIiQsL49atSOgxnCmMR_zJazKI">here</a>.</p>
GPU out of memory error

sandeep1: <p>“CUDA out of memory error”<br>
Getting this error here test data is an iterator.<br>
How to fix it?<br>
Here is the code where error is occuring?<br>
Can i take batches and test?</p>
<h1>get predictions for test data</h1>
<p>with torch.no_grad():<br>
preds = model(test_seq.to(device), test_mask.to(device))<br>
preds = preds.detach().cpu().numpy()</p>

ptrblck: <p>Your batch size might be too large, so you could try to lower it during the test run.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
What does xmp.MpSerialExecutor() really do?

random-ness: <p>Good day, I’m kinda new to using pytorch xla. In the colab tutorial provided <code>serial executor</code> was used. I was just wondering in what other scenario can it be used</p>
<pre><code class="lang-auto"> # Using the serial executor avoids multiple processes to
  # download the same data.
  train_dataset, test_dataset = SERIAL_EXEC.run(get_dataset)

  train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset,
    num_replicas=xm.xrt_world_size(),
    rank=xm.get_ordinal(),
    shuffle=True)
  train_loader = torch.utils.data.DataLoader(
      train_dataset,
      batch_size=FLAGS['batch_size'],
      sampler=train_sampler,
      num_workers=FLAGS['num_workers'],
      drop_last=True)
  test_loader = torch.utils.data.DataLoader(
      test_dataset,
      batch_size=FLAGS['batch_size'],
      shuffle=False,
      num_workers=FLAGS['num_workers'],
      drop_last=True)
</code></pre>
How to get/modify gradients of an encoder in a multi-output model with two losses?

nsmeele: <p>Hi!</p>
<p>I am working on a project and decided to use Pytorch for the first time. However, I am facing a challenge in modifying gradients for some parameters in the model. I have read some earlier posts that are somewhat related but I still cannot manage it to work… I hope you are able to help me out.</p>
<p>To give some context, suppose we have an encoder where its hidden representation is fed into two MLP classifiers (i.e. multi-output model). Classifier F addresses a multi-class classification task and is trained to minimize <code>loss_f</code>. Classifier A, on the other hand, uses the same hidden representation from the encoder to tackle a binary classification problem by minimizing <code>loss_a </code>.</p>
<p>In this setting, the parameters in both classifier F and A are trained by following the standard backpropagation with <code>.backward()</code> and <code>optim.step()</code>.</p>
<p>However, the parameters (<code>W_enc</code>) in the encoder must be updated by modifying the gradients as follows:</p>
<pre><code class="lang-auto">d_loss_f/d_W_enc  - d_loss_a/d_W_enc - [some modified gradient]
</code></pre>
<p>Where <code>[some modified gradient]</code> is a modified gradient that must be subtracted from <code>d_loss_f/d_W_enc</code>. So my question is:</p>
<ul>
<li>How can I implement this gradient modification to update the parameters in the encoder?</li>
</ul>
<p>I have read the following posts:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-get-the-gradient-of-parameters-for-different-losses-when-multiple-losses-used/20792">How to get the gradient of parameters for different losses when multiple losses used</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-modify-the-gradient-manually/7483">How to modify the gradient manually?</a></li>
</ul>
<p>But, in these cases, losses were added to each other or all parameters were adjusted the same way which is not my goal.</p>
<p>I hope someone can help me with this or at least point me in the right direction. If more information or any clarification is needed, please let me know! Many thanks in advance.</p>
Faster way to concat tensors with autoregressive model's outputs

seewoo5: <p>Hi, I’m trying to implement an autoregressive model which is a variant of memory-augmented neural network. (More precisely, I’m implementing a model called DKVMN <a href="http://papers.www2017.com.au.s3.amazonaws.com/proceedings/p765.pdf" rel="nofollow noopener">in this paper</a>). With a input of sequence length 100, the model basically do the read and write process 100 times and give 100 outputs in order. However, to compute BCE losses at once, I concat all these 100 outputs as one tensor (so it became <code>(batch_size, 100, 1)</code> tensor from <code>(batch_size, 1)</code> tensors) and this seems to give some computational bottleneck. I tried two approaches: 1) save each 100 tensors in a list and concat them later 2) concat 99 times for each output, and both took a lot of times. Maybe this is just a problem of the model itself (the I may just have to wait) but other RNN or LSTM models doesn’t take this much time for training. (Maybe these are highly optimized?) Is there any tip for boosting up the training time?<br>
Thanks in advance.</p>

googlebot: <p>cat/stack is pretty fast, you’re likely measuring asynchronous cuda calls incorrectly. Try the built-in <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.profiler.profile" rel="nofollow noopener">profiler</a>. RNNs are optimized (C++), yes. JIT may help you somewhat, but in general recurrent stuff is not fast.</p>
Size Mismatch m1: [8 x 512], m2: [1568 x 512]

datascguy: <p>Hello,</p>
<p>I am relatively new to the Field of RL and I’m using pytorch for the first time.</p>
<p>I’m trying to replicate the following code <a href="https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail" rel="nofollow noopener">https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail</a><br>
and make it run ion procgen benchmark (<a href="https://openai.com/blog/procgen-benchmark/" rel="nofollow noopener">https://openai.com/blog/procgen-benchmark/</a>).</p>
<p>I added the ‘appropriate’ pieces of code, but I still get a seed Error when I execute the main.py file with the following parameters:<br>
–env-name “procgen:procgen-coinrun-v0” --algo ppo --use-gae --lr 2.5e-4 --clip-param 0.1 --value-loss-coef 0.5 --num-processes 8 --num-steps 128 --num-mini-batch 4 --log-interval 1 --use-linear-lr-decay --entropy-coef 0.01</p>
<p>When I comment out the line related to the ‘env.seed(seed + rank)’ command, from the envs.py file, I get a size mismatch Error:</p>
<p>RuntimeError: size mismatch, m1: [8 x 512], m2: [1568 x 512] at /Users/soumith/mc3build/conda-bld/pytorch_1549312653646/work/aten/src/TH/generic/THTensorMath.cpp:940</p>
<p>I would appreciate any help to resolve this issue!</p>

ptrblck: <p>It seems to be a bit weird that a different seed creates a shape mismatch, but I’m not an expert in RL.<br>
Could you post the stack trace once you run into this error and explain how the seed might change the data or parameter shapes?</p>

datascguy: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> . Sorry for the late response. I managed to solve this error by changing the sizes of the CNN model.</p>
Is torch.max(x,y) differentiable?

Scott_Hoang: <p>Given x, and y are one-hot vectors matrixes obtaining from Gumbel-softmax, and z = torch.max(x,y), Is z still differentiable?</p>

albanD: <p>Hi,</p>
<p>Assuming that x and y are floating point gradients, then the element-wise max is differentiable yes. The gradient will flow back, for each value, to the element that was used.</p>
Hello comunity i have the problem

Gugga_Nation: <p>i have the follow problem.<br>
help please<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7adebf54aff21f57a175eb74692e32a319bef0e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7adebf54aff21f57a175eb74692e32a319bef0e" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7adebf54aff21f57a175eb74692e32a319bef0e_2_690x388.png" alt="error" data-base62-sha1="qcU0a9nkZYwqqTxrwOM9CWFIKmi" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7adebf54aff21f57a175eb74692e32a319bef0e_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7adebf54aff21f57a175eb74692e32a319bef0e_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7adebf54aff21f57a175eb74692e32a319bef0e_2_1380x776.png 2x" data-dominant-color="FBF7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1920×1080 160 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Pranavan_Theivendira: <p>Hi,</p>
<p>The problem occurs due to the input format that you are using. You have to feed input in the following format - <code>BxCxHxW</code>. You have used the channel as the last dimension(<code>BxHxWxC</code> - this is wrong).</p>
<p>Thanks</p>

Gugga_Nation: <p>I’ll see if it works</p>

Gugga_Nation: <p>Hi,</p>
<p>sorry, I still have the same problem this is the part of the image rescaling:</p>
<p>def process_img(self, image):<br>
i2 = np.array(image.raw_data)<br>
i2 = i2.reshape((self.im_height, self.im_width, 4))<br>
i2 = i2.mean(2)<br>
i2 = i2.astype(np.float32)<br>
i2 *= 1.0/255.0<br>
i2 = cv2.resize(i2, (84,84))<br>
i2 = np.reshape(i2, [1,84,84])<br>
self.front_camera = i2</p>
<p>self.front_camera is the one to introduce to the convolutional neural network please help me</p>
How to crop each element of a tensor with a window size at the same time without using for loop

qiminchen: <p>I have a tensor with a size of <code>3 x 240 x 320</code> and I want to use a window (window size <code>ws=5</code>) to slide over each pixel and crop out the sub-tensor centered at that pixel. The final output dimension should be <code>3 x ws x ws x 240 x 320</code>. So I pad the original tensor with window size and use <code>for loop</code> to do so.</p>
<pre><code class="lang-auto">import torch.nn.functional as F

image = torch.randn(1, 3, 240, 320)
image = F.pad(image, (ws // 2, ws // 2, ws // 2, ws // 2), mode='reflect')
patches = torch.zeros(1, 3, ws, ws, 240, 320)
for i in range(height):
    for j in range(width):
        patches[:, :, :, :, i, j] = image[:, :, i:i+ws, j:j+ws]
</code></pre>
<p>Are there any ways to do the cropping of each pixel at the sample time? Like without using the  <code>for loop</code> over each pixel? I feel like it’s pretty similar to <code>convolution</code> operation but I can’t think of wats to crop efficiently. Thanks in advance!</p>

hhaoao: <p>Take a moment to flip through the documents, there will be unexpected gains.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html#torch.nn.Unfold" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html#torch.nn.Unfold</a></p>

qiminchen: <p>great, thanks! I was able to get what I want using this</p>
<pre><code class="lang-auto">import torch.nn.functional as F

image = torch.randn(1, 3, 240, 320)
image = F.pad(image, (ws // 2, ws // 2, ws // 2, ws // 2), mode='reflect')
patches = F.unfold(a, kernel_size=ws).view(1, 3, ws**2, 240, 320)
</code></pre>
Periodic training loss

Jiaji_Huang: <p>I’m training a simple bidirectional LSTM model, taking sequence of words as input, and predicting a target variable for every word. The target variable is a float number in [0, 1]. The criterion is MSE loss. I used adam optimizer. Key code snippets are as follows (I omit model definition as it is a standard <code>nn.Module</code> and less relevant.)</p>
<pre><code class="lang-auto">optim = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0)
def compute_loss(output, target):
    output = output.flatten()
    target = target.flatten()
    loss = nn.MSELoss()(output, target)
    return loss

for epoch in range(10):
    for (input, target, n_samples) in data_loader.yield_batch(batch_size=128):
        model.zero_grad()
        output, hidden = model(input, hidden=None). # use zero initial hidden states in model forward pass
        loss = compute_loss(output, target)
        loss.backward()
        optimizer.step()
        print("training loss: {}".format(loss))
</code></pre>
<p>The training loss is visualized as following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6f471b7a07624d08d6e5de96d4c0c6659a48ac7.png" alt="adam" data-base62-sha1="so2g1tLsDoR4iHOQxsS1AYcRslF" width="320" height="240"><br>
There is a clear periodic behavior. I checked and found the huge oscillation appears at end of each epoch. Am I calling the adam optimizer in a wrong way? Or does pytorch implementation of Adam has some automatic tunning of learning rates, etc?</p>
Gradients do not update when I replace the loss value in this code

someAdjectiveNoun: <p>I would like to use weighted multi-task loss in my experiment. For the same, I found a script (<a href="https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb" rel="nofollow noopener">here</a>).</p>
<p>The only change I made in this script was to replace the squared loss with another loss value. However, this does not seem to be causing the problem where the <code>self.log_vars.grad</code> results in <code>None</code>. The standalone script works fine, but when I use it with my setup, it does not update grads. Yes, <code>list(model.parameters())</code> lists: <code>tensor([0., 0.], device='cuda:0', requires_grad=True)</code></p>
<ul>
<li>another change from the script is that my model setup is on GPU where <code>self.log_vars.grad = None</code>, but in the orignal script, it runs well on the CPU.</li>
</ul>

someAdjectiveNoun: <p>The problem is solved. Thanks to this <a href="https://discuss.pytorch.org/t/model-param-grad-is-none-how-to-debug/52634">post</a> and discussion between <a class="mention" href="/u/ljj7975">@ljj7975</a> &amp; <a class="mention" href="/u/ptrblck">@ptrblck</a> for hinting me towards checking if the computation graph was broken somewhere. It was exactly that. The computation graph was broken while calling loss.backward().</p>

Sharifi: <p>I had similar problem, but it was solved as I mentioned <a href="https://discuss.pytorch.org/t/grad-is-none-even-when-requires-grad-true/29826/8">here</a>:</p>
F.conv2d does not autograd

bruce: <p>dtype = torch.float<br>
device = torch.device(“cpu”)</p>
<p>corrupt_img = torch.tensor(corrupt_img,dtype=dtype,device=device)<br>
corrected_im = torch.randn(1,1,corrupt_img.shape[0],corrupt_img.shape[1],device=device,dtype=dtype,requires_grad=True)<br>
gt = torch.tensor(img)</p>
<p><span class="hashtag">#corrected_im</span> = corrected_im[None,None,:,:]<br>
corrupt_img = corrupt_img[None,None,:,:]</p>
<p>kernel = torch.tensor(kernel,dtype=dtype,device=device)<br>
kernel = kernel[None,None,:,:]</p>
<p>lr = 1e-5<br>
epoch = 1000</p>
<p>for i in range(epoch):<br>
conv_result = F.conv2d(corrected_im,kernel,padding=10)<br>
loss = (conv_result - corrupt_img).pow(2).sum() + torch.abs(corrected_im).sum()</p>
<pre><code>if i%10==0:
    print(i,loss.item())
    
loss.backward()
with torch.no_grad():
    corrected_im = corrected_im - lr*corrected_im.grad
    corrected_im.grad.zeros_()
</code></pre>
<p>I am finding a solution of compressed sensing problem using pytorch.to solve a optimization<br>
argmin_u  || kernel*u -b||^2 + ||u||_1<br>
I expressed the formula directly using F.conv2d ,and torch.tensor,but the backward doesn’t work for the u. the corrected_im.grad is a none type obeject is this code. However ,theoretically the autograd should work on this or is there any mistake in my convolution expression?</p>

albanD: <aside class="quote no-group" data-username="bruce" data-post="1" data-topic="93566">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ccd318/48.png" class="avatar"> bruce:</div>
<blockquote>
<p><code>corrected_im = corrected_im - lr*corrected_im.grad</code></p>
</blockquote>
</aside>
<p>I think the problem is this line: It changes the variable <code>corrected_im</code> to the result of the substraction. But since this substraction is done in a no_grad block, the result does not require gradient.<br>
I think you want to change it inplace no?<br>
<code>corrected_im -= lr * corrected_im.grad</code></p>
<p>Also I don’t think <code>.zeros_()</code> is a thing, you want <code>.zero_()</code> no?</p>

bruce: <p>Thx a lot! It really works.<br>
I still have one question: how could I use the inserted Adam optimizer in pytorch to solve my problem?<br>
looking forward to your answer!<br>
Thanks again</p>

albanD: <p>Hi,</p>
<p>You can use the provided optimizer by giving it your Tensors to optimize <code>(corrected_im,)</code> here if there is only one.<br>
Then you can just call <code>opt.step()</code> and it will use the content of <code>corrected_im.grad</code> to update <code>corrected_im</code>.</p>

bruce: <p>sorry for reply so late. Thank you for responding so patiently!</p>
Use Torchtext model in Flaskapp

EmreTokyuez: <p>Hello Pytorch Community</p>
<p>I am new to this whole framework and decided to do research on different models for Sentiment Analysis and their accuracy. For my pytorch models, I used this github repo: <a href="https://github.com/bentrevett/pytorch-sentiment-analysis" rel="nofollow noopener">https://github.com/bentrevett/pytorch-sentiment-analysis</a>.</p>
<p>I followed the <a href="https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/2%20-%20Upgraded%20Sentiment%20Analysis.ipynb" rel="nofollow noopener">LSTM</a>, <a href="https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/4%20-%20Convolutional%20Sentiment%20Analysis.ipynb" rel="nofollow noopener">CNN</a> and <a href="https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/6%20-%20Transformers%20for%20Sentiment%20Analysis.ipynb" rel="nofollow noopener">Transformer</a> tutorial.</p>
<p>I managed to benchmark the models and got my research done but I also want to build a flask app to allow people to test them out and compare them one on one. But I could not find good resources for deployment of torchtext models that utilize embeddings, vocab and so on.  As I am not very experienced with Pytorch, it is also hard for me to transfer tutorials that deploy normal pytorch models to my torchtext model.</p>
<p>Can anybody help me out with this?</p>
Evaluation in batches

sandeep1: <p>How can i evaluate test data in batches?</p>
<p>This is the evaluation code, i need to predict in batches. Please help</p>
<h1>get predictions for test data</h1>
<p>with torch.no_grad():<br>
preds = model(test_seq.to(device), test_mask.to(device))<br>
preds = preds.detach().cpu().numpy()</p>
<h1>model’s performance</h1>
<p>preds = np.argmax(preds, axis = 1)<br>
print(classification_report(test_y, preds))</p>
Libtorch, ParamState

freezek: <p>Hi, I want to realize optimizer transform(Adam --&gt; SGD) method during training, so I need some state parameters such as exp_avg…   From the document <a href="https://pytorch.org/cppdocs/api/classtorch_1_1optim_1_1_optimizer.html#_CPPv4N5torch5optim9OptimizerE" rel="nofollow noopener">https://pytorch.org/cppdocs/api/classtorch_1_1optim_1_1_optimizer.html#_CPPv4N5torch5optim9OptimizerE</a>  ,  I found state parameter from <code>ska::flat_hash_map&lt;std::string, std::unique_ptr&lt;OptimizerParamState&gt;&gt; &amp;state()</code>, but how can I access it ?</p>
<p>Please help me, thanks in advance!</p>