akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[c++] convertTo error with kCUDA device

username: 111547
cooked: <p>I am making segmentation with libtorch.</p>
<p>Input is 224<em>224</em>3 and output will 224<em>224</em>1 mask image.<br>
This is working well with CPU device, but when I try GPU it crashed at cv::convertTo  function.<br>
My code is like this.</p>
<pre><code class="lang-auto">torch::NoGradGuard no_grad;
segmentation-&gt;eval();
torch::Tensor val_loss;
for (torch::data::Example&lt;torch::Tensor, torch::Tensor&gt;&amp; batch : *data_loader_val) {
auto data = batch.data.to(device);
auto targets = batch.target.to(device);
				
try {
			torch::Tensor outputs = segmentation-&gt;forward(data);

			float* gt = targets.data_ptr&lt;float&gt;();
			float* pr = outputs.data_ptr&lt;float&gt;();
			char name[80];

			for (auto i = 0; i &lt; outputs.sizes()[0]; i++)
			{
						auto gttmp = gt + i * 224 * 224;
						cv::Mat gtmat(224, 224, CV_32FC1, gttmp);
						gtmat.convertTo(gtmat, CV_8UC1, 255);

						sprintf_s(name, "gt%d", i);
						cv::imshow(name, gtmat);

						auto prtmp = pr + i * 224 * 224;
						cv::Mat ptmat(224, 224, CV_32FC1, prtmp);
						ptmat.convertTo(ptmat, CV_8UC1, 255);

						sprintf_s(name, "pr%d", i);
						cv::imshow(name, ptmat);
			}
}
</code></pre>
<p>Could I have a clue for this problem?</p>

username: Matej_Kompanek
cooked: <p>I would guess that cv::Mat is trying to access memory on GPU (gt and pr are both referring to CUDA memory). Before calling ‘data_ptr()’, copy outputs and tensors back to CPU, e.g.:</p>
<pre><code class="lang-auto">		targets = targets.cpu().contiguous();
		outputs = outputs.cpu().contiguous();
</code></pre>

username: 111547
cooked: <p>Thank!!! I solved this problem!<br>
Does contiguous function help copy cuda memory to cpu memory?<br>
I tried targets.cpu(), outputs.cpu() but it doesn’t work.</p>

username: Matej_Kompanek
cooked: <p>Since you’re reading sequentially through the tensor’s memory, you need contigous tensor. There is a pretty good explanation what contigous does over here <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107">https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107</a></p>
Partial weight sharing

username: vltanh
cooked: <p>Hi, I want to write a weight share between an nn.Embedding(vocab, d) module and an nn.Linear(2d, vocab) with the two halves of the weight matrix of nn.Linear the same and equal to the weight matrix of nn.Embedding.</p>
<p>Is it possible to do so? By weight share I mean not just at initialization but for the whole training.</p>
<p>Many thanks.</p>
<p>P/s: I would like to also ask (to confirm) if doing</p>
<pre><code class="lang-auto">self.linear.weight = self.embedding.weight
</code></pre>
<p>in the model initialization is enough for weight sharing for the whole training or is it just at initialization?</p>
Conv3d, gpu,cuda

username: saran
cooked: <p>i am facing this error when i am runing my infer.py</p>
<p>RuntimeError: Could not run ‘aten::slow_conv3d_forward’ with arguments from the ‘CUDA’ backend. ‘aten::slow_conv3d_forward’ is only available for these backends: [CPU, Autograd, Profiler, Tracer].</p>

username: ptrblck
cooked: <p>I guess this error is raised, if you’ve installed a CPU-only libtorch library, so you would need to use the one with the CUDA bindings.</p>
Loss function ("reduction=sum") get better results than (reduction="mean")

username: PytorchBeginner
cooked: <p>Hi, I found a strange phenomenon and did not know the reason.</p>
<p>I am training a deep denoising autoencoder, which tries to recover the original image using a noisy version as the input. It should not be a difficult task. I found when setting “reduction=sum”, then I can get the expected result, though the final loss is not small enough. But if I use the default setting “reduction=mean”, then I fail to get a satisfactory result. In this situation, the autoencoder seems to have learned nothing, i.e., it just outputs the noisy version of the input (like copy operation) instead of recovering an intact one.</p>
<p>Any explanation and suggestion would be appreciated.</p>

username: ptrblck
cooked: <p>The difference in the loss reduction would create different loss ranges and thus also impact the gradient magnitudes, so you could also play around with the learning rate and try to find a sweet spot.</p>
How do I pass a list of tensors to my network?

username: Andrew_Legend
cooked: <p>I am training a Variational Autoencoder. I have a dataset of RGB images with resolution 256*256</p>
<pre><code class="lang-auto">images = torchvision.datasets.ImageFolder(root = 'data/', transform=transforms.Compose([transforms.ToTensor()]))
data_loader = torch.utils.data.DataLoader(images, batch_size=32, shuffle=True)
</code></pre>
<p>However, my batches are turned into a list of tensors. As a result I cannot pass it to my model for training.</p>
<p>What kind of adjustment should I do in order for my model to be able to train? Thanks in advance</p>

username: ptrblck
cooked: <p>The default collate function used in the <code>DataLoader</code> should create tensors from all passed samples, so I’m unsure why a list is returned unless you are using a custom <code>collate_fn</code>.<br>
Which PyTorch version are you using and could you post the folder structure, please?</p>
Normalizing before PCA Lowrank?

username: GabbyChan
cooked: <p>Hi! I would like to ask for advice on the PCA Lowrank (<a href="https://pytorch.org/docs/stable/generated/torch.pca_lowrank.html" class="inline-onebox" rel="noopener nofollow ugc">torch.pca_lowrank — PyTorch 1.9.0 documentation</a>)…</p>
<ul>
<li>
<p>Do I have to normalize my features along the feature dimension before?</p>
</li>
<li>
<p>I am a little confused because there is an input option where it said <em>center=True</em> … Does this option include whitening the input features?</p>
</li>
</ul>
<p>It would be nice if someone can point out how to normalize <code>A</code>… A is [*,m,n] .</p>
<p><code>m</code> = samples and <code>n</code> = features</p>
<pre><code class="lang-auto">U, S, V = torch.pca_lowrank(A)
k   = 3
A_k = torch.matmul(A, V[:, :k])
</code></pre>
ImportError libtorch_python.so: undefined symbol

username: sinuku
cooked: <p>Hi!<br>
I’m trying to use the Pytorch model in C++.<br>
I downloaded LibTorch(cxx11 ABI) additionally, and used it by adding its path.</p>
<p>If I use PyImport_Import function, The error message is as follows</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/sensor/Documents/test/build/segmentation.py", line 1, in &lt;module&gt;
    import torch, torchvision
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 190, in &lt;module&gt;
    from torch._C import *
ImportError: /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so: undefined symbol: _ZN5torch11distributed3rpc8RpcAgent12getDebugInfoEv
</code></pre>
<p>How to solve this error??</p>
<p>*my environment<br>
ubuntu : 18.04<br>
CUDA : 10.2<br>
python : 3.6.9<br>
torch : 1.7.0<br>
torchvision : 0.8.0</p>

username: Alex_Bui
cooked: <p>Have you managed to fix this bug?</p>
<p>I encounter the same one.</p>
Convert onehot encoded output to target

username: arya47
cooked: <p>Currently i have my targets one hot encoded st their values are:</p>
<pre><code class="lang-auto">tensor([[1, 0],
        [0, 1],
        [0, 1],
        [0, 1],
        [0, 1],
        [0, 1],
        [1, 0],
        [0, 1],
        [1, 0],
        [1, 0]])

</code></pre>
<p>for a batch size 10,<br>
how do i convert these to required format for CrossEntropy before training?</p>
<p>Thanks</p>

username: arya47
cooked: <p>solved</p><aside class="quote" data-post="2" data-topic="68321">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytocrh-way-for-one-hot-encoding-multiclass-target-variable/68321/2">PyTocrh way for one-hot-encoding multiclass target variable</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    You could use this code snippet to transform your class indices into a one-hot encoded target: 
target = torch.randint(0, 10, (10,))
one_hot = torch.nn.functional.one_hot(target)
  </blockquote>
</aside>
<JIT Trace> Redundant prim::ListUnpack and prim::ListConstruct

username: leslie-fang-intel
cooked: <p>I have a customized C++ operation <code>batch_score_nms</code> which returns a <code>vector&lt;at::Tensor&gt;</code> with size (Batchsize * 3). When I trace this graph, I get below output:</p>
<pre><code class="lang-auto">  %470 : Tensor[] = torch_ipex::batch_score_nms(%bboxes, %probs, %3, %2) # /home/lesliefang/pytorch_1_7_1/ssd-rn34/frameworks.ai.pytorch.cpu-models/inference/others/cloud/single_stage_detector/pytorch/utils.py:169:0
  %471 : Tensor, %472 : Tensor, %473 : Tensor = prim::ListUnpack(%470)
  %477 : Tensor[] = prim::ListConstruct(%471, %472, %473)
  return (%477)
</code></pre>
<p>I am a little confused when we need to ListUnpack and tensor list and pack it again for output? It seems  Redundant .</p>
NumPy and PyTorch give different results

username: Shivam_Sharma
cooked: <p>I am learning PyTorch from scratch and was following their official tutorials. I tried out the NumPy and Tensor only codes on <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="noopener nofollow ugc">this</a> article and thought it would be a fascinating experiment if I could use the same inputs in both of these codes. I expected the results to be the same since I removed the randomness from the codes but still both of them are giving me different values for weights.</p>
<p>Can someone please explain to me why this happens or where I am going wrong?</p>
How to use one-hot vectors with PyTorch's loss functionss

username: kodachrome
cooked: <p>I am trying to encode the alphabet into a numerical format and it appears there are two ways to go about this. Using class labels (0 1 2 … 25) or using a one-hot vector with each colum being the 26 letters. From the what I’ve read online, it would make sense to use a one-hot vector as there is no ordinality in letters e.g. A is not ‘close’ to B. However I am struggling to use one-hot vectors with PyTorch because all of the loss functions only take class labels. It is easy to convert a one-hot vector to class labels to be able to pass it to the loss functions, however would this hurt performance of the network? This is the architecture problem I am facing.</p>
<p>It may be worth metioning my situation. This is a many to one seq network using RNN layers and eventually the network goes through a linear layer with an output shape of 26.</p>

username: eqy
cooked: <p>This shouldn’t hurt the performance of the network, and in reality a one-hot formatting vs. a label wouldn’t be related to things like ordinality unless the loss function itself explicitly included ordinality in the loss computation (the typical cross-entropy loss doesn’t do this).</p>
<p>If you want to convert one-hot to class labels, a simple way to do this is using the max function.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; onehot_labels = torch.zeros(64, 26, dtype=torch.long)
&gt;&gt;&gt; for i in range(64):
...     onehot_labels[i,i%26] = 1
...
&gt;&gt;&gt; _, class_labels = torch.max(onehot_labels, axis=1)
&gt;&gt;&gt; print(class_labels)
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9,
       10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,  0,  1,
        2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
</code></pre>
<p>Note that having the architecture of the network producing a one-hot output while the loss function itself is taking a class label is not a problem, but rather an implementation detail of how the labels are being stored and passed to the loss function.</p>

username: kodachrome
cooked: <p>Thank you for the incredibly helpful answer. It makes me wonder, why you would use one-hot encoding in the first place? Storing class labels is much more space efficient.</p>

username: eqy
cooked: <p>My guess is for simplicity; as you noticed, the output of the network itself is one-hot encoded, so it can simplify things if this is the format that is used for the labels as well. In practice, storing labels this way on disk also wouldn’t really matter as compression will handle the long strings of zeros in one-hot encoded labels efficiently.</p>
will the outcome of autograd.Variable  converge？

username: rong_the
cooked: <p>I use the function autograd.Variable to process data， but the outcome of differentiation always change each time. I wonder if the result is ordinary or not.Does anyone know the problem?Any help you provide is appreciated.</p>

username: ptrblck
cooked: <p><code>Variable</code>s are deprecated since PyTorch <code>0.4</code>, so you should use tensors directly now.<br>
However, using it might not explain your issue, so could you add more information about your use case and, if possible, a code snippet showing this issue?</p>

username: rong_the
cooked: <p>I’m studying action recognition and now learn a github program pytorch-coviar,but I find that the result always change even for a same train model.After check the code ,I think the code line input_var = torch.autograd.Variable(data, volatile=True) (<a href="https://github.com/chaoyuaw/pytorch-coviar/blob/master/test.py" rel="noopener nofollow ugc">pytorch-coviar/test.py at master · chaoyuaw/pytorch-coviar · GitHub</a> line94) is the reason,because it will produce different number each time and finally make the result different.However,I don’t know if it is because function autograd.Variable can‘t converge or other unkonwn reasons.Could you please tell me something about it？</p>

username: rong_the
cooked: <p>don’t know whether you have received my messege or not.maybe the information that I present is not enough？if there are something that I should provide,please tell me,thank you</p>

username: ptrblck
cooked: <p>If you are concerned about the behavior of <code>Variable</code> (which is reasonable, as it’s deprecated), remove it and rerun the code.<br>
I couldn’t follow the entire explanation, but would recommend to also check, if <code>model.eval()</code> would solve the issue.<br>
If not, feel free to post an executable, minimal code snippet to reproduce the issue.</p>

username: rong_the
cooked: <p>Thank you for your help.After thinking about it, I found that the previous question is not accurate.What I want to find out is whether the pytorch function “autograd” can converge,When it try to differentiates a data（such as picture).In other words, when function autograd calculates the differentiation of the same data(matrix of an image or other), will it get the same result if it runs many times? Or it have some differences （errors） every time</p>

username: ptrblck
cooked: <p>Assuming you are using the same input as well as the same parameters in the model and also disable any randomness, it would come down to the determinism of the used operations.<br>
Take a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> to see how deterministic ops can be selected (if available).</p>
Attach 2 LSTM layers to ConvNet

username: bibekx
cooked: <p>I am trying to implement the model given by NVIDIA <a href="https://developer.nvidia.com/blog/deep-learning-self-driving-cars/" rel="noopener nofollow ugc">https://developer.nvidia.com/blog/deep-learning-self-driving-cars/</a></p>
<p>The current version of the model is given below, however, I want the model to look at 4 consecutive images to make a decision. I am thinking of attaching 2 LSTM layers but do not know how to…</p>
<pre><code class="lang-auto"># Regressor
class Oracle(nn.Module):
    def __init__(self):
        super().__init__()
        self.elu = nn.ELU(inplace=True)
        self.dropout1 = nn.Dropout(0.5) # Not given by NVIDIA, arbitrary choice
        #in, out, kernel_size, stride, padding
        self.conv1 = nn.Conv2d(3, 24, (5,5), 2) 
        self.conv2 = nn.Conv2d(24, 36, (5,5), 2) 
        self.conv3 = nn.Conv2d(36, 48, (5,5), 2) 
        self.conv4 = nn.Conv2d(48, 64, (3,3))
        self.conv5 = nn.Conv2d(64, 64, (3,3))
        
        self.fc1 = nn.Linear(64*18, 100) #NVIDIA says this must be 1164 instead of 1152 (64*18)
        self.fc2 = nn.Linear(100, 50)
        self.fc3 = nn.Linear(50, 10)
        self.fc4 = nn.Linear(10, 1)

    def forward(self, x):
        x = self.elu(self.conv1(x))
        x = self.elu(self.conv2(x))
        x = self.elu(self.conv3(x))
        x = self.elu(self.conv4(x))
        x = self.elu(self.conv5(x))
        #flatten the dims 
        x = x.view(x.size(0), -1)
        x = self.dropout1(x) ##
        x = self.elu(self.fc1(x))
        x = self.elu(self.fc2(x))
        x = self.elu(self.fc3(x))
        x = self.elu(self.fc4(x))
        return x
</code></pre>
<p>Your help is appreciated. Thanks!!</p>
Extract patches from image

username: InfT
cooked: <p>Good morning. I’ve a dataset with monochromatic images with 4096x256 pixel and I would like to split them in many patches of size 16x16. Every image is a tensor:<br>
torch.Size([1, 256, 4096])<br>
How can I do?</p>

username: eqy
cooked: <p>I believe this is a classic use case where Tensor.unfold can be used:</p><aside class="quote quote-modified" data-post="4" data-topic="16837">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837/4">How to extract smaller image patches (3D)?</a> 
  </div>
  <blockquote>
    For the second use case you could use Tensor.unfold: 
S = 128 # channel dim
W = 256 # width
H = 256 # height
batch_size = 10

x = torch.randn(batch_size, S, W, H)

size = 64 # patch size
stride = 64 # patch stride
patches = x.unfold(1, size, stride).unfold(2, size, stride).unfold(3, size, stride)
print(patches.shape)
&gt; torch.Size([10, 2, 4, 4, 64, 64, 64])

patches now containes [2, 4, 4] patches of size [64, 64, 64]. 
For the first use case: 
You could use pooling operators like nn.MaxPool3d an…
  </blockquote>
</aside>
Pytorch Tensor from Bytes

username: ed-muthiah
cooked: <p>Hello,</p>
<p>I’m  wondering what the fast way to convert from bytes to a pytorch tensor is.</p>
<p>I’ve found the reverse here:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.byte.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.byte.html</a></p>
<p>Thanks</p>

username: HDCharles
cooked: <p>You can use .to(torch.float) or .to(torch.uint8) to go in either direction.</p>
Missing documetnation

username: pinchazer
cooked: <p>I’m studying official <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial</a>. I was trying to search for AnchorGenerator in docs but, it’s absent. Why? And how can I find it?</p>
From torch.autograd.gradcheck import zero_gradients

username: jS5t3r
cooked: <p>0</p>
<p>This error comes after the upgrade of my PyTorch version from 1.8 to 1.9.0.</p>
<p>When using this line:</p>
<p><code>from torch.autograd.gradcheck import zero_gradients</code>, I get this error message: <code>ImportError: cannot import name 'zero_gradients' from 'torch.autograd.gradcheck'</code></p>
<p>The command:</p>
<p><code>zero_gradients(im)</code></p>
<p>is used.</p>
<p>What is the new command equivalent in PyTorch 1.9.0?</p>

username: soulitzer
cooked: <p>Unfortunately it was removed as part of a refactor to the code. I’m not sure there’s an equivalent you can just import, but since its relatively simple, you might want to just replicate it in your own code.<br>
This was the original function:</p>
<pre><code class="lang-auto">def zero_gradients(x):
    if isinstance(x, torch.Tensor):
        if x.grad is not None:
            x.grad.detach_()
            x.grad.zero_()
    elif isinstance(x, collections.abc.Iterable):
        for elem in x:
            zero_gradients(elem)
</code></pre>
Install torch Cuda 11.2 Ubuntu 20.04 GeForce RTX 3090

username: ilja_stas
cooked: <p>Hi everybody,</p>
<p>I have problems to use my GPUs with torch in Ubuntu 20.04 and my GeForce RTX 3090 and CUDA 11.2.</p>
<ul>
<li>
<blockquote>
<p>I installed according to <a href="https://pytorch.org/get-started/locally" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a> and adapt the cudatoolkit from 11.1 to 11.2: conda install pytorch torchvision torchaudio cudatoolkit=11.2 -c pytorch -c conda-forge</p>
</blockquote>
</li>
<li>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
1.8.0</p>
</blockquote>
</li>
<li>
<blockquote>
<p>I installed cuda by: <a href="https://developer.nvidia.com/cuda-downloads?target_os=Linux&amp;target_arch=x86_64&amp;target_distro=Ubuntu&amp;target_version=2004&amp;target_type=deblocal" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-downloads?target_os=Linux&amp;target_arch=x86_64&amp;target_distro=Ubuntu&amp;target_version=2004&amp;target_type=deblocal</a></p>
</blockquote>
</li>
<li>
<blockquote>
<p>nvidia-smi says:  | NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2</p>
</blockquote>
</li>
<li>
<blockquote>
<p>nvcc --version says: nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Sun_Feb_14_21:12:58_PST_2021<br>
Cuda compilation tools, release 11.2, V11.2.152<br>
Build cuda_11.2.r11.2/compiler.29618528_0</p>
</blockquote>
</li>
<li>
<blockquote>
<p>torch.cuda.is_available()<br>
False</p>
</blockquote>
</li>
</ul>
<p>Does anybody have a tip or an idea how I can use my GPU?</p>
<p>Thanks and best regards,<br>
Ilja</p>

username: ilja_stas
cooked: <p>That helped me a lot: <a href="https://medium.com/analytics-vidhya/install-cuda-11-2-cudnn-8-1-0-and-python-3-9-on-rtx3090-for-deep-learning-fcf96c95f7a1" class="inline-onebox" rel="noopener nofollow ugc">Install CUDA 11.2, cuDNN 8.1.0, PyTorch v1.8.0 (or v1.9.0), and python 3.9 on RTX3090 for deep learning | by Yifan Guo | Analytics Vidhya | Mar, 2021 | Medium</a></p>

username: dip4fish
cooked: <p>Unfortunately, the article is behind a paywall…</p>
Image Regression Technique

username: Henseljahja
cooked: <p>Is theres any documentation on how to load Image, with regression label in pytorch?</p>

username: Sterling_Ramroach
cooked: <p>Hello,</p>
<p>What do you mean by this?</p>
<p>Are you trying to do some image classification by treating it as a regression problem?</p>

username: Henseljahja
cooked: <p>basically, i just want to predict regression value, from an input image.</p>
Extract frame number and patch number for custom dataset

username: ZeroMax
cooked: <p>I have a custom dataset of 256x256px video frames. To limit my memory usage, I am using the filepath as a pointer. However, I would like to split these frames into 16x16 non-overlapping patches to feed into a Transformer, and as such am trying to compute the frame number as well as the patch number within the frame beforehand from the full frame * patch number.</p>
<p>Code:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    def __init__(self, directory='../data/*', get_subdirs=True, size=(16,16), max_ctx_length=4096):
        self.data = glob.glob(directory)
        if get_subdirs:
            data_temp = []
            for i in self.data:
                fata = glob.glob(i+"/*")
                fata.sort(key=lambda r: int(''.join(x for x in r if (x.isdigit()))))
                data_temp.extend(fata)

        self.data = data_temp
        self.max_ctx_length = max_ctx_length
        self.size = size
    def __len__(self):
        return len(self.data)*self.size[0]-self.max_ctx_length-1
    def __getitem__(self, key):
        frame_start = int(np.ceil(key / self.size[0]))
        frame_end = int(np.ceil((key+self.max_ctx_length+1) / self.size[0]))

        frames = self.data[frame_start:frame_end]

        patch_start = int(np.mod(key, self.size[0]))
        patch_end = int(np.mod(key+self.max_ctx_length+1, self.size[0]))
        print(patch_start, patch_end)
        patches = []

        for im in range(len(frames)):
            image = (Tvio.read_image(frames[im], mode=Tvio.ImageReadMode.RGB).float() / 255).unsqueeze(0)
            ppf = F.unfold(image, self.size, stride=self.size).transpose(1,2).split(1, 1)
            if im == 0:
                if not(im==len(frames)-1):
                    ppf = ppf[patch_start:]
                else:
                    ppf = ppf[patch_start:patch_end]
            elif im == len(frames)-1:
                ppf = ppf[:patch_end]
            patches.extend(ppf)
            print(len(patches))
        data_x = patches[:-2]
        print(len(data_x))
        data_y = patches[1:-1]
        return torch.cat(data_x, dim=1).squeeze(0), torch.cat(data_y, dim=1).squeeze(0)           

if __name__ == "__main__":
    dataset = Dataset()
    print(dataset.__getitem__(4)[0].shape)
</code></pre>
<p>Expected Output:<br>
A tensor of shape [4096, 768].</p>
<p>Actual Output:<br>
torch.Size([65279, 768])</p>
NVCC crash (core dump) with c++17

username: mirzadeh
cooked: <p>I have a wired issue in that if include &lt;torch/torch.h&gt; in a <code>.cu</code> file and compile it with NVCC and --std=C++17, NVCC will crash. No problem with C++14. This is using nvcc 11.2 and libtorch 1.9.</p>
<p>Is this a known issue? If so, what is the solution beyond dropping to C++14?</p>
Batch size is int but Sampler throws error ValueError: batch_size should be a positive integer value, but got batch_size=4

username: sajangohil
cooked: <p>I am using Sequential sampler and then batch sampler with batch size of 4 but get error:<br>
<code>ValueError: batch_size should be a positive integer value, but got batch_size=4</code></p>
<p>stack trace:</p>
<pre><code class="lang-auto">    self.batch_sampler = torch.utils.data.BatchSampler(self.sampler, self.batch_size, drop_last=False)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 217, in __init__
    "but got batch_size={}".format(batch_size))
ValueError: batch_size should be a positive integer value, but got batch_size=4
</code></pre>
<p>My code snippet:</p>
<pre><code class="lang-auto">self.sampler = torch.utils.data.SequentialSampler(self.dataset)
self.batch_sampler = torch.utils.data.BatchSampler(self.sampler, self.batch_size, drop_last=False)
</code></pre>
<p>self.dataset is a custom class which inherits torch.utils.data.Dataset</p>
<p>Am i doing something wrong?</p>

username: ptrblck
cooked: <p>The error message seems to be quite misleading.<br>
Which PyTorch version are you using and could you post an executable code snippet to reproduce this issue, so that we could debug it?</p>

username: sajangohil
cooked: <p>Hi, sorry for the inconvenience. I was creating an installable module and testing it in an environment. Initial version was not converting string to int, and after explicitly doing that, it works. (the installed version was not being updated as i forgot to use --force-reinstall and that’s why i thought it was not working). The error is still a bit misleading as it doesn’t specify the type type of passed value, but it works now.</p>

username: ptrblck
cooked: <p>Good to hear you’ve isolated the issue.<br>
Yeah, I believe that’s the “curse” of the flexibility using the <code>format</code> specifier in Python:</p>
<pre><code class="lang-python">print('{}'.format(4))
&gt; 4
print('{}'.format("4"))
&gt; 4
</code></pre>
Disabling all reduce in Distributed Data Parallel

username: DT6A
cooked: <p>Hello, I’m trying to setup distributed model training. <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">Distributed Data Parallel documentation</a> says that <code>torch.nn.parallel.DistributedDataParallel</code> performs all reduce operation by itself if I got it right. Is it possible to disable this functionality so I can call all reduce manually? Or in this case I must use something instead <code>DistributedDataParallel</code>?</p>

username: wayi
cooked: <blockquote>
<p>Is it possible to disable this functionality so I can call all reduce manually?</p>
</blockquote>
<p>Do you need to implement any customized logic in allreduce? If so, I will recommend <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" rel="noopener nofollow ugc">DDP comm hooks</a>, which provides an interface to implement customized allreduce.</p>
<p>Another option is <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync" rel="noopener nofollow ugc">no_sync context manager</a>, which will disable allreduce, then it’s your responsibility to run allreduce.</p>

username: DT6A
cooked: <p>Seems like no_sync is what I need. Thank you</p>
How to understand `groups` parameter of `Conv2d`

username: Delpha.T.Edwards
cooked: <p>Hello, I am trying to understand  parameter <code>groups</code> in <code>Conv2d</code>. I run the following code,</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

m = nn.Conv2d(3,12,3,1,1,groups=3)
fake_in = torch.randn(1,3,244,244)
print(f"input: {fake_in.size()}")
out = m(fake_in)
print(m.weight.size())
# torch.Size([12, 1, 3, 3])
print(out.shape)
# torch.Size([1, 12, 244, 244])
</code></pre>
<p>The weight shape is <code>torch.Size([12, 1, 3, 3])</code>, does this weight includes 3 groups?  and each group will generate <code>([1,4,244,244])</code> in this example?</p>

username: KFrank
cooked: <p>Hi Delpha!</p>
<aside class="quote no-group" data-username="Delpha.T.Edwards" data-post="1" data-topic="127331" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b3f665/48.png" class="avatar"> Delpha.T.Edwards:</div>
<blockquote>
<p>The weight shape is <code>torch.Size([12, 1, 3, 3])</code>, does this weight includes 3 groups?  and each group will generate <code>([1,4,244,244])</code> in this example?</p>
</blockquote>
</aside>
<p>Yes, this is correct.</p>
<p>In effect, you have three groups with <code>in_channels = 1</code> and<br>
<code>out_channels = 4</code>.  The <code>weight</code> tensor for the first group is<br>
<code>m.weight[0:4]</code> of shape <code>[4, 1, 3, 3]</code>, the weight tensors for the<br>
second and third groups are <code>m.weight[4:8]</code> and <code>m.weight[8:12]</code>,<br>
respectively, also of shape <code>[4, 1, 3, 3]</code>.  (Pytorch chooses to store<br>
the three per-group weight tensors as a single tensor of shape<br>
<code>[12, 1, 3, 3]</code> rather than as three separate weight tensors of<br>
shape <code>[4, 1, 3, 3]</code>.)</p>
<p>The three per-group convolutions produce three results, each of<br>
shape <code>[1, 4, 244, 244]</code>, that are then concatenated together along<br>
<code>dim = 1</code> to produce the actual result of shape <code>[1, 12, 244, 244]</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Delpha.T.Edwards
cooked: <p>Thanks, now I understand it.</p>
<p>For this example <code>m = nn.Conv2d(3,12,3,1,1,groups=3)</code>, is this called depthwise convolution? or the <code>channel_out</code> must be same with the <code>channel_input</code>?</p>
Manual seed DDP

username: char-t
cooked: <p>Quick one - is it necessary to use a manual seed when using DDP to ensure all local models initialise with the same parameters? I have been looking and seen examples that do and examples that don’t but nothing definitive.</p>
<p>Many thanks!</p>

username: ptrblck
cooked: <p>No, DDP shares the <code>state_dict</code> in the first iteration, so you won’t need to seed the code for the parameter initialization, but might need it for other use cases.<br>
From the <a href="https://pytorch.org/docs/stable/notes/ddp.html">DDP Internal Design</a>:</p>
<blockquote>
<p>The DDP constructor takes a reference to the local module, and broadcasts <code>state_dict()</code> from the process with rank 0 to all other processes in the group to make sure that all model replicas start from the exact same state. Then, each DDP process creates a local <code>Reducer</code>, which later will take care of the gradients synchronization during the backward pass. …</p>
</blockquote>
Weight and bias initilization

username: pramesh
cooked: <p>Hello,</p>
<p>The requirement is :<br>
create initialization(w0,b0)&lt;–init()<br>
i need to use the same weight for online model and global model at first epoch and need to do self update of weight and bias…<br>
I have done following but its not working…please support me …</p>
<p>I have done the initilization by following code,</p>
<pre><code class="lang-auto">def init_normal(m):
    if type(m) == nn.Linear:
        nn.init.kaiming_uniform_(m.weight)
        if m.bias is not None:
            nn.init.constant_(m.bias.data, 0.001)
</code></pre>
<p>Now i want to initilize weight and bias generated by the above function.i had traied with</p>
<pre><code class="lang-auto">model1=copy.deepcopy()
</code></pre>
<p>when i call the model1,the weight and bias is not updated…</p>

username: ptrblck
cooked: <p>To use the <code>init_normal</code> function, you would have to use it via <code>model.apply(init_normal)</code>.<br>
I’m unsure what the real use case is and what exactly is not working, so could you explain the use case and problems a bit more?</p>
What here is defining the channels in the initialization?

username: cazforshort
cooked: <p>I’ve been trying to adjust this code in order to switch the model from vqgan f=16 to f=8, but something seems wrong with the size of the code and it fails to initialize.</p>
<p>Heres the code where the error is:</p>
<pre><code class="lang-auto">sideX=256
sideY=256
class Pars(torch.nn.Module):
    def __init__(self):
        super(Pars, self).__init__()

        self.normu = torch.nn.Parameter(o_i2.cuda().clone().view(batch_size, 1024, sideX//16 * sideY//16))
        
        self.ignore = torch.empty(0,).long().cuda()

        self.keep = torch.empty(0,).long().cuda()

        self.keep_indices = torch.empty(0,).long().cuda()

    def forward(self):

      mask = torch.ones(self.normu.shape, requires_grad=False).cuda()
      mask[:, :, self.ignore] = 1
      normu = self.normu * mask
      
      normu.scatter_(2, self.ignore.unsqueeze(0).unsqueeze(0).expand(-1, 256, -1), self.keep.detach())


      return normu.clip(-6, 6).view(1, -1, sideX//16, sideX//16)
</code></pre>
<p>When I run Pars().Cuda() the shape is [1, 1024, 16, 16], but I think it should be [1, 256, 16, 16] since I get the error:</p>
<blockquote>
<p>RuntimeError: Given groups=1, weight of size [256, 256, 1, 1], expected input[1, 1024, 16, 16] to have 256 channels, but got 1024 channels instead</p>
</blockquote>
<p>I can’t figure out where that channel number of 1024 is being defined?</p>

username: cazforshort
cooked: <p>The answer was</p>
<blockquote>
<p>return normu.clip(-6, 6).view(1, -1, sideX//8, sideX//8)</p>
</blockquote>
CUDA out of memory with colab

username: aleemsidra
cooked: <p>I am working on a classification problem and using Google Colab for the implementation. I am using transfer learning and specifically using ResNet at the moment. I have 5955 images in the train set. When I am running my code, I face the following error:</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 1.53 GiB (GPU 0; 14.76 GiB total capacity; 12.24 GiB already allocated; 1.27 GiB free; 12.44 GiB reserved in total by PyTorch)</code></p>
<p>I have tried to clear out cache using</p>
<pre><code class="lang-auto">device = 'cuda'
import torch, gc
import os
gc.collect()
torch.cuda.empty_cache()
</code></pre>
<p>It does not seem to work either. I have a batch size of 512 for train loader, 256 for valid and test loader.</p>

username: bing
cooked: <p>1)Try to reduce batch size.<br>
What’s the image size you are using?</p>

username: Vrushank98
cooked: <p>Go to Runtime =&gt; Restart runtime<br>
Check GPU memory usage by entering the following command:<br>
!nvidia-smi<br>
if it is 00 MiB then run the training function again.</p>

username: aleemsidra
cooked: <p>Its 224x224. I reduced the batch size from 512 to 64. But I do not understand why that worked.</p>

username: bing
cooked: <p>Batch size is basically how many samples are being propagated in a network at a time, so it makes it a function of a GPU memory available. The higher the batch size, the higher the memory required.</p>