akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Complicated conv1d in parallel

erezinman: <p>Hi,<br>
I’m been trying to use the <code>torch.nn.functional.conv1d</code> function to perform multiple convolutions on the same input in parallel using the <code>groups=...</code> parameter - so far to no avail. Unlike other questions I’ve seen answered, I intend to perform these operations on the entire batch (i.e. not 1 kernel per 1 item in batch). I think it will be best explained in code:</p>
<pre><code class="lang-python"># Data Shape
n_batch = 7
n_in_channels = 5
n_time = 17
data = torch.rand(n_batch, n_in_channels, n_time)

# Convolution shape
n_convs = 11
n_out_channels = 3
kernel_size = 13
conv_weights = torch.rand(n_convs, n_out_channels, n_in_channels, kernel_size)
bias_weights = torch.rand(n_convs, n_out_channels)
</code></pre>
<p>Noticeably, in my case one can assume that the padding, stride and dilation parameters are trivially set.</p>
<p>The output which I wish to compute In parallel can be computed using a loop in the following code:</p>
<pre><code class="lang-python"># Out-data shape
out_data = torch.empty(n_batch, n_convs, n_out_channels, n_time)

for i in range(n_convs):
    out_data[:, i, :] = torch.conv1d(data, conv_weights[i], bias_weights[i])
</code></pre>
<p>I’ve been trying to use the <code>groups</code> parameter, but as far as I can see, it required me to copy the input data several times.<br>
I also tried to use the <code>conv_transpose1d</code> (at least for the special case where <code>kernel_size=1</code>), but also without success.</p>
<p>Is it possible?<br>
Thanks!</p>

ptrblck: <p>I think increasing the number of output channels would just work unless I miss some details.</p>
<p>Your code is not executable as <code>out_data</code> has a wrong shape in <code>dim3</code>. After fixing it, a simple conv seems to work:</p>
<pre><code class="lang-python"># Data Shape
n_batch = 7
n_in_channels = 5
n_time = 17
data = torch.rand(n_batch, n_in_channels, n_time)

# Convolution shape
n_convs = 11
n_out_channels = 3
kernel_size = 13
conv_weights = torch.rand(n_convs, n_out_channels, n_in_channels, kernel_size)
bias_weights = torch.rand(n_convs, n_out_channels)

# Out-data shape
out_data = torch.empty(n_batch, n_convs, n_out_channels, 5)

for i in range(n_convs):
    out_data[:, i, :] = torch.conv1d(data, conv_weights[i], bias_weights[i])
    

res = torch.conv1d(data, conv_weights.view(-1, n_in_channels, kernel_size), bias_weights.view(-1))
print((res - out_data.view_as(res)).abs().max())
# tensor(0.)
</code></pre>

erezinman: <p>Thank you very much. That’s embarrassingly simple.</p>
CUDA out of memory problem

mathwseg: <p>i tried to use those line</p>
<pre><code class="lang-auto">model.eval()
device = torch.device("cuda")
model.to(device)
</code></pre>
<p>but got<br>
<code>RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 5.80 GiB total capacity; 4.52 GiB already allocated; 4.56 MiB free; 4.53 GiB reserved in total by PyTorch)</code><br>
in this line<br>
<code> model.to(device)</code></p>

ptrblck: <p>As the error message describes your GPU doesn’t have enough free memory and cannot allocate new memory to push the model to the GPU.<br>
Try to kill other processes which might consume memory and rerun your script.</p>

mathwseg: <p>Thanks for replying , i checked the nvidia-smi and didn’t find any process using GPU</p>

ptrblck: <p>In that case your script seems to allocate the memory before the <code>model.to()</code> call and you would need to reduce the memory usage.</p>

mathwseg: <p>i’m using batch_size =1 how can i clear the memory before i got this problem , please</p>

mathwseg: <p>is there any help please</p>

ptrblck: <p>You could try to <code>del</code> unneeded tensors before creating the model.<br>
However, it depends on your use case if deleting tensors etc. from the device would actually work or if you would need them.</p>

mathwseg: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="156698">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>del</code> unneeded tensors</p>
</blockquote>
</aside>
<p>where exactly i should del the tensor , please<br>
the  error exactly in this line     <code>vec=model(preprocess(image).unsqueeze(0).cuda())</code> i used a counter  after vec1 and got the tensor for the first image then throw this error</p>

ptrblck: <p>It seems to error now moved from the <code>model.to()</code> operation to the actual forward pass.<br>
If the only data on the GPU are the model’s parameters and the input tensor, your model might be too large for your device. You could try to use e.g. <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

mathwseg: <p>thanks a lot for your time but can i extract the features using CPU and used it in my task or it will be a problem due to large size</p>

ptrblck: <p>I don’t know how high the peak memory usage for your entire workflow is, but I would assume that your system has more host RAM than the GPU (6GB), so it’s worth a try.</p>
Applying equal transformation on label as on image (e.g. RandomPerspective())

ptinn: <p>When we use <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.RandomPerspective.html" rel="noopener nofollow ugc">RandomPerspective()</a> as part of data augmentation, I notice the transformation is applied only to the image and not the labels that are also pixel location-dependent, which results in a misalignment between the two. Consider the following:</p>
<pre><code class="lang-auto">class RandomPerspective(object):
    def __init__(self, scale=0.5, prob=0.5):
        self.perspective_transformer = T.RandomPerspective(distortion_scale=scale, p=prob)

    def __call__(self, sample):
        image, labels = sample['image'], sample['labels']
        pil_image = F.to_pil_image((image).astype(np.uint8))
        image = (np.asarray(self.perspective_transformer(pil_image)))
   
        return {'image': image, 'labels': labels}
</code></pre>
<p>How does one–using this <code>RandomPerspective()</code> , <code>RandomHorizontalFlip()</code> or other similar modules–apply the same transformation to shift the coordinates of the label data to align with the transformed image?</p>

ptrblck: <p>Different approaches are discussed in e.g. <a href="https://github.com/pytorch/vision/issues/1406">this issue</a> or <a href="https://github.com/pytorch/vision/issues/9">this one</a>.<br>
My suggestion would be to use the functional transformation API, grab the parameters for each transformation, and apply it to the image and target as seen <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">here</a>.</p>
Fast masking/row-selection without copying

yousiyu: <p>My training loop looks like the following</p>
<pre><code class="lang-auto">for X, mask in dataloader(...):
    X = X[mask]  # only a small subset (of rows) of X is good for training, very slow
    X_cuda = X.cuda(non_blocking=True)
    prediction = model(X_cuda)
    ...
</code></pre>
<p>My X is very large. The masking/row-selection is taking a lot of time (because it copies the data instead of using shared storage) in each cycle so I cannot sufficiently use my GPU.</p>
<p>Is there a way to do X[mask] that avoids data copying?<br>
Thanks</p>

ptrblck: <p>I don’t think that’s possible as the number of output elements depends on the <code>mask</code> and is thus unknown before the actual values are available. I.e. you won’t be able to preallocate a tensor in a specific shape unless you waste memory use use the max. number of elements (<code>X.nelement()</code>).</p>

yousiyu: <p>Can X[mask] return a view or something?<br>
The following code returns a view instead of a copy right?</p>
<pre><code class="lang-auto">a, b = get_range(...)
view = X[a: b]
</code></pre>

ptrblck: <p>No, I don’t think masking can create a view, since the masked indices can be random.<br>
Yes, slicing the tensor will create a view, so if your mask uses a specific stride/indexing logic you might want to convert it to an indexing operation.</p>
How to compute union between two 2-D tensor

RobotGF: <p>Is there any API like tf.sets.union?<br>
or How to implement union between two 2-D tensor ?</p>

clockzhong: <p>What’s your definition of the union between two 2-D tensors?<br>
The two 2-D tensors represent two sets? What’s their relationship? There is no standard defintion on this “union operation”.</p>
Pytorch 1.12 with CUDA on Ubuntu20?

ptinn: <p>Hi,</p>
<p>Is there a specific way to install or upgrade a higher version &gt; 1.9.0 of PyTorch on Ubuntu 20 with CUDA?  The standard <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">installation command</a> on done my computer keeps the 1.9.0 version untouched:</p>
<pre><code class="lang-auto">Requirement already satisfied: torch in ./.local/lib/python3.8/site-packages (1.9.0+cu111)
Requirement already satisfied: torchvision in ./.local/lib/python3.8/site-packages (0.10.0+cu111)
Requirement already satisfied: torchaudio in ./.local/lib/python3.8/site-packages (0.9.0)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.8/site-packages (from torch) (4.2.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from torchvision) (1.22.3)
Requirement already satisfied: pillow&gt;=5.3.0 in ./.local/lib/python3.8/site-packages (from torchvision) (9.1.0)
</code></pre>
<p>How does one get to a stable version beyond this one?</p>
<p>P</p>

ptrblck: <p>Uninstall the PyTorch binaries and reinstall the latest stable ones or create a new virtual environment and install the latest release there.</p>

ptinn: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
How to make RandomRotation (Or any other random transform) fixed for DataLoader?

iso: <p>Hey there,<br>
I understand that the transforms we put on the dataset only happen when we actually call the data (for example, iterating over a dataloader). However, for creating a randomized dataset of any sort this will give different inputs (for the same initial images). Is there a way to make the dataset fixed? I guess seeding before each time we loop on the dataloader is possible but I’m looking for something more robust.</p>
<p>Example code:</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

transform = transforms.Compose([
                transforms.RandomRotation(100),
                 transforms.ToTensor()
             ])

mnist_train = datasets.MNIST("../mnist_train",download=True,train=True,
                             transform=transform )

loader = DataLoader(mnist_train,
                             batch_size=1,
                             shuffle=False)
</code></pre>
<p>If you will try to run this (the sum of the first input values), you should get different results each time due to the random transformation happening dynamically:</p>
<pre><code class="lang-auto">iter(loader).next()[0].ravel().sum()
</code></pre>
<p>I will want to pre-compute the transform somehow so the dataloader will be static.</p>

ptrblck: <p>Seeding is the right approach to let random transformations output deterministic results. Since it doesn’t seem to fit your use case you could iterate the <code>DataLoader</code> and store the randomly transformed samples. Afterwards you could then load these samples directly and disable all random transformations. Note that this approach would of course use more memory on your local SSD to store the data and the “randomness” would be limited to the number of epochs you’ve used to create the dataset.</p>

iso: <p>Thanks, well  then I might just go with seeding it beforehand for now. Yes, this is actually not what the transformations were made for, but here’s the case of wanting to create a brand new dataset utilizing pytorch’s transforms</p>
Normalisation in DCGANs which is not mean of 0.5 and std of 0.5

Chessnutter: <p>In the DCGAN tutorial (<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 1.12.0+cu102 documentation</a>), they use normalisation with means of (0.5, 0.5, 0.5) and standard deviations of (0.5, 0.5, 0.5). This happens to work nicely, as the minimum and maximum values of an image, 0 and 1, are both 1 standard deviation away from the mean. Therefore after normalisation, real images will be between -1 and 1, which is the co-domain of tanh.</p>
<p>If I instead use the ImageNet values for the mean and standard deviation, (0.485, 0.456, 0.406) and (0.229, 0.224, 0.225), suddenly normalised real images map to between (-2.11790393, -2.03571429, -1.80444444) and (2.2489, 2.4286, 2.6400).</p>
<p>I could multiply the output of the generator by (2.2489, 2.4286, 2.6400) and take the max of that and (-2.11790393, -2.03571429, -1.80444444), but this sounds like a terrible idea as it zeros the gradient for dark colours.</p>
<p>I could subtract the means from the output of the generator and divide by the standard deviation, but this defeats the point of normalised values in the first place, since the generator would be outputting an effectively unnormalised value</p>
<p>Does anyone know the way to get around this issue?</p>
Dynamic Pooling in PyTorch

nasim_nouri: <p>What is the best way to perform dynamic pooling (e.g. max pooling) on a Tensor? For instance:</p>
<pre><code class="lang-auto">import torch
A = torch.rand(4,10,3)
B = torch.LongTensor([[0,1],[4,4],[2,5],[7,7]])
</code></pre>
<p>Tensor <code>A</code> represents 4 examples of length 10-token, each token is represented with a vector of size 3. Tensor <code>B</code> contains 4 pairs of indices, each of which represents indices of the first and last token for each example that should be pooled. For instance, for the third example (i.e., <code>A[2,:,:]</code>), max-pooling should be operated on vectors <code>A[2,2,:]</code>, <code>A[2,3,:]</code>, <code>A[2,4,:]</code>, and <code>A[2,5,:]</code>.</p>
Train large image dataset in Pytorch

Jiahui_Tan: <p>I have a dataset including 12156*1954 images, which is so large that CPU/GPU memory can’t carry on, and training one epoch will take a very long time.</p>
<p>Is there any way I could succeed in training the PyTorch model?</p>
<pre><code class="lang-auto">class DataSet(Dataset):
    def __init__(self,fname, index):
        self.dataset = fname
        self.dataset_num = index
    
    def __len__(self):
        return len(self.dataset)
    
    def __getitem__(self,idx):
        datas = np.load(self.dataset[idx])
        x = datas['data'][self.dataset_num[idx]]
        x = x.astype(np.float32)
        x=torch.from_numpy(x)
        tens = x.reshape(100*100)
        inds = datas['inds'][self.dataset_num[idx]].tolist()
        return tens, inds
</code></pre>

ptrblck: <p>You could resize the samples to a smaller spatial size or use crops of your inputs to train the model and to avoid running out of memory.</p>

iso: <p>Adding to what <a class="mention" href="/u/ptrblck">@ptrblck</a> said above me, look into pytorch transforms. for a quick check you can resize a few images online and understand what is a reasonable new resolution that still perceives details on your dataset images.</p>
DataParallel divided into blocks?

Sam_Lerman: <p>For model-specific reasons, I have to apply DataParallel on multiple separate blocks, instead of on one single model. Then these blocks chain together.</p>
<p><code>Block1 -&gt; Block2 -&gt; etc.</code></p>
<p>Each are Pytorch modules.</p>
<p>Now the problem is, that DataParallel collates the parallelized batches at each intermediate block!</p>
<p><code>Block1 (Parallelized) -&gt; Collate -&gt; Block2 (Parallelized) -&gt; Collate -&gt; etc.</code></p>
<p>This collating is unnecessary and slow but an inadvertent byproduct of this modulation.</p>
<p>Is there any way to collate just-in-time or do some adaptive collate method that keeps the batches parallelized through all of the intermediate blocks and only collates at the end at <code>BlockN</code>?</p>

ptrblck: <p>I’m unsure what the exact use case is, but if you want to avoid the gather/scatter operations between the <code>DataParallel</code> blocks, you could just wrap all blocks into a <code>DataParallel</code> module.<br>
Could you explain why this wouldn’t work as it seems you have a specific limitation for this use case?</p>
Python 2.7 -- Torch 1.4.0 -- Cuda 10.1 -- cublas runtime error

Taiton: <p>Hello,<br>
Running the following code:</p>
<p>import torch<br>
import torch.nn.functional as F</p>
<p>theta = torch.Tensor([[[1, 1, 1], [1, 1, 1]]]).cuda()<br>
F.affine_grid(theta, torch.Size((1, 3, 2, 2)))</p>
<p>gives me the error:</p>
<p>File “/home/toni/PycharmProjects/testingGPU/venv/lib/python2.7/site-packages/torch/nn/functional.py”, line 2799, in affine_grid<br>
return torch.affine_grid_generator(theta, size, align_corners)<br>
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:368</p>
<p>I have checked all the compatibility issues and I believe there is the right match between:<br>
Python 2.7 – Torch 1.4.0 – Torchvision 0.5.0 – CUDA 10.1 – Ubuntu 18.04  (and I would like to keep using python2.7 for the current project)</p>
<p>I am trying to run it on RTX 3060, driver version 515.48.07</p>
<p>I’ve looked on many repositories and forums, but I could not find a solution (while keeping the PyTorch recommended configuration with torch 1.4.0, CUDA 10.1)</p>

ptrblck: <aside class="quote no-group" data-username="Taiton" data-post="1" data-topic="156798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taiton/48/45257_2.png" class="avatar"> Taiton:</div>
<blockquote>
<p>I am trying to run it on RTX 3060, driver version 515.48.07</p>
</blockquote>
</aside>
<p>Your Ampere GPU needs CUDA 11, so your CUDA 10.1 runtime with this old PyTorch version won’t work.<br>
Also, Python 2.7 is EOL so I would strongly recommend to the latest stable or nighlty release with the CUDA 11.6 runtime.</p>

Taiton: <p>nvidia-smi actually shows the version 11.7, but I installed the toolkit 10.1 (being the last one that supported python 2.7) because I thought there was a backward compatibility and could run toolkit 10.x with drivers for 11.x.</p>
<p>But thank you, my search is done…i ll try upgrading everything in the project to python 3 since there is no support for python2 in the 11.x cuda toolkit versions.</p>
<p>Should my 11.7 cuda version work with cuda toolkit 11.6 or 11.3?</p>

ptrblck: <aside class="quote no-group" data-username="Taiton" data-post="3" data-topic="156798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taiton/48/45257_2.png" class="avatar"> Taiton:</div>
<blockquote>
<p>Should my 11.7 cuda version work with cuda toolkit 11.6 or 11.3?</p>
</blockquote>
</aside>
<p>Yes, your driver should be compatible with both CUDA runtimes shipped in the PyTorch binary.</p>
Subsampling Only Spatial Dimentions in a 5D Tensor

Tm4: <p>I have a 5D tensor <code>x</code> (frames of a video) and I want to upsample the spatial size (the last two dimensions) of this tensor but when I use upsampling, the last three dimensions of the tensor are upsampled. For upsampling I use the following class:</p>
<pre><code>class Upsample(nn.Module):
    def __init__(self, scale_factor, mode, align_corners=False):
        self.interp = interpolate
        self.scale_factor = scale_factor
        self.mode = mode
        self.align_corners=align_corners

    def forward(self, x):
        x = self.interp(x, scale_factor=self.scale_factor, mode=self.mode)
        return x
</code></pre>
<p>And for example, the main class that I want to upsample a 5D tensor is as follows (I condensed my code):</p>
<pre><code>class Main(nn.Module):
   def __init__(self):
       super(Main, self).__init__()
       self.upsample = Upsample(scale_factor=2, mode='trilinear')

   def forward(self, x):
        x = self.upsample(x)
        return x
</code></pre>
<p>To be clearer, for example by applying upsampling on a tensor of <code>x=(2,4,3,10,20)</code>, the outcome based on the aforementioned class is <code>x=(2,4,6,20,40)</code> but I need to have <code>x=(2,4,3,20,40)</code>.</p>
<p>What is the problem and how can I solve this?</p>

ptrblck: <p>The issue is caused by the standard layout of a 5D tensor as <code>[batch_size, channels, depth, height, width]</code> and by specifying only a single <code>scale_factor</code>. This module then expects to apply the <code>scale_factor</code> too all 3 dims.<br>
If you want to skip the interpolation in the depth dimension, specify the <code>scale_factor</code> as a <code>tuple</code>:</p>
<pre><code class="lang-python">x = torch.randn(2, 4, 3, 10, 20)

up = nn.Upsample(scale_factor=(1, 2, 2), mode='trilinear')
out = up(x)
print(out.shape)
# torch.Size([2, 4, 3, 20, 40])
</code></pre>

Tm4: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  , Thanks a lot for your answer. Is it possible to include only the <code>batch_size</code> in the subsampling in addition to the spatial dimensions, too?</p>

ptrblck: <p>No, you cannot directly interpolate the batch dimension using the <code>Upsample</code> module or <code>F.interpolate</code>. However, you could <code>permute</code> the input tensor such that the batch dimension would align with a spatial or volumetric dimension, interpolate it, and <code>permute</code> it back.<br>
Note that this is not a typical use case since you are trying to interpolate “between samples”, so check if this is really what you want to do.</p>

Tm4: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  , Thanks a lot.</p>
Help with feature extracting pre-trained VisionTransformer model

Liano: <p>Can I finetune (in my case feature extracting) the pre-trained VisionTransformer model the same way as described in the pytorch Tutorial (<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation</a>) ?</p>
<p>In the tutorial only convolutional neural networks are used, so do I have to modify anything for the VisionTrasformer model (<a href="https://pytorch.org/vision/stable/models/vision_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">VisionTransformer — Torchvision 0.13 documentation</a>) or does it just work the same way?</p>

ptrblck: <p>The general fine-tuning logic would apply to all models, but you might need to change some model attributes (e.g. replacing the <code>.classifier</code> might look a bit different using other models depending how the internal modules are structured).</p>
How to JIT compile C++ Extension with libpng

ribofadam: <p>I want to compile a custom c++ extension that uses functionality from libpng. This is what my current <code>jit.py</code> file looks like:</p>
<pre><code class="lang-auto">from torch.utils.cpp_extension import load
from glob import glob
import os

os.environ["CC"] = "gcc-11"
os.environ["CXX"] = "g++-11"

sources = glob("src/*.cpp")
font_renderer = load(name="font_renderer", sources=sources, extra_cflags=["-std=c++17"], verbose=True)
help(font_renderer)
</code></pre>
<p>If I try to run this, I get an error with the first libpng function that I try and use, <code>undefined symbol: png_get_image_width</code></p>
<p>Since I want to <code>#include &lt;png.h&gt;</code> in one of my source files, how do I go about adding this simple dependency?</p>
<p>Thank you.</p>
How to set the right GPU

jS5t3r: <p>How to set the GPU at <code>iter</code>?  I want that <code>iter</code> initializes on <code>cuda:0</code> on the second line.</p>
<pre><code class="lang-auto">device = "cuda:0"
train_adv_loader_iter = iter(train_adv_loader) # How to set the GPU here? 
for i, (images, target) in enumerate( tqdm(train_loader)):
    data_adv = next(train_adv_loader_iter)

    print( data_adv.get_device() ) # cuda:2
    images_adv = data_adv[0].to(device)
    target_adv   = data_adv[1].to(device)
</code></pre>

a-parida12: <p>I set the visible device on the very top of the of the code.</p>
<pre><code class="lang-python">import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
</code></pre>
<p>other way could be you can pass in the device information into your data loader so that the correct device is used in that.</p>

jS5t3r: <p>Thanks. This is a very easy solution.</p>
Early stopping pytorch

Anita: <p>Hi,</p>
<p>I am trying to install a Github library and pytorchtools and I keep getting this error:</p>
<p>!pip install git+https://github.com/Bjarten/early-stopping-pytorch</p>
<p>Collecting git+https://github.com/Bjarten/early-stopping-pytorch<br>
Cloning <a href="https://github.com/Bjarten/early-stopping-pytorch" rel="nofollow noopener">https://github.com/Bjarten/early-stopping-pytorch</a> to /tmp/pip-req-build-txarset_<br>
Running command git clone -q <a href="https://github.com/Bjarten/early-stopping-pytorch" rel="nofollow noopener">https://github.com/Bjarten/early-stopping-pytorch</a> /tmp/pip-req-build-txarset_<br>
ERROR: Complete output from command python setup.py egg_info:<br>
ERROR: Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/nita/anaconda3/lib/python3.7/tokenize.py”, line 447, in open<br>
buffer = <em>builtin_open(filename, ‘rb’)<br>
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-txarset</em>/setup.py’<br>
----------------------------------------<br>
ERROR: Command “python setup.py egg_info” failed with error code 1 in /tmp/pip-req-build-txarset_/</p>
<p>Does anyone has previous experience on this. thank you in advance</p>

sparseinference: <p>Usually, <code>pip install</code>  requires a <code>setup.py</code> file in the directory containing the library to install.<br>
I haven’t used <a href="https://github.com/Bjarten/early-stopping-pytorch" rel="nofollow noopener">https://github.com/Bjarten/early-stopping-pytorch</a> but I don’t see any setup script there.</p>

Anita: <p>Yes that was the problem!</p>
<p>thank you for your feedback</p>

akhilesh_rai: <p>Was this resolved? Were you able to install?</p>

aneeka_azmat: <p>how did you solve it ? can you please tell me?</p>
Extract features from Mask R-CNN

Joysn: <p>I am following [1] to extract the features of the different layers. The Mask R-CNN model uses a resnet50 backbone, and there I want to add the feature extractors.</p>
<pre><code class="lang-auto">    train_nodes, eval_nodes = get_graph_node_names(model.backbone)

    nodes = {
        "body.conv1": "layer1",
        "body.maxpool": "layer2",
    }
    backbone = create_feature_extractor(model.backbone, return_nodes=nodes)
    model.backbone = backbone
    model.eval()
    out = model(img)
</code></pre>
<p>When executing this, the following error is thrown:</p>
<pre><code class="lang-auto">  File "C:\project/ml/visualize_model.py", line 184, in &lt;module&gt;
    out = model(img)
  File "C:\project\venv\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\project\venv\lib\site-packages\torchvision\models\detection\generalized_rcnn.py", line 98, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "C:\project\venv\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\project\venv\lib\site-packages\torchvision\models\detection\rpn.py", line 341, in forward
    objectness, pred_bbox_deltas = self.head(features)
  File "C:\project\venv\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\project\venv\lib\site-packages\torchvision\models\detection\rpn.py", line 50, in forward
    t = F.relu(self.conv(feature))
  File "C:\project\venv\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\project\venv\lib\site-packages\torch\nn\modules\conv.py", line 447, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\project\venv\lib\site-packages\torch\nn\modules\conv.py", line 443, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [256, 256, 3, 3], expected input[1, 64, 400, 512] to have 256 channels, but got 64 channels instead
</code></pre>
<p>[1] <a href="https://pytorch.org/vision/main/feature_extraction.html" class="inline-onebox" rel="noopener nofollow ugc">Feature extraction for model inspection — Torchvision main documentation</a></p>
Help with model architecture for a racing game

dht: <p>I’m working on a model for a racing game using pytorch. The model gets frame from the game as input and produces a controller state as output. The dataset consists of frames from the game and corresponding controller state (buttons state + joystick state)<br>
Until now I’ve been using a cnn connected to linear layers to predict the output. The problem is that this model cannot establish connection between buttons state output and the joystick state output so it can’t learn how to drift or aim items. (when the player drifts using b, the joystick output need to be updated accordingly)<br>
I thought about the following architecture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4588e9a60ca77945a35a5399bd595f96fadab2a.png" data-download-href="https://discuss.pytorch.org/uploads/default/d4588e9a60ca77945a35a5399bd595f96fadab2a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4588e9a60ca77945a35a5399bd595f96fadab2a_2_690x199.png" alt="image" data-base62-sha1="uiuXi72isuiaOLeuhdQgR956HlE" width="690" height="199" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4588e9a60ca77945a35a5399bd595f96fadab2a_2_690x199.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4588e9a60ca77945a35a5399bd595f96fadab2a_2_1035x298.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4588e9a60ca77945a35a5399bd595f96fadab2a_2_1380x398.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2182×630 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’m new to deep learning and I would like to know if someting like that even makes any sense because I haven’t seen anything like it before. If it does not make sense , can you suggest another model architectre?<br>
Thanks in advance.<br>
dht</p>
Need some guidance for data loading many small files

Messiah: <p>Hello,</p>
<p>I am working on a project in which many small files (~300K) containing .npy arrays are to be combined for training a deep learning model. Unfortunately, the files need to be combined into a unified dataset as each file does not represent a single sample, but rather a variable number of samples. With this being said, there is quite a significant bottleneck during data loading where each of these files must be iterated over and appended to a unified array in order to be in a suitable format for the DataLoader. I understand that this is not an ideal approach, as iterating over many small files can cause significant overhead. This is why my data loading can take upwards of 30 minutes to an hour depending on the size of the dataset.</p>
<p>I was wondering if anyone has experienced a similar circumstance and would be willing to explain how you dealt with it. One idea I have is to save a range of small files into single, larger files, such that there is much less smaller files, but a few larger files instead. This (I assume) would cause less overhead as there is less files to iterate through.</p>
<p>Does anyone know of any other advice that can be applied to help speed up my data loading procedure given this scenario? Any help is much appreciated.</p>
Binary_cross_entropy crashes when data on mps device

Podidiving: <p>Hello everyone.</p>
<p>I’m currently experimenting with <code>mps</code> accelerator on my m1 pro macbook. Encountered weird behaviour</p>
<p>This code works perfectly fine</p>
<pre><code class="lang-auto">device = torch.device("cpu")
model = nn.Sequential(
    nn.Linear(10, 20),
    nn.ReLU(),
    nn.Linear(20, 1),
    nn.Sigmoid()
)
model.to(device)
input = torch.randn(5, 10).to(device)
target = torch.randint(0, 2, (5, 1)).float().to(device)

F.binary_cross_entropy(
    model(input),
    target
)
</code></pre>
<p>but when I made a single change (“cpu” → “mps”):</p>
<pre><code class="lang-auto">device = torch.device("mps")
model = nn.Sequential(
    nn.Linear(10, 20),
    nn.ReLU(),
    nn.Linear(20, 1),
    nn.Sigmoid()
)
model.to(device)
input = torch.randn(5, 10).to(device)
target = torch.randint(0, 2, (5, 1)).float().to(device)

F.binary_cross_entropy(
    model(input),
    target
)
</code></pre>
<p>everything crashed.</p>
<p>My interpreter died with the following message</p>
<pre><code class="lang-auto">loc("reductionMeanTensor"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/b6051351-c030-11ec-96e9-3e7866fcf3a1/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":281:0)): error: invalid axes: 1
LLVM ERROR: Failed to infer result type(s).
[1]    8876 abort      ipython
</code></pre>
<p>Can anyone help me with this problem? Is this operation not supported by mps yet or is it my mistake?</p>
<p>P.s. I’ve encountered this only with <code>binary_cross_entropy</code> operation. I tried some more experiments (for ex. with <code>cross_entropy</code>) and they worked just fine.</p>
<p>My torch version is <code>1.12.0</code></p>
Only train certain rows of a linear layer

Niels_PyTorch: <p>Hi,</p>
<p>I’d like to train only several rows of a linear layer (the linear layer is a classification head on top of BERT for multi-label classification, and I’d like to only train the head further for 10 labels). I tried the following:</p>
<p><code>model.classifier.weight[other_labels_indices,:].requires_grad = False model.classifier.bias[other_labels_indices].requires_grad = False</code></p>
<p>In other words, I’d like to freeze the parameters of the classifier for the “other labels”, which already have good performance, and only train the parameters corresponding to the labels that are currently underperforming.</p>
<p>However, this gave me the following warning:</p>
<p><code>RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().</code></p>
<p>I then tried the following:</p>
<pre><code class="lang-auto">model.classifier.weight[other_labels_indices,:].detach_()
model.classifier.bias[other_labels_indices].detach_()
</code></pre>
<p>However, it seems that these parameters are still being updated. Am I doing something wrong?</p>

albanD: <p>Hi,</p>
<p>I am afraid you won’t be able to do that.<br>
The “requires_grad” property is associated with the whole Tensor. And cannot be set for subsets.</p>
<p>You have a few approaches here:</p>
<ul>
<li>Compute the gradient for the full layer and then zero-out the gradients before doing the optimizer update</li>
<li>If your optimizer still update the value when there is 0 gradient, then it is simpler to store the original value of the rows you want to save. Then do the full gradient computation and update with the optimizer and then write back the rows that you didn’t want changed.</li>
<li>The last option is to have a different Tensor (that is going to be your Parameter) that contains all the rows that you want learnt. And at every forward, write this Tensor into a bigger Tensor that also contains the frozen entries. And do training with that.</li>
</ul>

Niels_PyTorch: <p>Thanks! I tried the first approach and it seems to work.</p>

Diego_Garcia-soto: <p>Hi, I’m in a similar situation. These approaches work well, but they seem to be very slow.</p>
<p>My solution was to manually program the gradient descent, but it seems a bit limited, and wanted to use optimizers to use more advanced gradient descent algorithms.</p>
<p>Using the optimizer and zeroing the gradients of all the parameters I don’t want to update takes twice the time it used to take, and also I’ve done some tests and I think the part that takes long to run is actually zeroing all the gradients.</p>

Diego_Garcia-soto: <p>Actually, I was able to solve this.</p>
<p>Instead of zeroing the gradient of all gradients using a for loop, now I do it with a matrix multiplication on GPU, and now it’s much faster.</p>
<p>Now I basically do<br>
weights.grad = torch.mm(matrix, weights.grad)<br>
where the matrix is a matrix full of 0s and some 1s on specific positions to keep the gradients of the parameters I want.</p>
Error: no matching function for call to ‘InferShapes

jiapei-nexera: <pre data-code-wrap="console"><code class="lang-nohighlight">shape_type_inference.cpp:1989:47: error: no matching function for call to ‘InferShapes(std::__shared_ptr_access&lt;onnx_torch::ModelProto, __gnu_cxx::_S_atomic, false, false&gt;::element_type&amp;, onnx_torch::OpSchemaRegistry*&amp;, onnx_torch::ShapeInferenceOptions&amp;, torch::jit::ShapeDataMap*)’
 1989 |             onnx::shape_inference::InferShapes
</code></pre>
<p>Can anybody help??</p>
How Spatial Reduction works at Pyramid Vision Transformers

Oguzhan: <p>I am trying to understand how spatial reduction attention works but I could not find any source. You can see the formulas below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c66e4c587286a5ef3210290395449327630f41b0.png" data-download-href="https://discuss.pytorch.org/uploads/default/c66e4c587286a5ef3210290395449327630f41b0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c66e4c587286a5ef3210290395449327630f41b0_2_424x500.png" alt="image" data-base62-sha1="sjoQP6HUngZ1NLBOL4pLRBNQlZ6" width="424" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c66e4c587286a5ef3210290395449327630f41b0_2_424x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c66e4c587286a5ef3210290395449327630f41b0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c66e4c587286a5ef3210290395449327630f41b0.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">624×735 93.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and related paper: <a href="https://arxiv.org/pdf/2102.12122.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2102.12122.pdf</a></p>
<p>Can someone describe it, an example would be great.</p>

Oguzhan: <p>I think it works like Linformer, you can see it here: <a href="https://www.youtube.com/watch?v=-_2AF9Lhweo" class="inline-onebox" rel="noopener nofollow ugc">Linformer: Self-Attention with Linear Complexity (Paper Explained) - YouTube</a></p>
Simple Pixel-wise statistics for semantic Segmentation

joshuaeh: <p>I am trying to benchmark some later work using some common heuristics for identifying clouds. For example, one of the common statistics is that the Red-Blue ratio will be &gt; 0.7 for clouds.</p>
<p>I want to optimize this <code>Red / Blue &gt; X</code> value for X with the dataset I am working with, and then compute accuracy, IOU, etc. with ground truth masks. This is what I currently have:</p>
<pre><code class="lang-auto">class RBR_Model(nn.Module):
    """Custom PyTorch model for gradient optimization of RBR statistics"""

    def __init__(self):
        super().__init__()  # inherit from parent class
        self.RBR_threshold = torch.Tensor([0.7])

    def forward(self, img):
        print(img.shape)
        R = img[:,:,:,0]
        B = img[:,:,:,2]
        RBR = R / B
        RBR_mask = RBR &gt; self.RBR_threshold
        return RBR_mask
</code></pre>
<p>Apologies on the inexperience with pytorch.</p>
<p>TLDR: take image of input [H W C] and convert to [H W] with binary decision based on one value: the ratio of red and blue</p>

ptrblck: <p>The <code>self.RGB_threshold</code> won’t be trainable as the comparison is not differentiable and will detach the tensor (besides that you would also need to create <code>self.RGB_threhsold</code> as an <code>nn.Parameter</code>).<br>
Assuming you are comparing your feature against a single threshold, wouldn’t a ROC curve work to select it?</p>

joshuaeh: <p>Thanks for your reply-- My goal is to compare FCNN and Barlow Twin architectures to segment clouds, But establishing a baseline with a few heuristics that have been done in literature would be interesting and could contribute to a third bayesian model was my thought. I felt that to compare these heuristics to fitted models, I should fit these naive heuristics as well.</p>