akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about Transforms

Wise_One: <p>I have a dataset with 3 classes. To train a model, I want to apply different transforms for the 3 classes. How do I do that?</p>

ptrblck: <p>You can create a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and apply custom transformations in the <code>__getitem__</code> by using a condition on the target.</p>
Silly question about training the model

Tejan_Mehndiratta: <p>Hi, so if I trained my model for 10 epochs and then saved its state_dict(). (And closed the laptop). So after loading the state_dict() and running the model again for 10 epochs. Will it be the same thing as running the model for 20 epochs at once?</p>

sio277: <p>Yes. If you use the optimizer containing estimates of gradients, such as Adam and AdamW, also try to save and load its <code>state_dict()</code>.</p>

Tejan_Mehndiratta: <p>I am sorry, I don’t understand what you are saying.</p>

sio277: <p>This document might be helpful: <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.8.1+cu102 documentation</a>.</p>
Generating data in parallel from a pre-trained model

billiout: <p>I have a <code>Dataset</code> object that yields data generated from a pre-trained model that I load at the initialization of the dataset object. The model generating these data is placed on a separate GPU from the main model that will be optimized. Since the generation process is on a separate GPU, I would like to be able to generate more examples in parallel while the main model is performing an optimization step. I tried setting <code>num_workers</code> of the <code>DataLoader</code> object to a value greater than zero but I get the following error:</p>
<pre><code class="lang-auto">  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'module' object
</code></pre>
<p>Any suggestions on how to make this process faster in general?</p>
How could I use one dimension index to visit the parameter of model?

jerly-hjzhou: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c5b6442503797774753b9be7bb1c3a59799b110.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c5b6442503797774753b9be7bb1c3a59799b110" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c5b6442503797774753b9be7bb1c3a59799b110_2_690x475.png" alt="image" data-base62-sha1="hK70KbpWvsh82pBmsbPZCabaItW" width="690" height="475" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c5b6442503797774753b9be7bb1c3a59799b110_2_690x475.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c5b6442503797774753b9be7bb1c3a59799b110_2_1035x712.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c5b6442503797774753b9be7bb1c3a59799b110.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1098×757 59.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Does there exist some function to iterate the parameter in one dimension index?</p>

JuanFMontesinos: <p>I’m afraid the way it is coded modules contains submodules which contains submodules etcetera… Therefore the parent <code>nn.Module</code> is not aware of it.</p>
<p>Besides parameters is an iterable.<br>
The poor solution is saving the iterable into a  list like:<br>
<code>list(model.parameters())[1]</code><br>
The elegant solution is filter by name:</p>
<pre><code class="lang-auto">for name,param in model.parameters():
    if name == condition:
       do something
</code></pre>
<p>The last option is to directly query for it if you know the name:<br>
<code>model.decoder.weight</code> (for example)</p>

jerly-hjzhou: <p>Thank you for your reply. I did badly in explaining my meaning. I actually want to visit every single parameter. With model.parameters()[1], I expect to get the second parameter among the 11181642 parameters instead of a tensor with sizes of more than one. Do you know any ways to reach that?</p>
Convnet preserve dimensions

sugh7020: <p>Hi!<br>
I have a CNN:</p>
<pre><code class="lang-auto">class ConvNet(nn.Module):
    def __init__(self, num_classes):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 4, kernel_size=(1, 5), stride=1, padding=(2, 2)),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=(1, 3)))
        self.layer2 = nn.Sequential(
            nn.Conv2d(4, 8, kernel_size=(1, 5), stride=1, padding=(2, 2)),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=(1, 3)))
        self.layer3 = nn.Sequential(
            nn.Conv2d(8, 16, kernel_size=(1, 5), stride=1, padding=(2, 2)),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=(1, 3)))
        self.fc1 = nn.Flatten()
        self.fc2 = nn.Linear(2240, 64)
        self.fc3 = nn.Linear(64, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out
</code></pre>
<p>My input dimension(1 batch) is of shape:</p>
<blockquote>
<p>torch.Size([1, 1, 128, 40])</p>
</blockquote>
<p>I want to preserve the shape of (128, 40) before it goes into fc1. But I guess I’m doing the padding incorrectly since the shape after layer1 is:</p>
<blockquote>
<p>torch.Size([1, 4, 132, 13])</p>
</blockquote>
<p>Also, I need the convolutions and maxpool along each rows only. Can you all please suggest how I can do that?<br>
Also, for the input dimension to fc2(linearlayer → 2240 here) is it possible to define the value independent of the batch size?</p>
<p>Thanks a lot!</p>

JuanFMontesinos: <p>Well indeed if you use maxpool you need much more padding. There is a pytorch pad function which you can apply after each maxpool if that’s what you really want. Anyway I don’t find why would you like to maxpool if you want to preserve the shape. You may want to use dilated convolutions instead.</p>
Reshaping the MFCC

HitsujiAura: <p>Hello,</p>
<p>I used torchaudio to extract the MFCC from an audio so that it has the same amount of frames as the corresponding video I extracted - 525. However, the shape of the MFCC is (1, 20, 525).<br>
How can I reshape the data to get a dimension of (525, 20). So that I can manipulate it alongside the video frames?</p>
<p>Thank you for your help</p>

JuanFMontesinos: <p>You can use permute so that if you have M a tensor of shape (1,20,525) you can reach a tensor of shape (525,20) by doing<br>
<code>M[0].permute(0,1)</code></p>
Not able to understand Tensorboard nodes

Tejan_Mehndiratta: <p>Hi,<br>
My code is</p>
<pre><code class="lang-auto">class Tenet(nn.Module):
    
    def __init__(self):
        super(Tenet, self).__init__()
        
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=(3,3), padding=(3,3), stride=(2,2))

    def forward(self, x):
        
        conv1 = self.conv1(x)
        out = conv1.reshape(conv1.size(0), 32*32, 16)
        
        return out

tenet = Tenet()
rand = torch.randn(1,3,60,60)

out=tenet(rand)
</code></pre>
<p><code>conv1.shape</code> is <code> torch.Size([1, 16, 32, 32])</code><br>
and <code>out.shape</code> is <code>torch.Size([1, 1024, 16])</code></p>
<p>After I plotted the architecture on tensorboard, I got:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3d894647029e574f9fa8e858debe8c84143afa.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d3d894647029e574f9fa8e858debe8c84143afa" title="Screenshot from 2021-04-26 00-08-48"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d3d894647029e574f9fa8e858debe8c84143afa_2_690x494.png" alt="Screenshot from 2021-04-26 00-08-48" data-base62-sha1="diQfdKJOv7rWdHrZ1wHHxsTgfzk" width="690" height="494" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d3d894647029e574f9fa8e858debe8c84143afa_2_690x494.png, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3d894647029e574f9fa8e858debe8c84143afa.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3d894647029e574f9fa8e858debe8c84143afa.png 2x" data-dominant-color="F5F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-26 00-08-48</span><span class="informations">806×578 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t understand, what the nodes 28, 29, 30, 31, 34, and 35 are doing really.<br>
Help!</p>
Batched SVD_LOWRANK being much slower than loop implementation (both CPU and GPU)

EricAlcaide: <p>I’ve found the <code>torch.svd_lowrank</code> to be up to 2x slower in CPU and GPU for the batched implementation when compared to a loop implementation.</p>
<p>I guess if then the bacthed implemenation could be turned into a loop one so it is faster.</p>
<p>Simple test:</p>
<pre><code class="lang-python">import torch
p = torch.randn(7000, 3)
d = torch.cdist(p, p, p=2)
# optional
# d = d.to(torch.device("cuda:0"))
# loop implementation - faster
u,s,v = [], [], []
for i in range(5):
    u_, s_, v_ = torch.svd_lowrank(d)
    u.append(u_)
    s.append(s_)
    v.append(v_)
u = torch.stack(u, dim=0)
s = torch.stack(s, dim=0)
v = torch.stack(v, dim=0)
# batched implementation - 2x slower
u,s,v = torch.svd_lowrank(torch.stack([d]*5, dim=0))
</code></pre>
<p><strong>Note</strong>: i have found batched and loop implementations to be on par for low matrix sizes (n&lt;2000 for n x n matrices), but very different for big sizes (both in cpu and gpu)</p>
How to solve "RuntimeError: Address already in use" in pytorch distributed training?

keira_feng: <p>In the pytorch distributed training, I met a RuntimeError as following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "visual/distribution_train.py", line 387, in &lt;module&gt;
    main()
  File "visual/distribution_train.py", line 67, in main
    spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/public/home/fengm/.conda/envs/fm_pytorch_env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 167, in spawn
    while not spawn_context.join():
  File "/public/home/fengm/.conda/envs/fm_pytorch_env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 114, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/public/home/fengm/.conda/envs/fm_pytorch_env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/public/home/fengm/vehicle_reid/pytorch-pose-master/visual/distribution_train.py", line 74, in main_worker
    dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,world_size=args.world_size, rank=args.rank)
  File "/public/home/fengm/.conda/envs/fm_pytorch_env/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 354, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/public/home/fengm/.conda/envs/fm_pytorch_env/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 95, in _tcp_rendezvous_handler
    store = TCPStore(result.hostname, result.port, world_size, start_daemon)
RuntimeError: Address already in use
</code></pre>
<p>pytorch distributed initial setting is</p>
<pre><code class="lang-auto">torch.multiprocessing.spawn(main_worker, nprocs=8, args=(8, args))
torch.distributed.init_process_group(backend='nccl', init_method='tcp://110.2.1.101:8900',world_size=4, rank=0)
</code></pre>
<p>There are 10 nodes with gpu mounted under the master node. The master node doesn’t have GPU. I used the <strong>slurm system</strong> to submit my task and my task is randomly assigned to worker node. ‘110.2.1.101’ in init_method is the master IP.  I don’t kown whether is the init_method wrong?<br>
Is there anyone have met it before ? Who can help me to fix this bug?</p>

pietern: <p>What do you run in <code>main_worker</code> and where do the <code>world_size=4</code> and <code>rank=0</code> arguments to <code>init_process_group</code> come from? Are they hard coded, or do you list a single example?</p>
<p>The error itself means that multiple processes try to bind to the address and port, so I assume you are trying to run multiple processes with <code>rank=0</code>.</p>

HaFred: <p>Sorry to bring this up. I ran into the same issues, I wonder did you solve it?</p>

Yanli_Zhao: <p>you can set a free port</p>

M_Saiful_Bari: <p>Faced the same issue.</p>
<p>My Solution: It simply means that the GPU is already occupied under some other ddp training. Try deleting all the processes related to the running GPU and run the process again.</p>
Unable to exchange the weights of network

Mohsen_Saffari: <p>Hello,</p>
<p>I have defined my model. In forward pass of the neural network, i want to apply some changes on the outputs and the weights of layers. I could apply my operation on the weights but i can’t replace new weights with previous ones.<br>
To get the network’s weights i am using .state_dict().</p>
<p>Thank you!</p>

jerly-hjzhou: <p>This might be helpful. <a href="https://discuss.pytorch.org/t/how-to-assign-a-tensor-with-another-tensor/110644">https://discuss.pytorch.org/t/how-to-assign-a-tensor-with-another-tensor/110644</a></p>

Mohsen_Saffari: <p>Thanks for your reply Jerly!<br>
I exactly did the same as you did. But I don’t know what is the problem. i share part of my code here.</p>
<pre><code>    # Normalizing the weight tensor
    SD = MyResnet2.state_dict(self)
    W2 = SD['linear2.weight'].clone()
    norm_W2 = W2.norm(p=2,dim=1, keepdim=True)
    normalized_W2 = W2.div(norm_W2.expand_as(W2))
    
    # Replacing the Normalized weights with previous ones
    MyResnet2.state_dict(self)['linear2.weight'] = torch.nn.Parameter(normalized_W2)
    SD = MyResnet2.state_dict(self)
    W3 = SD['linear2.weight']
    Residue = normalized_W2 - W3 # This must be zero tensor
    print(Residue)
</code></pre>
<p>And Residue is non-zero!</p>

jerly-hjzhou: <pre><code class="lang-auto">    # Replacing the Normalized weights with previous ones
    SD = MyResnet2.state_dict(self)
    SD['linear2.weight'] = torch.nn.Parameter(normalized_W2)
    MyResnet2.load_state_dict(SD)
    W3 = SD['linear2.weight']
    Residue = normalized_W2 - W3 # This must be zero tensor
    print(Residue)
</code></pre>
<p>Change yours to this one maybe be work.</p>

Mohsen_Saffari: <p>Thanks for your reply!<br>
I figure out was was the problem. Before, I was applying my manipulation inside the neural network class. But now I am doing in the main loop of the network training.</p>
In RL: making sense of requires_grad=False

Zhihan_Yang: <p>I know there are topics with very similar title, but my question is about a special case.</p>
<p>I’m implementing an RL algorithm called soft actor-critic. There’s a step in the algorithm in which I need to update the weights of a network in order to minimize the loss computed from the output of a subsequent network.</p>
<p>Setting requires_grad=False for the subsequent network worked. But I wonder whether this is truly saving computational effort (the time difference is small so I can’t tell). In other words, I think PyTorch’s autograd engine still needs to compute the gradients for the subsequent network IN ORDER TO compute the gradients for the previous network.</p>
<p>Is this understanding correct?</p>

JuanFMontesinos: <p>Setting requires grad False just makes the engine not to update those weights but you still need to compute the gradients to backpropagate to the first one.</p>
Efficient way of accessing N multiples of k indexes and performing an operation over it

Pablo_Carneiro_Elias: <p>Hi all,</p>
<p>Im trying to implement a filter that runs over a spectrogram frequencies for each time frame and filters out inharmonic frequencies.</p>
<p>What I need is, given a time frame, for each frequency k in this timeframe, to access frequencies multiple of k (the harmonics), then perform some operation over them. So for frequency k, k=1,…,F, I need to access indexes, k, 2<em>k, 3</em>k,…, N*k (N in total), perform some operation and set the result to the position k (original frequency bin). Same calculation for all timeframes.<br>
In resume, it would be something like this:<br>
spec[k, :] = operation(spec[mutiple_indexes,:])</p>
<p>Basically its like running a operation sliding window starting in each frequency, but for each frequency this sliding window is more space (spaced by the frequency value).</p>
<p>I can do this iteratively, but Im wondering if there is a vectorized way of doing this.<br>
I was thinking about torch.unfold, but it can only work on sequential patches, not of increasing stride.</p>
<p>Any suggestions?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>I’m gonna assume you want to run 1D convolutions just in those specific bins rather than 2D convolutions around the harmonics.</p>
<p>If that were the case you can preset an slice operator which retrieves those harmonics, then you can run a 2D convolution with kernel size 1 for the frequency. With a proper padding you can keep the shape and then using the preset slice to “paste” back the values.</p>

JuanFMontesinos: <p>Kind of like this:</p>
<pre><code class="lang-auto">import torch

N = 1
T = 5
pad = (0,T // 2)
convolution = torch.nn.Conv2d(in_channels=1, out_channels=N, kernel_size=(1, T), padding=pad)

sp = torch.rand(256, 128)

preset_slice = [2*i for i in range(128)]  # Choose your own

harmonics = sp[preset_slice]

processed_harmonics = convolution(harmonics[None,None,...])

new_sp = sp.clone()
new_sp[preset_slice] = processed_harmonics[0,0]
</code></pre>
<p>In case of not using convolutions you can handle something similar with fold and unfold I woudl say</p>
Implication of multithreading with a model without mutexes in libtorch

Adesola_Adedewe: <p>So I am currently implementing distributed algorithms in libtorch. I purposely refused to do any form of synchronization to see if lib-torch does any. i don’t seem to have any troubles with multiple threads using the same pointer to a model for forward pass. how is this working under the hood. I would need some more clarification. thanks. is being copied, is there a better way to  optimize this?</p>
Ubuntu 20.04: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

mdeberts: <p>Hello,</p>
<p>everytime I try to transfer a tensor from my cpu to my gpu this error occurs:</p>
<p>RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</p>
<p>I installed Pytorch via the suggested command line on the website, and run a Nvidia 1060 6 GB.<br>
Can anyone tell me why it is not working?</p>
<p>nvidia-smi</p>
<p>Fri Apr 23 15:40:38 2021<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce GTX 106…  Off  | 00000000:26:00.0  On |                  N/A |<br>
| 27%   56C    P0    30W / 120W |    401MiB /  6075MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A       948      G   /usr/lib/xorg/Xorg                259MiB |<br>
|    0   N/A  N/A      1539      G   /usr/bin/gnome-shell               66MiB |<br>
|    0   N/A  N/A      2699      G   …gAAAAAAAAA --shared-files        8MiB |<br>
|    0   N/A  N/A      2817      G   …AAAAAAAA== --shared-files       60MiB |<br>
±----------------------------------------------------------------------------+</p>

ptrblck: <p>This error usually points to a broken driver/CUDA installation.<br>
Are you able to compile and run other CUDA code such as the CUDA samples?</p>

mdeberts: <p>I get this error by a simple command like</p>
<p>torch.cuda.FloatTensor(1000, 1000).fill_(0)</p>
<p>But</p>
<p>torch.cuda.current_device()<br>
torch.cuda.device(0)<br>
torch.cuda.device_count()<br>
torch.cuda.get_device_name(0)<br>
torch.cuda.is_available()</p>
<p>works.</p>

mdeberts: <p>Ok I solved the problem by:</p>
<p>a) using NVIDIA driver metapackage from nvidia-driver-460 (proprietary, tested)<br>
at Software &amp; Updates → Additional Drivers<br>
b) changing desktop from lightdm to gdm3 by<br>
sudo dpkg-reconfigure gdm3<br>
and rebooting</p>
<p>!DISCLAIMER!<br>
I had issues that I couldn’t pass the login screen using gdm3 before, command b) might get you into this kind of trouble. Anyhow for me it works with the listed driver.</p>
<p>Thanks for the feedback.</p>
Data Augmentation Transforms for Sequences on minority class

Jam: <p>I have a sequence of images, so with batch size 1, my <strong>getitem</strong> method output would be of size [1,9,3,256,256] with sequence length 9.<br>
The method currently looks like this:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):

        seq = self.data[idx]
        ims = np.empty((self.seq_length, 3, 256, 256), dtype=np.uint8)
        labels = np.empty((self.seq_length), dtype=int)

        for i, case in enumerate(seq.cases):
            ims[i] = imread(case.path)
            labels[i] = case.cls
                        
        # transform to tensors
        ims = torch.from_numpy(ims)
        labels = torch.from_numpy(labels)

        if self.transform: 
             ims = self.transform(ims)
            
        return ims, labels
</code></pre>
<p>Since my data is highly imbalanced, I’d like to perform transformations only on the minority class (labels==1). I know that I have to check for labels[i]==1 but the problem is that in one sequence, the labels can be 0 and 1. I am not 100% sure how torchvision transformations work. Will they change the length of my sequence or can I do it like this:</p>
<pre><code class="lang-auto">for i in range(len(ims)):
            if (labels[i] == 1) and self.transform: # check for minority class
                ims[i] = self.transform(ims[i])
</code></pre>

ptrblck: <p>Your code looks alright and the sequence length should not be changed, since you are reassigning the transformed tensors to <code>ims</code>. Do you see any issues using this approach?</p>

Jam: <p>Thanks for your reply! Before using data augmentation, my model was overfitting after a few epochs but with data augmentation it is now overfitting after the first epoch already. Do you think this is just caused by the augmentation methods I am using or can it be a result of the way I am applying the augmentation?</p>

ptrblck: <p>Your data augmentation might be too aggressive and might not represent the underlying data domain properly, so that the model learns the training set but gets worse in classifying the validation samples.</p>

Jam: <p>Thanks! Is there anything else I can do to prevent overfitting? I implemented the WeightedRandomSampler as well and now the loss is strictly decreasing for the first 10 epochs but then it always starts to overfit (also without using data augmentation). I am using a pretrained ResNet18 and also tried to freeze all except the last layer.</p>
Basic pytorch geometric architecture for graph signal classification

Laolga: <p>Hi! I’m new to PyTorch geometric, but my understanding is that all available examples are usually around node/graph classification while I’d like to do a signal classification.</p>
<p>So, the feature matrix X will have (n,m) dimensions, y will be (1,n) and edges then (2,m).</p>
<p>An example could be a feature matrix where for every author we have information about being involved in a certain paper (then <strong>n</strong> authors and <strong>m</strong> papers). Then we know which paper cited which other paper (adjacency matrix <strong>m*m</strong> or edge list 2 by m) and we want to predict the department for each author. This way we have a graph of features that is exactly the same for every author.</p>
<p>So, I could reformulate this as a graph classification problem and just represent each author as a graph, but this does not look very efficient, especially if there are a lot of papers.</p>
<p>Could someone point me towards a code example with a similar approach? Or tell me what can be modified in usual examples to make it work for my problem?</p>

ptrblck: <p>Your use case might be related to e.g. genre prediction for movies or maybe even the revenue prediction given the movies and genre. I’m not sure how closely related these topics are, but know that the “genre prediction” use cases have been used in the past, so you might be able to find some related work in this field.</p>
<p>That being said, your geometric approach could also certainly work.</p>

Yen-Hua_Huang: <p>An example implemented with Spektral (for TF, unfortunately) could do graph signal classification. You may find the code here <a href="https://github.com/danielegrattarola/spektral/tree/master/examples/other" rel="noopener nofollow ugc">graph_signal_classification_mnist.py</a>. One thing is that you may need to use self.flatten = Flatten() instead of using GlobalSumPool() of Spektral.</p>
<p>Even though I am keen to re-write the code by using PyTorch Geometric, I cannot find a dataloader similar to the one (i.e. <a href="https://graphneural.network/loaders/#mixedloader" rel="noopener nofollow ugc">mixedloader</a>) used by Spektral in the case with only one graph and multiple graph signal instances.</p>
Calculation of Angles between features

Angelina_Robert: <p>Hi,<br>
Can any one help to provide Pytorch implementation of eq. 3,4,5 given in the following pic. Here, Ci is the center of class and I believe it could be calculated by taking the mean of class. Could anyone also shed light on the statement in the paper  i.e. the dimension of ci (center of class i) and fik (kth sample of feature of class i)  are same, its a bit confusing to me.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b81896b11a1f216f1ca425e0b4bedf81d7d09c02.png" data-download-href="https://discuss.pytorch.org/uploads/default/b81896b11a1f216f1ca425e0b4bedf81d7d09c02" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b81896b11a1f216f1ca425e0b4bedf81d7d09c02_2_332x500.png" alt="image" data-base62-sha1="qgAxco0OJI5csOJHOeaIcJGMGIO" width="332" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b81896b11a1f216f1ca425e0b4bedf81d7d09c02_2_332x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b81896b11a1f216f1ca425e0b4bedf81d7d09c02_2_498x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b81896b11a1f216f1ca425e0b4bedf81d7d09c02.png 2x" data-dominant-color="E7E7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">518×778 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
App plugin id 'maven' not found

myuanqing: <p>Hi!</p>
<p>I am trying to build pytorch-android from source. However, I came across this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52908d51496dafa9f595d6a183478c382f1ae202.png" data-download-href="https://discuss.pytorch.org/uploads/default/52908d51496dafa9f595d6a183478c382f1ae202" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52908d51496dafa9f595d6a183478c382f1ae202_2_690x142.png" alt="error" data-base62-sha1="bMoQWduFfx9lg1nl9TzuC2gF6F4" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52908d51496dafa9f595d6a183478c382f1ae202_2_690x142.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52908d51496dafa9f595d6a183478c382f1ae202_2_1035x213.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52908d51496dafa9f595d6a183478c382f1ae202.png 2x" data-dominant-color="262525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1261×261 53.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
<p>I have found that shell execution is not friendly to my Windows OS, so I almost copyed the cmake command from build_android.sh to my cmd and finished the ‘cmake’ tasks. I am wondering if this step makes the build fail?</p>
<p>I have searched on the Internet that this kind of error may mainly because of not including the corresponding ‘classpath’ in the project level gradle. But I didn’t make any change to ‘build.gradle’.</p>
<p>Could anyone help me with this error please?</p>
RuntimeError: derivative for bincount is not implemented

kiranchari: <p>Hello torch gurus,</p>
<p>I need a backpropable implementation of bincount and looks like torch.bincount is not.</p>
<p>I am not sure how torch.bincount is implemented, is there any efficient alternative implementation of bincount (or work around) that I can backbrop through? My own vanilla implementation of bincount in torch is nowhere near as fast as torch.bincount for large arrays (~50k length).</p>
<p>Thanks</p>
Code stuck in multiprocessing

Aforwardz_Yu: <p>I have a model and serve it with Flask and it works fine. For accelerating the speed in batch input, I use celery with multi workers and load the model in each worker, but it stucked when one process got into “torch.layer_norm”, a C variable function in the torch package, and it also blocked other process</p>
<p>I wonder why this happen in multiprocessing and how to solve this problem</p>
<p>The “torch.layer_norm” is in “torch._C._VariableFunctions.pyi”</p>
How do I one-hot encode my outputs in a simple neural network?

srdg: <p>I have a dataset of 21 input features and 1 class label ranging from 0-3.<br>
I’m quite new to PyTorch and I simply don’t see how I can have multiple fully-connected layers with the OH-Encoded target list at the end. Here is what I have so far, after a day of experimentation -</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(21, 42),
            nn.ReLU(),
            nn.Linear(42, 21),
            nn.ReLU(),
            nn.Linear(21, 10),
            nn.ReLU(),            
            nn.Linear(10, 1),
            nn.ReLU()
        )

    def forward(self, x):
        logits = self.linear_relu_stack(x)
        return logits
</code></pre>
<p>If nothing else, I was planning on normalizing the target column as well, but of course that is a very crude idea. I would like to know how I can use one-hot encoded labels as my target.<br>
I’m using the <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" rel="noopener nofollow ugc">Quickstart tutorial</a> as a reference.</p>

KFrank: <p>Hi Soumik!</p>
<aside class="quote no-group quote-modified" data-username="srdg" data-post="1" data-topic="119283" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srdg/48/37517_2.png" class="avatar"> srdg:</div>
<blockquote>
<p>I have a dataset of 21 input features and 1 class label ranging from 0-3.<br>
I simply don’t see how I can have multiple fully-connected layers with the OH-Encoded target list at the end.<br>
…<br>
I would like to know how I can use one-hot encoded labels as my target.</p>
</blockquote>
</aside>
<p>It sounds like you wish to build a four-class classifier.</p>
<p>The most common approach would be to have the final layer of<br>
your model be a <code>Linear</code> layer with four output features (and no<br>
“activation” layer after it, so no final <code>ReLU</code>).</p>
<p>(As it stands, your final <code>Linear</code> layer has only one output feature<br>
so your model is not making predictions for four classes.)</p>
<p>You will likely use <code>CrossEntropyLoss</code> as your loss criterion.</p>
<p>The output of your model will have shape <code>[nBatch, 4]</code> and will be<br>
the <code>input</code> to <code>CrossEntropyLoss</code>.  The <code>target</code> passed to<br>
<code>CrossEntropyLoss</code> will have shape <code>[nBatch]</code> (<em>no</em> class dimension)<br>
and is <em>not</em> one-hot encoded.  Rather, <code>target</code> consists of integer<br>
class labels whose values are in <code>{0, 1, 2,  3}</code>.</p>
<p>So neither the output of your model nor your <code>target</code> will be one-hot<br>
encoded.</p>
<p>Best.</p>
<p>K. Frank</p>
Is Pytorch has similar `Status` class for indicating return result?

Stonepia: <p>I found that Tensorflow has the <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/status.h#L41" rel="noopener nofollow ugc"><code>Status</code></a> class , which is very useful for debugging and the return value is human-readable.</p>
<p>Do Pytorch has the similar class? For example, I could <code>return Status::OK()</code> instead of <code>return 0</code> for indicating a successful executing.</p>
Object Centric vs Scene Centric Dataset

Sahil_malik: <p>Hi Memebers,</p>
<p>Could you please help me to understand the difference between Object Centric vs Scene Centric dataset? Please correct me if I am wrong at some point listed in the following:</p>
<p>MNIST and Fashion-MNIST datasets are Object-Centric datasets because only 1 object that try to classify correctly whereas the Scene datasets are" Places397, MIT-Indoor67, etc. because there are multiple object in one image? Please correct me if i am wrong…</p>
<p>Is ImageNet dataset scene-centric or Object-Centric dataset? And what about Event8 scene dataset?</p>
<p>Thank you!</p>
Inplace error when gradient is required

kavka: <p>Following the official document of residual network, I write the following network class, however, when I run the code, I got the error</p>
<p>"one of the variables needed for gradient computation has been modified by an inplace operation "</p>
<p>‘’’’</p>
<pre><code>class ResBlock(nn.Module):
    def __init__(self, num_node, num_fc, activate = nn.ReLU(inplace=True)):
    super(ResBlock, self).__init__()
    self.act = activate
    self.linears_list = [nn.Linear(num_node, num_node) for i in range(num_fc)]
    self.acti_list = [self.act for i in range(num_fc)]
    self.block = nn.Sequential(*[item for pair in zip(self.linears_list, self.acti_list) for item in pair])
    
    'Xavier Normal Initialization'        
    for m in self.block:
        if isinstance(m, nn.Linear):
            nn.init.xavier_normal_(m.weight.data,  gain=1.0)

def forward(self, x):
    residual = x
    out = self.block(x)
    out += residual #### this part gives the error
    out = self.act(out)
    return out
</code></pre>
<p>‘’’’</p>
<p>However, if I change out += residual into out = out + residual, then everything is perfect.</p>
<p>But the “<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">vision/resnet.py at master · pytorch/vision · GitHub</a>” do the same thing.</p>
<p>I dont understand why its not working in my case.</p>

tom: <p>This depends on whether the thing you modify inplace is needed for the gradient (of the function that returned it - as here - or when you used it as input before modifying it). You have linear last which needs its output to do backward while resnet has batch norm last.</p>
<p>Best regards</p>
<p>Thomas</p>
Construct Tensor with Gradient from Tensors with Gradient

LivelyLiz: <p>I am trying to construct a tensor from other tensors (which requires a gradient). The new tensor should also be differentiable</p>
<p>So what I am trying to do is essentially:</p>
<pre><code class="lang-auto">import torch

x = torch.Tensor(1., requires_grad=True)
y = torch.Tensor(2., requires_grad=True)
z = torch.Tensor(3., requires_grad=True)
a = torch.Tensor([x*y, 1/(x*y), x/z])

print(a)  # output: tensor([1., 1.])
</code></pre>
<p>So for x, y, z I can get a gradient separately but not for “a”  although “a” is constructed of x, y, z.<br>
I know about <code>index_put</code> which seems to preserve the differentiability (at least in this example it shows a grad_fn):</p>
<pre><code class="lang-auto">target = torch.zeros([5,3])
indices = torch.LongTensor([[0,1], [1, 2], [2, 2], [3, 0], [4, 1]])
value = torch.ones(indices.shape[0], requires_grad=True)
target.index_put_(tuple(indices.t()), value)

print(target)
'''
output:
tensor([[0., 1., 0.],
        [0., 0., 1.],
        [0., 0., 1.],
        [1., 0., 0.],
        [0., 1., 0.]], grad_fn=&lt;IndexPutBackward&gt;)
'''
</code></pre>
<p>However, for my use case and not just a toy example like above it would be tedious to construct the tensor with <code>index_put</code> and matrix operations.</p>
<p>Is there a way to do this similar to my first example? Is there a function I just don’t know about? Or do I have to do it the long way?</p>

ptrblck: <p>Recreating a tensor detaches it from the computation graph, so you could use <code>torch.stack</code> and/or <code>torch.cat</code> to create <code>a</code>.</p>

LivelyLiz: <p>Thanks for your reply. This seems simpler than what I am doing right now.</p>
<p>Was experimenting a little and resorted to a solution where I would create a zero tensor and set the entries via indexing and that also seemed to work.</p>
<pre><code class="lang-auto">a = torch.zeros(1,3)
a[0,0] = x*y
a[0,1] = 1/(x*y)
a[0,2] = x/y
</code></pre>
How to Organize Binary Attribute Matrix into Diagonal Form

Qiyao_Wei: <p>Hi all,</p>
<p>I have a large (about 400 rows by columns) binary attribute matrix, where the elements are one if there is a connection between the corresponding row and column element, and zero if not. Currently I am having trouble visualizing the matrix. Displaying the matrix in a heatmap fashion or directly printing the matrix will both be unreadable due to the size of the matrix.</p>
<p>As a result, my idea is to reorganize the matrix (reorder the rows and columns) so that most of the ones are along the diagonal, allowing me to divide up the large matrix into block components (along the diagonal) and visualize separately. Therefore, I am curious whether such implementation has already been attempted before? Even better, I would really appreciate if someone could offer alternative visualization options for plotting a large matrix in heatmap form. Thanks a lot!</p>
Parameter vs tensor.requires_grad = True

raouf_ks: <p>Hello everyone ,<br>
I have a question regarding the autograd system  , why we have the following difference between these  two pictures  :</p>
<p>this one :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f63360aa504a7b4b177b3f30b240bcf096bf8b7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f63360aa504a7b4b177b3f30b240bcf096bf8b7c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f63360aa504a7b4b177b3f30b240bcf096bf8b7c.png" alt="image" data-base62-sha1="z7Zx5En2c7HJeLAPhnzDxWEdoew" width="690" height="325" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">841×397 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>VS</p>
<p>this one :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6d63c6187ee7a8b4827af19eb0a907172525a50.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6d63c6187ee7a8b4827af19eb0a907172525a50" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6d63c6187ee7a8b4827af19eb0a907172525a50.png" alt="image" data-base62-sha1="zdCsgFLSioMTE1FYEj6QqtCSdva" width="690" height="337" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">704×344 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JuanFMontesinos: <p>a nn.Parameters is a container aimed to create learnable tensors inside a nn.Module<br>
That’s why it keeps tracking the grad, as any other parameter in a nn.Module would do. Besides,<br>
You are instantiating the Parameter class with rand<em>0.3 as argument, meanwhile in the other case the variable which has  “requires_grad” is later submitted to another operation, thus, the resulting variable is not a leaf variable. Namely, w=3</em>rand(1) is not pointing to the same object that w=rand(1). In the former case you are pointing to the result of 3*rand(1) which is a different tensor than rand(1)</p>
<pre><code class="lang-auto">import torch
w=torch.rand(5).requires_grad_()
q=3*w
w.is_leaf
Out[8]: True
q.is_leaf
Out[9]: False
</code></pre>

raouf_ks: <p>thanks a lot  you are right  !</p>

dbsx: <p>Does it mean that we have to use nn.paramter (but not just tensor with requires_grad_) when building a custom network module layer?</p>

JuanFMontesinos: <p>Yes.<br>
That’s the difference between a buffer (a tensor that is static but not intended to change by backprop) and a parameter.</p>
<p>The latter is returned when you invoke model.parameters() too.</p>
How can I put the values of two tensor on the same scale?

S_M: <p>Hi, I have two tensors the range value of the first tensor is between 5 to 18 and the other tensor is 0.02 to 0.07. How can I change the scale value of the first tensor like the second tensor?<br>
thanks</p>

ptrblck: <p>You could normalize the data first and then shift and scale it to the desired range:</p>
<pre><code class="lang-python">x = torch.empty(1000).uniform_(5, 12)

y = (x - 5) / (12 - 5) # normalize to [0, 1]
print(y.min(), y.max())
&gt; tensor(0.0026) tensor(1.0000)

y = (y  * (0.07 - 0.02)) + 0.02 # scale to [0.02, 0.07]
print(y.min(), y.max())
&gt; tensor(0.0201) tensor(0.0700)
</code></pre>

S_M: <p>Hi ptrblck, thanks for your reply. Actually, I am using a knowledge distillation code where the teacher is Alexnet and the student is a simple network than Alexnet. when I print the last encoding vector of both networks I see a different range of values like 5 to 18 for the teacher and 0.02 to 0.07 for the student network. It makes a problem in my analysis. I confused!</p>
Detecting residual connections

vaibhavballoli: <p>Hi! I apologize in advance if this question is repeated, couldn’t find any similar queries.</p>
<p>Is it possible to detect skip connections in any model ? For example if a block in the model contains the below function as their forward pass.</p>
<pre><code class="lang-auto">def forward(self, x):
    out = self.layers(x)
    return x + out
</code></pre>
<p>Can I somehow detect that a particular set of block have this kind of forward pass with residual connection for any given model? I’m looking for any type of solution - using hooks, or graph or anything else ?</p>
<p>TIA</p>

albanD: <p>Hi,</p>
<p>In general I can’t think of a way to do this.<br>
Can you give a bit more context on why you want to do that?</p>

vaibhavballoli: <p>I was looking for a solution for pruning. If I prune layer_1 whose output say has a connection with layer_3, I wanted to identify this connection and prune layer_3 accordingly. Pruning here is the context of channel/filter pruning.</p>

braindotai: <p>Kind of a rought way to detect residual connection could be -</p>
<pre><code class="lang-python">def forward(self, x):
    self.out = self.layers(x)
    return x + self.out

outputs = model(features)
if (outputs - model.out).abs().mean() == model.layers(features).abs().mean():
   print('there is a residual connection')
</code></pre>
<p>Again not a reccomended way but should work in this particular example.</p>

ooodragon: <p>whilst looking at output.grad_fn.next_functions on autograd<br>
I do see some “Addbackward” gradient functions which is presumably residual connections (and even though it is not, I do think we should keep the channel dimensions for these tensors since we are adding. plz point me out if it is wrong)<br>
however, it seems this algorithm is quite daunting.<br>
My colleague who is highly skilled had quite a trouble making an algorithm to find these residual connections and it only works for resnet. (tested on mobilenet, efficientnet (timm), and so on)</p>
<p>I would love to see if there are any ideas or code implemented for this.<br>
Seems like knowing where the residual connection exists in the first place is the best attempt for this</p>
Dataloader Not working with larger dataset

willc: <p>I have tested my code and it trains fine when using a smaller subset of my data (40 items). But when I move to the full dataset it just gets stuck loading the data. I am using a batch size of 25 and the larger dataset has ~300 items. Each item in the dataset consists of 7 224x224 images.</p>

ptrblck: <p>Could you check, if you get any error message by iterating the <code>DataLoader</code> once using <code>num_workers=0</code>? Sometimes a worker process might die unexpectedly due to a loading issue and depending on your setup you might not see a valid error (saw this behavior in e.g. Jupyter notebooks)?</p>

willc: <p>With num_workers=0 I occasionally get an error after a long time which lead me to discover that some of the images were 2048x4096 (original images actually were 256x512, they were just cropped to 224x224). I created a new dataset with the downsampled images to 256x512 and that seemed to fix the issue. Thank you for the help</p>