akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Help me build a Pool CNN

Techloverguy: <p>I am trying to build a PoolCNN in Pytorch with the following architecture:</p>
<p>Input</p>
<p>1D convolution with 32 output channels and a kernel size of 3.</p>
<p>Relu.</p>
<p>1D adaptive max pool with 1 output size.</p>
<p>Fully connected layer with 25 output features</p>
<p>Output</p>
<p>Can somebody help me build that?</p>

Andrei_Cristea: <p>Your description is precise enough that you could probably have built this yourself. You didn’t specify a particular input size (number of channels and size of sequence) which is required to know how many input channels the first convolution needs, and how many input features the Linear layer ought to have. Here I’m assuming 1 input channel and then using LazyLinear rather than Linear to deal with the varying sequence length.</p>
<pre><code class="lang-auto">import torch
from torch import nn

model = nn.Sequential(
    nn.Conv1d(in_channels=1, out_channels=32, kernel_size=3),
    nn.ReLU(),
    nn.AdaptiveMaxPool1d(output_size=1),
    nn.LazyLinear(out_features=25),
)
</code></pre>
<p>Usage:</p>
<pre><code class="lang-auto">model(torch.randn(16, 1, 100))  # batch size of 16, 1 channel, size of 100
Output:
tensor([[[ 1.5257, -1.3829,  0.6808,  ...,  1.7156,  0.8902,  0.2436],
         [ 1.0326, -0.9974,  0.3288,  ...,  0.7250,  0.6279,  0.6433],
         [ 1.0429, -1.0055,  0.3362,  ...,  0.7457,  0.6334,  0.6349],
         ...,
         [ 1.2651, -1.1792,  0.4948,  ...,  1.1921,  0.7516,  0.4548],
         [ 1.1988, -1.1273,  0.4475,  ...,  1.0589,  0.7163,  0.5086],
         [ 1.3335, -1.2326,  0.5436,  ...,  1.3294,  0.7879,  0.3994]],
...
</code></pre>
Role of cumsum() in sine sweep generation in torchaudio

moabaom: <p>As part of audio processing tutorial of torchaudio:<br>
<a href="https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html</a></p>
<p>there is a method written for generating a sine sweep:</p>
<blockquote>
<p>def get_sine_sweep(sample_rate, offset=DEFAULT_OFFSET):<br>
max_sweep_rate = sample_rate<br>
freq = _get_log_freq(sample_rate, max_sweep_rate, offset)<br>
delta = 2 * math.pi * freq / sample_rate<br>
cummulative = torch.cumsum(delta, dim=0)<br>
signal = torch.sin(cummulative).unsqueeze(dim=0)<br>
return signal</p>
</blockquote>
<p>can someone explain whats the logic of making a cummulative sum of all frequencies before passing them to the sine function?</p>

moabaom: <p>Just found the answer:</p>
<p>Instantaneous frequency is the derivative of the phase (sine function’s argument). hence to go from instantaneous frequency to phase we integrate, ie. cumsum</p>
Difference in to(torch.float) and float() for a model and a tensor

paulus: <p>Background: When constructing a model in the development, I prefer to use high numerical precision, i.e., double, e.g., when initializing constant tensor attributes from numpy variables. For the actual use, the model can then be converted to a lower precision by <code>.float()</code>. Now I’m constructing a model that has a pre-computed complex-valued tensor attribute. Earlier I’ve used the split (real/imag) complex representation, but starting a new project with a new torch version I decided to give the complex dtypes a test.</p>
<p>This triggers the need for clarification:<br>
When calling <code>model.float()</code>, only the floating point attributes are converted to the given precision, but the complex-valued ones are not touched. Calling <code>model.to(torch.float)</code> <em>all</em> attribute buffers are converted to <code>float</code>, even the complex-valued. What I was expecting was that the <code>complex128</code> buffers were converted to <code>complex64</code>.</p>
<p>Reading the documentation of <code>torch.nn.Module</code>, this is what is claimed to happen. Just a bit confusing, when assuming <code>float()</code> and <code>to(torch.float)</code> to be the same, similar to what happens with <code>torch.Tensor</code>. Assuming that this difference in the behaviour is intended, what would be the best way to handle the situation?</p>
<ol>
<li>Construct the model using the correct precision from the start? This is probably the best option with the only drawback of minimal loss of development flexibility.</li>
<li>Overloading <code>.to()</code> in the model to handle the precision of the complex-valued buffers correctly?</li>
<li>Forget the complex data type and continue using the split representation? Which in turn makes using functions like FFT more difficult.</li>
</ol>
AttributeError: 'QFunctional' object has no attribute 'activation_post_process'

thekoshkina: <p>I’m trying to quantise a simple net wit QFunctional for adding 2 quantised tensors</p>
<pre><code class="lang-auto">class MinimalModel(Module):
  
    def __init__(self):
        super().__init__()
        self.quant_x = torch.quantization.QuantStub()
        self.quant_y = torch.quantization.QuantStub()
        self.x_y_add = torch.nn.quantized.QFunctional()

    def forward(self, x, y):
        y = self.quant_y(y)
        x = self.quant_x(x)
        result = self.x_y_add.add(y, x) # xi + mean
        return result

model = MinimalModel().cpu()
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
model = torch.quantization.prepare_qat(model)
model = torch.quantization.convert(model) 
x =torch.rand((1,48,2,2))
y =torch.rand( (1,48,2,2))
output = model(x, y)

</code></pre>
<p>I get an error:</p>
<blockquote>
<p>File “/home/guests/vira/repos/cross-platform-support/minimal_example_add.py”, line 102, in forward<br>
result = self.x_y_add.add(y, x) # xi + mean<br>
File “/usr/local/lib/python3.9/site-packages/torch/nn/quantized/modules/functional_modules.py”, line 187, in add<br>
r = self.activation_post_process(r)<br>
File “/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1181, in <strong>getattr</strong><br>
raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(<br>
AttributeError: ‘QFunctional’ object has no attribute ‘activation_post_process’</p>
</blockquote>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/thekoshkina">@thekoshkina</a> , if you replace <code>torch.nn.quantized.QFunctional</code> with <code>torch.nn.quantized.FloatFunctional</code>, it should work.</p>
<p><code>FloatFunctional</code> is what you want in your floating point model.  The framework will swap <code>FloatFunctional</code> to <code>QFunctional</code> during the <code>convert</code> step.</p>
<p>Side note, feel free to try out this workflow (<a href="https://pytorch.org/docs/master/quantization.html#prototype-fx-graph-mode-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a>) which can insert the quants/dequants for you automatically, and can handle functions and methods.</p>

thekoshkina: <p>Thank you! It worked</p>
How to shuffle a Tensor using Pytorch C++ API

mendozaflash66: <p>Hello,</p>
<p>Is there a way to shuffle the data in a tensor when using Pytorch C++ API without resorting to a loop to access each element randomly?</p>
<p>Thanks!</p>

tmaric: <p>This is what I am using, directly from my code, just disregard mesh, cc, and vf and use your own input data</p>
<pre><code class="lang-auto">// - Randomly shuffle cell center indices. 
    torch::Tensor shuffled_indices = torch::randperm(
        mesh.nCells(),
        torch::TensorOptions().dtype(at::kLong)
    ); 
    // - Randomly select 10 % of all cell centers for training. 
    long int n_cells = int(0.1 * mesh.nCells());
    torch::Tensor training_indices = shuffled_indices.index({Slice(0, n_cells)}); 
    
    // - Use 10% of random indices to select the training_data from vf_tensor 
    torch::Tensor vf_training = vf_tensor.index(training_indices);
    torch::Tensor cc_training = cc_tensor.index(training_indices);
</code></pre>
How to solve the runtime error - "RuntimeError: expected scalar type Long but found Float?"

Jeet: <p>After trying out several workaround to this problem , I am still stuck with this error message. I have a very basic Neural network called “Neuralnetwork”.</p>
<pre><code class="lang-auto">class Neuralnetwork(nn.Module):

    def __init__(self, in_features):

        super().__init__()

        self.disc = nn.Sequential(

            nn.Linear(in_features, 4), #change it to 4 neurons

            nn.LeakyReLU(0.01),

            nn.Linear(4, 1),

            nn.Sigmoid(),

        )

    def forward(self, x):

        return self.disc(x)
</code></pre>
<p>I have few samples to feed into the neural network. The samples are defined as follow -</p>
<pre><code class="lang-auto">samples = [torch.tensor([1,0,0,1],dtype=torch.long),

          torch.tensor([0.9,0.1,0.2,0.8],dtype=torch.long),

          torch.tensor([0.9,0.2,0.1,0.8],dtype=torch.long),

          torch.tensor([0.8,0.1,0.2,0.9],dtype=torch.long),

          torch.tensor([0.8,0.2,0.1,0.9],dtype=torch.long)]
</code></pre>
<p>So when I train the neural network with each of the sample fromt he samples, I get the error -</p>
<pre><code class="lang-auto">image_dim=4
disc=Neuralnetwork(image_dim)
criterion=nn.BCELoss()

for epoch in range(epochs):
     for sample in samples:
           disc_train=disc(sample)  ##This line is generating the error "RuntimeError: expected scalar type Long but found Float"
</code></pre>
<p>In order to look for a solution I have also taken the following measure but no luck -</p>
<pre><code class="lang-auto">sample= sample.type(torch.LongTensor)
disc_train=disc(sample)
</code></pre>

ptrblck: <p>Your model expects a floating point input (a few layers are exceptions such as e.g. <code>nn.Embedding</code> which expects integer types) so transform your input to a <code>FloatTensor</code> via:</p>
<pre><code class="lang-python">out = disc(sample.to(torch.float32))
</code></pre>
Torchtext min_freq: unexpected argument

Vishal_Ahuja: <p>My <code>torch</code> version is  <strong>1.10.2</strong>, and <code>torchtext</code> version is <strong>0.11.2</strong>. The error I am getting is from this statement:<br>
<code>vocab = torchtext.vocab.Vocab(counter, min_freq=1)</code></p>
<p><strong>Error: TypeError: <strong>init</strong>() got an unexpected keyword argument ‘min_freq’</strong>. <code>torchtext</code> is the latest version, so what could the problem be?</p>

jasnei: <p>you should use</p>
<blockquote>
<p>vocab = torchtext.vocab.vocab(counter, min_freq=1)</p>
</blockquote>
<p>please make sure not the Vocab.</p>
Do you have any information on a model that outputs labels and other information for object detection?

tanabe.d.zc32s: <p>Can you please let me know if you know of any examples of object detection models such as Faster r cnn that output object attributes (color, etc.) in conjunction with object labels and box location information?</p>

Michal_Bogacz: <p>Hello,<br>
I know the answer will sound generic, but if you have any kind of training dataset, so data is annotated with labels of your need. You might try to use any of RCNN models and just add a new head, which will be trained with the rest of the model frozen. Just use the backbone as the source for the features and build you topology on top of it.<br>
If you provide any additional info, based on the purpose of your model, and if you have the training datasets, I would be happy to follow up more.</p>
<p>Edit:<br>
I was looking for exact page in documentation:<br>
<a href="https://pytorch.org/vision/main/feature_extraction.html" rel="noopener nofollow ugc">Feature extraction for model inspection — Torchvision main documentation (pytorch.org)</a></p>

tanabe.d.zc32s: <p>Thank you for the reply.</p>
<p>Do you mean “new head” is a unit in a network like RPN Head or RoI Head？</p>
<p>The model I am tring to build is a model that detects plants from image and outputs the plant name and leaf age (degree of plant growth). The dataset is in PASCAL VOC format, with each plant box, label and leaf age (numerical value) recorded.</p>
<p>Best regards.</p>

Michal_Bogacz: <aside class="quote no-group" data-username="tanabe.d.zc32s" data-post="3" data-topic="151360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tanabe.d.zc32s/48/46394_2.png" class="avatar"> tanabe.d.zc32s:</div>
<blockquote>
<p>RPN Head or RoI Head</p>
</blockquote>
</aside>
<p>Exactly like this. Nothing stops you from creating any kind of new head :).<br>
Base the features on the backbone, and fine-tune the network for the problem.</p>

tanabe.d.zc32s: <p>Thanks for the advice.<br>
I will try again and come back here if I find something I don’t understand.</p>
<p>Thank you.</p>
Embedding within float type features

CyrilTT: <p>I am pretty new in Pytorch and is trying to build  a network with embedding for float type value.<br>
I am mixing some numerical features with the the category features so they are not all integers.</p>
<p>When I run the embedding, I get the following error: Expected tensor for argument <span class="hashtag">#1</span> ‘indices’ to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor</p>
<p>And if I change the type for those category features as int, the numerical features part will give me issues</p>
<p>Is there any chance that I can solve the issue?</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class EmbeddingAmodule(nn.Module):
    def __init__(self):
        super().__init__()
        self.Emb = torch.nn.Embedding(344, 16)
        
    def forward(self, x):
        x = self.Emb(x)
        return x

class EmbeddingBmodule(nn.Module):
    def __init__(self):
        super().__init__()
        self.Emb = torch.nn.Embedding(344, 16)
        
    def forward(self, x):
        x = self.Emb(x)
        return x

class Env(nn.Module):
    def __init__(self):
        super().__init__()
        self.env = nn.Linear(62,62)
        
    def forward(self, x):
        x = F.relu(self.env(x))
        return x
    
class Combinednetwork(nn.Module):
    
    def __init__(self, modelA1, modelB1,  modelenv):
        super().__init__()
        
        self.modelA1 = modelA1
        self.modelB1 = modelB1
        self.modelenv = modelenv
        
        
        self.fc2 = nn.Linear(262,256, bias=False)
       ...
        self.Logit = nn.Linear(64,1)
        
        
        
    def forward(self, xa1, xa2, xa3, xa4, xa5, xa6, xb1, xb2, xb3, xb4, xb5, xb6, xenv): 
        
        xa1 = self.modelA1(xa1)
        xa2 = self.modelA1(xa2)
...
        xb5 = self.modelB1(xb5)
        xb6 = self.modelB1(xb6)
        
        xenv = self.modelenv(xenv)
                
        x = torch.cat((xa1, xa2, xa3, xa4, xa5, xa6, xb1, xb2, xb3, xb4, xb5, xb6, xenv), dim=0)
        
...
        x = self.fc5(x)
        x = self.bn5(x) # layer 5
        value = self.Logit(x)
        
        return  torch.tanh(value)
</code></pre>
<pre><code class="lang-auto">params = {'batch_size': 64,
          'num_workers': 6}
max_epochs = 10

dataset = CustomIterableDatasetv1('cleaned.csv')
dataloader = DataLoader(dataset, **params, pin_memory=True)

modela = EmbeddingAmodule().to(device)
modelb = EmbeddingBmodule().to(device)
modelenv = Env().to(device)

model = Combinednetwork(modela,modelb, modelenv)


optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
criterion2 = torch.nn.MSELoss()
</code></pre>

ptrblck: <p>Your use case sounds similar to <a href="https://discuss.pytorch.org/t/why-doesnt-consider-floattensor-for-embedding-layer/151883/2">this one</a>.<br>
Since embeddings are working as lookup tables for categorical input features, you might be able to use an <code>nn.Embedding</code> for the <code>int</code> indices and e.g. a linear layer for the floating point features.</p>
Can convert model to onnx but can't perform jit script or trace

sreddy: <p>Does torch.onnx use jit script/trace to perform its conversion?</p>

ptrblck: <p>Yes, as described <a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting">here</a>.</p>

sreddy: <p>That is odd because for<br>
jit trace and script I get</p>
<blockquote>
<p>AssertionError: Expected Module but got &lt;class ‘method’&gt;</p>
</blockquote>
<p>with torch.onnx.convert everything runs perfectly. When I read the <a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting" rel="noopener nofollow ugc">doc</a> you shared carefully, it states that the converter does something <em>equivalent to jit trace</em>.</p>
<p>EDIT:<br>
Do you have any advice on debugging jit script/tracing?</p>
Quantization ERROR :embedding_bag_prepac

AlexWortega: <p>For this code</p>
<pre><code class="lang-auto">model.qconfig = torch.quantization.float_qparams_weight_only_qconfig

print(model.qconfig)
torch.quantization.prepare(model, inplace=True)
torch.quantization.convert(model, inplace=True)
</code></pre>
<p>I ve got this error</p>
<pre><code class="lang-auto">NotImplementedError: Could not run 'quantized::embedding_bag_prepack' with arguments from the 'QuantizedCUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::embedding_bag_prepack' is only available for these backends: [QuantizedCPU, BackendSelect, Python, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, Tracer, AutocastCPU, Autocast, Batched, VmapMode, Functionalize].
</code></pre>

AlexWortega: <p>Collab to reproduce <a href="https://colab.research.google.com/drive/1Tb7J4PvvegWOybPfUubl5O7m5I24CBg5?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/alexwortega">@AlexWortega</a> , this error message means that the kernel you are trying to run does not have an implementation on CUDA.  Can you try moving your model to CPU and trying on CPU?</p>
Getting filenames in Custom DataLoader

ageryw: <p>I want to build a custom dataloader which will use the class name and the image name as a permanent index. How can I get the filename and folder name of the current image in<code> __getitem__()</code>?</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, root):
        .......
        self.root = root
        self.dataset = datasets.ImageFolder(root=self.root,
                                        transform=transforms)
    def __getitem__(self, index):
        data, target = self.dataset[index]
        index = index
        return data, target, index

    def __len__(self):
        return len(self.dataset)
</code></pre>
<blockquote>
<p>My Dataset directory structure is as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b035151e2073c8f66e91eb05d3400e34b3d0bc76.png" alt="image" data-base62-sha1="p8NLkW8F8iDQvdfzFk3a9g6HRXg" width="370" height="403"></p>
</blockquote>
<blockquote>
<p>E.g For cat image 0 → index = cat0<br>
I would appreciate if there is any technique to extract the parent folder and file name of the current index in <code>__getitem__(self, index)</code>.</p>
</blockquote>
<p>Thank you.</p>

Andrei_Cristea: <p>Hello,</p>
<p>You can get it from <code>self.dataset.imgs</code>, like this:</p>
<pre><code class="lang-auto">import os
class MyDataset(Dataset):
    def __init__(self, root):
        self.root = root
        self.dataset = datasets.ImageFolder(root=self.root, transform=transforms)
    def __getitem__(self, index):
        data, target = self.dataset[index]
        index = index
        return {
            "data": data,
            "target": target,
            "index": index,
            "folder": os.path.dirname(self.dataset.imgs[index][0]),
            "filename": os.path.basename(self.dataset.imgs[index][0]),
        }

    def __len__(self):
        return len(self.dataset)

dataset = MyDataset("/path/to/data/")
dataset[0]
Output:
{'data': &lt;PIL.Image.Image image mode=RGB size=1920x1080 at 0x2C35E87A8D0&gt;,
 'target': 0,
 'index': 0,
 'folder': '/path/to/data',
 'filename': 'foo.png'}
</code></pre>

ageryw: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> This worked. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Is it faster to resize an entire dataset before using DataLoader or should I use .resize() per batch?

mesllo: <p>I was wondering whether due to the high-resolution images in this large dataset it would affect my performance if I’m performing a resize step every time in the DataLoader, compared to just resizing it entirely from before.</p>

ptrblck: <p>I think it generally depends how you are loading the data.<br>
Often you would be working with a large dataset, which either won’t fit entirely into the RAM or which would be wasteful to pre-load completely (long startup time; slow experiment iterations).<br>
In this case you won’t be able to resize the entire dataset at once unless you store each image/sample using the new resized shape.</p>
<p>However, if you are dealing with a smaller dataset and are already preloading the dataset, you might see a speedup if you are resizing it beforehand as it would reduce the transformation workload. The expected speedup again depends on the actual use case so you would need to measure it.<br>
Also, note that data augmentation approaches (such as random crops before resizing etc.) would not be possible anymore since your resize operation would be “static”.</p>

mesllo: <p>But I can still use a library such as PIL to manually resize this large high-res dataset offline, and then apply data aug transforms during training right? That’s what I mean, if I can save from having to do that one extra step and just do it once beforehand maybe there’s an advantage, but I’m not sure if it really matters since DataLoader uses workers to speed up training. Again I might be wrong here.</p>

ptrblck: <aside class="quote no-group" data-username="mesllo" data-post="4" data-topic="152529">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mesllo/48/48260_2.png" class="avatar"> mesllo:</div>
<blockquote>
<p>But I can still use a library such as PIL to manually resize this large high-res dataset offline, and then apply data aug transforms during training right?</p>
</blockquote>
</aside>
<p>Yes, assuming you would also resize the samples to a static shape during training and would not use a random transformation before.</p>
<aside class="quote no-group" data-username="mesllo" data-post="4" data-topic="152529">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mesllo/48/48260_2.png" class="avatar"> mesllo:</div>
<blockquote>
<p>if I can save from having to do that one extra step and just do it once beforehand maybe there’s an advantage, but I’m not sure if it really matters since DataLoader uses workers to speed up training.</p>
</blockquote>
</aside>
<p>I would recommend to profile both use cases to check if you would see a speed up. E.g. if your data loading pipeline is fast enough and would completely hide the latency by pre-loading the samples in the background using the larger images while the GPU is busy training the model, optimizing this pipeline would not yield any speedup.</p>
Best approach to quantize a network with Conv layer and GRU

Hari_Krishnan: <p>The <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">document</a> about quantization suggests that the Conv1D/2D/3D layers does not support dynamic quantization. While GRU and LSTM supports only dynamic quantization. So what is the best way to quantize models with such architecture ?</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/hari_krishnan">@Hari_Krishnan</a> ,</p>
<p>For now, what you mentioned above is correct.  In the future, there may be dynamic quantization of conv or static quantization of parts of GRU, but that does not exist at the moment.</p>
<p>You could try quantizing the convolutions statically and your GRU/LSTM layers dynamically.  If you are using FX graph mode quantization, you can specify which layers to quantize in which way and the framework will model the dtype transitions for you.  If you are using eager mode quantization, you’d have to model the dtype transitions yourself.</p>
Combining DDP with model parallelism in a specific way

Anshuman_Senapati: <p>Hi,</p>
<p>I have 4 GPU devices. Let my network have 10 layers {L0, L1, L2, L3} and {L4, L5, …, L10}. I want to do a hybrid of DDP and model parallelism.</p>
<ol>
<li>Each Li, where i=0,1,2,3 should go to GPU:i</li>
<li>A copy of {L4, …L10} has to be in each GPU. Hence the network in GPU:i should be {Li, L4, …, L10}.</li>
<li>In each pass, we want to gather the forward pass outputs from each GPU, compute a loss and backpropagate and synchronize gradients (although data passed through each of the GPUs is the same, so not “data parallel” in that sense just that the copy of a part of the model is same in all the devices).</li>
</ol>
<p>Issue: In the standard distributed documentation and model parallelism demos, the network is always sliced sequentially and given to the devices, for example {L0, L1} in GPU:0, {L2, L3} in GPU:1, and so on, while in this case a part of the model is different in each device and a part of the model is same.</p>
<p>Is there a starightforward way to implement this that I’m missing? And what all difficulties might I run into while doing so?</p>

Aadarsh_Sahoo: <p>I am also facing a similar issue. <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> any help with this?</p>

kumpera: <p>Have you tried to nest DDP inside each pipeline stage?</p>
<p>One issue you might run into is that you need to use DDP with grad accumulation and explicitly control that bit. You can do it with <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync" rel="noopener nofollow ugc">no_sync</a></p>
<p>cc <span class="mention">@jamesr66a</span> as he’s the expert in PP.</p>
Asking for code review

mopi47: <p>I am tryin to implement the following network and I need some advice and a someone to review the code I wrote<br>
here is a pic of the network<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2307729ac8d69273228d7fd0a0d6b0cfbc708df.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2307729ac8d69273228d7fd0a0d6b0cfbc708df" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2307729ac8d69273228d7fd0a0d6b0cfbc708df_2_521x500.png" alt="image" data-base62-sha1="tZq6xWL6ojCB9LcSKB1WG4Zxz6n" width="521" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2307729ac8d69273228d7fd0a0d6b0cfbc708df_2_521x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2307729ac8d69273228d7fd0a0d6b0cfbc708df.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2307729ac8d69273228d7fd0a0d6b0cfbc708df.png 2x" data-dominant-color="D5D7DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×663 72.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
here is the code I wrote so far<br>
<a href="https://github.com/hikkiMourad/detectron2/blob/main/projects/BrainSeg-rcnn/brainSeg/brainSeg.py" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/hikkiMourad/detectron2/blob/main/projects/BrainSeg-rcnn/brainSeg/brainSeg.py</a></p>
Datapipe reset?

aliutkus: <p>Hi there,<br>
I have a datapipe object, say <code>datapipe</code>. I want to get some shape out of it because it turns out I don’t know it beforehand.</p>
<p>For this, I do, say :<br>
<code>dim =  int(next(iter(datapipe)).shape[1]</code></p>
<p>Then, I do some training with a dataloader on top of this datapipe. This gets me as many of the following warning as I have workers:</p>
<blockquote>
<p>UserWarning: Some child DataPipes are not exhausted when <strong>iter</strong> is called. We are resetting the buffer and each child DataPipe will read from the start again.</p>
</blockquote>
<p>I guess it’s because my datapipe is sent anew to each worker.<br>
Do you see a way I could get rid of that warning gracefully ?</p>

nivek: <p>Currently, there isn’t a way to suppress that warning with <code>torchdata</code>'s API.</p>
<p>Depending on your use case, you can use <a href="https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings" rel="noopener nofollow ugc"><code>with warnings.catch_warnings()</code></a>.</p>
<p>Feel free to open a PR and suggest a way to suppress that warning. We will be happy to discuss.</p>

aliutkus: <p>thanks for the swift reply</p>

aliutkus: <p>is there maybe a way to reset manually ? I didn’t see that option</p>

nivek: <p>It should reset as needed if you call <code>iter(child_datapipe)</code> again. If you have a code snippet where something is not working as expected. I can have a look.</p>
nn.Embedding with a dynamic number of embeddings?

aliutkus: <p>Hi, I am wondering whether you already saw an implementation of <code>nn.Embedding</code> for which the number of embeddings could be dynamic, with the following behaviour:</p>
<p>if you give it an embedding index that is not in the current set of indexes for embeddings, allocate a new embedding for that.</p>
<p>if you would have an elegant way to do that in mind ? I guess it’s not trivial since the optimizer’s parameters would need to change, etc etc</p>
<p>thanks</p>
<p>antoine</p>
Setting gradient to zero manually?

jayz: <p>Hey,</p>
<p>for some predictions of my model, I want the optimizer not to consider these predictions in updating the parameters. Therefore, I thought of setting the gradient of the loss corresponding to these predictions to zero.</p>
<pre><code class="lang-python">def loss_fn(y_pred,y): 
     loss = some_function(y_pred,y)
     if y_pred==specific_prediction:
            loss.grad = None
     return loss
</code></pre>
<p>Would that be conceptually correct?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

kaizhao: <p>You can simply use a binary mask to filter out these losses of unexpected predictions.</p>
<pre><code class="lang-auto">loss = some_loss_function(y_pred, y)
loss = loss * binary_mask
loss = loss.sum() # or alternatively loss = loss.mean()
</code></pre>
<p>Where <code>binary_mask[i] = 0</code> for unexpected predictions and <code>binary_mask[i] = </code> otherwise.</p>
Train with video data

sky518303: <p>Hi, i am trying to train the network with video data. I build a dataloader which give [batch, frame, C, H, W].</p>
<p>I am trying to build a cnn-lstm based network, and regress the certain value from each frame. so, it’s basically many-to-many task like below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ea50cb991a26cf326937b139fada84528dc1ce7.png" alt="image" data-base62-sha1="8WbdFJfJGJ0VA0lHbOdQLmWCDIP" width="111" height="190"></p>
<p>as you know, i cannot use conventional convolution for input data [batch, frame, C, H, W]. do i have to use model for each frame? I have no idea how i can implement it… any suggestions or well-organized links?</p>
If else loop problem in nn.module

fllci: <p>Hello!</p>
<p>I am having a small problem in my code snippet below. Normally, when to end the if loop, I write pass. But, here neither pass nor break works. I get an invalid syntax. I do not understand why I cannot let the code else do nothing. Could anyone find my mistake here please? Thanks!</p>
<pre><code class="lang-auto">&gt; class ConvBlock(nn.Module):
&gt;     def __init__(self, in_channels, out_channels, down=True, use_act=True, **kwargs):
&gt;         super().__init__()
&gt;         self.conv = nn.Sequential(
&gt;             nn.Conv1d(in_channels, out_channels, padding_mode="reflect", **kwargs)
&gt;             if down
&gt;             else nn.ConvTranspose1d(in_channels, out_channels, **kwargs),
&gt;             nn.Dropout()
&gt;             if drop
&gt;             pass,
&gt;             nn.BatchNorm1d(out_channels, affine=True),
&gt;             nn.Mish(out_channels) if use_act else nn.Identity()
&gt;             )
</code></pre>

InnovArul: <aside class="quote no-group" data-username="fllci" data-post="1" data-topic="152537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fllci/48/12955_2.png" class="avatar"> fllci:</div>
<blockquote>
<pre><code class="lang-auto">&gt;             nn.Dropout()
&gt;             if drop
&gt;             pass,
</code></pre>
</blockquote>
</aside>
<p>Probably there is an <code>else</code> case missing here.<br>
You could do</p>
<pre><code class="lang-auto">nn.Dropout() if drop else nn.Identity(),
</code></pre>

fllci: <p>I can’t completely understand the operation that nn.Identity() does. But I know it will change the output.</p>

Matias_Vasquez: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Identity.html">Here</a> is the documentation where you can see that the output should be the same as the input.</p>
<p>But if you want to be sure, you can take a look at the source code and see that the output is the same as the input.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/bd5ec6c8b7b8d33689109fe258b73ef7a61b9a3c/torch/nn/modules/linear.py#L44-L45">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/bd5ec6c8b7b8d33689109fe258b73ef7a61b9a3c/torch/nn/modules/linear.py#L44-L45" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/bd5ec6c8b7b8d33689109fe258b73ef7a61b9a3c/torch/nn/modules/linear.py#L44-L45" target="_blank" rel="noopener">pytorch/pytorch/blob/bd5ec6c8b7b8d33689109fe258b73ef7a61b9a3c/torch/nn/modules/linear.py#L44-L45</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="44" style="counter-reset: li-counter 43 ;">
            <li>def forward(self, input: Tensor) -&gt; Tensor:</li>
            <li>    return input</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Nvprof and torch.cuda.Event time measurements are not consistent

manoj_kumar1: <p>I am trying to find out best API to use to measure inference time. So I came up with simple matrix multiplication operation and used different API’s to measure the time. But I see that all the API’s are giving different results and I am not able to figure out exact reason behind this. Can someone please help me in this. ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/2366c0ee29a45d96bbc92c6989c1011db52dbd70.png" alt="Screenshot from 2022-05-25 09-11-11" data-base62-sha1="53aRkgD7GV4JLLpUpWBMG3Z0z3W" width="592" height="214"></p>
<p>mat_shape            : Matrix of different shape<br>
profiler_emit_nvtx : Results from nvprof profile tool from NVIDIA<br>
py_cuda_events   : Results from torch.cuda.Event API<br>
py_time_sync       : Results from python time API with CUDA synchronization</p>
<p>I have share my code here <a href="https://github.com/manojec054/tvm-explore/blob/master/pytorch_benchmark_explore_v2.py" class="inline-onebox" rel="noopener nofollow ugc">tvm-explore/pytorch_benchmark_explore_v2.py at master · manojec054/tvm-explore · GitHub</a></p>
<p>How to run : <code>nvprof --profile-from-start off --print-gpu-summary  python pytorch_benchmark_explore_v2.py</code></p>
Multiprocessing inside forward on single GPU

Gaoyuan_Yang: <p>Hi, I want to run two lines in parallel inside forward function on single GPU.<br>
The two sub-processes are independent from each other. I want to run self.l1 and self.l2 simultaneously here. The minimum code is as follows:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.multiprocessing as mp
import time

mp = mp.get_context('spawn')

class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.l1 = nn.Linear(784, 100)
        self.l2 = nn.Linear(784, 100)

    def forward(self, x):

        processes = []
        num_processes = 2

        for i in range(num_processes):
            if i == 0:
                p = mp.Process(target=self.l1(x))
            else:
                p = mp.Process(target=self.l2(x))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()

       

  
if __name__ == '__main__':
    BATCH_SIZE = 64
    x = torch.randn(BATCH_SIZE, 784).cuda()
    mynet = Net().cuda() # ; torch.cuda.synchronize()
    mynet(x)
</code></pre>
<p>I found similar question: <a href="https://discuss.pytorch.org/t/parallelize-simple-for-loop-for-single-gpu/33701">for loop inside single GPU</a> and <a href="https://discuss.pytorch.org/t/how-to-write-parallel-for-loop-in-models-forward-method/51652">for loop inside forward</a>.<br>
There are two relevant questions : <a href="https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698">parallel over samples</a> and <a href="https://discuss.pytorch.org/t/parallel-execution-of-modules-in-nn-modulelist/43940">parallel execution</a>, the answers to these two questions are reformulated version of the original problem.<br>
Besides, I tried the nn.Parallel(block1, block2) inside this post <a href="https://discuss.pytorch.org/t/how-can-l-run-two-blocks-in-parallel/61618">two blocks in parallel</a> however it did not work.<br>
I know distribute model to multiple GPU is well documented, however I have only one GPU here.<br>
multiprocessing over model level is documented here <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">multiprocessing over model level</a></p>
<p>Any help would be appreciated, thanks a lot!</p>

YOGESH_SANKU: <p>Hello, Did u solve the problem</p>

smell_well: <p>hi, I have same problem, did you find a method to deal with it?</p>
nn.Modulelist in parallel

smell_well: <p>In my case, my modulelist size is large and need use ‘for’ to traverse. However the block in modulelist is small and execution fast so the gpu utils(on the same gpu) is low. All sub modules in modulelist have same input and structure. I had read the topic <a href="https://discuss.pytorch.org/t/parallel-execution-of-modules-in-nn-modulelist/43940" class="inline-onebox">Parallel execution of modules in nn.ModuleList</a> and <a href="https://discuss.pytorch.org/t/running-multiple-modules-in-parallel/58164" class="inline-onebox">Running multiple Modules in parallel</a>. But my case have a bit different, if there have some mothed to improve gpu utils?<br>
Here is a part of my code</p>
<pre><code class="lang-python">self.client_list = nn.ModuleList()
for client_idx in range(self.client_num):
    self.client_list.append(
         Client(args.input_dim, args.conv_dim, args.residual_channel, args.skip_channel, args.end_channel,
 args.out_dim, args.timesteps, args.base_model))


self.server_model = Sever(args.num_nodes, args.conv_dim, args.residual_channel, args.cheb_k, args.embed_dim, "graph_conv_model")


# forward
encoder_list = []

for idx, client in enumerate(self.client_list):
      encoder, skip = client.encoder(x[:, :, idx, :, :])
      encoder_list.append(encoder)
      skip_list.append(skip)

# encoder_list = torch.cat(encoder_list, dim=2)
graph_out = self.server_model(encoder_list, self.adj_matrix)  #
</code></pre>
"Could not get qualified name for class xyz: __module__ can't be None"

MiriamJo: <p>Hello!<br>
I am currently trying to convert the FFB6D pose estimation model to TorchScript.<br>
Since it has a lot of conditional flows, I have to jit.script it. However, I am totally new to Scripting and there aren’t a lot of tutorials out there…</p>
<p>I instantiated the model with the best checkpoint and supplied a batch size of 1 to my model with the data generator that is also used for training the network.</p>
<pre><code class="lang-auto">model.to('cpu')
for (batch_idx, batch) in enumerate(train_loader):
    scripted_module = torch.jit.script(model, batch)
print(scripted_module.graph)
</code></pre>
<p>Now whenever I try to run it it throws following error:</p>
<blockquote>
<p>File “/home/miriam/anaconda3/envs/ffb6d/lib/python3.6/site-packages/torch/jit/_script.py”, line 1362, in _get_overloads<br>
qual_name = _qualified_name(obj)<br>
File “/home/miriam/anaconda3/envs/ffb6d/lib/python3.6/site-packages/torch/_jit_internal.py”, line 1015, in _qualified_name<br>
raise RuntimeError(f"Could not get qualified name for class ‘{name}’: "<br>
RuntimeError: Could not get qualified name for class ‘conv2d’: <strong>module</strong> can’t be None.</p>
</blockquote>
<p>It’s a really big network, so i much rather send a link to the github page with the network file:<br>
<a href="https://github.com/ethnhe/FFB6D/blob/master/ffb6d/models/ffb6d.py" rel="noopener nofollow ugc">FFB6D.py Model</a></p>
<p>This is the file with the conv2d implementation where the error occurs:<br>
<a href="https://github.com/ethnhe/FFB6D/blob/master/ffb6d/models/pytorch_utils.py" rel="noopener nofollow ugc">Pytorch_Utils.py Conv2D</a></p>
`index_max_()` functionality? (and others)

marcman411: <p>I am currently trying to figure out a good <code>index_max_</code> solution that doesn’t require multiple copies of the data or a loop. This would work along the lines of <code>index_add_</code> except the underlying operation is a <code>max</code> instead of addition.  Does anyone have a solution?</p>
<p>Also, are there plans for more explicitly built-in functionality along the lines of <code>index_add_</code>, <code>index_copy_</code>, etc? Even better, it would be awesome to have some sort of lambda-style of overloading for these indexing operations.</p>
<p>In particular, I’m finding occasions where an <code>index_max_</code> or <code>index_avg_</code> would be useful. For example, if I have a list of indices corresponding to data (maybe nearest neighbor pointers or something similar) and I want to pool over the indices, I could use it.</p>
<p>For the averaging operation, I currently use <code>index_add_</code> on my data and also on a <code>torch.ones()</code> vector and then do the appropriate broadcasted division. I suppose it may be more efficient to do that on the backend though.</p>

justheuristic: <p>In case anyone else finds this: you can compute index max using scatter_reduce with amax reduction</p>
<pre><code class="lang-auto">import torch
input = torch.tensor([
    [1, 2, 3, 4, 5, 6, 3.5],
     [1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 3.75]])
index = torch.tensor([[0, 0, 0, 1, 1, 2, 2],
                      [2, 0, 1, 1, 1, 1, 1]
                     ])
torch.scatter_reduce(input, -1, index, reduce="amax", output_size=3)
# tensor([[3.0000, 5.0000, 6.0000],
#             [2.5000, 6.5000, 1.5000]])
</code></pre>
<p>Please refer to <a href="https://pytorch.org/docs/stable/generated/torch.scatter_reduce.html" class="inline-onebox" rel="noopener nofollow ugc">torch.scatter_reduce — PyTorch 1.11.0 documentation</a></p>
Difference between two kinds of distributed training paradigm

kstarxin: <p>From the document (<a href="https://pytorch.org/docs/stable/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.11.0 documentation</a>) we can see there are two kinds of approaches that we can set up distributed training.</p>
<p>The first approach is to use <code>multiprocessing.spawn()</code> approach within one python file.</p>
<p>The second approach is to use <code>torchrun</code> or <code>torch.distributed.launch</code>.</p>
<p>I observed that there are slight different for initialize the process group.</p>
<p>For the first approach, it is</p>
<pre><code class="lang-auto">dist.init_process_group("nccl", rank=rank, world_size=2)
</code></pre>
<p>For the second approach, it is</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend='nccl',
                                     init_method='env://')
</code></pre>
<p>As we can see the second approach specified the rank and world size, so what lead to such difference ?</p>
<p>I also tried to use <code>mp.spawn</code> to start distributed training without specifying the rank and world size, and it yield error says <code>RuntimeError: Address already in use</code>. So what lead to this problem ?</p>

pritamdamania87: <p>My guess is that torchrun automatically sets the appropriate rank and world_size for you. cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>  to confirm.</p>
<aside class="quote no-group" data-username="kstarxin" data-post="1" data-topic="152293">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/c37758/48.png" class="avatar"> kstarxin:</div>
<blockquote>
<p>I also tried to use <code>mp.spawn</code> to start distributed training without specifying the rank and world size, and it yield error says <code>RuntimeError: Address already in use</code>. So what lead to this problem ?</p>
</blockquote>
</aside>
<p>This is likely because your previous training run might still be running and using the old port. Or the port specified in <code>MASTER_PORT</code> is being used by something else.</p>

Kiuk_Chung: <p>Yes <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> is correct. <code>torch.multiprocessing.spawn()</code> is a more lower level API. In fact <code>torchrun</code> uses <code>torch.multiprocessing</code> under the hood to invoke multiple copies of the user’s main method when the main method is passed as a function.</p>
<p>Since <code>torchrun</code> is a higher level tool, it sets up all the context and environment variables that <code>torch.distribued.init_process_group()</code> expects so that as a user, you only need to pass the <code>backend</code> parameter. (FYI - you don’t even need to pass <code>init_method</code> since <code>env://</code> is the default one).</p>
<p>In short, use <code>torchrun</code> when you can, and use <code>torch.multiprocessing.spawn()</code> when you need to.</p>