akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What Dataset/DataLoader for DDP to train on sharded local dataset?

username: Olivier-CR
cooked: <p>Hi,</p>
<p>I’m working with a system (Amazon SageMaker Training) capable of spreading remote files homogenously across machines. Meaning that at the beginning of a distributed DDP training, I have the files of my dataset (eg images, text files) homogeneously spread on the disks of each DDP nodes, as represented in the diagram below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/542872fd2c3e0be601485a6c7c5325865fe01efd.png" data-download-href="https://discuss.pytorch.org/uploads/default/542872fd2c3e0be601485a6c7c5325865fe01efd" title="Capture d’écran 2022-02-10 à 11.48.13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/542872fd2c3e0be601485a6c7c5325865fe01efd_2_631x500.png" alt="Capture d’écran 2022-02-10 à 11.48.13" data-base62-sha1="c0uLHRRwO7mFS9hsVojnfQSDvU9" width="631" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/542872fd2c3e0be601485a6c7c5325865fe01efd_2_631x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/542872fd2c3e0be601485a6c7c5325865fe01efd_2_946x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/542872fd2c3e0be601485a6c7c5325865fe01efd_2_1262x1000.png 2x" data-dominant-color="F7F5F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2022-02-10 à 11.48.13</span><span class="informations">1312×1038 45.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What <code>Dataset</code> &amp; <code>DataLoader</code> settings shall I use to train DDP over this distributed dataset?</p>
<p>Most DDP example I saw work do a virtual sharding with the <code>DistributedSampler</code> in the <code>DataLoader</code>, but here my data is already phyiscally distributed…</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for dataloader</p>

username: ptrblck
cooked: <p>The <code>DistributedSampler</code> would create chunks using the dataset indices so that each worker only gets its corresponding subset. If you are using a local dataset on each worker, I would assume you should be able to load only this particular dataset in each process?</p>

username: VitalyFedyunin
cooked: <p>It feels like you can use the regular sampler, but you need to make sure that ALL datasets are equal-sized, otherwise, your distributed training will hang.</p>
Pairwise_distance memory allocation / functioning

username: S_whites
cooked: <p>I have an issue with the function pairwise_distance from torch.nn.functional.<br>
Unfortunately, I didn’t find any hint looking at the wiki/code implementation.<br>
I would like to know how much memory it uses to compute the distance.</p>
<p>I have a pretty large tensor of shape: A = [5056512×381] which occupies around 7GB in GPU. When I call F.pairwise_distance between a row of A (A[i]) and A my GPU goes out of memory and I don’t know why.<br>
Some notes:<br>
I launch: torch.cuda.empy_cache() right before the distance computation and I have the GPU almost free: 0.94GB allocated and 3GB reserved and my GPU has 24GB.</p>
<p>The error is the following:<br>
CUDA out of memory. Tried to allocate 7.18 GiB (size of tensor A) but 15.30 GiB was already allocated.<br>
It seems that allocates the memory of A 3 times! Any hint?<br>
(Since it is really slow to perform this computation in CPU I would like to keep it in GPU and do it in the shortest  time)<br>
Thank you very much, every suggestion is appreciated!</p>
About torch.nn.utils.clip_grad_norm

username: Diego
cooked: <p>Hello I am trying to understand what this function does. I know it is used to prevent exploding gradients in a model and I understand what the norm of a vector is and I’m guessing that this function ‘clips’ the norm of a vector to a specific maximum value. But I would like to know how this prevents the exploding gradient problem and what exactly does it do the the model parameters. Help would be greatly appreciated. Thanks!</p>

username: jpeg729
cooked: <p>In some cases you may find that each layer of your net amplifies the gradient it receives. This causes a problem because the lower layers of the net then get huge gradients and their updates will be far too large to allow the model to learn anything.</p>
<p>This function ‘clips’ the norm of the gradients by scaling the gradients down by the same amount in order to reduce the norm to an acceptable level. In practice this places a limit on the size of the parameter updates.</p>
<p>The hope is that this will ensure that your model gets reasonably sized gradients and that the corresponding updates will allow the model to learn.</p>

username: Diego
cooked: <p>I understand now. So since the norm is the distance of a vector from the origin you are forcing the gradient  to not go too far from the position it was the last time you ran model.zero_grad(). Did I get that right?</p>

username: jpeg729
cooked: <p>I think it is more correct to say that you are forcing the gradients to be reasonably small, which means that the parameter updates will not push the parameters too far from their previous values.</p>

username: Diego
cooked: <p>I see. So it’s similar to decreasing the learning rate?</p>

username: jpeg729
cooked: <p>But only selectively. Small gradients cause small updates and those will use the full learning rate.</p>
<p>Huge gradients will be squashed and the corresponding updates will be smaller.</p>

username: Diego
cooked: <p>So it’s similar to decreasing the learning rate only for big gradients to perform small updates all the time.</p>

username: jpeg729
cooked: <p>Yes. That is exactly what it does.</p>
<p>My previous answers assumed that lower layers would receive clipped gradients during the backpropagation. This is not the case. clip_grad_norm is applied after the entire backward pass.</p>

username: Diego
cooked: <p>So it just clips the resulting gradient right before performing one step. Correct?</p>

username: jpeg729
cooked: <p>Assuming you apply it after calling <code>loss.backward()</code> and before calling <code>optimizer.step()</code>, then yes.</p>

username: Diego
cooked: <p>Yes. That’s how I used it. Thanks a lot you have been a great help!</p>

username: Aiden1
cooked: <p>Thanks for the good thread, I’m wondering what is the intuition behind max_clip value? I’ve seen people use 1 or 10 etc, how to set your max value?</p>
<p>Thanks</p>
How do I design two networks which part of their layers are same

username: ZiQing_Li
cooked: <p>I want to design two networks and update them separately, but some of their layers are the same.</p>
<p>I have two plans. Can you tell me which one is better or give another better idea?</p>
<ol>
<li>Use two nn.module. Change the weights of the same layers of A network after B network updates.</li>
<li>Use one nn.module. In this situation, A and B can use the same layers.</li>
</ol>
Tips narrowing down problematic code snippet that is failing to compile to ONNX?

username: davidat
cooked: <p>So I’m trying to convert a <a href="https://github.com/burchim/EfficientConformer" rel="noopener nofollow ugc">relatively complex model</a> to ONNX. I’m currently getting the following error:</p>
<pre><code class="lang-auto">File ~/Library/Caches/pypoetry/virtualenvs/efficientconformer-bpIwAhA4-py3.9/lib/python3.9/site-packages/torch/onnx/symbolic_opset9.py:560, in transpose(g, self, dim0, dim1)
    558     return g.at("transpose", self, dim0_i=dim0, dim1_i=dim1, overload_name="int")
    559 else:
--&gt; 560     raise RuntimeError("Unsupported: ONNX export of transpose for tensor "
    561                        "of unknown rank.")
</code></pre>
<p>My current plan of action is to track which <code>.transpose()</code> call in Python is on a “tensor of unkown rank”, and somehow make that rank known to the JIT compiler (or possibly modify the JIT graph directly). But there are many <code>.transpose()</code> calls.</p>
<p>But I’m actually not sure how to narrow it down to which one it is, hence my question here!</p>
Does pytorch reset weights and biases when retraining a pretrain model

username: AIBOI
cooked: <p>Hi there, I was wondering if for instance a pre-trained resnet50 model’s weights and biases would reset when training it with a new set of data.</p>

username: Matias_Vasquez
cooked: <p>You might want to look into-fine tuning a model.</p>
<p>When you fine-tune a model with another dataset, you take advantage that the pretrained weights mean something already and are not just random numbers.</p>
<p>When you do this, you want to take a small learning rate in order to achieve the desired result.</p>
<p>So, to answer your question, yeah the weights do change but they are not ‘reset’ to something random.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Neurolabs Open Computer Vision Roles

username: patricieni
cooked: <p>Hi everyone,</p>
<p>I’m Patric and I’m one of the 3 co-founders and the CTO of Neurolabs and a Computer Vision Engineer at heart. At Neurolabs we’re shaping the way computers see the world by revolutionising the computer vision challenge with the use of Synthetic Data.</p>
<p>MIT Tech Review highlighted Synthetic data as one of the <a href="https://www.technologyreview.com/2022/02/23/1045416/10-breakthrough-technologies-2022/" rel="noopener nofollow ugc">Top 10 breakthrough technologies</a> of 2022.</p>
<p>I’m currently hiring for</p>
<ul>
<li>Senior Computer Vision Engineer</li>
<li>Senior Computer Graphics Engineer</li>
<li>Senior 3D Modeller</li>
</ul>
<p>We’re looking for people who are ambitious, curious and enjoy working on challenging problems!<br>
We’ve raised a pre-seed and are wrapping up a bigger seed-round, so it’s a great time to join!</p>
<p>Apply below!</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://jobs.lever.co/neurolabs-ai">
  <header class="source">

      <a href="https://jobs.lever.co/neurolabs-ai" target="_blank" rel="noopener nofollow ugc">jobs.lever.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:226/142;"><img src="https://lever-client-logos.s3.us-west-2.amazonaws.com/a796154b-b4f2-47b9-b4c2-9b812dc7285d-1643663541678.png" class="thumbnail" width="226" height="142"></div>

<h3><a href="https://jobs.lever.co/neurolabs-ai" target="_blank" rel="noopener nofollow ugc">Neurolabs jobs</a></h3>

  <p>Job openings at Neurolabs</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Changing requires_grad means creating new optimizer?

username: hamedB
cooked: <p>Hi.<br>
From this <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">link</a>, I read the thread about unfreezing during the training phase. However, in my case I want to freeze some parameter after some epochs.<br>
First, I have a nn.ParameterDict that includes a set of nn.Parameter parameters with specific key. First of all, whole of nn.ParameterDict parameter is trainable. Then after some epochs, during a for loop, I set the required_grad to False for some of the nn.Parameter parameters in nn.ParameterDict parameter. However, I see that they are still fine-tuned/trained while their required_grad is actually False.<br>
Should I define new optimization when freeze some parameters?<br>
Thank you</p>
Balance an imbalanced Dataset

username: datalord
cooked: <p>I have a dataset with data that falls into one of three labels/classes : A, B, C</p>
<p>class A has 3000 data points<br>
class B has 2000 data points<br>
class C has 1000 data points.</p>
<p>In every epoch of my training I want my Dataset to pick all 1000 data points of C and randomly select 1000 data points of B and 1000 data points of A .</p>
<p>As it stands in the code below, the Dataloader will parse through all 6000 data points. How do I modify the Dataset below to ‘balance’ the dataset so that I use a 1000 points from each class at every epoch?</p>
<pre><code class="lang-auto">A = np.random.rand(3000)
B = np.random.rand(2000)
C = np.random.rand(1000)

data_array = np.concatenate([A,B,C])
labels = np.concatenate([np.zeros_like(A), np.ones_like(B), 2*np.ones_like(C)])

class Imbalanced_Dataset(Dataset):
    def __init__(self, data_array, labels):
        self.data_array = data_array
        self.labels = labels
    def __len__(self):
        return len(self.data_array)

    def __getitem__(self, index):

        data_point = self.data_array[index]
        label = self.labels[index]

        return data_point,label
</code></pre>
<p>Edit: For clarity, the Dataset should sample randomly from A and B at every epoch. So the admitted batches from A and B will look different every epoch.</p>

username: blueeagle
cooked: <p>I think there is a problem in your <code>Imbalanced_Dataset</code> class.<br>
In your <code>__len__</code> method you are returning <code>len(self.dataset)</code>. However, <code>self.dataset</code> is undefined, I assume you mean <code>self.data_array</code>. This however leads to another problem, as a DataLoader will pick <code>len(self.data_array)</code> elements of your dataset per epoch. As far as I understand your question, you want it to only pick <code>number_of_classes*minimal_number_of_samples</code> element per epochs.<br>
Your <code>__len__</code> method should therefore look like that:</p>
<pre><code class="lang-auto">def __len__(self):
        number_of_elements_per_class = [len(np.where(self.labels==class_idx)[0]) for class_idx in np.unique(self.labels)]
        return len(number_of_elements_per_class)*min(number_of_elements_per_class)
</code></pre>

username: datalord
cooked: <p>Hi <a class="mention" href="/u/blueeagle">@blueeagle</a> , I’ve corrected some of the typos in the variable names.</p>
<p>This isn’t quite what I want. I want to ensure that for every epoch, I have 1000 randomly selected elements per class.</p>
<p>I don’t think changing the <code>__len__</code> function will help because if in my case I change the length to 3000, it will pick indices from 0 to 2999 and completely miss class B and class C.</p>

username: ejguan
cooked: <p>You can generate a random map from indices 0-2999 to the indices in A/B/C to achieve such feature<br>
[0-999] → a list of random indices from A [0-2999]<br>
[1000-1999] → a list of random indices from B [0-1999]<br>
[2000-2999] → indices from C [0-999]</p>

username: datalord
cooked: <p>This is a very creative solution. Thanks!</p>
Question about .clone() usage

username: ZiQing_Li
cooked: <p>In some codes, I see people do <code>.clone()</code> when operating on the same elements (e.g. <code>grid[:, :, 0] = 2.0 * grid[:, :, 0].clone()</code>). From my view, those operations won’t cause an in-place operation. So I have the following questions.</p>
<ol>
<li>Why do people use <code>.clone()</code> when operating on the same elements?</li>
<li>When should we use <code>.clone()</code>  except clone+detach &amp;&amp; avoid inplace operation?</li>
</ol>
How does masking work for Transformers? Are they really not seeing future values?

username: osllogon
cooked: <p>Hi everyone,</p>
<p>I was using a Transformer Encoder for predicting future values of a time series. As I wanted the transformer to take into account 200 past values and predicting the next 50, I was using a mask of (250x250) and the last 50 values of each row were -inf for hiding the future values. However, now I realized that when I put zeros on the last 50 values the Transformer crashes. I tried with the complete transformer (encoder + decoder) and the same thing happens. Does anyone know what can be happening or if I am doing something wrong?</p>
<p>Thanks in advanced to everyone</p>

username: osllogon
cooked: <p>Update: Now I see that a triangle mask, as the one is generally used in NLP, works but a square mask, as the one I created does not. It works as if a triangle mask and the squared were multiplied, as the positions when any of the two mask are zero are used by the transformer. I thought that you can use a mask of the shape that you want, if anyone can help I would really appreciate it,</p>
<p>Thank you all in advanced</p>
Why this code will cause an inplace operation?

username: ZiQing_Li
cooked: <p>The code is listed below.</p>
<pre><code class="lang-python">fake_w = torch.max(grid[:, :, 0])
fake_h = torch.max(grid[:, :, 1])
</code></pre>
<p>I used the <code>detect_anomally</code> to detect the forward error, and it told me that the line of fake_h had an in-place operation.</p>
<p>In my opinion, those two lines are the same. Why only the second line caused the error while the first line passed? And the second question is why the second line is an in-place operation?</p>

username: KFrank
cooked: <p>Hi ZiQing!</p>
<aside class="quote no-group" data-username="ZiQing_Li" data-post="1" data-topic="147164" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziqing_li/48/44710_2.png" class="avatar"> ZiQing_Li:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">fake_w = torch.max(grid[:, :, 0])
fake_h = torch.max(grid[:, :, 1])
</code></pre>
</blockquote>
</aside>
<p>This code does not perform an inplace operation (unless <code>fake_w</code> or<br>
<code>fake_h</code> refer to a view into a pytorch or are some similar alias).</p>
<p>Could you post a complete, runnable script that illustrates your issue?</p>
<p>Best.</p>
<p>K. Frank</p>

username: ZiQing_Li
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thanks for your quick reply! And here’s the full code of the function.</p>
<pre><code class="lang-python">        min_grid_ = torch.min(grid)
        fake_w = torch.max(grid[:, :, 0])
        fake_h = torch.max(grid[:, :, 1])
        grid[:, :, 0] = 2.0 * grid[:, :, 0].clone() / max(fake_w - 1, 1) - 1.0
        grid[:, :, 1] = 2.0 * grid[:, :, 1].clone() / max(fake_h - 1, 1) - 1.0
        max_grid = torch.max(grid)
        min_grid = torch.min(grid)
</code></pre>
<p>The shape of <code>grid</code> is [B, 3778, 2]. The error is listed below.</p>
<pre><code class="lang-auto">[W python_anomaly_mode.cpp:104] Warning: Error detected in MaxBackward1. Traceback of forward call that caused the error:
....................................
....................................
File "/home/xywang/xywang/hj-hassony2/meshreg/models/meshregnet.py", line 587, in gridsamplefun
    fake_h = torch.max(grid[:, :, 1])
</code></pre>

username: ZiQing_Li
cooked: <p>And I still have a question that in some codes why people do <code>.clone()</code> when operating on the same elements (e.g. <code>grid[:, :, 0] = 2.0 * grid[:, :, 0].clone()</code>)?</p>
When kernel size is 2 in cnn, with "same" padding, which side of the input is padded?

username: jayz
cooked: <p>Hey,</p>
<p>when we have a kernel with size that would result in an “odd” padding, which side would the larger padding standardly be applied?</p>
<p>The simplest case is kernel_size = 2. Consider a 1D input (X Y Z) of length 3. If we apply same padding, we would have to add a pad on either the left or the right side of the input:</p>
<p>P X Y Z &gt; (PX, XY, YZ)<br>
X Y Z P &gt; (XY, YZ, ZP)</p>
<p>The right hand side shows the sets of datapoints from the input XYZ that are received by the size 2 kernel in each stride step (where stride=1). “P” is the padding entry.</p>
<p>So, would P be on the left or the right side by convention?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

username: ptrblck
cooked: <p>Let’s perform your experiment with some pre-defined kernel values as <code>[[[0., 1.]]]</code> which would use the “right” pixel value of the current patch:</p>
<pre><code class="lang-python">conv = nn.Conv1d(in_channels=1, out_channels=1, kernel_size=2, stride=1, padding='same', bias=False)
with torch.no_grad():
    conv.weight.copy_(torch.tensor([[[0., 1]]]))

x = torch.randn(1, 1, 3)
print(x)
# tensor([[[ 0.7866,  1.1084, -1.3647]]])

out = conv(x)
print(out)
# tensor([[[ 1.1084, -1.3647,  0.0000]]], grad_fn=&lt;ConvolutionBackward0&gt;)
</code></pre>
<p>Based on this the right-hand side is padded.</p>

username: jayz
cooked: <p>Yes, this makes sense. Haven’t yet got the full experience with pytorch to come up with these handy small examples (just starting for three weeks or so), but will will note it down to answer these questions myself, in the future. Thanks for taking the time!</p>
<p>Best, JZ</p>
Data normalization help

username: Samuel_Bachorik
cooked: <p>Hi, i want to ask if this technique is normal,</p>
<p>Images for model normalized between -1,1 and labels for model normalized between 0,1.<br>
I see this in some documentation and I am not sure if this is normal.</p>

username: Matias_Vasquez
cooked: <p>Hi,</p>
<p>The normalization for images is usually done by subtracting the mean and the dividing by the standard deviation. This helps the neurons convert faster.</p>
<p>So yeah, -1,1 is a typical behavior, as well as 0,1.</p>
<p>However, I’m not sure how or why the labels are normalized, could you provide more context?</p>
<p>This is for BatchNormalization, but it is the same principle</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/351964acb877bc398a510f0a9cd74908d825d03e.png" alt="image" data-base62-sha1="7zJI4656XfJQDYyBDIOdP6DKx9c" width="300" height="200"><br>
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network<br>
training by reducing internal covariate shift. 3 2015. doi: 1502.03167</p>
Training performance (becomes worse) inversely proportional to image resolution (larger image size for inputs)

username: tsuijenk
cooked: <p>Hello everyone,</p>
<p>I am doing object segmentation. I have trained an excellent model with 40x40 images as input (say it has 3 objects in it).</p>
<p>Now, I am considering of training a model (call it model 2) with a larger image size as input, e.g. 160x160 images (and say it has 13-15 objects in it).</p>
<p>Surprisingly the model performance becomes much worse, despite I have made Model 2 “wider” and “deeper” to ensure that we have a better receptive field.</p>
<p>Indeed, what happened with the performance of model 2 is the following:<br>
Epoch 0: detected all 13-15 objects with some false positives around each object<br>
Epoch 1: retained those false positives, but the detection/localisation for those 13-15 objects begin to disappear<br>
Epoch 2: only false positives around the objects appeared, the actual objects themselves are not detected at all.</p>
<p>Has anyone experienced similar situation? Any thoughts are welcome! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thank you.</p>
Perform multi-class segmentation with pixel-labels, which only contain single class

username: lanfz_uestc
cooked: <p>For example, we have class number 3 in total.<br>
But the training set only contains classes like, [1,0,0],[0,1,0], and [0,0,1]<br>
What can i do to improve the performance?</p>

username: ptrblck
cooked: <p>The number of targets matches the number of classes, so I’m unsure what the issue is.<br>
In case you are struggling to get any improvements in your model training I would recommend to try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparameters first.</p>

username: lanfz_uestc
cooked: <p>But there is no traning samples containing multi-class.<br>
When inference, there may be [1,1,1] containing 3 class, it not work well.<br>
I try to use cutmix to increase the diversity of training samples, but it improved a little.</p>
No difference between resnet50 pretrained and not

username: mhai
cooked: <p>I’m training a classification model using resnet50 with pretrained and not pretrained one. However i don’t see much differences between them. Can anyone please explain to me?<br>
Thank you so much.<br>
My code:</p>
<pre><code class="lang-auto">dict_model = {
        'r18': models.resnet18(pretrained=args.pretrained),
        'r34': models.resnet34(pretrained=args.pretrained),
        'r50': models.resnet50(pretrained=args.pretrained),
        'r101': models.resnet101(pretrained=args.pretrained),
        'r152': models.resnet152(pretrained=args.pretrained)        
    }
    model_ft = dict_model[args.model]
  
</code></pre>
<p>Here is my result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/beee9825537d9a9da0a5f0db0dd001ba5a871647.png" data-download-href="https://discuss.pytorch.org/uploads/default/beee9825537d9a9da0a5f0db0dd001ba5a871647" title="Screenshot 2022-03-22 215728"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beee9825537d9a9da0a5f0db0dd001ba5a871647_2_690x391.png" alt="Screenshot 2022-03-22 215728" data-base62-sha1="rf3U8c14fC9G4B8YaSIkOdVTYnd" width="690" height="391" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beee9825537d9a9da0a5f0db0dd001ba5a871647_2_690x391.png, https://discuss.pytorch.org/uploads/default/original/3X/b/e/beee9825537d9a9da0a5f0db0dd001ba5a871647.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/beee9825537d9a9da0a5f0db0dd001ba5a871647.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-03-22 215728</span><span class="informations">889×505 48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue as the accuracy plot seems to show a better accuracy for the pretrained models? E.g. it seems you are reaching ~0.5 at step 1-2 using the pretrained models while training from scratch needs 5 steps or am I mixing up the plots?</p>
Concept of Head in A Neural Network (YoloV5)

username: Curimeowcat
cooked: <p>I am trying to understand the concept of “head”<br>
in <a href="https://github.com/ultralytics/yolov5/blob/ecc2c7ba73e71211b192cba69e255afad92de67a/models/yolov5x.yaml#L28" rel="noopener nofollow ugc">yolov5/models/yolov5x.yaml</a>, it would be great if you can explain it.</p>
<p>Specifically,<br>
(1) How is it different from simply an output from a neural network?<br>
(2) Does each head must have its corresponding labels so that we can compute losses? E.g. we I add a head to output class labels, I can use cross-entropy to train these addition classifier layers.<br>
(3) If I add an additional head, it is simply just some additional layers on top of previous layers?</p>
<p>Thanks in advance!</p>
How to store tensors as tensors instead of strings?

username: ADONAI_TZEVAOT
cooked: <p>I have 3 column in my pandas dataframe. Waveform, sample rate and score. Each entry in waveform is a n-dimensional torch tensor. Sample rate entries consists of simple scalar values and so are the scores.</p>
<p>I tried to store the tensors in a csv file but the waveform values are getting stored as strings. How to store torch tensors in csv file? Or is there a better alternative to csv ?</p>
<p>to solve the above issue I tried converting the waveform values to numpy nd array but still they get stored as strings into the csv.</p>
<p>Please help!</p>

username: JuanFMontesinos
cooked: <p>How do you save a waveform as a csv? Waveforms are  sets of n numbers. Even if you store them as int16 you still need to save N huge numbers. And csv are text files. So instead of taking 16 bits for a single integer you are using way more memory as each digit of the number will be a char.</p>
<p>You can just save .wav files. The format itself stores the sr as well and you can encode the score in the filename.</p>
<p>Optionally the optimal thing would be HDF5  files which allow to store everything independently as raw data.</p>
<p>But sincerely, .wav files + scipy is the best you can do here as it’s super fast to load and you can use memory map too.</p>

username: ADONAI_TZEVAOT
cooked: <p>How to use .wav files with scipy… are there any links you can guide me to please?</p>

username: JuanFMontesinos
cooked: <p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html</a></p>
<p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.write.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.write.html</a></p>
Installing PyTorch/Torchvision

username: Mem901
cooked: <p>Hello,<br>
I am trying to download Torchvision in an Anaconda environment. Pytorch is installed, but torchvision gives me an error:  “No module named torchvision”<br>
I’m using Windows 10.<br>
Please advise.<br>
I have done everything on the Google, Stackflow etc. ,nothing is working.<br>
My assignment is due Wednesday.<br>
Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Could you post the install command, the output of <code>conda list | grep torch</code> as well as your used Python version?</p>

username: Mem901
cooked: <p>Good afternoon!<br>
It’s fixed and this is what I used.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1950e792a7b2ba5163c27db0dfe605de15622a8c.png" alt="image" data-base62-sha1="3BXgZmtNi3ZZvRYbmN3coCJNfAg" width="472" height="148"><br>
For some reason,  it wasn’t working in my new anaconda environment.<br>
My python version is 3.9.7 and conda version 4.11.0.<br>
Thanks for responding.<br>
I will be in touch. I am using PyTorch all summer long. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
How do I test my parallel trained model on one gpu?

username: SteveXWu
cooked: <p>I trained a customized CNN model by using<br>
<code>model = torch.nn.DataParallel(model, device_ids=[0, 1, 2, 3])</code><br>
and save the whole model <code>torch.save(model, "./ours_5.pkl")</code><br>
Now I would like to load this model and test it on single GPU <code>model_net = torch.load(path, map_location="cuda:3")</code>.<br>
It gives me a RuntimeError:  module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3</p>
<p>How do I fix this?</p>

username: wanchaol
cooked: <p><a class="mention" href="/u/stevexwu">@SteveXWu</a> Thanks for posting.</p>
<p>If you save your DataParallel model and want to load it in a different env, you need to properly define the mapping of devices using the <code>map_location</code> option in torch.save/load, you can check the following pointers and see if they resolve your issue.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints</a></p>
<p>btw, we recommend using DDP instead of DataParallel <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 1.11.0 documentation</a></p>
<p>You can also check this similar issue <a href="https://discuss.pytorch.org/t/load-ddp-model-trained-with-8-gpus-on-only-2-gpus/73455/12" class="inline-onebox">Load DDP model trained with 8 gpus on only 2 gpus? - #12 by kazem</a></p>

username: rfmac
cooked: <p>I think the model should be loaded in the first device from the <code>device_ids</code> list.<br>
If I’m right, you <strong>must</strong> do one of the following options:</p>
<ol>
<li>Retrain your model with device <span class="hashtag">#3</span> in the first position: <code>model = torch.nn.DataParallel(model, device_ids=[3, 0, 1, 2])</code>
</li>
<li>Load the model in the device <span class="hashtag">#0:</span> <code>model_net = torch.load(path, map_location="cuda:0")</code>
</li>
</ol>
<p>Assuming you don’t want to retrain just to change the device, I think the best option would be to load in the device <span class="hashtag">#0</span> and then to transfer to device <span class="hashtag">#3</span>, with something like:<br>
<code>model_net = torch.load(path, map_location="cuda:0").device("cuda:3")</code></p>
<p>I’m not sure if it works, it’s just a guess based on what is written in the official docs, but I hope it helps you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Best regards,<br>
Rafael Macedo.</p>
Is my understanding of kernel stride correct in this example?

username: jayz
cooked: <p>Hello,</p>
<p>I have a matrix:</p>
<pre><code class="lang-auto">A1 A2 A3 A4 A5 ... 
B1 B2 B3 B4 B5 ... 
C1 C2 C3 C4 C5 ... 
D1 D2 D3 D4 D5 ... 
.. .. .. .. ..
</code></pre>
<p>I would like to apply a 2d conv kernel of size (2,3) to this matrix. The kernel should jump the rows with steps of 2, i.e. in the first stride it should encompass row A and B, in the second stride step it should consider row C and D, just like below (the kernel is represented by [.] and the matrix after &gt;&gt; shows how the kernel moves from left to right; not necessary, but I had fun typing these matrices <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> ;)):</p>
<p>First “Stride Step”:</p>
<pre><code class="lang-auto">[A1 A2 A3] A4 A5 ...  &gt;&gt;  A1 [A2 A3 A4] A5 ...    
[B1 B2 B3] B4 B5 ...  &gt;&gt;  B1 [B2 B3 B4] B5 ...    
 C1 C2 C3  C4 C5 ...  &gt;&gt;  C1  C2 C3 C4  C5 ...    
 D1 D2 D3  D4 D5 ...  &gt;&gt;  D1  D2 D3 D4  D5 ...  
</code></pre>
<p>Second “Stride Step”:</p>
<pre><code class="lang-auto"> A1 A2 A3  A4 A5 ...  &gt;&gt;  A1  A2 A3 A4  A5 ... 
 B1 B2 B3  B4 B5 ...  &gt;&gt;  B1  B2 B3 B4  B5 ... 
[C1 C2 C3] C4 C5 ...  &gt;&gt;  C1 [C2 C3 C4] C5 ... 
[D1 D2 D3] D4 D5 ...  &gt;&gt;  D1 [D2 D3 D4] D5 ... 
</code></pre>
<p>In my application, it is crucial that the kernel does not combine any rows other than the pairs AB and CD (i.e. for instance not BC). The setting that I chose was simply stride = (2,1) with ‘valid’ padding (i.e. padding = 0). Just to ensure, are these the correct settings?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

username: Matias_Vasquez
cooked: <p>Hi, yes this is correct!</p>
<p>If you use the following code, you can define your own kernel and see what is happening when you do the convolution.</p>
<p>In my case I put a 1 on the lower right corner of the matrix, but you can change it to whatever you want. When you do this, you can see in the image below that you get your desired result.</p>
<p>On the x axis you only advance by 1, whereas on the y axis you only use non-overlapping rows.</p>
<p>Hope this helps!</p>
<pre data-code-wrap="python"><code class="lang-python">your_matrix = torch.randn(1, 1, 4, 6)

kernel = torch.tensor([[0., 0., 0.],
                       [0., 0., 1.]]).unsqueeze(0).unsqueeze(0)

conv = nn.Conv2d(1, 1, (2, 3), (2, 1), bias=False)
with torch.no_grad():
    conv.weight = nn.Parameter(kernel)


print(your_matrix)
print(conv(your_matrix))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7a4e44c91ffa2e3c06dbd3e45fc9fe04f77b012.png" alt="image" data-base62-sha1="x3dzDbTuRiDXr6R63ww4M7322bw" width="549" height="134"></p>

username: jayz
cooked: <p>very handy trick! thanks for that! Best, JZ</p>
Dataset from pandas without folder structure

username: Erik_Sthlm
cooked: <p>Hi,</p>
<p>I’m a complete beginner trying to do image classification. My image data and label data comes from two parquet files, one for images and one for labels. I convert them to a big pandas dataframe (30000 rows x 900 columns plus 30000 rows x 1 column, where each row represents a 30 x 30 picture). However all the dataset examples I find use pictures stored in a tree/folder structure. How can I adapt to my case? Have anyone done it this way?</p>
<p>Regards,</p>
<p>Erik</p>

username: nivek
cooked: <p>You can create a custom <code>Dataset</code> with a <code>__getitem__</code> method that reads from your pandas dataframe.</p>
<p>The <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" rel="noopener nofollow ugc">example in this tutorial</a> may be helpful, replace the part of that is reading from file system with reading from your pandas dataframe instead.</p>
<p>Subsequently, you can pass that custom <code>Dataset</code> into <code>DataLoader</code> and begin your training.</p>

username: Erik_Sthlm
cooked: <p>Hi, thank you for replying. I have read the pytorch tutorial already. However I cannot figure out how to change the custom dataset class to suit my original data, as all the image and label data of different kinds are stored together in one giant dataframe.</p>

username: nivek
cooked: <p>Do you have additional context to provide? The basic idea is that when your <code>Dataset</code> receives an index, you want to read something from the pandas DataFrame and return a sample. For example:</p>
<pre><code class="lang-python">import numpy as np
import pandas as pd
from torch.utils.data import Dataset, DataLoader


class CustomDataset(Dataset):
    def __init__(self, dataframe):
        self.dataframe = dataframe

    def __getitem__(self, index):
        row = self.dataframe.iloc[index].to_numpy()
        features = row[1:]
        label = row[0]
        return features, label

    def __len__(self):
        return len(self.dataframe)


df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
                  columns=['label', 'feature_0', 'feature_1'])
data = CustomDataset(dataframe=df)
dataloader = DataLoader(data)
for sample in dataloader:
    print(sample)
</code></pre>

username: Erik_Sthlm
cooked: <p>Thank you for your code example, I find it easier to understand than the tutorial. I’ll try to provide more details later.</p>
How to invoke default gradient update of layer from inside an autograd function

username: iordanis
cooked: <p>As a <strong>non-running</strong> but conceptual example I am providing the following function.</p>
<p>I am trying to update that network’s parameters using default autograd but only with respect to that function’s i/o.</p>
<pre><code class="lang-python">class Square(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, net: nn.Module):

        ctx.save_for_backward(x)
        x=net(x.detach())
        return x**2


    @staticmethod
    def backward(ctx, grad_out):
        (x,) = ctx.saved_tensors
        net.backward(x)
        x=net(x)
        return grad_out * 2 * x
</code></pre>
Multi-gpu training of GANs

username: Israr_Bacha
cooked: <p>i have class for single gpu like:<br>
class mmodel():</p>
<pre><code class="lang-auto">def setgpu(self, gpu)
    self.gpu = gpu
    self.disA.cuda(self.gpu)
    self.disB.cuda(self.gpu)
   self.disA2.cuda(self.gpu)
   self.disB2.cuda(self.gpu)
   self.enc_c.cuda(self.gpu)
   self.enc_a.cuda(self.gpu)
   self.gen.cuda(self.gpu)
</code></pre>
<p>Now in main function:<br>
model = mmodel(opts)<br>
model.setgpu(opts.gpu)</p>
<p>I want to run this model on 2 GPU’s anyone can help me? do i need to change the batch size? my batch size is 2:</p>

username: ptrblck
cooked: <p>If you would like to use data parallel, you could simply warp your model to <code>nn.DataParallel</code> or <code>nn.DistributedDataParallel</code>. Have a look at <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">this tutorial</a> for more information.</p>

username: BRIJESH_SINGH
cooked: <p>What if I have to sub-models, say model1 and model2. Do I need to write as follows?<br>
model1 = nn.DataParallel(model1)<br>
model2 = nn.DataParallel(model2)<br>
…<br>
…</p>

username: ptrblck
cooked: <p>If both submodules are registered into the same parent module, you could call <code>nn.DataParallel</code> just on the panel. If not, then your approach should work.</p>