akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get name of files while loading images in Pytorch

Sina_Mirshahi: <p>I am loading images through DataLoader, to test them and get the accuracy, but the problem is Dataloader does not give me the file names so I understand which file was classified write and which one was wrong. how can I get the names? you can see the function I used</p>
<p>def get_all_images(data_location):<br>
data = datasets.ImageFolder(data_location, transform=test_transforms)<br>
classes = data.classes<br>
loader = torch.utils.data.DataLoader(data,batch_size = len(data))<br>
dataiter = iter(loader)<br>
images, labels = dataiter.next()<br>
return classes,images, labels</p>

someshfengde: <p>create custom Dataset object with help of Dataset class<br>
this code snippet might help you</p>
<pre><code class="lang-auto">import os
import pandas as pd
from torchvision.io import read_image

class CustomImageDataset(Dataset):
    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_csv(annotations_file)
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label
</code></pre>
<p>source for learning more about dataloader class<br>
<a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files</a></p>

Sina_Mirshahi: <p>Thank you for the reply and help, however can I do this instead?</p>
<p>to have “filenames = data.imgs”</p>
<p>def get_all_images(data_location):<br>
data = datasets.ImageFolder(data_location, transform=test_transforms)<br>
classes = data.classes<br>
filenames = data.imgs<br>
loader = torch.utils.data.DataLoader(data,batch_size = len(data))<br>
dataiter = iter(loader)<br>
images, labels = dataiter.next()<br>
return filenames,classes,images, labels</p>

someshfengde: <p>yes you can but It won’t be eficient<br>
making custom datasets for using them with dataloader is more efficient <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=10" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
CrossEntropyLoss Problem for multi-class classification

Thiru: <p>I’m trying to train a classification model: multi-class classification<br>
The loss function used is <strong>nn.CrossEntropyLoss()</strong></p>
<p>The batch size is 8 and the number of classes to classify is 6.</p>
<p><strong>The target labels aren’t one-hot encoded</strong> - (Checked it)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fd2dbf154f978fe3ea4459e760e80fba19e41c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fd2dbf154f978fe3ea4459e760e80fba19e41c4" title="Screenshot from 2021-11-02 09-43-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd2dbf154f978fe3ea4459e760e80fba19e41c4_2_690x388.png" alt="Screenshot from 2021-11-02 09-43-59" data-base62-sha1="2fYVopkHP0rBC85Tm6aDRzDsfeQ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd2dbf154f978fe3ea4459e760e80fba19e41c4_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd2dbf154f978fe3ea4459e760e80fba19e41c4_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd2dbf154f978fe3ea4459e760e80fba19e41c4_2_1380x776.png 2x" data-dominant-color="2D2E28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-11-02 09-43-59</span><span class="informations">1920×1080 361 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This image shows how the target is read from the dataframe and the shape of the target is printed in terminal.</p>

ptrblck: <p>The target tensor seems to have an additional dimension and its shape looks like <code>[batch_size, 1]</code>. Remove <code>dim1</code> for a multi-class classification via <code>label = label.squeeze(1)</code> and it should most likely work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

Thiru: <p>Thank you I figured it out</p>
How can I get scheduled graph which minimizes memory usage?

ares: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7bca04c2793cf2724195fcc81eed585b55ac018.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7bca04c2793cf2724195fcc81eed585b55ac018" title="screen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7bca04c2793cf2724195fcc81eed585b55ac018_2_690x382.png" alt="screen" data-base62-sha1="qdpvq3YQyc4mHwOiQyRd1eC6OO4" width="690" height="382" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7bca04c2793cf2724195fcc81eed585b55ac018_2_690x382.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7bca04c2793cf2724195fcc81eed585b55ac018_2_1035x573.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7bca04c2793cf2724195fcc81eed585b55ac018.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">screen</span><span class="informations">1217×674 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hi, Everyone!</p>
<p>I’m attempting to optimize computation graph in Python with Glow. I have CPU and GPU devices. So,</p>
<ol>
<li>I want to get ops schedule that Glow creates and also I want to change it;</li>
<li>I want to obtain kernel fusing that Glow builds and I want to manage it;</li>
<li>I want to see where each operation is performed (on CPU or on GPU) and I want to control it;</li>
</ol>
<p>This is what I tried to do. I took a simple computation graph from <a href="https://github.com/pytorch/glow/blob/master/torch_glow/examples/basic_example.py" rel="noopener nofollow ugc">https://github.com/pytorch/glow/blob/master/torch_glow/examples/basic_example.py</a> :<br>
<strong>def foo(a, b):</strong><br>
**    c = a.mul(b)**<br>
**    a = c.mul(c)**<br>
**    a = c.mul(a)**<br>
**    d = c.div(a)**<br>
**    return d**</p>
<p>I got (as I understood) original schedule and tensor placement (without Glow) of this graph, like on screen, by this line:<br>
<strong>foo.graph_for(x, y)</strong></p>
<p>But when I launch Glow (as I understood, it is launched by the line <strong>torch_glow.enableFusionPass_DO_NOT_USE_THIS()</strong>) I obtain error:<br>
<strong>module ‘glow.torch_glow’ has no attribute ‘enableFusionPass_DO_NOT_USE_THIS’</strong></p>
<p>Please, tell me how I can find out at least something about 1), 2) or 3).<br>
I would be grateful for any advice because I’m new to Glow:)</p>
Ensemble Trained Segmentation Models

Giulia_Carvalhal: <p>Hey community,</p>
<p>I would like to ensemble some segmentation trained models pre-trained with slightly different datasets and create a new segmentation model with those. How can I do that? I saw a few examples with classifier problems, but I did not see any with segmentation.</p>
<p>Best Regards,<br>
Giulia</p>

JamesDickens: <p>Do you mean like the same framework (like a DeepLabv3 or PSPNet) pre-trained on different datasets, followed by fine-tuning all of them on a shared dataset? For instance if you pre-trained on Pascal VOC 2012 with one model, and MS COCO Stuff with another, then they have different classes, and hence different network head weight shapes, but if you want to only use the classes (20 or so) from Pascal VOC, then you could do this, testing on either Pascal VOC or MS COCO stuff filtering the categories of the latter to be the desired 20.</p>
<p>Then there’s the question of how you combine the predictions. I guess you could simply weight the logits from each model at inference time, and use an argmax operation computed pixel-wise across classes.</p>

JamesDickens: <p>I notice a lot of the contest winners in semantic segmentation use ensemble models, so if you wanna check out the technical reports of the leaderboards for MS COCO Stuff or Pascal VOC 2012 or Cityscapes, I bet you could find some generic descriptions of how they do it. Often they train different models for different image scales, and then at test time resort to multi-scale predictions by averaging the interpolated logits to the desired output size, i.e. model A predicts a mask for an image of size 600 times 600, model B for an image of size 800 times 800, etc… then you need a final prediction of size 500 times 500 which you can get by bilinear/bicubic interpolation to that final size, and average the prediction values at each pixel.</p>

Giulia_Carvalhal: <p>Thank you for your reply,</p>
<p>Actually, I am dealing with the UNet model for multiclass segmentation task. The datasets are quite the same, same size and pixels, only the grid change a little bit. I have 25 types of ground-truth/mask and 4 different shapes of tumour that I can add to it. But I can not repeat the ground-truth types. I was thinking to use the ensemble approach because my dataset is very small (n=25). I saw this paper <a href="https://arxiv.org/abs/1711.01468" class="inline-onebox" rel="noopener nofollow ugc">[1711.01468] Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation</a> discussing it, but I did not fully understand.</p>
<p>Best,<br>
Giulia</p>

JamesDickens: <p>What is your dataset?</p>

Giulia_Carvalhal: <p>Medical images, breast phantoms.</p>
Dataloader's split

bingqing_liu: <p>how can i split the dataloder.</p>
<pre><code class="lang-auto">dataloader=tud.DataLoader(dataset,batch_size=batch_size,shuffle=True)
</code></pre>
<p>i want to split it into 3 parts</p>

someshfengde: <p>I assume you have to split your dataset into train test and valid format</p>
<p>for that instead of doing it with the dataloader you have to do it beforehand</p>
<pre><code class="lang-auto"># scikit learns train test split 
from sklearn.model_selection import train_test_split

# I assume you are having some x , y formatted labels 
X_train , y_train = train_test_split(X,y)
# here you can also give out percentage of train and test split 
# you have default as 80% of data for train and 20% of data for test 

</code></pre>
<p>more about train test split</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">
  <header class="source">
      <img src="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html/../../_static/favicon.ico" class="site-icon" width="" height="">

      <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" target="_blank" rel="noopener nofollow ugc">scikit-learn</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" target="_blank" rel="noopener nofollow ugc">sklearn.model_selection.train_test_split</a></h3>

  <p>Examples using sklearn.model_selection.train_test_split: Release Highlights for scikit-learn 0.23 Release Highlights for scikit-learn 0.23, Release Highlights for scikit-learn 0.24 Release Highligh...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>splitting the dataset into train_test_and split beforehand is good way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
enjoy coding <img src="https://discuss.pytorch.org/images/emoji/apple/computer.png?v=10" title=":computer:" class="emoji" alt=":computer:"></p>
How to load .pt in C++/windows

zhangkun: <p>dear all</p>
<p>i am using yolov5. the .pt is trained in python. now i want to apply the .pt in c++/windows. how to do?</p>
<p>i have tried the demo in <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#depending-on-libtorch-and-building-the-application" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 1.10.0+cu102 documentation</a>. however, it does not work.</p>
<p>thanks</p>

zhangkun: <p>i have load .pt in c++. the .pt should be transformed.</p>
Can't install latest pytorch version with pip

MaltheG: <p>Hey</p>
<p>I’m trying to install the latest pytorch version using pip.</p>
<p>I am running the following command as described on the pytorch webpage:</p>
<pre><code class="lang-auto">pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
</code></pre>
<p>However, I get the following error:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cu113 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.10.0+cu113
</code></pre>
<p>Is this a problem on my end or something with the latest relase?</p>
<p>I am using Python 3.10 with pip 21.3.1 on Windows 10</p>

MaltheG: <p>I fixed the issue by downgrading to python version 3.9.7. I guess pytorch isn’t upgraded for 3.10 yet</p>
tensorboard/tensorboardX not showing all steps

yegane: <p>I am making logs with tensorboardX on my training, and I am trying to visualize it from tensorboard, but it only shows it up to certain point. I am using tensorboard 2.7, and my system is ubuntu 18. I have changed from Windows to Ubuntu, and in windows I was able to see all the steps. I have 66k steps uptil now, but I can only see it 11,039. I used also this tag: <code>--max_reload_threads 4</code> , and <code>--samples_per_plugin images=99999999,</code> and <code>--samples_per_plugin scalars=999999999</code> but it didn’t help. My logfile is nearly 4GB. I had no problem loading 13GB logfile in Windows with the same system. I also disabled --load_fast=false, and now it shows 10,799 steps, and it display this error each few seconds: <code>E1102 13:43:21.874801 139799442822912 plugin_event_multiplexer.py:235] Unable to reload accumulator '.': [Errno 5] Input/output error</code></p>
Pytorch Vs Keras - pytorch accuracy a lot lower

David_Draper: <p>I am trying to learn PyTorch just using a basic sequential model and comparing against keras - I am getting a widely different accuracy on the titanic dataset.</p>
<p>“”“model_2 = torch.nn.Sequential(torch.nn.Linear(7, 16),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(16, 16),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(16, 1),<br>
torch.nn.Sigmoid())”""</p>
<p>Keras:<br>
“”“model = Sequential()<br>
model.add(Dense(16, activation=‘relu’, input_shape=(7,)))<br>
model.add(Dense(16, activation=‘relu’))<br>
model.add(Dense(1, activation=‘sigmoid’))”""</p>
<p>colab notebook:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1mPRmk0DU3W6pyO2ZV1zw63pdUxB6t3xH#scrollTo=62X_pAxe9-rc">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211029-060057-RC01_406381293" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1mPRmk0DU3W6pyO2ZV1zw63pdUxB6t3xH#scrollTo=62X_pAxe9-rc" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1mPRmk0DU3W6pyO2ZV1zw63pdUxB6t3xH#scrollTo=62X_pAxe9-rc" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please be nice still learning!</p>

ptrblck: <p>I can’t run the notebook, as the data is missing.<br>
However, based on the code I’m not sure how the learning rate is set in Keras.<br>
In PyTorch you are using <code>lr=0.0001</code>, while I guess Keras might be using their default of <code>0.001</code> as given <a href="https://keras.io/api/optimizers/adam/">here</a>?<br>
If so, change it in PyTorch to the same value.</p>
<p>Also, I would recommend to remove the <code>sigmoid</code> and use <code>nn.BCEWithLogitsLoss</code> for a better numerical stability.</p>

David_Draper: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you - I missed that I thought it was 0.0001 as a default and that has brought the accuracy up. Will test with Logits now! Thanks again!</p>
Which can speed up?

bingqing_liu: <p>i want to ask which one is better in the aspect of speed.</p>
<ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">device=torch.device("cuda")
class fc_layer(nn.Module):
    def __init__(self):
        super(fc_layer,self).__init__()
        self.linear1=nn.Linear(emb_size,emb_size)
    def forward(self,x):
        y=self.linear1(x)
        a=np.ones((2,3))
        a=torch.tensor(a,dtype=torch.float32).to(device)
        y=y+a
        return y
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">device=torch.device("cuda")
class fc_layer(nn.Module):
    def __init__(self):
        super(fc_layer,self).__init__()
        self.linear1=nn.Linear(emb_size,emb_size)
    def forward(self,x):
        y=self.linear1(x)
        a=torch.ones(2,3).to(device)
        y=y+a
        return y
</code></pre>
<ol start="3">
<li>
</li>
</ol>
<pre><code class="lang-auto">device=torch.device("cuda")
class fc_layer(nn.Module):
    def __init__(self):
        super(fc_layer,self).__init__()
        self.linear1=nn.Linear(emb_size,emb_size)
    def forward(self,x):
        y=self.linear1(x)
        a=torch.ones(2,3,device=device)
        y=y+a
        return y
</code></pre>
<p>by the way, is there any faster ways?</p>

ptrblck: <ol>
<li>Will use two copies: when creating the tensor (as you are not using <code>from_numpy</code>) and the data transfer to the GPU</li>
<li>Only the transfer</li>
<li>Will create the tensor on the device directly, so I would use this approach.</li>
</ol>
<p>Of course you won’t notice a significant difference given the current shapes, but I assume that this code snippet is only to show the different approaches.</p>

bingqing_liu: <p>okay,thank you.Concerning the speed,i have one more question:</p>
<pre><code class="lang-auto">class net(nn.Module):
    def __init__(self):
        super(net,self).__init__()
        self.linear=nn.Linear(2,2)
    def forward(self,x):
        for i in range(2):
            x=self.linear(x)
        return x
</code></pre>
<p>in the above code <code>for i in range(2)</code>,the loop is controlled by cpu or gpu?if cpu,for the sake of  speed,should i use the following way?i wonder if there is a special controller in GPU to do some easy control,so cpu can be free of computation in most time rather than control gpu all the time?</p>
<pre><code>def forward(self,x):
    x=self.linear(x)
    x=self.linear(x)
    return x</code></pre>

tom: <p>While there is the new (in torch 1.10) cuda graphs API to deal with the very case, I would not worry about it too much until you find your GPU is underutilized from you using Python for loops.</p>
<p>The Python overhead is more or less constant per operation. The <em>Thomas rule of thumb</em> for this is that if your operands have hundreds of elements, the Python overhead is likely not a crucial bottleneck. On the other hand, if you have many operands that are much smaller, the mere Tensor administration overhead (in C++, too) is likely to be a problem, too.</p>
<p>Best regards</p>
<p>Thomas</p>
Clipping parameter values during training/ testing

Afradi: <p>I’m working on GAN, There are upper and lower bounds in the pixel values of the images.  I have applied clipping and normalization during the training, Should I want to clip or  normalize them during the testing as well?</p>

tom: <p>So it is hard to comment on very concrete situations from somewhat more general descriptions, but here are some thoughts:</p>
<p>In general, it is a good idea to keep the difference between training and testing smallish.</p>
<p>In practice, the two most notable exceptions are</p>
<ul>
<li>dropout, which has an interpretation as taking a mean during evaluation where during training we sample from a distribution,</li>
<li>and batch norm, where we don’t like the discrepancy between training and testing, so for example transformers use layer norm which, among other things, eliminates the train-test discrepancy even if it make evaluation quite more costly.</li>
</ul>
<p>For other things, like spectral norm, we do don’t do all the compute as during training, but make sure the same weight is used in evaluation as in training.</p>
<p>From your description, it would sound like you are more in the spectral norm-like case. If you actually clipped/normalized weights, they might keep that so you would not need to do it over and over again, if you worked on the activations (intermediate results of the computation), you would keep it.</p>
<p>Best regards</p>
<p>Thomas</p>

Afradi: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="135571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>spectral norm</p>
</blockquote>
</aside>
<p>Thank you for your suggestions. I will try them…</p>
Eval mode: feed different inputs, get the same outputs

marsggbo: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d550ec8eb9bb5d215eb7b03bbe3bf9aca216f07.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d550ec8eb9bb5d215eb7b03bbe3bf9aca216f07" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d550ec8eb9bb5d215eb7b03bbe3bf9aca216f07.png" alt="image" data-base62-sha1="b2702nTFdkkvgwlCSfn8viwLkI7" width="271" height="500" data-dominant-color="2E2F28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">356×656 6.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/768793302654b02d288f526fd82bd5268353b264.png" data-download-href="https://discuss.pytorch.org/uploads/default/768793302654b02d288f526fd82bd5268353b264" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/768793302654b02d288f526fd82bd5268353b264.png" alt="image" data-base62-sha1="gUyObLaFIzBnh6TD8HMZ7bIi9da" width="266" height="500" data-dominant-color="2E3029"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">332×623 6.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I change the model to <code>eval</code> mode, the class prediction will be the same.  I tested <code>resnet18</code> and <code>resnet50</code>.</p>
<p>Any explanation for this result? In my own experiment, I always get the 10% evaluation accuracy on the CIFAR-10 dataset if I set <code>eval</code> mode.</p>
Dataset: Loading from the same file with more than a single thread

Maximal: <p>Hey,</p>
<p>I’d like to write a dataset which reads examples from one single binary file and use it in combination with the standard Dataloader with more than a single worker.<br>
I tried to simply inherit my custom Dataset torch.utils.data.Dataset like this:</p>
<pre><code>class MyDataset(torch.utils.data.Dataset):
    def __init__(self):
        self.data_file = open('data.bin', "rb")
        self.transform = ....

    def __getitem__(self, index):
        self.data_file.seek(index * NUM_BYTES)
        data = self.data_file.read(NUM_BYTES)
        data_np = np.fromstring(data, dtype='uint8').reshape(DATA_SIZE, order="F")
        data_transformed = self.transform(data_np)
        return data_transformed , 0 
</code></pre>
<p>This works fine for a single thread, but when I set the number of workers larger than 1, some of the data is just wrong. It almost appears to be the case that the data loader is mixing different data points. I guess this is due to the fact that read and seek are not a single atomic operation. At first I simply tried to use the multiprocessing Lock, so one thread would acquire the lock before seek and release if after the read, but this didn’t really change anything.<br>
I can obviously also move open into getitem, in which case everything works fine with multiple workers, but open is actually quite slow in comparison to seek and read, so I’d prefer not do it like this<br>
So how would I correctly synchronize different workers?<br>
Thanks!</p>

Frank-Jing: <p>When using multi workers to load data, you’d better to reserve file-descriptor in workers(i.e in <strong>getitem</strong>() method), not in main process(i.e. in <strong>init</strong>() method).<br>
You can refer this post(and it’s reply): <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/14" class="inline-onebox">DataLoader, when num_worker &gt;0, there is bug - #14 by piojanu</a></p>
Libtorch tensor.to(torch::kCPU) took a lot of time

x1290148: <p>I spent a lot of time converting tensor to mat.<br>
My algorithm time is only 20ms, while the conversion takes 80ms.</p>
<pre><code class="lang-auto">cv::Mat tensor2Mat(torch::Tensor &amp;i_tensor)
{
	int height = i_tensor.size(0), width = i_tensor.size(1);
	i_tensor = i_tensor.to(torch::kCPU);
	cv::Mat o_Mat(cv::Size(width, height), CV_32F, i_tensor.data_ptr());
	return o_Mat;
}
</code></pre>
<p>Is there any other solution?<br>
CPU:Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz<br>
GPU:1080Ti<br>
Windows 10</p>

ptrblck: <p>Pushing a CUDATensor to the CPU is a synchronizing operation in the default setup and will thus accumulate the time unless you manually synchronize the code.<br>
You could try to use the <code>non_blocking</code> option, but would still need to wait until the data transfer is done and based on your code you are directly using the CPUTensor afterwards, so you wouldn’t gain any perf. improvements.</p>
Training multiple pytorch models concurrently leading to longer training time for each model

jzhao004: <p>I’m currently training two variations of the following model (<a href="https://github.com/TengdaHan/DPC" class="inline-onebox" rel="noopener nofollow ugc">GitHub - TengdaHan/DPC: Video Representation Learning by Dense Predictive Coding. Tengda Han, Weidi Xie, Andrew Zisserman.</a>).</p>
<p>I’ve noticed that when training both models concurrently (each on a dedicated GPU), the training time for each model increases by ~25%, as compared to when training only one model at a time.</p>
<p>Currently, GPU usage is &gt;90% for both models.CPU usage is ~60% and RAM usage is ~25% when both models are trained concurrently.</p>
<p>Here are the profiling results (sorted by total CPU time, total CUDA time, CPU memory usage, and CUDA memory usage).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee3c9328f5939264af42fd6ef378b2bc6902e77c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee3c9328f5939264af42fd6ef378b2bc6902e77c" title="Profiler"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee3c9328f5939264af42fd6ef378b2bc6902e77c_2_474x500.png" alt="Profiler" data-base62-sha1="xZxpYwwTbxwxdNPWzbTvT97zxDe" width="474" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee3c9328f5939264af42fd6ef378b2bc6902e77c_2_474x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee3c9328f5939264af42fd6ef378b2bc6902e77c_2_711x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee3c9328f5939264af42fd6ef378b2bc6902e77c_2_948x1000.png 2x" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Profiler</span><span class="informations">2008×2118 2.05 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your screenshots are a bit hard to decipher, however based on some values I could read it seems that the CUDA operations are using approx. the same amount of time, while CPU ops are a bit slower.<br>
This could point towards a CPU-bound workload and the host might not be fast enough to run ahead and schedule the GPU work fast enough if you start multiple processes.</p>

jzhao004: <p>Thank you for your reply! I’ve updated the screenshot to a higher resolution version.</p>
How to install torchrun?

Olivier-CR: <p><a href="https://pytorch.org/docs/stable/elastic/run.html#" rel="noopener nofollow ugc">This doc</a> encourages to use <code>torchrun</code>. But doesn’t tell how to install it <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"><br>
How to install and get started with <code>torchrun</code>?</p>

cbalioglu: <p><code>torchrun</code> is part of PyTorch v1.10. If you are running an older version,  <code>python -m torch.distributed.run</code> command serves the same purpose.</p>

Olivier-CR: <p>thanks <a class="mention" href="/u/cbalioglu">@cbalioglu</a> ! nice to meet you here, long time no see <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

Olivier-CR: <p><a class="mention" href="/u/cbalioglu">@cbalioglu</a> shall I run <code>python -m torch.distributed.run</code> once for the whole cluster? or once per node? or once per GPU? also asked here <a href="https://discuss.pytorch.org/t/how-to-map-processes-to-gpu-in-ddp-and-how-to-launch-the-ddp-cluster/135253" class="inline-onebox">How to map processes to GPU in DDP and how to launch the DDP cluster?</a></p>

mrshenli: <p>It should be once per node IIUC</p>
How to differentiabl-y alter weights in a torch.nn.Module

DonnyHe: <p>hey i have a question that might be weird.</p>
<p>given a model, either initialized with some weights or pretrained, I want to do the following:</p>
<ol>
<li>access weight of torch.nn.Module model m, <em>w0</em>
</li>
<li>
<em>w1 = f(w0, u)</em>, do modification of weight based on aux var <em>u</em>
</li>
<li>compute grad w.r.t <em>u</em>, namely <em>dL(m(w1)))/dx</em>
</li>
</ol>
<p>Neither</p>
<ul>
<li>model.state_dict<br>
or</li>
<li>model.parameters(),</li>
</ul>
<p>works, because state_dict does not allow gradient to pass through, and you can’t assign value to parameters in model.parameters() as model.parameters() are rather copies of the parameter, assign to them does not change the value of parameters used by the model.</p>
<p>any help would be appreciated, thanks in advance!</p>
How to freeze part of the pretrained resnet50 using pytorch

nikiguo93: <p>Hi how can I freeze part of a pretrained resnet50 and trained the remaining part of the net using my custom dataset?</p>

ptrblck: <p>You could iterate the parameters you would like to freeze and set their <code>.requires_grad</code> attribute to <code>False</code>:</p>
<pre><code class="lang-python">for name, param in model.layer1.named_parameters():
    param.requires_grad = False

for name, param in model.layer2.named_parameters():
    param.requires_grad = False

...
</code></pre>
<p>Depending how the model was created you could also directly use submodules as <code>model.submodule.prameters()</code>.</p>
Saving bert features

Omama: <p>I have two Bert models. I detached the classifier layer and kept the pre_classifier and drop out layers. Then I created an ensamble that has the classifier layer.</p>
<p>So the output of the two BERT models will be concatenated on the ensamble, then will be input to the classifier layer.</p>
<p>The problem is that I’m getting errors about running out of memory. So how can I save the output of each model to the csv file, then concatenate the two files and run the ensamble model for classification?</p>

ptrblck: <aside class="quote no-group" data-username="Omama" data-post="1" data-topic="135615">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/ad7895/48.png" class="avatar"> Omama:</div>
<blockquote>
<p>So how can I save the output of each model to the csv file, then concatenate the two files and run the ensamble model for classification?</p>
</blockquote>
</aside>
<p>If you store the outputs, load them afterwards and train the classifier, this might not be an ensemble model training anymore, since you would only train the classifier (but it also depends on your definition of ensemble).<br>
If that’s your use case, then you could either <code>detach()</code> the outputs from both BERT models or use <code>torch.save</code> to store the activations (with their target).</p>
<p>On the other hand, if you want to train the model end-to-end, you would need to keep both BERT models, their computation graph, as well as the classifier in memory.</p>
Prevent updating weight for specific element of weight matrix

afsaneh_ebrahimi: <p>I’m trying to set one element of weight to 1 and then hold it until the end (prevent it from updating in the next epochs). I know I can set requires_grad but I just want it for one element?</p>

KFrank: <p>Hi Afsaneh!</p>
<aside class="quote no-group" data-username="afsaneh_ebrahimi" data-post="1" data-topic="135661" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/afsaneh_ebrahimi/48/39738_2.png" class="avatar"> afsaneh_ebrahimi:</div>
<blockquote>
<p>I know I can set requires_grad but I just want it for one element?</p>
</blockquote>
</aside>
<p>The short answer is to set the specific element in question back to<br>
1 after calling <code>opt.step()</code>.</p>
<p>As you’ve recognized, <code>requires_grad</code> applies to an entire tensor,<br>
not separately to individual elements.  It is also true that an <code>Optimizer</code><br>
applies to entire tensors, and not to individual elements.  There are<br>
a number of approaches to “freezing” individual elements of a tensor<br>
that is otherwise being optimized, but simply resetting the desired<br>
values after running <code>opt.step()</code> is probably the most straightforward.</p>
<p>Best.</p>
<p>K. Frank</p>
Kernel size in unit

Aleena_Suhail: <p>if I try to change the kernel size of cnn in the following code it gives me error maybe because the dimensions of skip connection does not match x, How can I amend the code for kernel size 4</p>
<p>import torch<br>
import torch.nn as nn<br>
import torchvision.transforms.functional as TF</p>
<p>class DoubleConv(nn.Module):<br>
def <strong>init</strong>(self, in_channels, out_channels):<br>
super(DoubleConv, self).<strong>init</strong>()<br>
self.conv = nn.Sequential(<br>
nn.Conv2d(in_channels, out_channels, 4, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(out_channels, out_channels, 4, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
)</p>
<pre><code>def forward(self, x):
    return self.conv(x)
</code></pre>
<p>class UNET(nn.Module):<br>
def <strong>init</strong>(<br>
self, in_channels=3, out_channels=1, features=[64, 128, 256, 512],<br>
):<br>
super(UNET, self).<strong>init</strong>()<br>
self.ups = nn.ModuleList()<br>
self.downs = nn.ModuleList()<br>
self.pool = nn.MaxPool2d(kernel_size=2, stride=2)</p>
<pre><code>    # Down part of UNET
    for feature in features:
        self.downs.append(DoubleConv(in_channels, feature))
        in_channels = feature

    # Up part of UNET
    for feature in reversed(features):
        self.ups.append(
            nn.ConvTranspose2d(
                feature*2, feature, kernel_size=2, stride=2,
            )
        )
        self.ups.append(DoubleConv(feature*2, feature))

    self.bottleneck = DoubleConv(features[-1], features[-1]*2)
    self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

def forward(self, x):
    skip_connections = []

    for down in self.downs:
        x = down(x)
        skip_connections.append(x)
        x = self.pool(x)

    x = self.bottleneck(x)
    skip_connections = skip_connections[::-1]

    for idx in range(0, len(self.ups), 2):
        x = self.ups[idx](x)
        skip_connection = skip_connections[idx//2]

        if x.shape != skip_connection.shape:
            x = TF.resize(x, size=skip_connection.shape[2:])

        concat_skip = torch.cat((skip_connection, x), dim=1)
        x = self.ups[idx+1](concat_skip)

    return self.final_conv(x)
</code></pre>
<p>def test():<br>
x = torch.randn((2, 1, 512, 512))<br>
model = UNET(in_channels=1, out_channels=1)<br>
preds = model(x)<br>
print(preds.shape)<br>
print(x.shape)<br>
assert preds.shape == x.shape</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
test()</p>
Training GANs using automatic mixed precision？

hlkzh: <p>Hi, guys.<br>
I encountered an overflow problem when deploying the GAN model(trained in fp32) to fp16 precision. And I decided to give it a try after seeing <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples" rel="noopener nofollow ugc">amp</a>. I’m wondering how G and D are trained in amp context. (Only G needs to be deployed in fp16 while D is only used when training)</p>

ptrblck: <p>The linked doc shows an example how to use <code>amp</code> for <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers">multiple models, losses, and optimizers</a>.<br>
Did you follow these steps or did you use a custom approach?</p>
How to set nn.conv2d weights

ramin272: <p>For some predefined operation i need to fix the parameters of a convolution  layer.<br>
when I have this layer</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(1, 5, kernel_size=1, stride=1, padding=0,  bias=False)
</code></pre>
<p>the code works fine, however when I set the weights of constitutional filter  as</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(1, 5, kernel_size=1, stride=1, padding=0,  bias=False)
self.conv1.weight = torch.nn.Parameter(torch.ones((1, 1, 5)))
</code></pre>
<p>I received following error</p>
<blockquote>
<p>RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]</p>
</blockquote>
<p>Any idea how can i solve this problem?</p>

LeviViana: <p>The dimensions are not correct: you are assigning a <code>[1, 1, 5]</code> tensor to the weights, whereas <code>self.conv1.weight.size()</code> is <code>torch.Size([5, 1, 1, 1])</code>.</p>
<p>Try:</p>
<pre><code class="lang-python">self.conv1.weight = torch.nn.Parameter(torch.ones_like(self.conv1.weight))
</code></pre>
<p>and it will work !</p>

G.M: <p>A great way to know what the shapes of the weight should be is to print the shape of the weights.</p>

fn246.c: <p>I want to <strong>set and fix</strong> weights of nn.conv1d, so this layer in network has fixed parameters and is <strong>NOT learnable</strong>.<br>
Is        self.conv1.weight = torch.nn.Parameter(torch.ones_like(self.conv1.weight))<br>
makes the weights fixed?</p>

ptrblck: <p>No, <code>nn.Parameter</code>s require gradients by default, so you would need to set its <code>requires_grad</code> attribute to <code>False</code>.</p>

fn246.c: <p>I want to set <strong>initial weights</strong> and they start learning from that initial weights, so is it true to use bellow script:<br>
self.conv1.weight = torch.nn.Parameter(torch.ones_like(self.conv1.weight),requires_grad=<strong>True</strong>)<br>
?</p>

afsaneh_ebrahimi: <p>Is there any way to set requires_grad=False just for one element of weight?</p>

ptrblck: <p>No, that’s not possible as you can change the <code>requires_grad</code> attribute for an entire tensor only.<br>
An alternative approach would be to either set the gradients to zero for the desired elements after the <code>backward()</code> operation and before the <code>step()</code> call or to recreate the parameter from different tensors (which use different <code>requires_grad</code> attributes) via <code>torch.cat</code> or <code>torch.stack</code>. However, the latter approach might be cumbersome depending how complicated the tensor creation would be.</p>
Triangular_solve transpose keyword argument

gncs: <p>The keyword argument <code>transpose</code> of the <code>triangular_solve</code> function is confusing me. In the document, it says:</p>
<blockquote>
<p><strong>transpose</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a><em>,</em> <em>optional</em>) – whether A should be transposed before being sent into the solver.</p>
</blockquote>
<p>In the code below, <code>x_1</code> makes sense to me, but <code>transpose</code> doesn’t seem to be doing anything in <code>x_2</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

u = torch.tensor([
    [2.0, 0.0],
    [4.0, 3.0],
])

x = torch.tensor([
    [2.0],
    [3.0],
])

b = u.T @ x

x_1 = torch.triangular_solve(b, u.T)
print(x_1.solution)

x_2 = torch.triangular_solve(b, u, transpose=True)
print(x_2.solution)

x_3 = torch.triangular_solve(b, u, transpose=False)
print(x_3.solution)
</code></pre>
<p>Output:</p>
<pre data-code-wrap="python"><code class="lang-python">tensor([[2.],
        [3.]])
tensor([[8.],
        [3.]])
tensor([[8.],
        [3.]])
</code></pre>
<p><code>torch==1.10.0</code> and <code>numpy==1.21.3</code></p>

tom: <p>I think you want <code>upper=False</code> when you pass in <code>u</code> (as opposed to <code>u.T</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

gncs: <p>Thank you for looking into this, Thomas.</p>
<ol>
<li>Does that mean <code>upper=False</code> transposes the matrix?</li>
<li>What is <code>transpose</code> for then?</li>
</ol>

tom: <p><em>First</em>, upper=False/True will decide whether PyTorch will look at the upper or lower “half” (including diagonal) of the matrix. You pass in a lower lower triangular matrix, you want upper=False because otherwise it’ll look like a diagonal matrix to PyTorch.</p>
<p><em>After</em> this, PyTorch will consider transpose=False/True and transpose it. But if, as in the example you posted, PyTorch gets a diagonal matrix (because the upper half is only the entries in the diagonal), you’ll get the same result whether you transpose or not. If you pass in a matrix that is not diagonal, transpose will make a difference, just as you expect.</p>
<p>Best regards</p>
<p>Thomas</p>

gncs: <p>Thank you for the detailed response, Thomas.</p>
<p>While I don’t think that the function arguments and their behaviors are intuitive or well-explained in the documentation, I now understand what is going on underneath the hood thanks to your explanation.</p>

tom: <p>Yeah, I stared down the documentation, implementation, and tests for quite a while before I got it.<br>
I even found a bug or two in the test coverage before noticing that it is the upper not matching the input. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If you have ideas how to improve the documentation, I think it might well be worth it.<br>
It’s a bit tricky with all the options.</p>
Using matplotlib during training ( Avoiding overhead due to processing on CPU)

fschiffers: <p>During my optimization loop I have the need to visualize a lot of my training output using matplotlib and other tools. Hence, I need to copy data from GPU to CPU and subsequently use matplotlib to plot (and save) data which takes quite a while.</p>
<p>This became so exhaustive that a large amount of time is now spent on plotting these intermediate results while the GPU utilization drops down to 0. I’ve been looking into flags like non_blocking, pin_memory etc., but they seem to focus mostly on CPU → GPU memory transfers.</p>
<p>Is there any way that plotting and training can be done asynchronously ? Ideally, I’d like to continue training on the GPU while the CPU is used to do “minor tasks” like plotting, saving on disk etc.</p>
<p>Thanks</p>

googlebot: <p>you’ll have to program this, preferably as off-process jobs. for plotting, tensorboard is an example of this approach; if your plots are simple line plots, you can use it (pytorch has a supporting module).</p>

fschiffers: <p>Are there any examples how this could be done with off-process jobs ? I have never implemented something like this in Python.</p>
<p>For the type of logging information: I have a few line plots, but I’m also in need to show a bunch of images (plt.imshow and subplots). I already have Tensorboard-logging in place, but I’d like to have an additional source of intermediate output information.</p>

googlebot: <p>well, there is a lot of ways to “send off” data arrays, the choice also depends on your “renderer”, i.e. browser/gui thread loop, incremental/reprocessing etc.</p>
<p>you can try just doing matplotlib stuff in a python thread, I’m not sure how badly this will affect training (python’s global lock is the potential problem).</p>