akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Understanding issues: 'one of the variables needed for gradient computation has been modified by an inplace operation'

username: bstan
cooked: <p>Hey I am pretty new to PyTorch.</p>
<p>When I am not using detach().clone() in the following line</p>
<pre><code class="lang-auto">iou = evaluate_reconstruction(predictions_reconstruction.detach().clone(), voxels)
</code></pre>
<p>I am getting this error message:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 32, 32, 32]], which is output 0 of MeanBackward1, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)</p>
<p>Even though I know how to fix this problem, I don’t really get why this happens. Could somebody explain this more in detail? Any tutorial pages, etc.?</p>
<p>Thank you!</p>
<p>Code:</p>
<pre><code class="lang-auto">for batch_idx, batch in tqdm(enumerate(train_dataloader)):
      _, renderings, class_labels, voxels = batch
      renderings, class_labels, voxels = renderings.to(device), class_labels.to(device), voxels.to(device)

       # Predict and estimate loss
       predictions_classification, predictions_reconstruction = model(renderings.float())

       train_loss_classification = criterion_classification(predictions_classification, class_labels)
       train_loss_running_classification += train_loss_classification.item()

       if predictions_reconstruction != None:
               train_loss_reconstruction = criterion_reconstruction(predictions_reconstruction, voxels)
               train_loss_running_reconstruction += train_loss_reconstruction.item()
               train_loss = args.loss_coef_cls * train_loss_classification + args.loss_coef_rec * train_loss_reconstruction
               train_loss_running += train_loss.item()
               iou = evaluate_reconstruction(predictions_reconstruction.detach().clone(), voxels)
               train_reconstruction_iou += iou
       else:
               train_loss = train_loss_classification
                
       # Backprob and make a step
       optimizer.zero_grad()
       train_loss.backward()
       optimizer.step()
</code></pre>
PyTorch Slack Invitation

username: newbieee
cooked: <p>Hi guys,</p>
<p>I have a question about the Torch forum (PyTorch Slack Invitation). Do you receive a response about the status of the application/regestration in any case - or only if you get invited? I’m asking for a buddy who applied a few weeks ago and hasn’t heard back. (He is to nice to ask himself <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> )</p>
<p>Kind regards</p>
PyTorch Reparametrization Method For Gamma, Dirichlet, Von Mises distribution

username: Hamed_Mahdavi
cooked: <p>Does anyone know that which algorithm PyTorch exactly uses for Gamma, Dirichlet, and Von Mises distributions? I searched in the PyTorch repository and didn’t find anything readable.</p>

username: Hamed_Mahdavi
cooked: <p>In case anyone is concerned: I found out that PyTorch uses this method <a href="https://arxiv.org/abs/1806.01851" rel="noopener nofollow ugc">https://arxiv.org/abs/1806.01851</a> according to the <a href="https://papers.nips.cc/paper/2018/file/92c8c96e4c37100777c7190b76d28233-Paper.pdf" rel="noopener nofollow ugc">Implicit Reparametrization Gradient</a> paper.</p>

username: DonnyHe
cooked: <p>Hey Hamed - did you find out through the code or through reading the paper? If through the code can you please give a github reference?</p>

username: Hamed_Mahdavi
cooked: <p>Hi,</p>
<p>I found out this fact from this github issue which is mentioned by <a class="mention" href="/u/fritzo">@fritzo</a>:<br>
<a href="https://github.com/tensorflow/probability/issues/51#issuecomment-395262574" rel="noopener nofollow ugc">https://github.com/tensorflow/probability/issues/51#issuecomment-395262574</a></p>

username: DonnyHe
cooked: <p>thanks for replying. how is your experience using some of the distributions? i tend to find them unstable close to the optimum.</p>

username: DonnyHe
cooked: <p>the reason I say it is unstable close to the optimum is because I have a same version of code in TFP, and it works much better than the same one with torch distributions.</p>

username: Hamed_Mahdavi
cooked: <p><a class="mention" href="/u/donnyhe">@DonnyHe</a> Sorry for my late answer, I didn’t test my code on TFP, but I can test it in the upcoming days</p>
How to convert model with custom C++ CUDA layer to TorchScript

username: RocketFlash
cooked: <p>How to convert model with custom C++ CUDA layer to TorchScript? I have a model with Mixed C++/CUDA layer created like here: <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.9.0+cu102 documentation</a> .<br>
I want to convert the whole model to TorchScript to use it in production. Is it possible to convert it? If yes, what is the best and proper way to do it?</p>
How do I update my ImageFolder dataset in pytorch?

username: HarshRangwala
cooked: <p>I am working on a dataset where I am required to find accuracy of classes with less than 20 samples. So first I used pytorch’s ImageFolder to get all the images in the folders.</p>
<pre><code class="lang-auto">dataset = ImageFolder('/content/drive/MyDrive/data/dataset')
</code></pre>
<p>Now to get classes with less than 20 samples I use:</p>
<pre><code class="lang-auto">def get_class_distribution(dataset_obj):
    count_dict = {k:0 for k,v in dataset_obj.class_to_idx.items()}
    
    for element in dataset_obj:
        y_lbl = element[1]
        y_lbl = idx2class[y_lbl]
        count_dict[y_lbl] += 1
            
    return count_dict
# print("Distribution of classes: \n", get_class_distribution(dataset))
class_distribution = get_class_distribution(dataset)

sampled_classes = [classes  for (classes, samples) in class_distribution.items() if samples &lt;= 20]
</code></pre>
<p>I get the list of classes correctly but my doubt is how do I proceed further for inference? How do I convert/update it to ImageFolder so that I can use the filtered dataset in the following code:</p>
<pre><code class="lang-auto"># Test model performance for classes with less than 20 samples.

y_pred_list = []
y_true_list = []
with torch.no_grad():
    for x_batch, y_batch in tqdm(data_loader):
        x_batch, y_batch = x_batch.to(device), y_batch.to(device)
        y_test_pred = model(x_batch)
        _, y_pred_tag = torch.max(y_test_pred, dim = 1)
        y_pred_list.append(y_pred_tag.cpu().numpy())
        y_true_list.append(y_batch.cpu().numpy())
</code></pre>

username: eqy
cooked: <p>Depending on how much data there is, it might be simpler to just filter out data elements at inference time (with a moderate penalty on performance due to smaller batch sizes).</p>
<pre><code class="lang-auto"># sketch of idea, might need refinement depending on specific shape
def filter_data(x_batch, y_batch, skip_classes):
    for class in skip_classes:
        y_batch[y_batch==class] = -1
    mask = y_batch &gt;= 0
    filtered_x_batch = x_batch[mask,:]
    filtered_y_batch = y_batch[mask]
    return filtered_x_batch, filtered_y_batch
...
# Test model performance for classes with less than 20 samples.

y_pred_list = []
y_true_list = []
with torch.no_grad():
    for x_batch, y_batch in tqdm(data_loader):
        x_batch, y_batch = x_batch.to(device), y_batch.to(device)
        x_batch, y_batch = filter_data(x_batch, y_batch, skip_classes)
        y_test_pred = model(x_batch)
        _, y_pred_tag = torch.max(y_test_pred, dim = 1)
        y_pred_list.append(y_pred_tag.cpu().numpy())
        y_true_list.append(y_batch.cpu().numpy())
</code></pre>
Signal to Signal processing

username: daniel1
cooked: <p>Hello all,</p>
<p>I’m new to deep learning and am looking for some advice on models and processes to be using for my task.</p>
<p>I have sequential data consisting of eight variables at every time step as input; and the output is one variable, also at every time step.</p>
<p>I am starting with a simple RNN using LSTMs. Is there any other models or processes that I should be aware of that may improve performance?</p>
<p>Thanks very much!</p>
Docker, AssertionError: can only join a child process

username: Qinsheng_Zhang
cooked: <p>I am running <a href="https://github.com/nicolas-chaulet/torch-points3d" rel="noopener nofollow ugc">open source code</a> inside docker environment.  It is quite strange to encounter the error inside the docker containers, while others run on PC do not encounter the issue.</p>
<pre><code class="lang-auto">Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7fae49c835f0&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.7/multiprocessing/process.py", line 138, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
</code></pre>
<p>The parameters to create a container</p>
<pre><code class="lang-auto">docker run --shm-size 64G --gpus all -d -v /xxx:/xxx --network host --name xxx_name --cpus 40 xxx_image
</code></pre>

username: ptrblck
cooked: <p>It seems that an error is raised, the <code>DataLoaderIter</code> dies and this multiprocessing error is raised instead.<br>
Could you run the code with a single worker (<code>num_workers=0</code>) and check, if another error might be raised?</p>

username: Qinsheng_Zhang
cooked: <p>If with one single worker, there is no error.</p>

username: akos.kiss
cooked: <p>Same here, code worked with num_workers=0, 1, 2, but saw a lot of these errors when num_worers&gt;=3.<br>
For me increasing the shared memory size (–shm-size 256M → 1G) solved the problem, now works fine with num_workers=12.</p>
A100 is slower than 1080ti with pytorch

username: deeriny
cooked: <p>First, I found out that the A100 supports:</p>
<ul>
<li>cuda 11.0</li>
<li>nvidia driver 450</li>
</ul>
<p>So, when using the A100, I adjusted the environment as follows.</p>
<ul>
<li>cuda 11.0</li>
<li>cudnn 8005 (I checked by <code>torch.backends.cudnn.version()</code>)</li>
<li>nvidia driver 450 (more detailed, 450.119.04)</li>
<li>pytorch 1.8.0</li>
</ul>
<p>I installed pytorch using,<br>
<code>pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html </code></p>
<p>However, the time for training my model takes too long.<br>
For example, the model used in the experiment is a complex model with a size of 7.2M.<br>
Using the 1080ti took 1600 seconds for 1 epoch (18,480 data), but it takes 2000 seconds with the A100.</p>
<p>I doubt this is a cudnn version problem, and I want to change cudnn to 8.0.2 or 8.0.3.<br>
How can I change only the cudnn version in pytorch?</p>
<p>Or is there any other solution to this problem?<br>
I’ve already tried <code>torch.backends.cudnn.benchmark = True</code> to solve this problem, but it was twice as slow.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="deeriny" data-post="1" data-topic="124913">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeriny/48/39617_2.png" class="avatar"> deeriny:</div>
<blockquote>
<p>I doubt this is a cudnn version problem, and I want to change cudnn to 8.0.2 or 8.0.3.<br>
How can I change only the cudnn version in pytorch?</p>
</blockquote>
</aside>
<p>I think it is indeed a cudnn problem, since you would be running into <a href="https://github.com/pytorch/pytorch/issues/50153">this issue</a> in the <code>1.8.0</code> release (so you should update to <code>1.9.0</code> or the nightly release) and also might want to build PyTorch from source using the latest cudnn release (8.2.1).</p>

username: deeriny
cooked: <p>Thank you for the reply!</p>
<p>Can I run what you suggested using this code? :</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
</code></pre>
<p>If so, I did it the first time. However, the A100 speed was slow, so I lowered the pytorch version to 1.8.0 and cuda version to 11.0.<br>
Should I try again?</p>

username: ptrblck
cooked: <p>Your command should install the latest <code>1.9.0</code> release, which would ship with the missing cutlass kernels in cudnn8.0.5. However, you might still get a better performance by building from source with cudnn8.2 as said before.</p>
Row wise combinations

username: lkp411
cooked: <p>Is there a way to efficiently, in a vectorized manner, compute the combinations along each rows of a 2D tensor individually?</p>
<p>For example:</p>
<pre><code class="lang-auto">a = torch.tensor([[2, 5, 6], [7, 9, 4]])
result = torch.combinations(a, 2, dim=1)
</code></pre>
<p>results should look like</p>
<pre><code class="lang-auto">torch.Tensor([[[2, 5], 
                       [2, 6],
                       [5, 6]],

                      [[7, 9],
                       [7, 4],
                       [4, 9]]])
</code></pre>
<p>This seems like a natural thing to be able to do, but I haven’t been able to figure out a way to achieve this without looping. I’d appreciate any help and pointers. Thanks!</p>

username: lkp411
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know if there’s a way to achieve this?</p>

username: KFrank
cooked: <p>Hi lkp411!</p>
<aside class="quote no-group" data-username="lkp411" data-post="1" data-topic="125082" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/90ced4/48.png" class="avatar"> lkp411:</div>
<blockquote>
<p>Is there a way to efficiently, in a vectorized manner, compute the combinations along each rows of a 2D tensor individually?</p>
</blockquote>
</aside>
<p>Yes, the idea would be to construct the combinations of a 1D vector<br>
of indices and then index into your multidimensional tensor, swapping<br>
the dimensions around appropriately.</p>
<p>Like this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; a = torch.tensor([[2, 5, 6], [7, 9, 4]])
&gt;&gt;&gt; a.T[torch.combinations (torch.arange (a.shape[1]), 2)].permute (2, 0, 1)
tensor([[[2, 5],
         [2, 6],
         [5, 6]],

        [[7, 9],
         [7, 4],
         [9, 4]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: lkp411
cooked: <p>Oh that’s clever. Thanks a lot <a class="mention" href="/u/kfrank">@KFrank</a> !!</p>
Multiple Single Object Localization

username: Ahmed11
cooked: <p>This is my first time creating a model tasked with predicting <strong>multiple</strong> locations of an object (<strong>single class</strong>) in an image. My dataset consists of X and Y values. X values are images, and the Y values are lists of bounding boxes (list of lists of bounding boxes for each image).</p>
<p>I want the model to predict <strong>all</strong> bounding boxes in a single passed image.</p>
<p>It seems that <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial is the closest to my use case.</p>
<p>But I have two questions:</p>
<ol>
<li>
<p>Since I only have one class, if I delete/modify lines of code in the tutorial that are responsible for the classification task, would that mean that the model (both regression and classification) included in the tutorial would not be suitable for my use case?</p>
</li>
<li>
<p>I dont think the pre-trained model on COCO would be suitable, since my images are scanned text pages, is that right? Should I instead train the model (say, resenet50) on my dataset?</p>
</li>
</ol>
<p>Thank you all for your help and time.</p>
Loading a Tensor from file in batches

username: yacinem
cooked: <p>Hi everyone,</p>
<p>I have a tensor stored in a file (initially it was a dataset stored in a matlab .mat file that I read using scipy and then stored as a tensor using torch.save). the data represents RGB images stored in a tensor of shape (N * C * H * W) where N = number of training examples/images, C = number of channels, H * W =  size of the images.</p>
<p>I load the tensor using torch.load, but the problem is that I want to load these images in batches, so that I don’t have to load the whole file to memory for training (actually I don’t even have enough memory to load it all).</p>
<p>I read the Pytorch tutorial about loading custom datasets but in the tutorial they load the whole tensor file containing image names/paths (stored as .jpg files) then load the images one at a time (I can’t do this because my images are all stored in a single tensor).</p>
<p>Thanks a lot !</p>
<p>TL;DR: how to load a tensor from disk to memory in batches? (the tensor file is saved with torch.save)</p>

username: ptrblck
cooked: <p>Based on <a href="https://discuss.pytorch.org/t/memory-mapped-tensor/8954/2">this older post</a> it seems that you could use a <code>Storage</code> to load the data in chunks.<br>
However, I don’t see an <code>offset</code> argument, so I guess the proper way would be to use <a href="https://numpy.org/doc/stable/reference/generated/numpy.memmap.html?highlight=memmap#numpy.memmap"><code>np.memmap</code></a> and load chunks of a numpy array (assuming you could store it via numpy).</p>

username: yacinem
cooked: <p>I think the memmap idea is a good solution for my problem, I’ll try to create a Dataset object that uses it under the hood.</p>
<p>Thanks !</p>

username: yacinem
cooked: <p>additionally to <a class="mention" href="/u/ptrblck">@ptrblck</a> 's answer, I also used another method for my 2nd dataset: I made a Python script that reads my entire dataset and then saves each training example in a single file, then in my DataLoader I implement the loading of one training example. So, now, the DataLoader will take care of loading just “batch_size” number of examples to memory, instead of the whole dataset.</p>
Reduced features by AutoEncoder

username: kountaydwivedi
cooked: <p><strong>UPDATE:</strong><br>
My apologies. I gave a full day just to understand that the output of bottleneck layer (the last layer in the Encoder container) is actually a COMPRESSED feature set that somehow tries to match the distribution of the original feature set. Thus, to sum up-<br>
If you originally have 1000 features set, and you want output features to be around 100, then you will get this by using Autoencoder. However, the output 100 features would not be a subset of 1000 input features. They will be all new, compressed form of the initial, 1000 input features.</p>
<hr>
<p>Hi all,<br>
I am new to Pytorch. I want to know when we train an autoencoder, what are the outputs of the ENCODER container of it?<br>
As I know the Autoencoder is used for Dimension reduction. So, after the encoder part gets executed, do we get a set of indices that correspond to relevant columns from the dataset, or do we get something entirely different?<br>
…<br>
I am actually using an Autoencoder for feature extraction, and this knowledge will help me quite a lot. I need to extract features from the ENCODER container and I don’t know how to do it.</p>
<p>Thanks in advance.</p>
After export onnx model, two inputs are missing

username: zhonhel
cooked: <p>Hi everyone，</p>
<p>I export a model by <strong><code>torch.onnx.export</code></strong> function. But I give 8 args, so the forward function needs 8 args, but the output graph only have 6 input nodes.</p>
<p>I know maybe the other 2 inputs doesn’t make sense to final outputs, so they don’t appear in the output graph.</p>
<p>My question is now I don’t know which 6 args are needed in 8 args.</p>
<p>BTW, the code is from github, so I don’t know why the 2 args not be used now.</p>
The future of caffe2

username: Titusium
cooked: <p>Hello everyone,<br>
I’m currently working on deploying models in C++ through the Caffe2 API. But regarding the torch c++ API, I’m wondering what’s the future of caffe2 in the development of Pytorch.<br>
Is it still worthy to work with caffe2 or should I move to libtorch ?<br>
Also a second question is what do you use or advice for the deployment of detection models in a C++ project.</p>
<p>Thank you everyone for your feedback !! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>I don’t have any numbers about the usage of Caffe2 vs. PyTorch in different deployment scenarios, it seems that at least <a href="https://ai.facebook.com/blog/pytorch-builds-the-future-of-ai-and-machine-learning-at-facebook/">some teams are migrating to PyTorch</a> so you could check for similar articles from other teams.</p>
Low Volatile GPU-util

username: yuhaoooo
cooked: <p>I am training a small model with 256 kb parameters. And I set the dataset loader with num_works = 16, pin_memory = True and batch_size = 128. However, the GPU-util is still very low with 0.007%. Could you give any advice to improve the GPU-util?</p>

username: eqy
cooked: <p>Is that the batch size that is the maximum possible? Have you profiled or timed parts of the code to see what the bottleneck is?</p>
I don't really understand the difference between kernel as weight and convolution kernel

username: Keiku
cooked: <p>What does “kernel” in <code>trained_kernel</code> in the code below mean? Is it different from the kernel often expressed in convolution? Also, is the operation of <code>torch.stack ([torch.mean (trained_kernel, 1)] * 6, dim = 1)</code> a general operation? Is there a reference somewhere?</p>
<pre><code class="lang-python">from torchvision.models import resnet34

# get resnet34 model with 6 channels
def get_model(pretrained=False):
    model = resnet34(pretrained=pretrained)
    model.fc = nn.Linear(512, n_class)
    trained_kernel = model.conv1.weight
    new_conv = nn.Conv2d(6, 64, kernel_size=7, stride=2, padding=3, bias=False)
    with torch.no_grad():
        new_conv.weight[:,:] = torch.stack([torch.mean(trained_kernel, 1)]*6, dim=1)
    model.conv1 = new_conv
    return model
</code></pre>

username: eqy
cooked: <p>I think in most cases “kernel” and “weight” are used interchangeably for convolutions. For example, “kernel size” usually specifies the spatial dimensions of the weight tensor.</p>
<p>As for the <code>torch.stack ([torch.mean (trained_kernel, 1)] * 6, dim = 1)</code> here, I don’t think that is very common, but it is a simple way to reinitialize the first layer for a model when it was trained for three channel input images and the new task has six channel input images. My guess would be that it might help prevent changing the downstream layers too much (compared to completely random reinitialization) during finetuning. Then again, I wonder if batchnorm might do something similar already.</p>
Style Transfer produces white noise

username: Podidiving
cooked: <p>Hi!<br>
I’m playing with <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 2.2.0+cu121 documentation</a><br>
Running this demo on colab (torch version == <code>1.9.0</code>)<br>
I changed images (style is <a href="https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/inputs/starry_night_google.jpg" rel="noopener nofollow ugc">https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/inputs/starry_night_google.jpg</a> and content is <a href="https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/inputs/hoovertowernight.jpg" rel="noopener nofollow ugc">https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/inputs/hoovertowernight.jpg</a>)<br>
Also set <code>num_steps</code> to 1500<br>
After a while (about 1400 steps) image degrades to white noise</p>
<p>Below is Content and Style losses graph to show this.</p>
<p>Why is it happening?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b775e610e2b0800af7e6350881d435f14e526c0d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b775e610e2b0800af7e6350881d435f14e526c0d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b775e610e2b0800af7e6350881d435f14e526c0d_2_690x403.png" alt="image" data-base62-sha1="qaXYnBfoVpXw7u5P65DzBFGb0lf" width="690" height="403" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b775e610e2b0800af7e6350881d435f14e526c0d_2_690x403.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b775e610e2b0800af7e6350881d435f14e526c0d_2_1035x604.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b775e610e2b0800af7e6350881d435f14e526c0d.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1278×748 71.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
LBFGS throws RuntimeError

username: Podidiving
cooked: <p>Hello, everyone.</p>
<p>I’m trying to run NeuralStyleTransfer tutorial from <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 1.9.0+cu102 documentation</a><br>
My torch version is <code>1.9.0</code></p>
<p><code>run_style_transfer</code> function throws next error: <code>RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.</code><br>
Traceback shows, It is happening inside the lbfgs optimizer (in <code>_gather_flat_grad</code> function)<br>
Is there any way to handle this error without changing optimizer itself? Plus, can anyone explain, why this error occurs?<br>
Thank in advance</p>

username: Podidiving
cooked: <p>Got it. You should do <code>input_img = content_img.clone().contiguous()</code> to make it work</p>
Run RPC over MPI for Parameter Server DRL

username: Matthew_Landen
cooked: <p>I am currently developing an drl framework that can run on a cluster with mpi. i am able to perform synchronous training using DDP over MPI. Now, I want to explore a different structure using a parameter sever and MPI. I saw that RPC would be the right tool, but I cannot figure out how/if rpc can run with mpi.</p>
<p>I saw this <a href="https://gist.github.com/froody/7bb2b5971944833cd201ef675c0622fb" rel="noopener nofollow ugc">example</a>, but it only works when all ranks are running on the same node. Is there a way to accomplish this with pytorch alone or is an additional tool needed?</p>

username: Yanli_Zhao
cooked: <p>you do not have to run rpc on MPI, pytorch distributed provides gloo and nccl backends, you can pass ‘gloo’ or ‘nccl’ to init_process_group().</p>
<p>for rpc, to get better performance, you can use tensor pipe as backend option</p>
How can I check whether DDP is working properly or not?

username: oasjd7
cooked: <p>How can I check whether DDP is working properly or not?</p>
<p>I compared the speed in the environment below, and DDP was 2 times slower than DP.<br>
Can such a case exist? Is it a code problem?<br>
:+ Is all_reduce mandatory?</p>
<ul>
<li>Train dataset: 3,000 images</li>
<li>ResNet-101</li>
<li>Batch_size: 32</li>
<li>2 GPUs</li>
<li>workers: 8</li>
</ul>

username: Yanli_Zhao
cooked: <p>you can compare DDP training convergency or evaluate its accuracy or loss.</p>
<p>in terms of performance, 2 times slower is a little surprised… depends on your codes and hardware</p>
The same parameters, different results for multiple trainings for cross-entropy loss

username: mark_eu
cooked: <p>Hi,<br>
I noticed that multiclass classification on the same dataset gives different results for each training of bidirectional LSTM. I am mostly concerned for weighted avg f1 and weighted avg recall in the classification report. I just <strong>re-run</strong> the same Jupyter notebook and get a different result. For example, the first time I get a weighted avg f1 of 0.17 and the next training gives 0.26. I use the classic</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
</code></pre>
<p>Also, I use</p>
<pre><code class="lang-auto">torch.manual_seed(SEED) 
</code></pre>
<p>and the</p>
<pre><code class="lang-auto">BucketIterator.splits
</code></pre>
<pre><code class="lang-auto">train_iterator, valid_iterator = BucketIterator.splits(
    (train_ds, val_ds), 
    batch_sizes         =  (train_batch_size, valid_batch_size),
    sort_key            =   lambda x:len(x.text),
    sort                =   False, 
    sort_within_batch   =   True, 
    device              =   device)
</code></pre>
<p>I would like to hear your opinion.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Setting the seed alone might not be sufficient to get deterministic results and you would have to disable all non-deterministic algorithms as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>.</p>
What is causing my results to change every time I initialize my model and use it for inference?

username: Luca_Guarro
cooked: <p>I have defined my model in the following way:</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self, pre_classifier_init, classifier_init):

        super(Net, self).__init__()
        self.pre_classifier = nn.Linear(768, 768)
        self.classifier = nn.Linear(768, 2)
        self.dropout = nn.Dropout(0.1)
        self.pre_classifier.weight.data.copy_(pre_classifier_init.weight.data)
        self.classifier.weight.data.copy_(classifier_init.weight.data)

    def forward(self, x, labels = None):
        x = self.pre_classifier(x)
        x = nn.ReLU()(x)
        x = self.dropout(x)
        logits = self.classifier(x)
        loss = None
        if labels is not None:
          loss_fct = CrossEntropyLoss()
          loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

        return SequenceClassifierOutput(
            loss = loss,
            logits = logits
        )

net = Net(model.pre_classifier, model.classifier1)
net.eval()
</code></pre>
<p>As you can see, I am passing parameters to the constructor to set the weights for the 2 layers in the model each time I instantiate it; as well as putting the model in evaluation mode to turn off the effects of the dropout layer. With this setting, I would thus expect there to be no randomness involved and for my results to be deterministic. However the performance of my model varies each time I instantiate it so there still must be some randomness going on.</p>
<p>I confirm this by running this code beforehand:</p>
<pre><code class="lang-auto">seed = 0
torch.manual_seed(seed)
</code></pre>
<p>So where is this randomness coming from?</p>

username: ptrblck
cooked: <p>The non-deterministic results could be created by e.g. cublas as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> so you could set <code>torch.use_deterministic_algorithms(True)</code> and see, if non-determinsitic algorithms would be used.</p>

username: Luca_Guarro
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the reply. I discovered that I was only setting the weights upon instantiation but I also needed to set the biases by adding these lines:</p>
<pre><code class="lang-auto">        self.pre_classifier.bias.data.copy_(pre_classifier_init.bias.data)
        self.classifier.bias.data.copy_(classifier_init.bias.data)
</code></pre>
<p>With these lines, my results become deterministic.</p>

username: ptrblck
cooked: <p>Good to hear it’s working now!<br>
A small addition: don’t use the <code>.data</code> attribute, as it could yield unwanted side effects. Wrap the assignment in a <code>with torch.no_grad()</code> instead and assign the new values to the parameters.</p>
Rescale image data [0, 255] to [-1, 1]

username: Florentino
cooked: <p>Hello, as the title says, when I use <code>torchvision.transforms.Compose</code>, is there a way to do the tensor rescaling or redistribution method to complete it after the Compose fuction ingredient <code>transforms.ToTensor()</code>?</p>

username: eduardo4jesus
cooked: <p>Given that <code>torchvision.transform.ToTensor</code> <a>already standardize the data to <code>[0, 1]</code></a>, you could just multiply that by 2 and subtract 1 on your input layer.</p>

username: Florentino
cooked: <p>Ok, thank you for reminding what <code>ToTensor</code> have done.</p>
Inference speed of BertForSequenceClassification

username: Zvi_Boshernitzan
cooked: <p>Not sure if this is a better question for the transformers group, but moving from pytorch 1.3.0 to pytorch 1.9.0 ( transformers 2.3.0 to 4.7.0, simpletransformers 0.15.7 to 0.61.6 ) , and profiling my bert inference model, i found that the (python-profiled) rate limiting step was using two different calls in pytorch:</p>
<p>in 1.3.0, it was {method ‘matmul’ of ‘torch._C._TensorBase’ objects};<br>
in 1.9.0, it was {built-in method torch._C._nn.linear}, and was taking 4x the time.</p>
Replacing cross_entropy loss with smooth_l1_loss

username: treadstone
cooked: <p>I ma trying to replace torch.nn.functional cross_entropy loss with smooth_l1_loss.<br>
Below line of code I have used  for this-</p>
<pre><code class="lang-auto">loss = F.smooth_l1_loss(logits, label)
</code></pre>
<p>However I am getting error like-  UserWarning: Using a target size (torch.Size([729])) that is different to the input size (torch.Size([729, 27])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. RuntimeError: The size of tensor a (27) must match the size of tensor b (729) at non-singleton dimension 1</p>
<p>I know that both loss function need input in seperate tensor format. But how to solve this problem. Any work around for replacing cross_entropy with smooth_l1_loss or huber loss?</p>

username: pascal_notsawo
cooked: <p>I don’t understand why you want to do this kind of replacement, since these are two functions commonly used for different kind of problems : classification vs regression.</p>
<p>But if you want, you can just give an acceptable shape to your tensors : this by using a single neuron at the last layer, so that <code>logits</code> will be compatible with <code>label</code> in terms of dimension.</p>
Libgcc_s.so.1 must be installed for pthread_cancel to work pytorch

username: ruotianluo
cooked: <p>Not sure if anyone got this problem too or familiar with this.</p>
<p>I keep getting this error when the program crash, and it also hides the original error trace so that it’s hard for me to debug.</p>
<p>It only happens after I upgraded to 1.6.0.</p>

username: seyeeet
cooked: <p>I also got this error with pytorch 1.9+cu102</p>
Module List Activation Function

username: drbeethoven
cooked: <p>Hello,</p>
<p>I am trying to create a method that allows me to create a nn of variable number of layers and size. As such, I am using a module list.<br>
However, I notice that when I used “nn.Linear” before using a module list, I would have to specify Sigmoid in between layers and soft max at the end. The soft max I can just put at the end, but how do I put a sigmoid layer between each module within the module list?</p>
<p>Thank you so much!</p>

username: KFrank
cooked: <p>Hi James!</p>
<aside class="quote no-group" data-username="drbeethoven" data-post="1" data-topic="124975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b9bd4f/48.png" class="avatar"> drbeethoven:</div>
<blockquote>
<p>how do I put a sigmoid layer between each module within the module list?</p>
</blockquote>
</aside>
<p>I’m not entirely sure what you’re asking or how you intend to use<br>
your <code>ModuleList</code>, but note that a <code>torch.nn.ReLU</code> is a <code>Module</code><br>
so you can include it in your <code>ModuleList</code>, in between, for example,<br>
some <code>torch.nn.Linear</code>s.</p>
<p>Best.</p>
<p>K. Frank</p>

username: drbeethoven
cooked: <p>That makes sense, thank you Frank! I was originally trying to create both the ReLu and linear layer when initializing the module list, but found this method more helpful if anyone is curious:</p>
<p>self.fc2 = nn.ModuleList()</p>
<p>for _ in range(initHiddenWeights.shape[0]):<br>
self.fc2.append(nn.Linear(initHiddenWeights.shape[1], initHiddenWeights.shape[2]))<br>
self.fc2.append(nn.ReLU())</p>
Model Binarisation

username: chhaya_kumar_das
cooked: <p>Hi Everyone.<br>
My objective is to obfuscate my model so that the clients can’t look into model and the inference code. If I package my model and code to standalone executables using Glow, will this objective be achieved? I did see that Glow supports AOT compilation. Will this solve the problem?</p>

username: jfix
cooked: <p>It should do a decent job, you’ll end up with a binary for running the model itself along with binary weights to load. Assuming you strip out any debug info then they’d have to understand the model based on machine code.</p>
Torch.tile is my bottleneck

username: jmschrei
cooked: <p>Hello!</p>
<p>I’m writing a custom operation that involves running one example through a layer, tiling it to be the full batch size, and then only calculating sparse deltas and applying them. In practice, I’m finding that the tiling operation (<code>torch.tile(X_ref, (X.shape, 1, 1))</code>) is the bottleneck and can take up to 90% of the time of this operation when I run it on a CPU. That being said, it’s not <em>slow</em>, per-say, it’s just that the other operations are very fast. I was wondering if there’s a faster way to do tiling along a single dimension than <code>torch.tile</code>. I’ve already tried <code>X_ref.expand(X.shape, -1, -1).contiguous()</code> but it’s roughly the same amount of time.</p>
<p>Thanks for your time.</p>
Loading custom object detection data

username: U1353279
cooked: <p>I’m wanting to train  a SSD-Mobilenet model using my own dataset.</p>
<p>My dataset is labelled, below is the structure of my data;</p>
<p>Dataset</p>
<blockquote>
<p>JPEGImages</p>
<blockquote>
<p>0001.jpeg<br>
0002.jpeg…</p>
</blockquote>
</blockquote>
<blockquote>
<p>Annotations</p>
<blockquote>
<p>0001.XML<br>
0002.XML</p>
</blockquote>
</blockquote>
<p>Almost all tutorials i can find either use built in datasets or datasets containing a csv file.</p>
<p>Any ideas on how i can load the above structure into pytorch,I’ll be using torchvision.</p>
<p>Thanks</p>

username: Flock1
cooked: <p>You can get a list of files using <code>os.listdir</code>. Then, you can write a custom dataloader like this:</p>
<pre><code class="lang-auto">class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):
        
        self.files = files
        
    def __getitem__(self, I):

        #read the images here, convert to torch format and return


    def __len__(self): 
        
        return &lt;number of images&gt;
</code></pre>
<p>Then, use an instance like <code>train_dl = data_gen(image_files)</code> and finally use</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
        train_dl, batch_size= , shuffle=True, num_workers= , pin_memory=True)
</code></pre>

username: U1353279
cooked: <p>Thanks for a prompt reply. I’ll give it a shot now.</p>

username: U1353279
cooked: <p>Below is teh example code i’m trying to adapt, but the issue is i don’t have a csv file my annoattaions are simple in teh annotation folder as .xml???</p>
<p>class data_Gen(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self,csv_file,root_dir,transform):<br>
self.annotations=pd.read_csv(csv_file)<br>
self.root_dir=root_dir<br>
self.transform=transform</p>
<pre><code>def __getitem__(self,I):
    img_path=os.path.join(self.root_dir,self.annotations.iloc[index,0])
    image=io.imread((img_path))
    y_label=torch.tensor((int(self.annotations.iloc[index,1])))
    
    if self.transform:
        image=self.transform(image)
        
    return(image,y_label)</code></pre>

username: Flock1
cooked: <p>You can read xml files in python. Look for library that does it and get the annotations from there and return it as <code>y_label</code></p>

username: U1353279
cooked: <p>Thanks and really sorry for the late reply, I just noticed your answer</p>
Dist.broadcast and dist.send do not work for more than 2 ranks

username: tc22
cooked: <p>I’m running a piece of code (attached below) that is very similar to the one listed here: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Distributed Applications with PyTorch — PyTorch Tutorials 1.9.0+cu102 documentation</a> - in particular, the section about sending a tensor that is incremented by 1. However, instead of having only 2 ranks, I am using 5, and instead of using send, I am broadcasting the incremented tensor from the source (rank 0) to the remaining ranks, but the script doesn’t terminate (note - I have also tried using send to all the other ranks and it hasn’t worked). I’m pretty sure the line that uses dist.broadcast is the source of the issue, as the script never gets past that line. I’m not sure if this is because of some fundamental misunderstanding (as I’m new to distributed training) so it would be appreciated if someone could shed some light onto this.</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp

def run(rank, size):
    tensor = torch.zeros(1)
    if rank == 0:
        tensor += 1
        # Send the tensor to process 1
        dist.broadcast(tensor=tensor, src=0)
    else:
        # Receive tensor from process 0
        print("receiving")
        dist.recv(tensor=tensor, src=0)
    print('Rank ', rank, ' has data ', tensor[0])

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 5
    processes = []
    mp.set_start_method("spawn")
    for rank in range(size):
        p = mp.Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>

username: Yanli_Zhao
cooked: <p>you need to call ‘dist.broadcast()’ instead of dist.recv() in recv ranks</p>

username: tc22
cooked: <p>Oh I see, thanks for the clarification!</p>
In pytorch, is there pdf(),logpdf () function for distribution?

username: ElleryL
cooked: <p>I couldn’t find through the documentations. I was looking for something similar to scipy.stats such that</p>
<pre><code class="lang-auto">from scipy.stats import multivariate_normal
mvn = multivariate_normal(np.zeros(2),np.identity(2))
mvn.pdf(np.array([[0,1],[1,0]]))
</code></pre>
<p>I can directly pass torch.Tensor object into scipy.stats but it only return numpy object and requires me to transform back.</p>

username: richard
cooked: <p><a href="https://pytorch.org/docs/master/distributions.html?highlight=distributions#module-torch.distributions" class="onebox" target="_blank">https://pytorch.org/docs/master/distributions.html?highlight=distributions#module-torch.distributions</a></p>
<p>It looks like probs() and log_probs() are what you’re looking for</p>

username: phan_phan
cooked: <p>It seems that it now disappeared ?<br>
So to get <code>dist.prob(x)</code> I am doing</p>
<pre><code class="lang-auto">dist.log_prob(x).exp()
</code></pre>
<p>Is that the best way ?</p>

username: Hippolyte_Dubois
cooked: <p><code>log</code> is base 10 logarithm, you should use this:</p>
<pre><code class="lang-auto">10**dist.log_prob(x)
</code></pre>

username: phan_phan
cooked: <p>Thank you for your answer however I am pretty sure log_prob returns the natural logarithm.<br>
For example you can see in the code for the <a href="https://pytorch.org/docs/stable/torch.html#torch.log" rel="nofollow noopener">uniform distribution</a> that it uses <a href="https://pytorch.org/docs/stable/torch.html#torch.log" rel="nofollow noopener">torch.log()</a> which is the natural logarithm.</p>

username: agg-shambhavi
cooked: <p>Does this give similar results as scipy module?</p>
Getting Nan Loss when using wide_resnet_fpn or resnext_fpn bacbone with FasterRCNN

username: Sahil_Goyal
cooked: <p><strong>Nan loss</strong> appears only in the case of using <em>wide_resnet_fpn</em> or <em>Resnext_fpn</em> as a backbone whereas classic resnets with fpn are working properly as backbone in FRCNN. But the pytorch-vision has mentioned that we can use all of them in the below model .  Any idea?</p>
<p><strong>Error</strong></p>
<pre><code class="lang-auto">Epoch: [0]  [  0/457]  eta: 0:26:59  lr: 0.000032  loss: 2.0617 (2.0617)  loss_classifier: 0.6785 (0.6785)  loss_box_reg: 0.5333 (0.5333)  loss_objectness: 0.6902 (0.6902)  loss_rpn_box_reg: 0.1597 (0.1597)  time: 3.5431  data: 0.6946  max mem: 6346
Loss is nan, stopping training
{'loss_classifier': tensor(nan, device='cuda:2', grad_fn=&lt;NllLossBackward&gt;), 'loss_box_reg': tensor(nan, device='cuda:2', grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(nan, device='cuda:2', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;), 'loss_rpn_box_reg': tensor(nan, device='cuda:2', grad_fn=&lt;DivBackward0&gt;)}

</code></pre>
<p><strong>Model</strong></p>
<pre><code class="lang-auto">from torchvision.models.detection.retinanet import  retinanet_resnet50_fpn
from torchvision.models.detection import FasterRCNN

def FRCNN_resnetfpn_backbone(backbone_name='resnet101', pre_trained=True):

    # Reference: https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py

    backbone = resnet_fpn_backbone(backbone_name, pre_trained)
    """
    resnet_fpn_bacbone:
    Args:
        backbone_name (string): resnet architecture. Possible values are 'ResNet', 'resnet18', 'resnet34', 'resnet50',
             'resnet101', 'resnet152', 'resnext50_32x4d', 'resnext101_32x8d', 'wide_resnet50_2', 'wide_resnet101_2'
        pretrained (bool): If True, returns a model with backbone pre-trained on Imagenet
        norm_layer (torchvision.ops): it is recommended to use the default value. For details visit:
            (https://github.com/facebookresearch/maskrcnn-benchmark/issues/267)
        trainable_layers (int): number of trainable (not frozen) resnet layers starting from final block.
            Valid values are between 0 and 5, with 5 meaning all backbone layers are trainable. default=3
        returned_layers (list of int): The layers of the network to return. Each entry must be in ``[1, 4]``.
            By default all layers are returned.
        extra_blocks (ExtraFPNBlock or None): if provided, extra operations will
            be performed. It is expected to take the fpn features, the original
            features and the names of the original features as input, and returns
            a new list of feature maps and their corresponding names. By
            default a ``LastLevelMaxPool`` is used.
    """
    model = FasterRCNN(backbone,
                       num_classes=2)
    return model
</code></pre>
<p>Thanks a lot!  help!</p>

username: Sahil_Goyal
cooked: <p>Actually, it is related to the version of torchvision. Earlier I was using <code>0.8.0a0</code>, changing to<code> 0.9.0a0</code> resolved the issue</p>