akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Filename print twice

rui_zhou: <p>Hi, I always get naming error when loading my own dataset:<br>
why the filename print twice?<br>
FileNotFoundError: [Errno 2] No such file or directory: ‘./dataset/train_GT./dataset/train/32.png’<br>
My folder structure:<br>
––dataset<br>
——train<br>
———0.jpg<br>
———1.jpg<br>
——train_GT<br>
———0.png<br>
———1.png<br>
is there something wrong with the filename? and I define GT_path as follows:<br>
` def <strong>getitem</strong> (self, index):</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>image_path = self.image_paths[index]</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td>filename = image_path.split(’_’)[-1][:-len(".jpg")]</td>
</tr>
<tr>
<td></td>
<td>GT_path = self.GT_paths + filename + ‘.png’`</td>
</tr>
</tbody>
</table>
</div><p><a href="https://github.com/LeeJunHyun/Image_Segmentation" rel="noopener nofollow ugc">https://github.com/LeeJunHyun/Image_Segmentation</a><br>
With reagards.Any help would be so grateful!</p>

AndreasW: <p>why you are split with (’<em>’)<br>
with<br>
filename = image_path.split(’</em>’)[-1][:-len(".jpg")]<br>
you get --&gt; ./dataset/train/32</p>
<p>and not with (’/)’<br>
filename = image_path.split(’/’)[-1][:-len(".jpg")]</p>
Dataloader error when num_workers > 0

sr_frost: <p>Hello. I am trying to load a video frame by frame using <code>imageio</code>, but facing an weird error when <code>num_workers &gt; 0</code>. This is a short snippet to reproduce the error.</p>
<pre><code class="lang-auto">from skvideo import datasets
import imageio
import torch
from torch.utils.data import Dataset, DataLoader

class VideoDataset(Dataset):
    def __init__(self, video_path):
        super().__init__()
        self.video_reader = imageio.get_reader(video_path)
        self.metadata = self.video_reader.get_meta_data()
        self.nframes = self.metadata["nframes"]

    def __getitem__(self, idx):
        img = self.video_reader.get_data(idx)
        img = torch.from_numpy(img)

        return img

    def __len__(self):
        return self.nframes
</code></pre>
<pre><code class="lang-auto">video_path = datasets.bikes()
video_data = VideoDataset(video_path)
video_loader = DataLoader(video_data, batch_size=4, num_workers=4)

for imgs in video_loader:
    print(imgs.shape)
</code></pre>
<p>when I try to iteratively print the shape of the loaded tensors, it gets stuck mid way and throws an error:</p>
<pre><code class="lang-auto">torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
torch.Size([8, 272, 640, 3])
---------------------------------------------------------------------------
CannotReadFrameError                      Traceback (most recent call last)
&lt;ipython-input-20-453bfa4da85a&gt; in &lt;module&gt;()
----&gt; 1 for imgs in video_loader:
      2     print(imgs.shape)
</code></pre>
<p>Note that the code works fine when  <code>num_workers=0</code>. I can’t figure out what’s going wrong.</p>

AndreasW: <pre><code class="lang-auto">class VideoDataset(Dataset):
    def __init__(self, video_path):
        super().__init__()
        self.video_path = video_path
        self.video_reader = imageio.get_reader(video_path)
        self.metadata = self.video_reader.get_meta_data()
        self.nframes = self.metadata["nframes"]

    def __getitem__(self, idx):
        video = imageio.get_reader(self.video_path)
        img = video.get_data(idx)
        img = torch.from_numpy(img)

        return img

    def __len__(self):
        return self.nframes
</code></pre>
<p>then it works</p>
MLP pytorch implementation

Elidor: <p>Hi everyone,</p>
<p>doing a bit of research on the forum and looking at various codes I got a doubt about implementing an MLP in pytorch.<br>
In particular, I have often seen two implementations for an MLP. The first is simply:</p>
<pre><code class="lang-auto">from torch import nn

MLP = nn.Linear(
            in_features=... ,
            out_features=...,
        )
</code></pre>
<p>the second one also includes the activation function and the dropout:</p>
<pre><code class="lang-auto">from torch import nn

class MLP(nn.Module):
    def __init__(self, in_features, out_features, activation='ReLU', dropout=0.3):
        super(MLP, self).__init__()
        self.layers = nn.Sequential()
        act_fn = getattr(nn, activation)
      
        self.layers.add_module('fc_{}'.format(i),nn.Linear(in_features, out_features))
        self.layers.add_module('{}_{}'.format(activation, i), act_fn())
        self.layers.add_module('dropout_{}'.format(i), nn.Dropout(dropout))
        
        in_features = out_features

    def forward(self, x):
        return self.layers(x)
</code></pre>
<p>my question is whether these two implementations, under the hood, are the same or not. By default, does pytorch’s <code>Linear layer</code> already have its own (default) activation function and a (default) dropout?</p>
<p>If they weren’t the same, as I suspect, what is the most faithful implementation for an MLP?</p>
<p>Thank you all!</p>

chinmay5: <p>The linear layer only does a linear transformation and does not include a non-linearity inbuilt. So, the second one is actually creating a MLP in true sense.</p>

Elidor: <p>OK thank you very much!</p>
How to make model parameters as a function of a tensor?

gaseln: <p>I have a simple neural net.</p>
<pre><code class="lang-auto">class my_nn(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(4, 5)
        self.w = torch.Tensor(25)
    
    def forward(self, x):
        return torch.tanh(self.fc(x))
</code></pre>
<p>And I want parameters of my_nn object (weights and a bias) to be a function of <code>self.w</code> (say, the Tensor of flattened and concatenated parameters is equal to <code>self.w ** 2</code>). While backpropogating I would like to get the gradient of the net with respect to <code>self.w</code>, not <code>my_nn</code> parameters. How can one do it?</p>
Adversarial Training model mode

chinmay5: <p>I have a basic question about the Adversarial training using PyTorch. I understand that the model for adversarial example generation should be <strong>eval()</strong>  as suggested by documentation. However, if we are performing adversarial training, in each epoch, we would need to generate these adversarial examples. What should be the mode here?</p>
<p>I am confused because Adversarial example generation should be in eval mode. However, during training, layers such as dropout and Batch Norm would behave differently. I am a bit confused about this step and any help would be really appreciated</p>
Enhance.histogram input and output?

simonjaq: <p>I can’t wrap my head around kornia.enhance.histogram.</p>
<p>I want to use it to compare histograms of two images in a differentiable way.<br>
But I don’t really get what the funtion expects as input and what the output means. Also I don’t really understand the bandwidth and epsilon parameters.</p>
<p>With torch.histc I’m doing the following to get my desired result. An 1D Tensor of the histogram of R channel of the image.</p>
<pre><code class="lang-auto">histoA = torch.histc(input_A[0,0,:,:], bins=255, min=0, max=1)
</code></pre>
<p>Input_A is a RGB image in n,c,h,w format.</p>
DDP training slows down at night

kaka_zhao: <p>I‘m training a DDP model with 2x2080ti.<br>
I found that Rank0 had always slowed down at night.<br>
The batch time of Rank0 can always increase at about 23:00 and decrease at about 8:00.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b74bb2897b21edd3aaad79ed69c14a3519e427b2.jpeg" alt="screen" data-base62-sha1="q9vyB7VEdei5DMjG6HBCMnjUWxY" width="469" height="302"></p>
<p>How to solve this problem? Thanks</p>

mrshenli: <p>Hey <a class="mention" href="/u/kaka_zhao">@kaka_zhao</a>, this is very interesting finding. We didn’t have any time-based algorithm in DDP. Is there any other user share the same cluster with you? I wonder if it is possible that some recurrent job is kicked off every day at 23:00, which will compete for resources on network/GPU with your job?</p>

kaka_zhao: <p>Thanks for your reply. I have found a process about Xorg would be running at night. Now I think that this problem may be attributed to my remote access via VNC.</p>
What is the correct way to use DS and DL in predict?

optimoose: <p>I have some code for a predict method (shown below) that I got to work after some trial and error. Is there some way to simplify this code? I am running the predict method in batches because if I don’t too much memory is used up. Still, I do not understand why I have to use the innermost loop. Instead of the innermost loop I tried <code>batch_preds = self.model.forward(xb)</code>, but that fails. Why? Is there a better way to do this? Here is the code:</p>
<pre><code>def predict(self, X):
    if self.gpuid is not None:
        device = torch.device(f"cuda:{self.gpuid}")
    else:
        device = torch.device("cuda")
    self.model.to(device)
    self.model.eval()
    with torch.no_grad():
        X = torch.tensor(X).float().to(device)
        predict_ds = TensorDataset(X)
        if self.predict_by_batch:
            predict_dl = DataLoader(predict_ds, batch_size=self.batch_size)
            preds = []
            for xb in predict_dl:
                for x in xb:
                    batch_preds = self.model.forward(x)
                    batch_preds = batch_preds.to('cpu')
                    preds.extend(list(batch_preds.numpy()))
            preds = np.asarray(preds)
        else:
            preds = self.model.forward(X).to('cpu').numpy()
    return np.squeeze(preds)
</code></pre>
<p>I should add that it seems to run slowly.</p>

optimoose: <p>So I am really new at pytorch and did not really know where to start. For my problem all of the data fits into memory so just using simple indexing of the batches worked. That runs very fast.</p>

ptrblck: <p>The inner loop should not be necessary. What kind of error are you seeing?</p>
Applying same name for layers not giving different weights

arjun_pukale: <pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2,2)
    def forward(self,x):
        x = self.fc1(x)
        x = self.fc1(x)
        return x
</code></pre>
<p>Here is my model, as you can see I have defined fc1 just once and using it in my forward function twice sequentially.<br>
So ideally i must get separate weights for the 2 layers in my forward function, but I am getting weights for just 1 layer. Please help!!!</p>
<pre><code class="lang-auto">for name,param in model.named_parameters():
    print(name,param, param.shape)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">fc1.weight Parameter containing:
tensor([[ 0.0108, -0.2179],
        [ 0.5695, -0.1553]], requires_grad=True) torch.Size([2, 2])
fc1.bias Parameter containing:
tensor([ 0.4658, -0.6482], requires_grad=True) torch.Size([2])
</code></pre>

pascal_notsawo: <p>I don’t see why this is a problem.<br>
Please have a look at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">documentation</a> if you haven’t already done so.</p>

ptrblck: <p>Why should you get two different weights? In your code snippet you are reusing the layer, which only contains one set of parameters.</p>
Multicore CPU parallelization

jlopez: <p>I need to parallelize the training of a ANN using n cores of a CPU not GPU, is that possible to achieve this in Pytorch, all the parallelization examples that I have seen here use GPU’s…</p>

ptrblck: <p>You could use <code>torch.set_num_threads(int)</code> to define the number of threads used for intraop parallelism and <code>torch.set_num_interop_threads(int)</code> for interop parallelism (e.g. in the JIT interpreter) on the CPU. Also, the env vars <code>OMP_NUM_THREADS</code> and <code>MKL_NUM_THREADS</code> might be useful.</p>
<p>More information are given in the <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html">CPU threading docs</a>.</p>
Insert a tensor into another tensor with different location depending on the index

kkjh0723: <p>I want to insert a small tensor B into another larger tensor A but the insert location changes over an index.(I’m not sure how to explain exactly…)<br>
For example,</p>
<pre><code class="lang-auto">a = torch.zeros((2,8,10))
b = torch.rand((2,8,2))
for i in range(8):
    a[:,i,i+0:i+2] = b[:,i,:]
</code></pre>
<p>Suppose the first dimension is batch dimension and the second dimension is time. Then I want to insert <code>b</code> into <code>a</code> but the location in third dimension changes over time. The location is not necessarily depending on the time.</p>
<p>Is there any tensor operation that can replace the loop which makes it faster in GPU?</p>
Adapting 2d gaussian conv weights at each pixel

sbelharbi: <p>hi,<br>
let’s say we have an image (single plane), and a corresponding matrix V. we want to smooth the image using a 2d gaussian kernel at each pixel (i, j). the gaussian kernel is parametrized with the mean that is the pixel value (i, j) of the image and standard deviation that is the corresponding value in the matrix V at the position (i, j).</p>
<p>the image could be single or minibatch.</p>
<ol>
<li>is there an efficient way to do this?</li>
<li>can we use one single filter that re-parametrize itself dynamically with respect to V?</li>
</ol>
<p>thanks</p>

ptrblck: <p>This might not be the most efficient way, but you could <code>unfold</code> the image to create image patches (same as the im2col operation), create the filters for all patches (since each patch would need a separate filter), and use a matrix multiplication to compute the output. Once this is done you could reshape the result back to the expected 4-dimensional shape.<br>
Note that this approach would use a lot of memory, since you are explicitly creating the patches and filters.</p>
Weighted Random Sampler

shakeel608: <blockquote>
<p>WeightedRandomSampler<br>
Can someone please explain how does <code>WeightedRandomSampler</code> work.<br>
It is confusing ?<br>
I have 5 imbalanced classes with count say [100, 20, 167, 700, 500,].<br>
How shall I choose weights for it .<br>
Could someone please explain it with a detailed example<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
</blockquote>

ptrblck: <p>The <code>WeightedRandomSampler</code> expects weights for each sample and uses it to draw the corresponding sample. <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">Here</a> is an example how to use it.</p>

shakeel608: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you but I still have one confusion<br>
for a batch_size = 50/100<br>
If minority class has 100 and majority class has 1000 samples respt.<br>
Does it draw these 50/100 samples  repeatedly during batch processing from minority one. I mean<br>
Does the data loader takes the same 50/100 samples from minority class after the first batch (as there is no sample point left now) but there are still a lot sample points left now for majority class</p>

ptrblck: <p>In the default setup (<code>replacement = True</code>), this would be the case and the sampler would oversample the minority class, i.e. draw the same samples multiple times (and augment them if a transformation is defined in your <code>Dataset</code>).</p>

shakeel608: <p>Thank you patrick<br>
I totally got it now</p>
Non-Zero Values for Zero Point in Symmetric Quantization

leimao: <p>From PyTorch official quantization tutorial:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In the following setting, we did <code>torch.per_tensor_symmetric</code>. However, the zero point for <code>QuantizedConv2d</code> is 63 instead of 0. At first, I thought it is due to that the kernel size is <code>1 x 1</code>. We could not use a scale factor of 0 and a zero point of 0 for symmetric quantization.</p>
<pre><code class="lang-auto">QConfig(activation=functools.partial(&lt;class 'torch.quantization.observer.MinMaxObserver'&gt;, reduce_range=True), weight=functools.partial(&lt;class 'torch.quantization.observer.MinMaxObserver'&gt;, dtype=torch.qint8, qscheme=torch.per_tensor_symmetric))
Post Training Quantization Prepare: Inserting Observers

 Inverted Residual Block:After observer insertion

 Sequential(
  (0): ConvBNReLU(
    (0): ConvReLU2d(
      (0): Conv2d(
        32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32
        (activation_post_process): MinMaxObserver(min_val=inf, max_val=-inf)
      )
      (1): ReLU(
        (activation_post_process): MinMaxObserver(min_val=inf, max_val=-inf)
      )
    )
    (1): Identity()
    (2): Identity()
  )
  (1): Conv2d(
    32, 16, kernel_size=(1, 1), stride=(1, 1)
    (activation_post_process): MinMaxObserver(min_val=inf, max_val=-inf)
  )
  (2): Identity()
)
..........Post Training Quantization: Calibration done
Post Training Quantization: Convert done

 Inverted Residual Block: After fusion and quantization, note fused modules:

 Sequential(
  (0): ConvBNReLU(
    (0): QuantizedConvReLU2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.1516050398349762, zero_point=0, padding=(1, 1), groups=32)
    (1): Identity()
    (2): Identity()
  )
  (1): QuantizedConv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), scale=0.17719413340091705, zero_point=63)
  (2): Identity()
)
Size of model after quantization
Size (MB): 3.631847
..........Evaluation accuracy on 300 images, 66.67
</code></pre>
<p>Later, in some of my own experiments, I found for some <code>QuantizedConv2d</code> whose kernel size is <code>3 x 3</code>, there could be non-zero zero points for symmetric quantization as well. How to understand the zero point in this context? Thank you.</p>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [6, 3, 5, 5], but got 3-dimensional input of size [3, 48, 48] instead

baglan_zhubatkanov: <pre><code class="lang-auto">from PIL import Image
from IPython.display import display
import matplotlib.image as mpimg
img = mpimg.imread('train/train2/plastic472.jpg')
imgplot = plt.imshow(img)
image= Image.open('train/train2/plastic472.jpg')
plt.show()
im= transform(image)
net = ConvNet()
net.load_state_dict(torch.load(PATH))
outputs = net(im)
</code></pre>
<p>I tried to test my NN, but I got this error, no such errors were while training</p>

pchandrasekaran: <p>Could you post your NN code?</p>
Can I set unequal load on 2 GPUs using nn.DataParallel?

Megh_Bhalerao: <p>Hello all,<br>
I am doing <strong>distributed training</strong> using <code>model  = nn.DataParallel(model, device_ids=[0, 1])</code>. Is there any option to indicate how much <strong>fraction of memory</strong> usage should which GPU or something like that? Because both the GPUs which I have <strong>might not be equally occupied</strong> and hence, when I do distributed training, one might <strong>OOM</strong>.<br>
Thanks,<br>
Megh</p>

osalpekar: <p>I don’t believe there is any such option. In DataParallel (and DistributedDataParallel), every GPU will have a replica of the model for local training, and every GPU will see input batches of the same size. Thus the amount of memory used by each rank (each GPU) in distributed training is approximately the same. If one of your GPUs is OOMing, you can try to:</p>
<ul>
<li>reduce the batch size (though this will reduce the batch size for every rank, even on the GPU with enough memory)</li>
<li>use an optimizer that stores less local state (such as SGD as opposed to Adam)</li>
</ul>

Megh_Bhalerao: <p>Thank you <a class="mention" href="/u/osalpekar">@osalpekar</a>. But, if the batch size remains the same across the multiple GPUs, where is the <em>distribution</em> of training happening? I mean which part of the <strong>memory load on a single GPU</strong> is getting split across multiple GPUs? Please let me know if I am missing something.</p>

osalpekar: <p>Each Data Loader pairs with one DDP instance. So if you define a batch size of 64 in your data loader, each replica trains on a size-64 batch. When training on 2 GPUs, this essentially means you train on 64*2=128 samples in one iteration. Each replica performs the forward pass independently on their separate batches, then in the backward pass, they communicate gradients computed with each other and take the average of them, so each replica has gradients as if they trained on an entire 128-sample batch themselves. Finally, each replica performs the optimization step using the averaged gradients, so they end up with the same model weights at the end of each iteration.</p>
<p>In essence, the speed-up comes from the fact that with <code>n</code> GPUs, you are able to train on <code>n</code> times as much data in each iteration. For this reason, the memory load on a single-node model and a distributed model are essentially the same (there may be small overheads for synchronizing the gradients but these are unlikely to influence which model/hyperparameters are chosen).</p>

Megh_Bhalerao: <p>Thanks for the detailed explanation! That makes sense.</p>
Trouble using indexing over the mini-batch

Jake_Sokol: <p>I am having trouble implementing this expression using indexing, and would like to know how to implement it without iterating over the mini-batch.</p>
<ul>
<li>max_args has shape (batch_size, num_channels)</li>
<li>each chunk has shape (batch_size, x, y, y)</li>
<li>x.shape[2] == x.shape[3] == y</li>
</ul>
<pre><code class="lang-auto">out = []
        for i, chunk in enumerate(chunks):
            temp = []
            for j in range(max_args.shape[0]):
                temp.append(chunk[j, max_args[j, i], :, :].view(1, 1, x.shape[2], x.shape[3]))
            out.append(torch.cat(temp, 0))
out = torch.cat(out, 1)
</code></pre>
Why doesn't setting random seed give same performance across runs?

Megh_Bhalerao: <p>Hello everyone,</p>
<p>I am training some deep learning models using <code>pytorch</code>, which also includes the usage of <code>numpy</code>. Since the randomisation is not truly random and it is <strong>pseudo-random</strong>, why aren’t the numbers i.e. accuracy etc same across <strong>different runs</strong>?</p>
<p>I am doing <code>torch.cuda.manual_seed(seed_val)</code> to set the random seed.</p>
<p>I mean, even if I do not set some random seed, there should be some default random seed according to which my code must run and give same results across different runs. Is there something more to it?</p>
<p>Please do let me know if something is not clear.<br>
Thanks,<br>
Megh</p>

ptrblck: <aside class="quote no-group" data-username="Megh_Bhalerao" data-post="1" data-topic="103774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/megh_bhalerao/48/13547_2.png" class="avatar"> Megh_Bhalerao:</div>
<blockquote>
<p>I mean, even if I do not set some random seed, there should be some default random seed according to which my code must run and give same results across different runs. Is there something more to it?</p>
</blockquote>
</aside>
<p>By default the system PRNG is used, if I’m not mistaken, so your code will not be deterministic if you don’t properly seed it manually.</p>
<p>Since you are using 3rd party libraries, such as numpy, I would recommend to also seed them.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> for more information.</p>

Megh_Bhalerao: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a>. Yes, I have gone through the docs.<br>
Thanks again,<br>
Megh</p>
What is the difference between these training methods in Pytorch?

Kingsmove: <p>Hi there, I am a 3-month freshman who is doing small NLP projects with Pytorch.<br>
Recently I am trying to reappear a GAN network introduced by a paper, using my own text data, to generate some specific kinds of question sentences.</p>
<blockquote>
<p>Here is some background… If you have no time or interest about it, just kindly read the following question is OK.<br>
As that paper says, the generator is firstly trained normally with normal question data to make that the output at least looks like a real question. Then by using an auxiliary classifier’s result (of classifying the outputs), the generator is trained again to just generate the specific (several unique categories) questions.<br>
However, as the paper do not reveal its code, I have to do the code all myself. I have these three training thoughts, but I do not know their differences, could you kindly tell me about it?<br>
If they have almost the same effect, could you tell me which is more recommended in Pytorch’s grammar? Thank you very much!</p>
</blockquote>
<p>Suppose the discriminator loss to generator is loss_G_D, the classifier loss to generator is loss_G_C, and loss_G_D and loss_G_C has the same shape, i.e. [batch_size, loss value], then what is the difference?<br>
1.</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss_G_D = loss_func1(discriminator(generated_data))
loss_G_C = loss_func2(classifier(generated_data))
loss = loss_G+loss_C
loss.backward()
optimizer.step()
</code></pre>
<ol start="2">
<li></li>
</ol>
<pre><code class="lang-auto">optimizer.zero_grad()
loss_G_D = loss_func1(discriminator(generated_data))
loss_G_D.backward()
loss_G_C = loss_func2(classifier(generated_data))
loss_G_C.backward()
optimizer.step()
</code></pre>
<ol start="3">
<li></li>
</ol>
<pre><code class="lang-auto">optimizer.zero_grad()
loss_G_D = loss_func1(discriminator(generated_data))
loss_G_D.backward()
optimizer.step()

optimizer.zero_grad()
loss_G_C = loss_func2(classifier(generated_data))
loss_G_C.backward()
optimizer.step()
</code></pre>
<p>Additional info: I observed that the classifier’s classification loss is always very big compared with generator’s loss, like -300 vs 3. So maybe the third one is better?</p>
SyncBatchNorm models and converting back to BatchNorm

blackberry: <p>Is there a way to create the new CNN models with <code>SyncBatchNorm</code>, in <code>torchvision.models</code>, apart from using the <code>convert_sync_batchnorm</code> function?</p>
<p>And, is there a simple way to convert the trained models back to regular <code>batchnorm</code>, for backward compatibility?</p>
DataParallel: will modifications to `model.module` broadcast to all GPUs?

zeakey: <p>In this case I made some modifacations to the parameters of a <code>DataParallel</code> wrapped model.</p>
<p>Such as</p>
<pre><code class="lang-auto">model = resnet50()
model = nn.DataParallel(model).cuda()
# several parameter reinitialization tricks here
model.module.conv1.weight.data = xxx
</code></pre>
<p>In my understanding there is a copy of the original model in each GPU device.</p>
<p>My question is: ‘will the aforementioned modifications to the <code>nn.DataParallel</code> wrapped model be broadcasted to all GPUs’?</p>
<p>And how about if I change the architecture of the model (such as replacing a conv3x3 in the original<br>
model with dilated convolution)?</p>
<hr>
<p>If it won’t be broadcasted to all other GPUs, how can I broadcast the modifications to all other divices<br>
INSTANTLY?</p>

ptrblck: <p>Would it be possible to apply your manipulations to the model before wrapping it to <code>DataParallel</code>?<br>
I think this would be the cleaner approach.</p>

zeakey: <p>Because I will dynamicaly change the module to perform some kind of <code>online network pruning</code>,it’s impossible to apply the manipulations before wrapping to <code>DataParallel</code>.</p>

ptrblck: <p>Would re-wrapping into <code>nn.DataParallel</code> work in this case? Or are you manipulating the model during training?</p>

zeakey: <p>Yes I’m manipulating the model during training (after each epoch).</p>
<p>So I think re-wrapping the model to <code>nn.Parallel</code> would be okay.</p>
<pre><code class="lang-auto">model = nn.DataParallel(model)
# manipulate the model
model = model.module
model.conv1 = xxxxx
# re-wrap
model = nn.DataParallel(model)
</code></pre>
<p>Is this right?</p>

ptrblck: <p>Re-wrapping after the epoch should be alright.<br>
However, I would recommend to create some dummy example and make sure the manipulation an re-wrapping is really working, e.g. set all parameters to zero and check the parameters in the next iteration for these values.</p>

kwanUm: <p>Does the re-wrapping technique works well for you? I’d imagine it’s slow to use as wrapping with dataparallel each time copies ALL its weights to other GPUs, rather than copying just the ones needed when pruning the weights.</p>

blackberry: <p>I have a similar question: I am changing the <code>requires_grad</code> of module parameters after wrapping with <code>DataParallel</code>. In <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc"><code>DistributedDataParallel</code></a>, which is now recommended instead of <code>DataParallel</code>, there is a warning that says “<strong>don’t do it!</strong>”. But I do not see the same warning for <code>DataParallel</code>. I agree that the proper way is to re-wrap it after the change, but that is a little more code.</p>
Memory efficient way to create block triangular matrices

wasabi: <p>Hi guys, I want my weight matrix to be block triangular matrix (and maintain that way during training), what will be the memory-efficient way to achieve this? Currently, I’m creating a mask using torch.triu (and Kronecker product with an all-one matrix of my block size), then register_hook using this mask matrix. However, my weight matrix is quite large so I would like to avoid storing the mask matrix.</p>
<p>There’s a straightforward way that I can implement the triangular matrix multiplication using a list, but are there any existing APIs I can use to avoid that path? Thanks.</p>
How to multiply array elements based on repeated elements in a second array

drelyea_blackberry: <p>I have what seems like a simple problem, but I cannot find answers for it anywhere. If I have two arrays, I want to multiply/combine the elements of one of them based on whether or not the elements in the other are sequential or repeated. For instance,</p>
<pre><code class="lang-auto">array_with_repeated_elements = tensor([1, 2, 0, 0, 2, 2, 2, 1, 0, 0])
# could just as well be [a, b, c, c, d, d, d, e, f, f]
array_to_be_multiplied = tensor([1., 3., 5., 2., 2., 7., 2., 4., 3., 4.])

desired_output = tensor([1, 3, 10, 28, 4, 12])
</code></pre>
<p>In numpy, this can be done easily:</p>
<pre><code class="lang-auto">first_index_of_each_sequence = np.hstack([0,np.where(array_with_repeated_elements[1:] != array_with_repeated_elements[0:-1])[0]+1])
# this creates array([0, 1, 2, 4, 7, 8])
desired_output = 1-np.multiply.reduceat(array_to_be_multiplied, first_index_of_each_sequence)
</code></pre>
<p>I can’t seem to do this in pytorch. The best guess I have is this monster:</p>
<pre><code class="lang-auto">first_index_of_each_sequence = torch.cat([torch.LongTensor((0,)), torch.where(array_with_repeated_elementst[1:] != array_with_repeated_elementst[0:-1])[0]+1, torch.LongTensor((len(array_with_repeated_elements),))])
# makes tensor([0, 1, 2, 4, 7, 8, 10])
size_of_each_sequence = first_index_of_each_sequence[1:] - first_index_of_each_sequence[0:-1]
# makes tensor([1, 1, 2, 3, 1, 2])
full_length_array_of_ascending_index_elements = torch.arange(len(size_of_each_sequence)).repeat_interleave(size_of_each_sequence)
desired_output_base = torch.zeros(len(size_of_each_sequence))
# makes tensor([0, 1, 2, 2, 3, 3, 3, 4, 5, 5])
desired_output_base.index_add_(0, full_length_array_of_ascending_index_elements, torch.log(array_to_be_multipliedt))
# does what I want in log space, but ew if I ever have a zero
desired_output = torch.exp(desired_output_base)
# duh
</code></pre>
<p>Does anyone have any ideas on how to do this nicely? The easy numpy implementation suggests I’ve missed something in pytorch…</p>

drelyea_blackberry: <p>That answer wouldn’t work for sequences with zeros or near-zeros in them (also really large numbers, but they’d render this whole thing a bit moot). I need a solution which won’t have numerical errors. That’s why I built the monstrosity up above.</p>
<p>In numpy, this is trivial. Is it not trivial in pytorch? Again, seems like I’ve missed some function.</p>
How does PyTorch utilize multiple GPU?

wasabi: <p>Hi guys, I was reading this DataParallel tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a> and I have several questions, suppose I have two gpus: gpu A and gpu B:</p>
<ol>
<li>How is the model allocated on A and B? Based on my experiments it seems that a copy of the model will be made on both A and B. That is, if my model has size 1G, then both A and B will use 1G memory. Is this correct?</li>
<li>How is the data allocated on A and B? From the tutorial, it seems you only need to use nn.DataParallel on the model, and not on the data?</li>
<li>How does batch size affect memory allocation? My gpu can allocate using a small batch size (say 10), but run out of memory on large batch size (say 30).</li>
</ol>
<p>Appreciate any answers/references. Thanks.</p>
PyTorch Dataloader and training

aleks_ban: <p>Hello everyone,</p>
<p>I am brand new to using PyTorch and just trying to implement some older computer vision white papers to get a hang of how this all works. I had two main questions which I will ask and then elaborate on below, any guidance would be much appreciated.</p>
<ol>
<li>
<p>How to use the PyTorch DataSet class and Data loader to train my network if the data set is a folder containing a few thousand images.</p>
</li>
<li>
<p>How do the images get passed to the model class I create?</p>
</li>
</ol>
<p>I am implemented the semantic segmentation model known as SegNet. I have been able to figure out how to build my model at least in terms of building the constructor and forward definition. I am now trying to build my train.py, but am really struggling with using the PyTorch data loader for image data sets contained in a folder directory and not a csv setup. Can anyone point me to an example that shows me how this works? And possibly an explanation? All the examples I find use CSV files where as I just have a few thousand .png files for training and testing.</p>
<p>I am also confused on how the images are passed to the forward function in my class “class SegNet(nn.Module):”. I never see a direct pass of the images to the network model in most examples and just wondered how they get there. What I mean is most examples I see call the class in the the train.py script and then load the data but is there some method to passing the data directly to the forward definition? I never see a directly call to the forward function where some variable storing the dataset is passed. I’m pretty new to all this so any help would be much appreciated cause I might be thinking about this all wrong.</p>

AndreasW: <p>Hi,<br>
this tutorial <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> have a example for such a dataloader which you are sreaching.</p>
<p>Greets</p>
Failing to pass a simple tensor into a simple deep nn

Eric_Cartman: <p>I’m failing to pass a simple tensor into a simple deep nn.</p>
<p>What am I doing wrong?</p>
<p>ERROR MESSAGE:</p>
<p>“RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x2 and 1x128)”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2da01323ed4b0d15bbaa094c411f358dcb0a6b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2da01323ed4b0d15bbaa094c411f358dcb0a6b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2da01323ed4b0d15bbaa094c411f358dcb0a6b1.png" alt="image" data-base62-sha1="pwc3SCKP7pAapMMeuUOnxuuo1eF" width="407" height="500" data-dominant-color="2C2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">668×819 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>import numpy as np</p>
<p>import torch as t</p>
<p>import torch.nn as nn</p>
<p>import torch.nn.functional as F</p>
<p>import torch.optim as optim</p>
<p>class DNN(nn.Module):</p>
<pre><code>def __init__ (self,input=1,h1=128,output=1):

    super().__init__()

    self.input = input

    self.h1 = h1

    self.output = output

    self.fc1 = nn.Linear(input,h1)

    self.fc2 = nn.Linear(h1,output)

def forward(self,kiwi):

    out = F.relu(self.fc1(kiwi))

    out = self.fc2(out)

    return out
</code></pre>
<p>dnn = DNN()</p>
<p>data = t.tensor([1,2])</p>
<p>dnn.forward(data)</p>

torch_bearer: <p>Try sending a 1x1 tensor instead of 1x2</p>
<p>data = <code>t.tensor([1])</code> instead of <code>t.tensor([1, 2])</code></p>

Eric_Cartman: <p>i did it but another error has emerged:</p>
<p>“RuntimeError: expected scalar type Long but found Float”</p>

torch_bearer: <p><a class="mention" href="/u/eric_cartman">@Eric_Cartman</a></p>
<p>This appears to be a issue with data types.<br>
Try sending <code>t.tensor([1.0])</code></p>
<p>Usually to test a network, I send random data using <code>t.randn(shape)</code><br>
In this case, <code>shape</code> = <code>(1,  )</code></p>

Eric_Cartman: <p>I did this modifications and everything went fine:</p>
<p>input = 2</p>
<p>data = t.tensor([1,2],dtype = t.float)</p>
<p>Thanks for your attention, <a class="mention" href="/u/torch_bearer">@torch_bearer</a> !!!</p>
Ecosystem form does not respond when I hit Submit Project

benedekrozemberczki: <p>I filled out the form but hitting the Submit Project button does not give any response about the submission being successful.</p>
<p>Is this intended or a glitch?</p>
<p>Bests,</p>
<p>Benedek</p>

ptrblck: <p>Are you still facing this issue?</p>

benedekrozemberczki: <p>Yes, it is still unresolved. I wanted to submit this project:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/benedekrozemberczki/pytorch_geometric_temporal" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://repository-images.githubusercontent.com/275276531/14c5bf00-bc54-11ea-9650-befadb36c47b" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/benedekrozemberczki/pytorch_geometric_temporal" target="_blank" rel="noopener nofollow ugc">benedekrozemberczki/pytorch_geometric_temporal</a></h3>

<p>A Temporal Extension Library for PyTorch Geometric - benedekrozemberczki/pytorch_geometric_temporal</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> are you aware of any issues submitting projects to the ecosystem?</p>

jspisak: <p>No not aware of any issues. Let me try a test and see. If there is still an issue, we can just connect directly and get this figured out…</p>

jspisak: <p>I ran a quick test and things look fine and i received a submission email. Please try again to submit and i’ll take a look when it comes in: <a href="https://pytorch.org/ecosystem/join">https://pytorch.org/ecosystem/join</a></p>
<p>thanks and apologies for any issues!</p>

benedekrozemberczki: <p><a class="mention" href="/u/jspisak">@jspisak</a> Just submitted the form again.</p>

benedekrozemberczki: <p><a class="mention" href="/u/jspisak">@jspisak</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> What happened in the end? Did you receive anything?</p>

jspisak: <p>I ran a test and the form seemed to work fine (i.e. i received an email with the info). Yours did not come through through. Feel free to send me an email directly and I can take a look. <a href="mailto:jspisak@fb.com">jspisak@fb.com</a>. So sorry for this!</p>

benedekrozemberczki: <p><a class="mention" href="/u/jspisak">@jspisak</a> I have sent you an e-mail with the answers to the questions from the form.</p>
Network using Dropout giving different results with eval() than with train()

jlph: <p>I am experiencing unexpected changes to the results of my network when I switch between model.eval() and model.train(). The model gives good results while model.training=True (~88% accuracy on test data) but when I switch to model.eval() the accuracy is reduced to ~55% (the same test data).<br>
The network aims to predict one of two outputs A or B. Model architecture is: 278 inputs, 2 hidden layers of 90 neurons, 2 outputs. Optimiser: SGD. Activation: Tanh. Dropout is used with default arguments.</p>
<p>Here is the breakdown by outcome, A and B:</p>
<p>with model.train():<br>
percentage correct predicting A: 91%<br>
percentage correct predicting B: 79%</p>
<p>with model.eval():<br>
percentage correct predicting A: 42%<br>
percentage correct predicting B: 98%</p>
<p>There is a large swing towards predicting B rather than A when eval() mode is turned on.</p>
<p>There are a number of inputs to the model that are either 0 or 1, one of the inputs will be 1 at a time, alongside other inputs of varying value - 34 of 278 inputs are of this binary nature. Could using inputs that are 0 a lot of the time with dropout be the cause of these changes in predictions? When dropout is not used this does not occur; the results are the same in eval mode as they are in training mode.</p>
<p>Any help would be appreciated.</p>
Masking one dimension

Anton_Z: <p>Suppose I have a tensor A with the following shape:</p>
<pre><code class="lang-auto">torch.Size([5, 16, 5000, 3])
</code></pre>
<p>I also have a mask of the same shape:</p>
<pre><code class="lang-auto">torch.Size([5, 16, 5000, 3])
</code></pre>
<p>If I apply this mask M directly to the tensor A via</p>
<pre><code class="lang-auto">A = A[M]
</code></pre>
<p>I end up with a flattened tensor with single dimension.</p>
<p>However, I would like to mask out only along dimension 2. In other words, I would like to get a tensor of the shape</p>
<pre><code class="lang-auto">torch.Size([5, 16, 5000 - N, 3])
</code></pre>
<p>where N is the number of entries for which mask M is False.</p>
<p>What is the way of doing this?</p>

phan_phan: <aside class="quote no-group" data-username="Anton_Z" data-post="1" data-topic="103889">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anton_z/48/30069_2.png" class="avatar"> Anton_Z:</div>
<blockquote>
<p>However, I would like to mask out only along dimension 2.</p>
</blockquote>
</aside>
<p>If you want to mask out only in dimension 2, then you must have a mask that matches dimension 2.<br>
In your example, the mask must be shaped <code>torch.Size([5000])</code></p>
<p>Only then could you do <code>A = A[:, :, M]</code></p>

Anton_Z: <p>Thank you!</p>
<p>So there is no way to vectorize it along batch and channels dimensions?</p>
<p>I am going to use precomputed masks during training (A tensor is trainable, while M is not).</p>
Libtorch: how to reshape to a dynamic dims

pytorch_lzwhard: <p>I have a <strong>dynamic shape</strong> maybe ( 1, …,). then How to view a tensor to this shape in <strong>C++</strong> with libtorch.<br>
I know <code>tensor.view({1, 3, 4, 4})</code>, but this is static with pre-known dim.<br>
how about <code>tensor.view(dynamic_shape)</code>? thank you!</p>

ptrblck: <p>I’m not sure I understand the question correctly. You could still define the <code>dynamic_shape</code> during the runtime, which would make it “dynamic”, wouldn’t it?</p>

pytorch_lzwhard: <p>thank u for your replay, I mean vector as input shape</p>
Does BatchNorm statistics need to be updated in Transfer Learning?

Donato_Caf: <p>Hi, I’m a little bit confused about when to update the Batch Norm statistics gamma and beta.</p>
<p>I’m doing transfer learning on a pre-trained model on the VGGFace2 Dataset.</p>
<p>My new dataset is quite similar (it contains only faces), so should I update the statistics or set Batch Norm layers to eval() ?</p>
Why the inference speed differs a lot between windows and linux on pytorch-cpu

lxtong01: <p>why the inference speed differs a lot between windows and linux on pytorch-cpu?<br>
linux is faster than windows using the same pytorch cpu version without cuda?<br>
why why why!</p>
<p>really important to me!<br>
hope i can get the answer</p>

googlebot: <p>There is some inherent windows problem with either L2 cache or memory page management, this difference is not specific to pytorch. Also, your builds may be not identical in regards to MKL, avx, openmp threading.</p>
Implementig new distributions

Youcef_Moualek: <p>Hello.<br>
How can I implement the Generalized gamma distribution and the Gompertz distribution using TransformedDistribution (if it’s possible) ?</p>

googlebot: <aside class="quote no-group">
<blockquote>
<p>Generalized gamma distribution</p>
</blockquote>
</aside>
<p>Probably not possible, would imply a bijective transform from 2 to 3 param family</p>
<aside class="quote no-group">
<blockquote>
<p>Gompertz distribution</p>
</blockquote>
</aside>
<p>Not sure that a transformation into this exists.</p>
<p>You can just write log_prob functions manually, for example adapting <a href="https://github.com/scipy/scipy/blob/ab9e9f17e0b7b2d618c4d4d8402cd4c0c200d6c0/scipy/stats/_continuous_distns.py#L2916" rel="noopener nofollow ugc">scipy.stats code</a></p>
How to compile with NNPack or MKLDNN

liangstein: <p>Compile pytorch from the latest git library. How to set the variables to let cmake detect NNPack or MKLDNN , I found there are variables like NO_MKLDNN  WITH_MKLDNN  MKLDNN_LIBRARY  MKLDNN_LIB_DIR  MKLDNN_INCLUDE_DIR .  How to pass these variables to cmake?</p>
<p>Does NNPack or  MKLDNN speed up for double precision?</p>

liangstein: <p>Top Up!<br>
I really need to improve the inference performance on CPU. After I found the benckmark of mkldnn, I want to have a try. However, compiling pytorch with mkldnn seems not native, because I have to struggle with variables like NO_MKLDNN,  WITH_MKLDNN, MKLDNN_* …  I wish compiling pytorch with mkldnn will be much easier, just as easy as linking mkl or openblas.</p>

mingfeima: <p>hi, Xiao Liang. you need to install MKLDNN from conda first and then follow the installation steps from README.<br>
conda install -c mingfeima mkldnn</p>
<p>Performance speedup you can get strongly depends on your CPU type. The optimization from MKLDNN is mostly for Xeon CPU. We also have some <a href="https://github.com/intel/pytorch#bkm-on-xeon" rel="nofollow noopener">BKM</a> for better CPU performance. i upload some benchmark numbers at <a href="https://github.com/mingfeima/convnet-benchmarks/tree/master/pytorch" rel="nofollow noopener">link</a>. you can take a look.</p>
<p>The optimization job is still working in progress and we will continuously boost up cpu performance. Conv performance should be much faster with MKLDNN, rnn optimization is currently under review.</p>
<p>If you run on Xeon CPU, NNPACK is definitely not as performant as MKLDNN, we double checked this before.</p>
<p>Let me know if you have trouble compiling with MKLDNN.</p>

lxtong01: <p>hi，did you solve the problem? how to install pytorch with nnpack on cpu?<br>
really looking forward to your reply</p>