akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How does batch normalization work with multiple GPUs

username: weedwind
cooked: <p>I am going to use 2 GPUs to do data parallel training, and the model has batch normalization. I am wondering how pytorch handle BN with 2 GPUs. Does each GPU estimate the mean and variance separately? Suppose at test time, I will only use one GPU, then which mean and variance will pytorch use?</p>

username: Mata_Fu
cooked: <p>Do you have the answer right now?</p>

username: Andrew_Paint
cooked: <p>According to the document of PyTorch, the batch norm performs over mini-batch, namely, per GPU</p>

username: liyz15
cooked: <p>See <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.SyncBatchNorm" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.SyncBatchNorm</a>, with DistributedDataParallel and SyncBatchNorm then BN can be performed on multiple GPUs.</p>
Proper way to use 2D CNN with video input

username: Theocharis
cooked: <p>Hello,i was wondering if the following is the right way to process a video with 2D CNN:</p>
<pre><code class="lang-auto">self.cnn=models.alexnet(pretrained=True)
self.cnn.classifier[-1]=Identity()
...
batch_size, timesteps, C, H, W = x.size()
c_in = x.view(batch_size * timesteps, C, H, W)
c_out = self.cnn(c_in)
</code></pre>
<p>This seems to work perfect with AlexNet,but when i use a bigger network(say googlenet or resnet) it’s performance downgraded.In addition limited data is not the matter, as there are papers using such big networks with the same Dataset.The only thing to blame from my pov could be the fact that Alexnet doesn’t have BatchNorm, whilst the rest of them do.<br>
(My batch_size = 1 and i don’t use temporal padding)</p>

username: ptrblck
cooked: <p>Are you processing <code>c_out</code> somehow afterwards?<br>
Note that different models might use different names for the “last” layer, e.g. <code>model.fc</code> instead of <code>model.classifier</code>.<br>
A batch size of 1 with batch norm layers is usually not a good idea. You could try to replace the batch norm layers by e.g. instance norm, change the momentum argument, remove them, etc.</p>

username: Theocharis
cooked: <p>Thank you for your reply,<br>
c_out is then processed by an RNN (CNN+RNN+Log_Softmax and is trained with CTC loss).As far as different names are concerned this is definitely not the problem.I am going to try replacing batch norm layers and i 'll let you know.<br>
Thanks again.</p>
I want to analyze the PyTorch code implemented with AVX instructions

username: jiseongg
cooked: <p>I’m trying to study the low level implementation of some computations in neural network. I built the PyTorch from source with  following command.</p>
<pre><code class="lang-bash">DEBUG=1 NO_CUDA=1 python setup.py develop
</code></pre>
<p>I’ve succeeded to debug in C++ source level for <code>torch.add</code> function with cgdb. (I set the breakpoint at <code>at::native::add</code> function) But what I want to analyze is the SIMD instructions. I know PyTorch have added specialized AVX and AVX2 intrinsics for Tensor operations. AVX is what I’m interested in. (not AVX2) But I cannot go inside to that level.</p>
<ul>
<li>aten/src/TH/vector/AVX.cpp</li>
<li>aten/src/ATen/cpu/vec256/*</li>
</ul>
<p>I think the source files above are the candidates, but setting the breakpoints on some functions inside them failed. The code I’m using is the simple add, sub operation and the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">CNN codes in PyTorch tutorial.</a> But both don’t implement the function in those source files.</p>
<p>I’m curious which PyTorch codes(written in Python) implement AVX instructions. And I want to know if I missed some options when I built it.</p>
<p>Actually, I’ve googled about it, but I cannot figured it out. A lot of advice with respect to this kind of topic deal with cmake, but I couldn’t understand it because I’m not well aware of it.</p>

username: tom
cooked: <p>This could be due to inlining (at least when AVX is detected).<br>
Most of the action w.r.t. using AVX is going on in <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native/cpu" rel="nofollow noopener">ATen’s native/cpu</a> directory. This also uses the TensorIterator mechanism a lot. This is defined in the <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native/" rel="nofollow noopener">ATen native directory</a> and provides a common infrastructure for plain+SIMD CPU and GPU ops and does most pointwise things and also reductions.<br>
I would not look at the (Torch legacy) aten/src/TH* too much, as we’re working on moving things over to the fancy new way.</p>
<p>Best regards</p>
<p>Thomas</p>
Dataparallel pytorch share tensor

username: razvanc92
cooked: <p>I’m having a model which I want to run in parallel on multiple gpus. The code works well, but I’m having a problem. I’m doing something that requires an input and an adjacency matrix (which is static). I’ve tried to save the adjacency matrix under the model but it only saves it on the first model only. I’ve tried to pass it as a parameter in the forward function but it splits it in half. Any ideas how I could do this?</p>

username: ptrblck
cooked: <p>Could you try to register this matrix as a buffer using <code>self.register_buffer</code>?</p>

username: razvanc92
cooked: <p>Seems to be working. Thank you!</p>
Whether after updating param.grad, the grad is cleared automaticly

username: Yannis
cooked: <p>“log_alpha” is a variable(I think), every time I update its data with its grad, the grad will change into None.<br>
code as follow:<br>
1 alpha_loss = -torch.mean(self.log_alpha * (log_pis.double() + self._target_entropy))<br>
2 <span class="hashtag">#self</span>.log_alpha.retain_grad()<br>
3 alpha_loss.backward()<br>
4 grad = self.log_alpha.grad<br>
5 self.log_alpha = self.log_alpha - grad * 3e-4<br>
but if I use line 2 code, no error. So I’m a little confused about the grad clearing mechanism.<br>
&amp; I’m not sure if the grad will be added all the time.</p>

username: ptrblck
cooked: <p>It seems you are replacing <code>self.log_alpha</code> in your example.<br>
Have a look at these example, where we 1) replace <code>x</code>, 2) assign to a new tensor and 3) modify <code>x</code> inplace:</p>
<pre><code class="lang-python"># 1) replace tensor
x = torch.randn(2, requires_grad=True)
loss = torch.mean(x)
loss.backward()
print(x.grad)
&gt; tensor([0.5000, 0.5000])
grad = x.grad
x = x - grad
print(x.grad)
&gt; None

# 2) assign to other tensor
x = torch.randn(2, requires_grad=True)
loss = torch.mean(x)
loss.backward()
print(x.grad)
&gt; tensor([0.5000, 0.5000])
grad = x.grad
y = x - grad
print(x.grad)
&gt; tensor([0.5000, 0.5000])
print(y.grad)
&gt; None

# 3) inplace
x = torch.randn(2, requires_grad=True)
loss = torch.mean(x)
loss.backward()
print(x.grad)
&gt; tensor([0.5000, 0.5000])
grad = x.grad
with torch.no_grad():
    x.sub_(grad)
print(x.grad)
&gt; tensor([0.5000, 0.5000])
</code></pre>

username: Yannis
cooked: <p>I get it~ your explanation is so clear, thank you.<br>
If I update x with loss.backward() but without zero_grad(),  because I don’t use an optimizer. In this way, whether the grad of x will be restored and added all along？</p>

username: ptrblck
cooked: <p>It depends on the approach you are using. E.g. in 3) you would accumulate the gradients.</p>

username: Yannis
cooked: <p>Well, thank you so much~</p>
__getitem__ is called multiple times

username: lorenzo_fabbri
cooked: <p>I’m using a custom <code>Dataset</code> where the path to the images is created from a Pandas DataFrame. Out of curiosity I added a <code>print(idx)</code> to the <code>__getitem__</code> function and I noticed that it’s called twice (it prints two different indices) if I use 1 worker. If I use multiple workers, it’s called even more times. The batch size is 1, though.<br>
Am I missing something? Shouldn’t I get just one image? Moreover, it returns just one image, independently of the number of workers (as it should be).</p>

username: Nagesh
cooked: <p>Could you please share the code?</p>

username: lorenzo_fabbri
cooked: <p>It’s rather difficult to understand what it does without having the Pandas DataFrame (which I cannot share, I guess). But here’s the class:</p>
<pre><code class="lang-auto">class Data(Dataset):

    def __init__(self, mode, df, img_dir, site, transform):
        self.mode = mode
        self.df = df
        self.img_dir = img_dir
        self.site = site
        self.transform = transform
    
    def path_channel(self, channel, idx):
        experiment = self.df.loc[self.df.index[idx], 'experiment']
        plate = self.df.loc[self.df.index[idx], 'plate']
        well = self.df.loc[self.df.index[idx], 'well']

        path = os.path.join(self.img_dir, experiment, f'Plate{plate}', 
                            f'{well}_s{self.site}_w{channel}.png')
        
        return path
    
    def __getitem__(self, idx):
        print(idx)  # With 1 process and batch size 1, printed twice (different items)

        # Iterate over channels of one image (from file)
        all_channels = [np.array(Image.open(self.path_channel(ch, idx)), 
                                dtype=np.float32) for ch in range(1, 7)]
        img = np.stack([ch for ch in all_channels], axis=2)
        
        if self.mode == 'train':
            label = self.df.loc[self.df.index[idx], 'label'].astype('int32')

            return img, label
        elif self.mode == 'test':

            return img
    
    def __len__(self):
        return self.df.shape[0]
</code></pre>

username: ptrblck
cooked: <p>Each worker will create a batch and call into your <code>Dataset</code>'s <code>__getitem__</code>.<br>
For <code>num_workers=0</code>, the main thread will be used to create the batch. For <code>num_workers=1</code> you will use another additional process to fetch the next batch.</p>
Why the training and the Test goes in the same loop

username: OBouldjedri
cooked: <p>Hi I am new in pytorch,  I was wondering why should we use the same epoch loop for train and test</p>
<p>for epoch in range(1, args.epochs + 1):<br>
model.train_(train_loader, epoch)<br>
model.test_(test_loader, epoch)<br>
and how is that different from using 2 loops ? thank you</p>
<p>for epoch in range(1, args.epochs + 1):<br>
model.train_(train_loader, epoch)</p>
<p>for epoch in range(1, args.epochs + 1):<br>
model.test_(test_loader, epoch)</p>

username: ptrblck
cooked: <p>Usually you would like to validate your model after each training epoch to get a signal about your model’s ability to generalize, i.e. how high the validation accuracy is (or how low the validation loss gets). Using these validation metrics you could apply e.g. early stopping in order to stop the training once your model starts to overfit.<br>
If you train for some epochs and try to validate (or test) your model afterwards, you will just get the final validation metrics. Running the validation (or test) for several epochs sequentially doesn’t make sense, as the metrics won’t change and you’ll end up with the same values.</p>

username: OBouldjedri
cooked: <p>Thank you for your reply but why for the validation we focus only on accuracy best score to save the model what about the loss  , also some implementations uses<br>
for data, target in test_loader:</p>
<p>instead of</p>
<p>for batch_idx, (data, target) in enumerate(test_loader):</p>
<p>why ?</p>
<p>Thank you</p>

username: mmisiur
cooked: <p>In validation phase we care mostly about the general performance of the model rather than the loss. During training phase tracking training loss is very useful to see how the model actually process the data, but what is most important loss is strictly tied to backpropagation and updating weights, thus calculating loss is obligatory for training, but not for validation.<br>
But sometimes if you want to know better how the model behaves under the validation data it is also useful to keep track of test loss.</p>
<p>In the code you provided the only difference is that in</p>
<blockquote>
<p><code>for data, target in test_loader:</code></p>
</blockquote>
<p>you simply load batch (data and target labels) and don’t keep track of the index of the batch, but in</p>
<blockquote>
<p><code>for batch_idx, (data, target) in enumerate(test_loader):</code></p>
</blockquote>
<p>you load batch but also batch index which is sometime useful to keep track of progress or whatnot. You can check why the author went with keeping track of batch indexes simply by looking where he/she used it later in code.</p>
<p>Hope I helped</p>

username: OBouldjedri
cooked: <p>Thank you you saved the day</p>

username: OBouldjedri
cooked: <p>Thank you I appreciate it</p>
Is my custom loss function correctly defined?

username: cijerezg
cooked: <p>I define my custom loss as follows:</p>
<pre><code class="lang-auto">def custom loss(target,output,mask):
    output = torch.neg(output)
    loss = torch.add(target,output)
    loss = torch.pow(loss,2)
    loss = torch.mul(loss,mask)
    loss = torch.mean(loss)
</code></pre>
<p>I’ve seen many people use a class. But I also read a function would be fine. Would that loss work? target,output and mask are Variable.</p>
<p>Subsequently, in my training loop, I call loss.backward(). The program runs but I just want to confirm my implementation of the loss is correct.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Yes, your approach should work just fine.<br>
The advantage of using a class is that it can hold internal attributes you would have to pass otherwise to your functional approach.<br>
E.g. using <code>nn.CrossEntropyLoss</code>, you could define the <code>weight</code> or <code>reduction</code> argument while creating an instance of this class. If you are using the functional method (<code>F.cross_entropy</code>), you would need to pass these arguments in each call.<br>
While this might be cumbersome in some cases, it might make the code cleaner e.g. if you need to change the <code>weight</code> argument often.</p>
<p>Both APIs have some advantages and disadvantages and you can chose whatever feels right and makes the code clean. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Hey, I have one question about matrix operations

username: cstsunfu
cooked: <p>My target is pooling(max, min, avg, etc.) every sequence to one vector.</p>
<p>For example, given two sentences “I love Outman. I love Superman, too.”, after encoding, the first sentence length is 5(include punctuation) second is 6. I want to pool these two sentences to 2 vectors. And these sentences must encode in one round.</p>
<p>A simple idea is using the ‘scatter_’ function, but I have no idea about this.</p>
<pre><code class="lang-auto"># To simplify this question, I don't use batch.
# suppose we have the sentences embedding, 5+6=11 means the sum of the lengths of the sentences, 20 means the embedding dim.

sentence_embedding = torch.Tensor(11, 20)
# I have the split flag like this:
sentence_split_flag = torch.LongTensor([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0])

....
# After some oprations(depends on the pooling method), I want to get:
pooled_sentence_embedding = torch.Tensor(2, 20)
</code></pre>
<p>And I’m sorry for my poor English, if you are perplexed for anything, I will try my best to explain it.</p>

username: spanev
cooked: <p>Hi, If I understand correctly want you want to do, you could do this:</p>
<pre><code class="lang-auto">sentence_embedding = torch.Tensor(11, 20)

# We the equivalent indices instead of flags
# sentence_split_flag = torch.LongTensor([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]) 
# which you can also get with:
# sentence_split_idx = (sentence_split_flag == 1).nonzero().view(-1)
sentence_split_idx =  torch.LongTensor([0, 5])

pooled_sentence_embedding  = sentence_embedding.index_select(dim=0, index=sentence_split_idx)
</code></pre>

username: cstsunfu
cooked: <p>Thanks for your reply.</p>
<p>I want to pool all word of one sentence to one, not just get the first.<br>
And in the sentence_split_flag, 1 means the start of one sentence.</p>
<pre><code class="lang-auto"># the sentence segment could be represented like this :
sentence_segment = torch.LongTensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
</code></pre>

username: spanev
cooked: <p>My bad, I misunderstood.</p>
<p>So I’m not sure of the optimal way of doing it.<br>
But maybe doing something like this might help you:</p>
<pre><code class="lang-auto">segments_sizes = (5, 6)
pooled_segs = []
for seg in torch.split(sentence_embedding, segments_sizes, dim=0):
    pooled_segs.append(pooling_op(seg))
pooled_sentence_embedding = torch.cat(pooled_segs, dim=0)
</code></pre>

username: cstsunfu
cooked: <p>Yes, it works. But I think there is a more efficient method. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><br>
Maybe I could scatter the sentences which dim is (11, 20) to (2, 6, 10), which sentence length is smaller than 6 should be padded. Then pooling (2, 6, 10) to (2, 1, 10), and then reshape to (2, 10).<br>
So I want to know one quick method to get the scatter index from sentence_split_flag or sentence_segment.</p>
Running cuda in dataloader is slow

username: hubert0527
cooked: <p>Hi,</p>
<p>I face an unsolvable problem and looking for any advice here…</p>
<p>In my use case, I have a special model M that processes the input images in the dataloader. And the model is quite huge, so it always requires GPU execution speed up.<br>
In this case, I have two solutions:</p>
<ol>
<li>Straightforward one. Use the main thread of the dataloader only. (It works, but very slow.)</li>
<li>Run the dataloader with multiple workers. Use <code>torch.multiprocess.set_start_method("spawn")</code> to let the child processes to acquire cuda. (15 times slower that 1…)</li>
</ol>
<p>As execution time is critical in my case, so I keep finding faster implementation.<br>
I originally believe that 2. should be faster.<br>
But in reality, it runs 15 times slower than running with main thread only.<br>
It seems like “spawn” backend significantly slows down the child process creation and the dataloader does not reuse the used child process (I can see the GPU memory usage goes ups and downs).</p>
<p>Any suggestion would be a great help!<br>
Sincerely thanks!</p>
<p>P.S.<br>
I also tried to use <code>M.share_memory</code> to share the model M among child processes, but it seems do not affect the execution speed at all.</p>

username: colesbury
cooked: <p>Yeah, I would avoid (2). Accessing the GPU from dataloader workers is the path to ruin.</p>
<p>You don’t have to do all your preprocessing in the dataloader. For example, you can do your file load and CPU pre-processing in the data loader, but do the GPU operations afterwards:</p>
<pre><code class="lang-auto">for sample_cpu in dataloader:
   sample_gpu = preprocess_gpu(sample_cpu)
   train(sample_gpu) # or whatever
</code></pre>
<p>In general, the way to make GPU operations fast is:</p>
<ol>
<li>batch operations</li>
<li>avoid CPU-GPU synchronizations</li>
<li>make sure the underlying ops are efficient</li>
</ol>

username: hubert0527
cooked: <p>Thanks for your reply!</p>
<p>But my use case (I’m trying model-based data augmentation) strictly constraints that each sample is processed differently.<br>
In this case, as I can’t process the samples in batch-wise, I believe that running multiprocessing is the last hope for me ;(</p>
Confusion about loss.backward() and how it updates gradients

username: RobbinDG
cooked: <p>Hello!</p>
<p>I’m an experienced programmer, familiar to neural nets (theoretically) but new to pytorch. I’m trying out pytorch in <code>C++</code> and am currently writing the learning stage of my simple network (1 hidden layer).</p>
<p>I have an optimizer and have calculated a loss tensor, all according to the tutorial found <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">here</a>. The tutorial then calls <code>d_loss_fake.backward()</code>. I did some digging and this apparently updates the gradients of the network. I am confused however about how it achieves this, as I could not find any relations between the loss tensor and the network. Concretely: how does <code>.backward()</code> know which weights to update if there is no connection between the loss tensor and the network?</p>
<p>Sorry for the beginner question. I hope this isn’t a duplicate, I’ve been googling for a couple of hours now. Thanks in advance!</p>

username: andreaskoepf
cooked: <p>Maybe a starting point is <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/autograd.html</a></p>
<p>Behind the scenes PyTorch tracks all operations on tensors with <code>requires_grad == true</code> and builds a computation graph during the forward pass. It knowns how the loss value was calculated and can automatically back-propagate the gradient step by step from the loss (or any scalar model output) to the model parameters.</p>
Std::system_error when distributed training using gloo

username: hanb302
cooked: <p>Trainning resnet101 for 19k steps (3 nodes), I got “Unexpected poll revent: 25 on socket: 9: Software caused connection abort”, what can I do to fix this?</p>
<p>thanks</p>

username: abh31
cooked: <p>I am facing the same issue with 2 node training on a different model: “Unexpected poll revent: 25 on socket: 90: Software caused connection abort”. I am using the torch.distributed.launch utility.<br>
Anyone able to resolve it? Or have any idea what might be causing it?</p>
Indexing spans of various length

username: Pramodith
cooked: <p>I need to extract phrases that span different lengths from a pre-trained BERT model. For example<br>
let’s say 2 sentences are : “Pytorch is a great library” and “Deep learning with pytorch is fun” I need to extract the vectors of “great library” and “pytorch is fun” after passing those sentences through BERT.</p>
<p>I’m stumped at how I can efficiently index the output of BERT if a batch of sentences is passed to it since the spans are of variable length. Is there anyway to extract these values without using a for loop.</p>
<p>PS: I’m aware that since BERT used Byte Pair Encoding the tokens won’t necessarily remain to be those exact words, I just wanted to simplify the example.</p>
Unexpected 0 Loss in training a heatmap-predict network

username: chauvinfish
cooked: <p>During my training process, the loss rapidly decline to 0,so do the output heatmaps.but they are quite diffrent to the target heatmap(ground truth).I wonder how could this be happening. i checked the Loss function,and switched to a simpler model but nothing changed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f3ccb552f31ff1702b6e54bf125cf2a535af63fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3ccb552f31ff1702b6e54bf125cf2a535af63fc" title="report"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3ccb552f31ff1702b6e54bf125cf2a535af63fc_2_690x410.png" alt="report" data-base62-sha1="yMKBS08mC1K37OoQ9fXsSi1aALa" width="690" height="410" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3ccb552f31ff1702b6e54bf125cf2a535af63fc_2_690x410.png, https://discuss.pytorch.org/uploads/default/original/2X/f/f3ccb552f31ff1702b6e54bf125cf2a535af63fc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f3ccb552f31ff1702b6e54bf125cf2a535af63fc.png 2x" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">report</span><span class="informations">1008×600 41.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Just as the picture shows.<br>
The prediction is a N<em>C</em>W*H tensor ,comparing with the heatmap transformed form the dataset original bounding box’s center points.<br>
the Channels correspoding to the object classes(areoplanes,blablabla)<br>
the targetmap with object in it looks brighter because i set the min value 0.1<br>
i wish the model stop outputing all zeros.Dont care about this trick, it makes no difference.</p>

username: ptrblck
cooked: <p>Which loss function are you using and what kind of target distribution are you expecting?<br>
Could you print the number of unique values in your target, where the loss is zero via</p>
<pre><code class="lang-python">print(target.unique(return_counts=True))
</code></pre>
Pytorch disable gradients globally

username: vikigenius
cooked: <p>Is there a way to disable gradients globally ?</p>
<p>I typically evaluate my models in a jupyter notebook.</p>
<p>Instead of having to repeatedly use with torch.no_grad blocks, can I disable gradients for the entire notebook ?</p>

username: andreaskoepf
cooked: <p>As long as the evaluation in jupyter happens on the same thread you probably can use <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_grad_enabled" rel="nofollow noopener">torch.set_grad_enabled(False)</a>.</p>
Can't iter(Dataloader Object).BrokenPipeError

username: Sumching
cooked: <p>Sorry for my terrible English…</p>
<p>I run the code from 60mins tutorial,like this.</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./datasets’, train=True, download=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)<br>
dataiter = iter(trainloader)</p>
<p>And return a error.</p>
<p>File “C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py”, line 705, in runfile<br>
execfile(filename, namespace)<br>
File “C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py”, line 102, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)<br>
File “C:/Users/flow_/Documents/cifarten/cf10.py”, line 36, in <br>
dataiter = iter(trainloader)<br>
File “C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 451, in <strong>iter</strong><br>
return _DataLoaderIter(self)<br>
File “C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 239, in <strong>init</strong><br>
w.start()<br>
File “C:\ProgramData\Anaconda3\lib\multiprocessing\process.py”, line 105, in start<br>
self._popen = self._Popen(self)<br>
File “C:\ProgramData\Anaconda3\lib\multiprocessing\context.py”, line 223, in _Popen<br>
return _default_context.get_context().Process._Popen(process_obj)<br>
File “C:\ProgramData\Anaconda3\lib\multiprocessing\context.py”, line 322, in _Popen<br>
return Popen(process_obj)<br>
File “C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in <strong>init</strong><br>
reduction.dump(process_obj, to_child)<br>
File “C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
BrokenPipeError: [Errno 32] Broken pipe</p>

username: ptrblck
cooked: <p>There is an issue regarding multi-processing on Windows machines, since apparently Windows subprocesses will import (i.e. execute) the main module at start, which will result in recursively creating subprocesses.<br>
Try to protect your code in <code>if __name__ == '__main__'</code>.</p>
<p>Also you could check, if this is the error by setting <code>num_workers=0</code> and running it again.</p>

username: hktxt
cooked: <p>I got similar error on Ubuntu, which is: module multiprocessing.util’ has no attribute '_flush_std_streams<br>
how to fix it?</p>

username: ptrblck
cooked: <p>The method is defined in the Python lib <code>multiprocessing</code>, and was not introduced in PyTorch.<br>
Could you check if you can update the multiprocessing lib or re-install it?</p>

username: Sumching
cooked: <p>Thank you. I fixed it.</p>

username: Layyy_Ma
cooked: <p>That works. Thanks a lot.</p>

username: Siyuan_Chai
cooked: <p>Thank you for your explanation! The problem seems to disappear if I run the same code (without <code>if __name__ == '__main__'</code>) in Jupyter notebook in Windows.</p>
How to speed up lstm in libtorch?

username: k3833653
cooked: <p>I convert a 2 layer bilstm model to jit sciript model, and the program run correctly. If I want to speed up the inference on cpu backend, what can I do for the lstm operation. Is there any  configuration like multi-thread helpful?</p>

username: dambo
cooked: <p>There is the C++/CUDA LLTM (not LSTM) implementation here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="nofollow noopener">pytorch/extension-cpp</a></h3>

<p>C++ extensions in PyTorch. Contribute to pytorch/extension-cpp development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Why can you call model without specifying forward method

username: Bruce_Ho
cooked: <p>I am not sure this is a pytorch question or a python question. In</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/TropComplique/lda2vec-pytorch/blob/master/utils/training.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/TropComplique/lda2vec-pytorch/blob/master/utils/training.py" target="_blank" rel="nofollow noopener">TropComplique/lda2vec-pytorch/blob/master/utils/training.py</a></h4>
<pre><code class="lang-py">import numpy as np
import torch
from torch.autograd import Variable
import torch.optim as optim
import math
from tqdm import tqdm
from torch.utils.data.dataset import Dataset
from torch.utils.data import DataLoader
from .lda2vec_loss import loss, topic_embedding


# negative sampling power
BETA = 0.75

# i add some noise to the gradient
ETA = 0.4
# i believe this helps optimization.
# the idea is taken from here:
# https://arxiv.org/abs/1511.06807
# 'Adding Gradient Noise Improves Learning for Very Deep Networks'
</code></pre>

  This file has been truncated. <a href="https://github.com/TropComplique/lda2vec-pytorch/blob/master/utils/training.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>line 127, why is it ok to call model(doc_indices, pivot_words, target_words) without specifying forward() method name?</p>

username: SimonW
cooked: <p>python question.</p>
<p>When you call <code>model(...)</code>, you are actually calling <code>model.__call__(...)</code>. As you can see <a href="https://github.com/pytorch/pytorch/blob/d3f98b5ffc991d22df9b2783ec43a5148b824e6d/torch/nn/modules/module.py#L471-L497">here</a>, the <code>__call__</code> method on <code>nn.Module</code> eventually calls <code>forward</code> along with taking care of tracing and hooks.</p>

username: chsafouane
cooked: <p>Would you mind explaining what do you by ‘tracing and hooks’ ?</p>
Error using TensorBoard in PyTorch 1.2

username: Roberto_Carlos_Cruz
cooked: <p>I am having the following problem when trying to use TensorBoard in PyTorch 1.2:</p>
<pre><code class="lang-auto"> from torch.utils.tensorboard import SummaryWriter 
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.
</code></pre>
<p>I understood that I could use TensorBoard with PyTorch 1.2 without problems!</p>
<p>I am using Anconda and I have installed version 1.14 of TensorBoard, so I don’t know what  the problem is!!:</p>
<pre><code class="lang-auto">tensorboard               1.14.0           py37hf484d3e_0    anaconda
cpuonly                      1.0                0    pytorch
pytorch                       1.2.0             py3.7_cpu_0  [cpuonly]  pytorch
torchvision                 0.4.0             py37_cpu  [cpuonly]  pytorch
</code></pre>
<p>I tried with “pip uninstall tb-nightly” but it didn’t work for me. Anyone have the same problem?. I come from Keras / TF and I’m starting with PyTorch (I like PyTorch!!!), so maybe I’m missing out!</p>

username: 19951024
cooked: <p>I meet the same error as you. I uninstall tensorboard and then install tensorflow, finally it worked. Bless for you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Roberto_Carlos_Cruz
cooked: <p>It seems that I had some conflict between the TensorBoard of my system (ubuntu 18.04) and the one I had in Anaconda. I reinstalled everything using pip3 (not the anaconda pip3!!) directly and it already works. Thank you a lot. Problem solved!</p>
Autograd `Function` for conv layers

username: Sia_Rezaei
cooked: <p>I’m trying to write a “Conv2DFunction” in the same spirit as the <code>class LinearFunction(Function)</code> here: <a href="https://pytorch.org/docs/master/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/extending.html</a><br>
But, I can’t quite wrap my head around how to calculate gradients in the <code>backward</code> method, because with convolution kernels you can’t use simple matmuls to calculate gradients. Any hints or help is appreciated.<br>
Thanks!</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/backprop-over-kernels-in-a-cnn/49353/2">this post</a> which gives more details on the backward pass in conv layers.</p>

username: Sia_Rezaei
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, in the simple case of one input channel, one output channel, and default values for other parameters, it is relatively easy. There are a couple of good blog posts covering it; like this one: <a href="https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c" rel="nofollow noopener">https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c</a></p>
<p>But, when you have padding, dilation, stride, and input/output channels, it makes for a really mind twisting exercise!</p>
Very Large model on single GPU

username: aaodds71
cooked: <p>Hi, I have a very big model that consists of different modules and I can’t reduce the mini-batch size. Also, I only have one GPU. So what are my options here? on this forum, I found out about accumulating gradient. will it help me? but I prefer doing normal SGD calculation. Is there any way to transfer some tensors and variables to CPU between modules and when I want to backpropagate transfer them back to GPU one by one or do backpropagation on CPU?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>I think the easiest way would be to use <a href="https://pytorch.org/docs/stable/checkpoint.html">torch.utils.checkpoint</a> to trade memory for compute.<br>
Manually pushing the values to the host and back to the device would probably make your training really slow.<br>
Accumulating gradients won’t help, as the same memory would be needed.</p>

username: mtbrandy
cooked: <p>Check out Large Model Support for PyTorch:</p>
<p>Wiki: <a href="https://github.com/mtbrandy/pytorch/wiki/Large-Model-Support" rel="nofollow noopener">https://github.com/mtbrandy/pytorch/wiki/Large-Model-Support</a><br>
Code: <a href="https://github.com/mtbrandy/pytorch/tree/v1.1.0-LMS" rel="nofollow noopener">https://github.com/mtbrandy/pytorch/tree/v1.1.0-LMS</a></p>
<p>This is the contribution proposal for the LMS feature that is currently available in the Watson Machine Learning solution.  For information about that solution, see <a href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.1/navigation/wmlce_getstarted_pytorch.html" rel="nofollow noopener">https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.1/navigation/wmlce_getstarted_pytorch.html</a></p>
Are these two neural network structures equivalent?

username: vainaijr
cooked: <pre><code class="lang-auto">class ModelOne(nn.Module):
  def __init__(self):
    super().__init__()
    self.weights = nn.Parameter(torch.randn(300, 10))
    self.bias = nn.Parameter(torch.zeros(10))
  def forward(self, x):
    return x @ self.weights + self.bias
</code></pre>
<pre><code class="lang-auto">class ModelTwo(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(300, 10)
    
  def forward(self, x):
    return self.linear(x)
</code></pre>
<p>if so, then why does</p>
<pre><code class="lang-auto">mo = ModelOne()
[len(param) for param in mo.parameters()]
</code></pre>
<p>give<br>
[300, 10]</p>
<p>while</p>
<pre><code class="lang-auto">mt = ModelTwo()
[len(param) for param in mt.parameters()]
</code></pre>
<p>give<br>
[10, 10]</p>

username: vainaijr
cooked: <p>it turns out that</p>
<pre><code class="lang-auto">[param.size() for param in mo.parameters()]
</code></pre>
<p>gives<br>
[torch.Size([300, 10]), torch.Size([10])]</p>
<p>while</p>
<pre><code class="lang-auto">[param.size() for param in  mt.parameters()]
</code></pre>
<p>gives<br>
[torch.Size([10, 300]), torch.Size([10])]</p>
<p>transpose when using nn.Linear</p>

username: spanev
cooked: <p><code>nn.Linear</code> is <a href="https://github.com/pytorch/pytorch/blob/1bf1970fe240f19ce8a7f6a26ec2c4358451069d/torch/nn/functional.py#L1372" rel="nofollow noopener">transposing the weights</a> before multiplication so the two networks are equivalent.<br>
To match the params shapes and the behaviour of nn.Linear you have to do:</p>
<pre><code class="lang-auto">class ModelOne(nn.Module):
  def __init__(self):
    super().__init__()
    self.weights = nn.Parameter(torch.randn(10, 300))
    self.bias = nn.Parameter(torch.zeros(10))
  def forward(self, x):
    return x @ self.weights.t() + self.bias
</code></pre>
How to filter a signal using transfer function in Pytorch? #25180

username: Ahmed_m
cooked: <p>Hi,</p>
<p>I am trying to implement a 1D convolution operation with <code>F.conv1d</code>. The current usage of this function is to provide the weights of the filter directly in the time domain. However, for some DSP purposes, it is more effective to apply the filtering process in terms of the rational transfer function of the filter. In other words, to provide the coefficients of numerator and denominator of the transfer function and the function applies the filtering process accordingly.</p>
<p>This is already provided in <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lfilter.html" rel="nofollow noopener">scipy</a> as well as <a href="https://www.mathworks.com/help/matlab/ref/filter.html" rel="nofollow noopener">Matlab</a>.</p>
<p>I think it is possible to do the same with Pytorch, but I am still struggling to grasp all the details to achieve this … could you please give any tips?</p>
<p>Many thanks in advance<br>
Best</p>
"set_storage is not allowed on Tensor created from .data or .detach()" error when using "self.lstm.flatten_parameters()"

username: Zheng_Chen
cooked: <p>I am using LSTM with data parallel. What happens it first gives a warning</p>
<blockquote>
<p>RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().<br>
self.num_layers, self.dropout, self.training, self.bidirectional)</p>
</blockquote>
<p>The according this post, I added the following in the forward function before calling the LSTM.</p>
<pre><code class="lang-auto">self.lstm.flatten_parameters()
</code></pre>
<p>But this time I got this error,</p>
<blockquote>
<p>set_storage is not allowed on Tensor created from .data or .detach()</p>
</blockquote>
<p>The error occurs at the last line of the <code>flatten_parameters</code> function in the PyTorch source code,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3e2328a51fa7293764d0bf1e148b0cc536e53a32.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e2328a51fa7293764d0bf1e148b0cc536e53a32" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3e2328a51fa7293764d0bf1e148b0cc536e53a32.png" alt="image" data-base62-sha1="8RGVGMsiet7lvynXCdIs2Qgmn2a" width="594" height="499" data-dominant-color="474B46"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×873 45.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: christopherkuemmel
cooked: <p>Hi Zheng Chen, did you manage to solve this issue? I’m running in similar problems when calling</p>
<blockquote>
<p>self.gru.flatten_parameters()</p>
</blockquote>
<p>The error occurs at inference while executing all code within</p>
<blockquote>
<p>with torch.no_grad():</p>
</blockquote>

username: christopherkuemmel
cooked: <p>Could solve this error by updating my dependencies. Upgrading/reinstalling my conda environment from PyTorch <strong>1.1.0</strong> to Pytorch <strong>1.2.0</strong> resolved this issue.</p>
Assigning every instance of siamese network to separate GPU

username: Tzeviya
cooked: <p>I have a Siamese Network with a triplet loss function at the end. E.g.:</p>
<pre><code class="lang-auto">&gt; class Siamese(nn.Module):
&gt;     def __init__(self, ae_net):
&gt;         super(Siamese, self).__init__()
&gt;         self.ae_net = ae_net
&gt; 
&gt;     def forward(self, x1, x2, x3, hidden):
&gt;         
&gt;         a = self.ae_net(x1, hidden)
&gt;         b = self.ae_net(x2, hidden)
&gt;         c = self.ae_net(x3, hidden)
&gt;         return a, b, c
</code></pre>
<p>The network that repeats itself (i.e., <code>self.ae_net</code>) is an LSTM with inputs of varying lengths, so I’m not sure I can use <code>nn.DataParallel</code>.  I was wondering if there was a way to assign the implementation of every instance of <code>self.ae_net</code> to a different GPU, so that they would be calculated in parallel.</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>In your example, you use the hidden state sequentially for each network one after the other. Is that what you want? Because if you want that, then you cannot really run them in parallel on different GPUs as they will need to wait for the previous one to finish.</p>

username: Tzeviya
cooked: <p>Hi, thanks for your answer! The hidden state that is used for <code>a</code>, <code>b</code> and <code>c</code> is the same one, i.e., it’s just an initialized tensor used three times separately (isn’t it?), so I don’t think it’s a problem to use them in parallel.</p>

username: albanD
cooked: <p>Ho right it is, my bad.</p>
<p>It is a bit tricky for Siamese network as you need to accumulate the gradients for all tree runs.<br>
One simple way to do this is to make three copies of your network, one on each device, then send each copy to it’s respective device.<br>
After each backward, you will need to accumulate the gradients by hand then share the new values by hand.<br>
This is going to be tricky to do very efficiently, and you might not get a large improvement for using multiple gpus because of the synchronisation needed between devices.</p>
Expected behavior of a VAE trained on single example

username: docsjoo
cooked: <p>A quick question. What is the expected behavior of a variational autoencoder that is trained on only a single example, over and over?<br>
I thought I would use this as a sanity check on my architecture, but I’m not seeing the outcome I was expecting, and am wondering if I missed anything.</p>
<p>What I expect to happen is for the decoder to gradually learn to ignore the sampled latent, since there is no variation in the target - the latent is just noise, essentially. As this starts to occur, less and less of a gradient gets passed back through the encoder from the target, since the output is now more and more independent of the latent. Thus the only gradient affecting the encoder is that from the KL divergence, and the encoder starts to learn to output a Gaussian posterior independently of the (constant) input, and the KL loss approaches 0.</p>
<p>Is this reasoning correct?</p>
<p>I wasn’t planning on asking for help debugging my code, just my reasoning <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> - but if the above is correct, what general reasons could there be for the failure? A few possibilities come to mind: regularization losses that interfere with the decoder’s freedom to ignore the latent; lack of a bias term in the decoder; a too-high learning rate that prevents reaching the local optimum.</p>
How to visualize colorful feature map

username: SkyeLu
cooked: <p>Hi, I have feature map in shape of [num_channels, height, weight], how can I visualize it and create picture like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/566a0f266e4af864b13d9868033cc752f966ee82.png" alt="1564572227(1)" data-base62-sha1="cksib4KJYpn4hj3ViL1txwHMDce" width="189" height="158"></p>

username: ptrblck
cooked: <p>You could use <code>matplotlib.pyplot.imshow</code> to visualize these tensors:</p>
<pre><code class="lang-python">x = torch.randn(10, 100, 100)

f, axarr = plt.subplots(x.size(0))
for ax, x_ in zip(axarr, x):
    ax.imshow(x_.numpy())
</code></pre>

username: SkyeLu
cooked: <p>Thanks a lot for your reply! But can I draw these 10 channels into 1 image rather than 10 images respectively? Besides, I want to save it as a  ‘png’ file.</p>

username: ptrblck
cooked: <p>How would you like to visualize 10 channels in your image?<br>
The usual image libraries work with a single channel image (grayscale) or 3 channel images (RGB).</p>

username: SkyeLu
cooked: <p>Oh I got it. Sorry for my stupid question. Thank you again.</p>

username: ptrblck
cooked: <p>I don’t think it’s a stupid question and there might still be some “reasonable” mapping between 10 channels to 3 channels, but I’m not aware of any at the moment.</p>
Using Data Augmentation to choose one class label

username: Cythonista
cooked: <p>In Pytorch<br>
I want to use Data Augmentation to choose one class label.<br>
Somebody tell me how to, please.<br>
For example,<br>
0 label class &lt;- Use Data Augmentation<br>
1 label class &lt;- Not using</p>

username: ptrblck
cooked: <p>You could add a condition in your <code>Dataset</code>'s <code>__getitem__</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data, target, transform=None):
        ...
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        if y == 0:
            x = self.transform(x)
        return x, y
    
    def __len__(self):
        return len(self.data)
</code></pre>
<p>You could also pass different transformations and apply them based on the current target label.</p>

username: Cythonista
cooked: <p>Dear ptrblck</p>
<p>What beautiful code this is !<br>
I see a lot.</p>
<p>Thanks for your help!</p>