akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataParallel performance in C++

sharvil: <p>It’s well-established that <code>DataParallel</code> performs poorly in Python. Does anyone have anecdotal experience with <code>DataParallel</code> in C++? I assume that it can achieve performance on par with <code>DistributedDataParallel</code> since there’s no GIL contention but it would be really helpful to hear real experiences with it before invest in switching from Python to C++.</p>

ptrblck: <p>Take this information with a grain of salt, as I didn’t run DataParallel in C++, but I don’t think the worse performance of DataParallel compared to DDP is due to the GIL, but probably the communication overhead needed for the scatter/gather of the model parameters and data.</p>
Typo in Deep Learning with PyTorch

dstrube: <p>I’m enjoying reading the book “Deep Learning with PyTorch”. If I found a typo in it, do you know what would be the best way to contact the person who would correct it?</p>

tom: <p>Hi David,</p>
<p>so I’m not entirely sure about how and if the essential excerpts will be updated, but if you send them to me (in PM or at <a href="mailto:tv@lernapparat.de">tv@lernapparat.de</a>), I’ll check that we’re fixing them for <a href="https://www.manning.com/books/deep-learning-with-pytorch" rel="nofollow noopener">the “proper” book</a>, it might be just in time to sneak in typo corrections before going to print, and also forward it to the people who extracted the essential excerpts.</p>
<p><a class="mention" href="/u/elistevens">@elistevens</a> and <a class="mention" href="/u/lantiga">@lantiga</a> are on the forums, too, they’ll be delighted to hear that the book is useful, too.</p>
<p>Best regards</p>
<p>Thomas</p>

carlo_david: <p>hi <a class="mention" href="/u/tom">@tom</a> when will be the release of the final book? thanks</p>

tom: <p>Hi,</p>
<p>so we are done writing the book we would write. But we, with the help of the editors, we are making it a bit more the book that we would like to have written. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Manning has put “Summer” in the estimate, but I’d be hoping we actually make May. The February MEAP has the deployment chapter and many of the revisions we made to Part I.<br>
I would expect the next MEAP release to include everything except the copyediting.</p>
<p>Best regards</p>
<p>Thomas</p>

carlo_david: <p>when is the next MEAP release?</p>

tom: <p>Hey, I’m replying late, but so I think we more or less have handed in everything, so there might not be a next MEAP release but the proper book instead. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>Turns out, it’s today. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Now the book has the final shape, except for the editorial enhancements that didn’t make it in this last MEAP.</p>

carlo_david: <p>awesome book, i’m currently reading it now <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
How to select random indeces from batch?

hadaev8: <p>I have very few data and want to store it all in gpu.<br>
Still it more than i can use in minibatch.<br>
What is best way to select random sub batch on every step?</p>

ptrblck: <p>You could randomly permute the indices and then slice the batch:</p>
<pre><code class="lang-python">batch = torch.arange(10).view(10, 1)
# select 5 samples randomly
idx = torch.randperm(10)[:5]
mini_batch = batch[idx]
</code></pre>
How to reduce memory usage?

thancaocuong: <p>Hi everyone, I caculate loss from multi stage of hougrass net, I found that the memory usage of caculating loss is very big, so I can not train the model with batchsize greater than 4. The output of the model is list of tensor output from each stage, so I compute and sum up them with groud truth heatmaps. So how can I reduce memory usage by modifying my code? I mean can I caculate the loss with memory efficient? you can see my code below. thank you. note that the size of dense_map and weight is 34x96x96, and center_map is 1x96x96</p>
<pre><code class="lang-auto">def _train_epoch(self, epoch):
        self.model.module.train()
        train_loader_bar = tqdm(range(len(self.train_loader)),
                                desc="Epoch {}".format(epoch), position=0
                                )
        self.train_loss.reset()
        root_loss_sum = 0.0
        dense_loss_sum = 0.0
        for img, center_map, dense_map, weight in self.train_loader:
            img = img.to(self.device)
            center_map = center_map.to(self.device)
            dense_map = dense_map.to(self.device)
            weight = weight.to(self.device)
            center_pred, dense_pred = self.model(img)

            root_loss = self.root_criterion(center_pred[0], center_map)
            dense_loss = self.dense_criterion(
                                              dense_pred[0]*weight,
                                              dense_map*weight
                                              )
            total_loss = root_loss + self.dense_loss_weight * dense_loss
            root_loss_sum += root_loss.item()
            dense_loss_sum += dense_loss.item()
            for s in range(1, len(dense_pred)):
                root_loss = self.root_criterion(center_pred[s], center_map)
                dense_loss = self.dense_criterion(
                                                  dense_pred[s]*weight,
                                                  dense_map*weight
                                                  )
                total_loss = total_loss + root_loss \
                                        + self.dense_loss_weight * dense_loss
                root_loss_sum += root_loss.item()
                dense_loss_sum += dense_loss.item()
            self.train_loss.update(total_loss.item())
            self.optimizer.zero_grad()
            total_loss.backward()
            self.optimizer.step()
            train_loader_bar.update(1)
        root_loss_avg = root_loss_sum / len(self.train_loader)
        dense_loss_avg = dense_loss_sum / len(self.train_loader)
        return root_loss_avg, dense_loss_avg, self.train_loss.avg

</code></pre>
Categorical channel in image segmentation

asand: <p>Hello,</p>
<p>I have an existing Unet network which is trained to find “interesting stuff” in CT brain scans. The brain scans I have are decomposed from a single input channel to 5 input channels. Now I want to add an extra channel containing a brain atlas (outlining the anatomy of the brain). This channel will contain categories, say, 0 to 10, but the fact that 0 &lt; 5 has no meaning.<br>
What is the preferred way to add such a channel? I guess I want to add some kind of embedding of the categorical channel. Does anyone have an example of this?</p>
<p>Kind regards,<br>
Andreas</p>
How is my Conv1d dimension reducing when I have padding?

shamoons: <p>My <code>conv module</code> is:</p>
<pre><code class="lang-auto">        return torch.nn.Sequential(
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=1,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=2,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=4,
                padding=1
            ),
            torch.nn.ReLU()
        )
</code></pre>
<p>And in <code>forward</code>, I have:</p>
<pre><code class="lang-auto">down_out = self.downscale_time_conv(inputs)
</code></pre>
<p><code>inputs</code> has a <code>.size</code> of <code>torch.Size([8, 161, 24])</code>. I’d expect <code>down_out</code> to have the same size, but instead it has: <code>torch.Size([8, 161, 23])</code></p>
<p>Where did that last element go?</p>

ptrblck: <p>The output shape is given in the <a href="https://pytorch.org/docs/stable/nn.html?highlight=conv1d#torch.nn.Conv1d">formula in the docs</a>.<br>
Based on this formula, the output is expected, as the layers will return:</p>
<pre><code class="lang-python">L_out = math.floor((L_in + 2 * padding - dilation * (kernel_size - 1) - 1) / stride + 1)

# 1st layer
(24 + 2 * 1 - 1 * (2-1) - 1) / 1 + 1 = 25
# 2nd layer
(25 + 2 * 1 - 2 * (2-1) - 1) / 1 + 1 = 25
# 3rd layer
(25 + 2 * 1 - 4 * (2-1) - 1) / 1 + 1 = 23
</code></pre>
How to adjust moving average window in BatchNorm

94Andrew: <p>Basically, in BatchNorm2D layer, running_mean and running_var are calculated by moving average of all seen batch_mean and batch_var.<br>
However, i want to only see just few batch statistic window to generate running statistic.<br>
For example, running statistics at step 5 are calculated by moving average of batch statistic at only steps 3, 4, 5. Also, batch statistics at only steps 8, 9, 10(Not all steps 1~10) are involved in running statistics at step 10.</p>
Iterating over dataloader twice

Ameen_Ali: <p>Hello<br>
Let’s say I have a dataloader (loader) which returns data of the following shape : [BatchSize , 1024]<br>
I have 10000 samples and I want to build a score matrix of the shape (10000 , 10000) such that entry [i,j] denotes the score between sample i and sample j, so far I did it in the following way (setting batchsize = 1)</p>
<pre><code class="lang-auto">   d = numpy.zeros((10000 , 10000))
   for i, sample_i  in tqdm(enumerate(loader)):
     for i, sample_j  in tqdm(enumerate(loader)):
       d[i][j] = calculate_somre_score(sample_i , sample_j)
</code></pre>
<p>But this takes a lot of time and i’m sure there is some good way to do it whihc takes less time.<br>
Thanks<br>
Best</p>

ptrblck: <p>You could try to increase the batch size of both <code>DataLoader</code>s and index <code>d</code> as <code>d[i*bs:(i+1)*bs]</code>.<br>
Also, do you need to compute the full matrix, i.e. is</p>
<pre><code class="lang-python">calculate_some_score(a, b) != calculate_some_score(b, a)
</code></pre>
<p>?</p>

Ameen_Ali: <p>Hi,<br>
I didn’t get your answer , how does that compare each sample with each other possible sample in the dataset?<br>
and the score calculation function is not commutative.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="Ameen_Ali" data-post="3" data-topic="77728">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ameen_ali/48/15528_2.png" class="avatar"> Ameen_Ali:</div>
<blockquote>
<p>the score calculation function is not commutative</p>
</blockquote>
</aside>
<p>That is what I was asking about, so thanks for clarification. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>In that case, you could potentially speed it up with the batched approach.</p>

Ameen_Ali: <p>can you elaborate more on the batched approach?<br>
thanks</p>

ptrblck: <p>If your loss function accepts batches of inputs, you could assign the batch of scores to <code>d</code> with a range indexing: <code>[i*bs:(i+1)*bs]</code>.<br>
This would avoid looping one by one in both loops and could calculate the loss for multiple sample pairs.</p>

Ameen_Ali: <p>Yes , but that does not compare each sample with <strong>All Samples In The Dataset</strong>, it only compare each sample with every other possible sample shifted by batchsize.</p>

ptrblck: <p>You are right. The outer loop could use a single sample (and <code>expand</code> it to a matching batch size), while the inner loop could use batches.<br>
This would calculate (left is outer loop sample, right inner loop samples):</p>
<pre><code class="lang-python">[0] - [0]
[0] - [1]
[0] - [3]
# next batch
[0] - [4]
[0] - [5]
[0] - [6]
# next outer loop iteration
[1] - [0]
[1] - [1]
[1] - [2]
...
</code></pre>
Using more gpu memory (virtual memory)

lnBrTgAaaL: <p>I am training a cnn model on Nividia RTX 2080 with a memory of 8 GB on Cuda.<br>
And my ram is 32 GP.<br>
I always get this error : “CUDA out of memory. Tried to allocate 5.96 GiB (GPU 0; 8.00 GiB total capacity; 596.29 MiB already allocated; 5.70 GiB free; 600.00 MiB reserved in total by PyTorch)”<br>
is there a way to use a kind of virtual memory or something like that so I can train it on the GPU?</p>

TinfoilHat0: <p>Perhaps decrease your batch size?</p>

lnBrTgAaaL: <p>The patch size is already 5</p>
Parallelization error

silvester: <p>I parallelize the model using</p>
<p>net = torch.nn.DataParallel(net)</p>
<p>but I get the error</p>
<p><code>RuntimeError: expected device cuda:1 but got device cuda:0</code></p>

TinfoilHat0: <p>A naive guess but could it be the case  that you only have a single GPU in your system?</p>
Function or module to retrieve the attributes layer by layer for a neural network

Sai_krishna: <p>Would like to know or find any function to retrieve the attributes such as stride, padding, dilation, groups, benchmark kind of values for each of the layer like convolution layer. Trying to kind of parse the model in pytorch. I could successfully read the values of weights, bais from each layer using function p = Model.parameters(), then using p.data_ptr().</p>
<blockquote>
<p>p.data_ptr()</p>
</blockquote>

ptrblck: <p>You can directly access these parameters via:</p>
<pre><code class="lang-python">conv = nn.Conv2d(
    in_channels=1,
    out_channels=3,
    kernel_size=(3, 4),
    stride=(5, 6),
    dilation=(7, 8),
)

print(conv.in_channels)
print(conv.out_channels)
print(conv.kernel_size)
print(conv.stride)
print(conv.dilation)
</code></pre>

Sai_krishna: <p>Thank you for the reply, but im trying to traverse through each node of the model graph and came to a point of checking if the node is an operator. Now i want to copy all the inputs values(weights, bais, in_channels,kernel_size, etc) of each layer.</p>
<pre><code class="lang-auto">for (auto it = nodes.begin(); it != nodes.end(); ++it)
	{

		std::cout &lt;&lt; " " &lt;&lt; std::endl;
		auto node = it-&gt;second-&gt;node();

		auto inp_size = it-&gt;second-&gt;node()-&gt;inputs();
		std::cout &lt;&lt; "inp_sizeL:/n" &lt;&lt; inp_size.size() &lt;&lt; std::endl;
		if (it-&gt;second-&gt;node()-&gt;maybeOperator()) {
			//std::cout &lt;&lt; *node.getparameters() &lt;&lt; std::endl;```
So at this point, i would like to read all the inputs to each node. Is there any function like **getAttributes** or get_attributes("mods") to retrieve all the values. Thank you.
</code></pre>
Calculating OPs in a Linear Layer?

Navdeep_Kumar: <p>I wanted to calculate the flops in a linear layer for a given input and output size.<br>
For example, a Linear layer defined as torch.nn.Linear(3, 2) and given an input tensor x = torch.randn(4, 384, 3, dtype=torch.float16, device = device), will first do a matmul for the input tensor, which does 4 * 384 * 3 * 2 * 2 opearations.<br>
Then bias addtion takes  place and it will do another 4 * 384 * 2 operations.<br>
Is the calculation right? Is it what the linear layer acually does or are there some other optimizations?</p>
RGB value ranges 0-1 after rescaling, How do I normalize images?

Wei_Wong: <p>I wrote a class to rescale images, but the RGB value became ranging from 0 to 1 after preocessing. What happened to the RGB which intuitively should be ranging from 0-255 ? Following are the Rescale class and  the RGB values after rescaling.</p>
<p>Question:</p>
<ol>
<li>
<p>Do I still need a Min-Max Normalization, map the RGB value to 0-1?</p>
</li>
<li>
<p>How do I apply transforms.Normalization, where do I put the Normalization, before or after the Rescale, how do I calculate the mean and variance, use the RGB value ranging from 0-255 or 0-1？</p>
</li>
</ol>
<p>Thanks for your time!</p>
<pre><code class="lang-auto">class Rescale(object):
    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size
        
    def __call__(self, sample):
        image, anno = sample['image'], sample['anno']
        
        # get orginal width and height of image
        h, w = image.shape[0:2]
        # if output_size is an integer
       
        if isinstance(self.output_size, int):
            if h &gt; w:
                new_h, new_w = h * self.output_size / w, self.output_size
            else:
                new_h, new_w = self.output_size / h, w * self.output_size / h
                
        # if output size is a tuple (a, b)
        else:
            new_h, new_w = self.output_size
        new_h, new_w = int(new_h), int(new_w)
        
        image = transform.resize(image, (new_h, new_w))       
        return {'image': image, 'anno': anno}
</code></pre>
<pre><code class="lang-auto">[[[0.67264216 0.50980392 0.34503034]
  [0.67243905 0.51208121 0.34528431]
  [0.66719145 0.51817184 0.3459951 ]
  ...
  [0.23645098 0.2654311  0.3759458 ]
  [0.24476471 0.28003857 0.38963938]
  [0.24885877 0.28807445 0.40935877]]

 [[0.67465196 0.50994608 0.3452402 ]
  [0.68067157 0.52031373 0.3531848 ]
  [0.67603922 0.52732436 0.35839216]
  ...
  [0.23458333 0.25195098 0.36822142]
  [0.2461343  0.26886127 0.38314558]
  [0.2454384  0.27233056 0.39977664]]

 [[0.67707843 0.51237255 0.34766667]
  [0.68235294 0.5219951  0.35553024]
  [0.67772059 0.52747687 0.35659176]
  ...
  [0.24485294 0.24514568 0.36592999]
  [0.25407436 0.26205475 0.38063318]
  [0.2597007  0.27202914 0.40214216]]

 ...

 [[0.2957549  0.37119608 0.35928922]
  [0.30588235 0.37932353 0.36563725]
  [0.31078569 0.38772197 0.36392157]
  ...
</code></pre>
<p>The following is RGB values before scaling</p>
<pre><code class="lang-auto">[[[172 130  88]
  [172 130  88]
  [172 130  88]
  ...
  [ 63  74 102]
  [ 65  76 106]
  [ 67  77 112]]

 [[173 131  89]
  [173 131  89]
  [173 131  89]
  ...
  [ 65  74 103]
  [ 64  75 105]
  [ 63  73 108]]

 [[173 131  89]
  [174 132  90]
  [174 132  90]
  ...
  [ 63  72 101]
  [ 62  71 102]
  [ 61  69 105]]

 ...
</code></pre>
CUDA out of memory sometimes

jokerWRN: <p>I encountered a strange error. When I run a program, I get an error that is cuda out of memory. When I run the same program on the same machine in a few days, the error disappears. I just change num_workers from 10 to 25, i don’t think it will offer a help to this problem. Can anyone explain why this kind of thing happens？Thanks in advance.</p>

TinfoilHat0: <p>Each worker is a process that copies its data from the main process afaik. So, it is normal that the GPU memory usage to increase with number of workers.</p>

ptrblck: <p>As long as you are loading the data into the system RAM (not the GPU memory), the number of workers shouldn’t yield a CUDA OOM issue.</p>
<p>Is this error reproducible with <code>num_workers=25</code> or are you seeing this issue som3times even with 25 workers?</p>
Strategy for k fold cross validation?

Aptha_Gowda: <p>My single fold model performed well(unseen test data) with a train validation slit of 80%,20% respectively.<br>
Here are more details about the single fold model.<br>
Seed  = 42, batch_size = 16, Epoch = 15, StepLR (step_size=5,factor=0.1), TTA = 6,<br>
Image_shape( 320x320), LR = 0.0005.</p>
<p>Now I want to implement StratifiedKfold with n=5. I implemented with the same combination above to the StratifiedKfold and it performed worse than the single fold model(unseen test data).<br>
I know the problem is with Epoch, StepLR( step_size) and TTA.</p>
<p>My question is,</p>
<ol>
<li>Whether I should use TTA for each fold or not? ( because I heard that averaging 2 weaker models is better than 2 best model something like that)</li>
<li>What should be my epoch range? or should I follow epoch=15?</li>
<li>What should be the step_size for stepLR?</li>
</ol>

ptrblck: <ol>
<li>I think averaging uncorrelated outputs could yield a performance gain. Generally you could take the average of a lot of weak models, if their performance is at least better than a random guess.</li>
</ol>
<p>For 2. and 3. I would refer to <a class="mention" href="/u/rasbt">@rasbt</a>’s post on <a href="https://sebastianraschka.com/faq/docs/evaluate-a-model.html">cross validation</a>.</p>
Normalization of input data

cbd: <p>Data is normalize between 0 to 1 and given to CNN. If the same data is normalize between -1 to 1 and given to same CNN, Is it produce same output?</p>

ptrblck: <p>If you pass <code>data1</code> (normalized to <code>[0, 1]</code>) and <code>data2</code> (normalized to <code>[-1, 1]</code>) to the <em>same</em> model, you won’t get the same outputs.<br>
However, you could be successful in training models using both approaches.<br>
This would of course still give you slightly different outputs, even if the accuracy of both models is comparable.</p>

cbd: <p>I am little confuse in normalizing data. Suppose i am restoring the image by the formula</p>
<p>out =  (cnn_input_image - b) / cnn_output_image.</p>
<p>cnn_input_image and b = actually range are between 0 to 255 but normalize between -1 to 1<br>
cnn_output_image             = <strong>actually range is between 0 to 1</strong> not  between 0 to 255.</p>
<p>So numerator are in the range of 0 to 255 (which is normalized to -1 to 1) and denominator is in the range of 0 to 1.</p>
<p>The question is</p>
<ol>
<li>how to restore the image using cnn_output_image and which activation function is<br>
appropriate at last layer?</li>
<li>Is the normalization require of cnn_output_image?</li>
<li>Should it requires to take absolute value of denominator if it range -1 to 1</li>
</ol>

ptrblck: <p>I’m not sure to understand the question completely.<br>
Are <code>cnn_input_image</code>, <code>b</code>, and <code>cnn_output_image</code> all image tensors, i.e. di they have the same shape?<br>
If so, wouldn’t you divide by zero, if <code>cnn_output_image</code> is normalized to <code>[0, 1]</code>?</p>

cbd: <p>cnn_input_image( normalize between -1 to 1) and cnn_output_image are image tensor and “b” is scalar(constant value like 0.8) . So operation is like</p>
<p>image_tensor (out=1,3,256,256) = image_tensor(cnn_input_image=1,3,256,256) - scalar (b=1,3,1,1)/ image_tensor(cnn_output_image = 1,3,256,256).</p>
<p>cnn_input_image is subtracted from some constant value and after that it will divide by output of CNN which is image tensor.</p>

ptrblck: <p>So the division might still divide by zero, since <code>cnn_output_image</code> might take arbitrary values.</p>
<p>Back to the questions:</p>
<ol>
<li>If you want to remove the operations, you could use <code>(out * cnn_output_image) + b</code>
</li>
<li>How are you normalizing the output? Usually you would normalize the input and let the model train.</li>
<li>I would suggest to at least prevent the potential division by zero.</li>
</ol>
Adding package to support non-GPU accelerators

almeetb: <p>Hi, I want to add a package that allows tensors and modules to be sent to a non-GPU accelerator similar to <code>pytorch/xla</code> by creating a new device type</p>
<p>I believe this device would need its own allocator but I am not sure of what memory management primitives are required for this to work.</p>
<p>Appreciate any help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why is the default for non_blocking False

davidtvs: <p>I was reading up on the <code>non_blocking</code> parameter from the <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.to" rel="nofollow noopener"><code>to()</code></a> method and I see that it defaults to <code>False</code> but in this discussion - <a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234">Should we set non_blocking to True?</a> - it’s stated that setting it to <code>True</code> improves performance in the right conditions and that there shouldn’t be any negative side effects when used outside of those conditions.</p>
<p>Any particular reason as to why <code>non_blocking</code> still defaults to <code>False</code>?</p>
How can I more efficiently create this tensor without two for loops?

ProGamerGov: <p>Is it possible to create a tensor like this in a more efficient manner?</p>
<pre><code class="lang-auto">height, width = 128,128

w = torch.arange(0, width)
h = torch.arange(0, height)
tensor = torch.zeros((width, height, 2))
for vi in w:
        for vj in h:	
            tensor[i][j] = torch.tensor([float(vi) / height - 0.5, float(vj) / width - 0.5])	
tensor = tensor.reshape(width * height, 2)
</code></pre>

albanD: <p>Hi,</p>
<p>The following will generate the same Tensor without the loop:</p>
<pre><code class="lang-auto">full_size = (width, height)
w_exp = w.unsqueeze(1).expand(full_size).true_divide(height) - 0.5
h_exp = h.unsqueeze(0).expand(full_size).true_divide(width) - 0.5

tensor = torch.stack((w_exp, h_exp), -1)
tensor = tensor.reshape(width * height, 2)
</code></pre>
<p>Hope this helps</p>

ProGamerGov: <p>It works! Thanks for the help!</p>
Tensorboard tutorial - running code in different cells

desmond13: <p>I am new to PyThorch and I am trying to go through the tutorials of the official page.</p>
<p>After some struggle, I am able to run the tutorial about Tensorboard, <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">this tutorial </a>.</p>
<p>I am on an Ubuntu 16.04 machine and I am using virtualenvironment. I am using a jupyter notebook.</p>
<p>I would like to know one last thing that is not clear to me. It seems that if I don’t put all the code in one cell the code does not get executed well. Basically what happens is that I can’t see any output if I do a print.</p>
<p>Is this normal?</p>
<p>If you want to give a look to my notebook I copied it <a href="https://drive.google.com/open?id=1AJpLIfbnlW1u61J4PJNigmtCXeeqcVtx" rel="nofollow noopener">here</a></p>
<p>Thanks a lot for your help.</p>
Per tensor symmetric quantiation

kirilllukorovich: <p>Hello !<br>
I can’t understand where I have the error, in the configuration I write that I want fake per_tensor_<strong>symmetric</strong> quantization, but when I display the picture of the graph, he writes that I have a FakeQuantizePerTensor<strong>Affine</strong>Backward node.</p>
<p>What am I doing wrong?<br>
Thank you in advance!</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torchviz import make_dot

model = nn.Sequential()
model.add_module('W0', nn.Linear(8, 16))
model.add_module('tanh', nn.Tanh())
model.add_module('W1', nn.Linear(16, 1))
model.qconfig = torch.quantization.QConfig(activation=torch.quantization.FakeQuantize.with_args(observer=torch.quantization.MinMaxObserver, 
                                                            quant_min=0,
                                                            quant_max=255,
                                                            qscheme=torch.per_tensor_symmetric,
                                                            dtype=torch.quint8,
                                                            reduce_range=False),
                   weight=torch.quantization.FakeQuantize.with_args(observer=torch.quantization.MinMaxObserver, quant_min=-128, quant_max=127,
                                                   dtype=torch.qint8, qscheme=torch.per_tensor_symmetric, reduce_range=False))
torch.quantization.prepare_qat(model, inplace=True)
x = torch.randn(1,8)
make_dot(model(x), params=dict(model.named_parameters()))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d47912ed94e09a02158837b17bf1e28d38918a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d47912ed94e09a02158837b17bf1e28d38918a7" title="Снимок экрана от 2020-04-22 17-07-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d47912ed94e09a02158837b17bf1e28d38918a7_2_481x500.png" alt="Снимок экрана от 2020-04-22 17-07-20" data-base62-sha1="k9OECwu35bwPkTDRMCY0a1Tfxbx" width="481" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d47912ed94e09a02158837b17bf1e28d38918a7_2_481x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d47912ed94e09a02158837b17bf1e28d38918a7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d47912ed94e09a02158837b17bf1e28d38918a7.png 2x" data-dominant-color="EEF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана от 2020-04-22 17-07-20</span><span class="informations">696×722 41.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

dskhudia: <p>FakeQuantizePerTensorAffineBackward handles both symmetric and asymmetric. In case of symmetric the zero_point would be set to 0. So your example looks fine to me.</p>
RBM and Auto-encoder comparison

rostom007: <p>Hi.</p>
<p>im trying to implement a recommender system using RBM and deep auto-encoder architectures separately.RBM try to predict ratings for not rated movies(1 like/0 no),and AE try to predict the exact rating of the movie(from 1 to 5 stars).<br>
i used the RMSE and MAE metrics for comparison.however i noticed that they are note necessarily used for the same thing,RBM use binary inputs to predict features accuracy (%),and AE use inputs to predict the user rating for a specific movie . so we can not compare them using RMSE and MAE ?(in my case i get MAE of RBM 0.24 (75%accuracy) and with AE0 i get 0.77 (if user gives rate 4 stars we systeme can give between 3 and 4) not exactly the same job .what do u say?</p>
Tensorboard in virtualenvironment - No module named 'tensorboard'

desmond13: <p>I am new to PyThorch and I am trying to go through the tutorials of the official page.</p>
<p>I am struggling to understand how to run Tensorboard in a python notebook. I was trying first to do it in a google colab and understood that it is maybe better to first try to run it in a local python notebook. I am following <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">this tutorial </a>.</p>
<p>I am on an Ubuntu 16.04 machine and I am using virtualenvironment. I did not find a good guide to set up a python notebook (.ipynb) yet.</p>
<p>For now, when I arrive at the command:<br>
<code>from torch.utils.tensorboard import SummaryWriter</code></p>
<p>I get the error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~/deepLearning/lib/python3.5/site-packages/torch/utils/tensorboard/__init__.py in 
      1 try:
----&gt; 2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:

ImportError: No module named 'tensorboard'
</code></pre>
<p>Anyone would be able to help me? Thanks!</p>

TinfoilHat0: <p>Well you dont seem to have installed tensorboard.</p>
<p>Try running</p>
<blockquote>
<p>pip install tensorboard</p>
</blockquote>

desmond13: <p>Thanks for your answer. I had tensorboard installed. However, what I did was</p>
<pre><code class="lang-auto">pip uninstall tensorflow
pip uninstall tensorboard
</code></pre>
<p>I deactivated my virtualenv, reactivated it</p>
<pre><code class="lang-auto">pip install tensorflow
pip install tensorboard
</code></pre>
<p>And now it is working fine</p>
Can't wrap two model in the same class

silvester: <p>I define a class with model initializer. When I wrap one model with that class<br>
it looks fine, but with two model instances, it does not converge.</p>
<p>Code example:</p>
<pre><code class="lang-auto">def wrapper(nn.Module):

    def __init__(self, model):
          self.model = model
    def forward(self, x):
         return self.model(x)



model1 = net1()

w_model1 = wrapper(mode1)


model2 = net2()

#when I include this line, w_model1 does not converge.
w_model2 = wrapper(model2)
</code></pre>

albanD: <p>Hi,</p>
<p>One thing to keep in mind here is that when you create net2(), you draw random samples. So it has the same effect as changing the random seed for the training of net1. If you net1 model is not very stable, it is possible that it only converge for some seeds.</p>
<p>Otherwise, I can’t really see how this could effect it. If it is not the point above, you would need to provide a sample script that shows the behavior so that we can help you further.</p>
Torch.multiprocessing for model inference

assyl_coop: <p>I am trying to use torch.multiprocessing for inference.</p>
<p>I followed tutorial from pytorch documentation: <a href="https://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors" rel="nofollow noopener">https://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors</a><br>
My code is below</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

model = MyModel(num_classes=2)
model.load_state_dict(torch.load(path_to_model'))
model = model.to(device)
model.eval()

def inference(model):
    torch.set_num_threads(1)
    results = []
    for img in imgs*100:
        result = model(img)
        print(result[1].data)
        results.append(result[1].data.cpu())
    torch.set_num_threads(1)
    return results

processes = []

if __name__ == '__main__':
    mp.set_start_method('spawn', force=True)
    model.share_memory()
    
    st = time.time()
    for i in range(4):
        p = mp.Process(target=inference, args=(model,))
        p.start()
        processes.append(p)
    
    for p in processes: p.join()
    print(time.time() - st)
</code></pre>
<p>It launches, but works too fast if model even didt get the images to infer.</p>
<p>So, I’m trying to get results from function inference using multiprocessing.</p>
<p>What am I doing wrong?</p>

terryyizhong: <p>Have you have any idea now? Is it faster to use multiprocessing on inference?</p>
<p>I get confuse on this to and below topic may help<br>
<a href="https://discuss.pytorch.org/t/multiprocessing-cuda-memory/18951">Multiprocessing CUDA memory</a></p>

assyl_coop: <p>Nope, but I decided to move forward of multiple instances of microservices. MPI + gunicorn</p>

Paulo_Mann: <p>Hey <a class="mention" href="/u/assyl_coop">@assyl_coop</a>, can you further explain this MPI + gunicorn solution? I need to solve the same problem. I am also trying to use multiprocessing for inference using CPU</p>
Torch.distributed for windows 7/10

Liron_Mor_Yosef: <p>hello,<br>
there is any way to run pytorch distributed on windows?</p>
<p>i see at pytorch main page that there is version for windows but when i tried to used it,<br>
i get that  torch.distributed.is_available() is False</p>
<p>Thanks!</p>

Liron_Mor_Yosef: <p>i tried to use windows 7 with torch 1.3.1 and 1.5</p>

mrshenli: <p>Currently, <code>torch.distributed</code> does not support Windows yet. Just created a poll/issue to track how many people would need this feature: <a href="https://github.com/pytorch/pytorch/issues/37068" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/37068</a></p>
Unequal sampling from two different datasets

tvdhout: <p>Hi all!</p>
<p>I have two datasets, dataset 1 has  ±400 samples and dataset 2  ±1000 samples. I want to train my network such that it sees one sample from dataset 1 for every 5 samples of dataset 2 (roughly).</p>
<p>With a batch size of 32, each batch containing 5 samples from dataset 1 and 27 from dataset 2 would work. How would I go about setting up a DataLoader to achieve this? I want to definitely train on all 1000 samples from dataset 2 every epoch, and just randomly sample however much I need from dataset 2 to achieve that distribution.</p>
<p>I looked at ConcatDataset but this just combines the two into one large dataset and samples from that. I saw DataLoader has a <code>sampler</code> parameter; could I achieve what I want with a <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#WeightedRandomSampler" rel="nofollow noopener">WeightedRandomSampler</a>?</p>
<p>Any help is appreciated!</p>

ptrblck: <p>One approach would be to create separate <code>DataLoader</code>s with <code>batch_size=5</code> (small dataset) and <code>=27</code> (large dataset), respectively.<br>
In the outer loop you would iterate the large dataset to make sure you are sampling all data points.<br>
You could create an <code>iterator</code> for the small dataset via <code>small_iter = iter(small_loader)</code> before entering the loop and sample the batch from the small dataset.<br>
Once you have both batches, you could concatenate them via <code>data = torch.cat((small, large), dim=0)</code> to create your final batch.</p>
<p>I think the <code>WeightedRandomSampler</code> wouldn’t strictly work, as you need to sample <em>all</em> samples from the large dataset.</p>

tvdhout: <p>Thanks! smart solution</p>
How to calculate gradient of the whole dataset?

ttgump: <p>Hi,<br>
I am trying to implement a model with two losses, say loss1 and loss2. Loss1 can be calculated per batch like regular SGD algorithm, but loss2 need the whole dataset to be calculated (for example, the average of all data’s reconstructions of the autoencoder). How to implement this model in pytorch? Thanks.</p>

TinfoilHat0: <p>Use two data loaders where batch size for one is the size of the training set.</p>
Loss does not drop

zyz1996: <p>Only when batch_size = 1, the loss will drop, otherwise, the loss will oscillate. Use lstm network, is this normal?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/893911e46b5834e21642c5218037f232c691de22.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/893911e46b5834e21642c5218037f232c691de22" title="QQ截图20200422101251"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/893911e46b5834e21642c5218037f232c691de22.jpeg" alt="QQ截图20200422101251" data-base62-sha1="jzVGtDM5sb3MOZXkSN2HKEuZQ5A" width="493" height="500" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20200422101251</span><span class="informations">532×539 32.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c957947c0b39f0a666d6296b12c9c23f4e2cbfb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7c957947c0b39f0a666d6296b12c9c23f4e2cbfb" title="QQ截图20200422101310"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c957947c0b39f0a666d6296b12c9c23f4e2cbfb.jpeg" alt="QQ截图20200422101310" data-base62-sha1="hM7s2Uy9XBuBS8tyJWuWWnO1vMD" width="690" height="319" data-dominant-color="2E2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20200422101310</span><span class="informations">771×357 21.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5029db10632b18241417c2f4d0ef2dbcbfba13ca.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5029db10632b18241417c2f4d0ef2dbcbfba13ca" title="QQ截图20200422101207"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5029db10632b18241417c2f4d0ef2dbcbfba13ca_2_553x499.jpeg" alt="QQ截图20200422101207" data-base62-sha1="br9S8c99yDqFoSKFvJ2fI9whpmW" width="553" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5029db10632b18241417c2f4d0ef2dbcbfba13ca_2_553x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5029db10632b18241417c2f4d0ef2dbcbfba13ca.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5029db10632b18241417c2f4d0ef2dbcbfba13ca.jpeg 2x" data-dominant-color="C0D7E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20200422101207</span><span class="informations">635×574 32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pythorch tutorial about Tensorboard

desmond13: <p>I am new to Pythorch and I am trying to go through the tutorials of the official page.</p>
<p>I am struggling to understand how to run Tensorboard in a google colab as explained in <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="noopener nofollow ugc">this tutorial</a>. I tried to look here: <a href="https://stackoverflow.com/questions/47818822/can-i-use-tensorboard-with-google-colab" class="inline-onebox" rel="noopener nofollow ugc">tensorflow - Can I use TensorBoard with Google Colab? - Stack Overflow</a><br>
and I ended up using <a href="https://stackoverflow.com/a/57791702/2761849" rel="noopener nofollow ugc">this solution</a>:</p>
<pre><code class="lang-auto">%load_ext tensorboard
%tensorboard --logdir logs
</code></pre>
<p>Tensorboard opens up correctly in the notebook but I get the following message: <em>No dashboards are active for the current data set.</em><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b1ddc9551119120776e0f914010d94e24e8a953.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b1ddc9551119120776e0f914010d94e24e8a953" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1ddc9551119120776e0f914010d94e24e8a953_2_690x332.png" alt="image" data-base62-sha1="hz8HQkDB5DZzVgzy1rjcDqiAMcH" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1ddc9551119120776e0f914010d94e24e8a953_2_690x332.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1ddc9551119120776e0f914010d94e24e8a953_2_1035x498.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1ddc9551119120776e0f914010d94e24e8a953_2_1380x664.png 2x" data-dominant-color="D2C7BD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1476×712 76 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Can you help me understanding what I am doing wrong? Maybe I should not use tensorboard in google colab? Thanks!</p>

TinfoilHat0: <p>Log files should be locally accessible. Im not sure if you can upload files to Colab from your machine but if that’s the case, you should first do so.</p>
How to concatenate sentence level features with word level features

bash94: <p>How can I concatenate sentence level embeddings with word level embeddings for padded sequences?<br>
<code>embedded_seqs</code> has to be a <code>PackedSequence</code> before I feed it to the RNN.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, vocab_size, word_embed_dim,
                 num_labels, labels_embed_dim,
                 hidd_dim, padding_idx=0):
        
        super(Model, self).__init__()
        self.w_embedding = nn.Embedding(vocab_size, word_embed_dim, padding_idx=padding_idx)
        self.l_embedding = nn.Embedding(num_labels, labels_embed_dim)
        self.rnn = nn.GRU(word_embed_dim + labels_embed_dim, 
                         hidd_dim, batch_first=True, bidirectional=True)
        
    def forward(self, input_seqs, input_lengths, input_labels):

        embedded_seqs = self.w_embedding(input_seqs)
        
        embedded_labels = self.l_embedding(input_labels)

        # How can I concatenate embedded_seqs with embedded_labels?

        packed_seqs = pack_padded_sequence(embedded_seqs, input_lengths, 
                                           batch_first=True)
        
        output, h_t = self.rnn(packed_seqs)
        ......
</code></pre>
Name 'Model' is not defined

111296: <p>I used pytorch 1.1.0 ,torchvision 0.3.0 and cudatoolkit 10.0.When I typed this “optimizer = torch.optim.SGD(Model.parameters(), lr=learning_rate)”,it appeared name ‘Model’ is not defined.</p>

ptrblck: <p>How did you define the <code>Model</code> instance?<br>
Note that <code>Model</code> is only a variable you’ve used to create the model, e.g. via:</p>
<pre><code class="lang-python">Model = ResNet()
optimizer = torch.optim.SGD(Model.parameters(), lr=1e-3)
</code></pre>

111296: <p>I type this code:</p>
<p>import torch<br>
import torch.nn as nn<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
from torch.autograd import Variable<br>
input_size=1<br>
output_size=1<br>
learning_rate=0.001</p>
<p>xtrain=np.array([[2.3],[4.4],[3.7],[6.1],[7.3],[2.1],[5.6],[7.7],[8.7],[4.1],[6.7],[6.1],[7.5],[2.1],[7.2],[5.6],[5.7],[7.7],[3.1]],dtype=np.float32)<br>
ytrain=np.array([[3.7],[4.76],[4.],[7.1],[8.6],[3.5],[5.4],[7.6],[7.9],[5.3],[7.3],[7.5],[8.5],[3.2],[8.7],[6.4],[6.6],[7.9],[5.3]],dtype=np.float32)<br>
plt.figure()<br>
plt.scatter(xtrain,ytrain)<br>
plt.xlabel(‘xtrain’)<br>
plt.ylabel(‘ytrain’)<br>
plt.show()<br>
class LinearRegression(nn.Module):<br>
def <strong>init</strong>(self,input_size,output_size):<br>
super(LInearRegression,self).<strong>init</strong>()<br>
self.linear=nn.Linear(input_size,output_size)<br>
def forward(self, x):<br>
out = self.linear(x)<br>
return out<br>
criterion = nn.MSELoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>

ptrblck: <p>You have to create an instance of <code>LinearRegression</code>:</p>
<pre><code class="lang-python">model = LinearRegression(input_size, output_size)
</code></pre>
<p>before using the model in some way.</p>
Dynamic graphs on mobile

Etienne_Perot: <p>is it possible to keep a dynamic computational graph on mobile?<br>
i see an opportunity for RNN, if you process arbitrary-length input, you cannot do the whole computation at once on your neural accelerator (think a video stream for example).</p>
<p>When porting to ONNX a RNN, i managed to port it Feed-Forward style meaning the RNN do 1/ N steps with hidden states and give back output and updated hidden states. The bottleneck of this approach is that you need to transfer a lot of data between host &amp; accelerator (for instance when using ConvLSTM for large video)</p>
<p>Is there a way to simply run pytorch code on mobile side without passing by this feed-forward hack?</p>
CPU usage high when inferencing on CPU

kstn_hut_hat: <p>I have a pre-trained model using pytorch. (train and eval on CPU)<br>
My laptop has 12 cores.<br>
When I inferenced on CPU, I saw system monitor. CPU usage of all cores is about 50%. Can I adjust the CPU usage for 2-4 cores (each core 100% is ok)?</p>
<p>Thank you.</p>
How can i find tensor inside of tensor of different shape

jubick: <p>Let’s say i have tensor A of shape (1024) and tensor B of shape (5,20,1024) and tensor B contains tensor A.<br>
How can i find index of tensor A in tensor B?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">a = torch.zeros(1024)
b = torch.randn(5, 20, 1024)
b[2, 17].fill_(0.)

idx = (a == b).all(dim=2).nonzero()
</code></pre>

jubick: <p>Once again thank you very much mister <a class="mention" href="/u/ptrblck">@ptrblck</a></p>