akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Exiting Training function, does not free memory for validation, which runs out of memory

username: tueboesen
cooked: <p>I have a code that looks like this:</p>
<pre><code class="lang-auto">    for epoch in range(epochs):
        aloss_t,aloss_E_t,aloss_F_t,MAE_t,Fps_t,Fts_t, t_dataload_t, t_prepare_t, t_model_t, t_backprop_t = use_model(model, dataloader_train, train=True, max_samples=1e6, optimizer=optimizer, device=device, batch_size=batch_size)
        aloss_v,aloss_E_v,aloss_F_v,MAE_v,Fps_v,Fts_v,t_dataload_v, t_prepare_v, t_model_v, t_backprop_v = use_model(model, dataloader_val, train=False, max_samples=100, optimizer=optimizer, device=device, batch_size=batch_size)
</code></pre>
<p>where use_model is a function that can either be used for training or validation over a dataset. The model runs fine during training and uses about 70% of my gpu memory with batchsize=40, however when it exists the training loop and switches to validation the memory does not get freed, and hence it runs out of memory when running the validation.</p>
<p>I thought the memory would be automatically freed when I left the training function, but apparently not, how do I ensure that this happens?</p>

username: sio277
cooked: <p>Python object defined in local scope is freed outing the scope (no pointer to it). Check if your GPU tensors are saved to some objects in a global scope while/after training. Note that even if you remove Torch GPU tensors, the memory is not released to OS but kept in a pool, for a faster realloc of the future tensors (<a href="https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/2" class="inline-onebox">About torch.cuda.empty_cache() - #2 by albanD</a>). So even if your nvidia-smi memory usage looks full, but is still available.</p>

username: tueboesen
cooked: <p>Based on your post I went back and took another look at it, and I narrowed down my issue. I was wrong in the memory not being freed it is instead the eval mode of my code that is requiring more memory than my training mode.</p>
<p>The problem is that in eval mode I’m still running the code with autograd, which is needed since I explicitly need to get gradient information even in eval mode. However I guess this gradient information is accumulating, so I need a way to zero it after each iteration.<br>
Using zero_grad on the optimizer doesn’t solve the problem, guess I need to use it on each variable individually</p>
Example for DistributedDataParallel?

username: spadel
cooked: <p>I usually like the PyTorch Documentation but I think for DistributedDataParallel it is terrible. I already cancelled several approaches to make it work and at the moment I am really depending on it. So some help would be amazing!</p>
<p>In all the tutorials I have seen so far, they never showed a way to specify the GPU’s I want to use. I am working on a server with 10 GPU’s, but they are not mine and I usually can only use 4 when I am lucky. So I somehow need to specify which exact GPU’s I would to use - this is covered nowhere in the tutorials.</p>
<p>In the official dokumentation [<a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a>] it says as follows:</p>
<pre><code class="lang-auto">torch.cuda.set_device(i)
torch.distributed.init_process_group(
backend='nccl', world_size=N, init_method='...')
model = DistributedDataParallel(model, device_ids=[i], output_device=i)
</code></pre>
<p>It says further:</p>
<blockquote>
<p>In order to spawn up multiple processes per node, you can use either <code>torch.distributed.launch</code> or <code>torch.multiprocessing.spawn</code> .</p>
</blockquote>
<p>But the documentation over there is even more complicated… Is there no easier way to do it? Can someone please help me out here or refer to a useful example? Thanks a lot!!</p>
Multihead attention has an extra linear layer?

username: Alexander_Soare
cooked: <pre><code class="lang-auto">mha = nn.MultiheadAttention(512, 4, bias=False)
[p.shape for p in mha.parameters()]
</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">[torch.Size([1536, 512]), torch.Size([512, 512]), torch.Size([512])]
</code></pre>
<p>So I see indices 1 and 2 as comprising an “extra” linear layer.</p>
<p>Following Attention Is All You Need, I wasn’t aware of the extra linear layer as part of multi-head attention.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1ac1b8f8550255b3489f089f3939b3016957bdfa.png" alt="image" data-base62-sha1="3OHsHdYMmGNi6u36Qrwx7EQuvF0" width="151" height="215"></p>
<p>Although they do have the blue Feed Forward box, I see that as a separate component.</p>
<p>So what’s going on in the PyTorch implementation? Is there an easy way to drop it?</p>

username: Alexander_Soare
cooked: <p>Well, after some thinking about it, I realised I forgot to consider $W^O$ from section 3.2.2 in the paper! I believe that’s it right?</p>
Why Pytorch C++ documentation is so bad when compared to Python

username: witl
cooked: <p>I wonder why the C++ Pytorch library is not well documented as the Python is, with examples, etc.</p>
<p>Even the full examples in the repo are not updated.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The main reason is that most of our users are using the python API and not so many use the c++ API.<br>
Also, the core contributor that was handling all the c++ API stuff has moved to other stuff, so there is no more core contributor to work on that.<br>
And having a full API with high quality documentation is a lot of work…</p>
<p>That being said, we are very happy to take any contribution to improve the doc, API or anything related to the c++ API!</p>
Softmax implementation

username: HmmRfa
cooked: <p>I want to reimplement Softmax so I can customize it.</p>
<p>I followed <a href="https://discuss.pytorch.org/t/how-to-implement-the-exactly-same-softmax-as-f-softmax-by-pytorch/44263/6">this</a> post by ptrblck.</p>
<p>Two questions:<br>
There is a lot of discussion about numeric stability (see <a href="https://stackoverflow.com/questions/34968722/how-to-implement-the-softmax-function-in-python" rel="noopener nofollow ugc">here</a> for example). Is this the case in the provided solution?<br>
Why is it necessary to substract the max of x?</p>
<p>Thanks for you help!</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="HmmRfa" data-post="1" data-topic="118049">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ecb155/48.png" class="avatar"> HmmRfa:</div>
<blockquote>
<p>Why is it necessary to substract the max of x?</p>
</blockquote>
</aside>
<p>this is exactly for numerical stability, exp(x) never overflows on a non-positive tensor x, exp(0)=1 (constant) is always included in the denominator, and underflow is more stable (worst case output is 0 instead of Infinity)</p>

username: Eta_C
cooked: <p>Note that <code>softmax(x) = exp(x)/sum(exp(x))</code>. If we have <code>x = [1, 10, 1000, 10000, 10000000]</code>, <code>exp(x)</code> would be too large so that our computer may couldn’t store it (may return <code>inf</code>). After substracting the max of <code>x</code>, <code>x</code> is in the interval <code>(-inf, 0]</code> and <code>exp(x)</code> is in the interval <code>(0, 1]</code>.</p>
Gathering dictionaries with NCCL for hard example mining

username: jteuwen
cooked: <p>When hard example mining, it is important to keep track of the data indices to be able to set the proper weights in either the loss function or the sampler. For this purpose, my Dataset outputs a dictionary including the index that outputs the sample, e.g., <code>{'image_index': idx, 'image': image, 'target': target}</code>.</p>
<p>The collate function then merges the indexes to a <code>Double</code> tensor, so far so good. When I am now evaluating the training set in multi gpu setting, I store the loss and the indices in two dictionaries (as data types are different) and attempt to merge these dictionaries across these GPUs to GPU 0 which I can then use to compute proper weights for the next epoch.</p>
<p>However, NCCL does not seem to support <code>gather</code>. I get <code>RuntimeError: ProcessGroupNCCL does not support gather</code> I could copy the data to the CPU before gathering and use a different process group with gloo, but preferable I would want to keep these tensors on the GPU and only copy to the CPU when  the complete evaluation is done. Is there a way around this so I can gather anyway (or another approach)?</p>

username: ekurtic
cooked: <p>Could you accomplish that with <code>all_gather</code>? All GPUs will receive it, but you can process it only on rank-0 GPU.</p>
Generating a perfect image example for a CNN network with using autograd

username: kamil4u
cooked: <p>Hello,</p>
<p>I would like well understand how a autograd works. I decided to make a simple experiment: generating noise image (from human point-of-view), which returns high confidence for a target class, similar on how FGSM attacks work. I do the following steps:</p>
<ul>
<li>
<p>loading trained CNN model: <code>net</code></p>
</li>
<li>
<p>setting target: <code>target = torch.tensor([5])</code></p>
</li>
<li>
<p>generating random/noised image:</p>
</li>
</ul>
<pre><code class="lang-auto">image = torch.rand((1, 3, 32, 32))`
image.requires_grad = True
</code></pre>
<ul>
<li>
<p>passing it through the model: <code>output = net(image)</code></p>
</li>
<li>
<p>calculating loss:</p>
</li>
</ul>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss()
cost = criterion(output, target)
</code></pre>
<ul>
<li>
<p>calculating grad:<br>
<code>grad = torch.autograd.grad(cost, image, retain_graph=False, create_graph=False)[0]</code></p>
</li>
<li>
<p>modifying original noised image:<br>
<code>ideal_image = image - grad.sign() #.clamp(min=0.0, max=1.0)</code></p>
</li>
</ul>
<p>There are some simplifying in codes.</p>
<p>I expect that the new image will be classified by the network with 100% confidence - as the class 5 (my target). In the FGSM attack, an eps parameter is added, but here I wanted to create the perfect image example for my network, so eps=1. However, it was not going to happen - the network returns nearly a random top class.</p>
<p>What I did wrong? I thought that if I subtracted the grad calculated from the loss to the noised image, then for the network point of view the new image will be the ideal example of the target class. Maybe, it works differently?</p>
<p>Thanks</p>
About License for "c10.lib" and "libiompstubs5md.dll"

username: suenaga.shohei
cooked: <p>Dear Pytorch License contact,</p>
<p>I am writing this email to know the license for Pytorch.<br>
It seems that Pytorch includes following unknown license files.<br>
・torch/lib/c10.lib:<br>
・torch/lib/libiompstubs5md.dll</p>
<p>nuget repository url:<br>
<a href="https://www.nuget.org/packages/Opteamizer.Torch-Debug" class="inline-onebox" rel="noopener nofollow ugc">NuGet Gallery | Opteamizer.Torch-Debug 2.0.0</a><br>
<a href="https://www.nuget.org/packages/Opteamizer.Torch-Shared" class="inline-onebox" rel="noopener nofollow ugc">NuGet Gallery | Opteamizer.Torch-Shared 2.0.0</a></p>
<p>Could you tell me the license for these files?</p>
<p>Sincerely,</p>
<p>Shohei Suenaga</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a>, who might know the licenses of these files.</p>

username: jspisak
cooked: <p>We have a uniform BSD style license across the project. You can find it here: <a href="https://github.com/pytorch/pytorch/blob/master/LICENSE" class="inline-onebox">pytorch/LICENSE at master · pytorch/pytorch · GitHub</a></p>

username: suenaga.shohei
cooked: <p>Thank you very much!!<br>
I understand the license.</p>
<p>Please close the topic.</p>
Torchvision and torchaudio for Windows missing on PyPI?

username: 1enn0
cooked: <p>Hello everybody!</p>
<p>I’m a little puzzled by the current status of the PyTorch wheels for Windows on PyPI. I’m aware the “official” way of installing PyTorch through pip on Windows is to use the index from pytorch.</p>
<p>I noticed the wheel for the CPU build of the main <code>torch</code> package is available on PyPI, while the win wheels for <code>torchvision</code> and <code>torchaudio</code> packages are not.</p>
<p>I cannot find any recent helpful information about this, only some discussions in GitHub issues (mainly <a href="https://github.com/pytorch/vision/issues/1774" rel="noopener nofollow ugc">this</a> and <a href="https://github.com/pytorch/pytorch/issues/24310#" rel="noopener nofollow ugc">this</a>) about binary size constraints from PyPI. Looking at the <code>torchvision</code> and <code>torchaudio</code> whls from the pytorch index, this should not be an issue, though, as they are both &lt; 1 MB.</p>
<p>Also, the <code>torchvision</code> builds <em>used</em> to be pushed to PyPI (up until version 0.5.0) which makes me wonder even more why they are not anymore. On the other hand, <code>torchaudio</code> has never had any Windows wheels pushed.</p>
<p>I understand the binary size problems related to the CUDA builds with NVIDIA not providing a global CUDA PyPI package but I do not really see a reason why the CPU builds could not be pushed to PyPI?</p>
<p>Does somebody have more information about whether this is intended or if it’s just a bug?</p>

username: 1enn0
cooked: <p>Just as a quick update:</p>
<p>Starting with the</p>
<ul>
<li>
<a href="https://pypi.org/project/torch/" rel="noopener nofollow ugc"><code>torch</code></a>: 1.8.0,</li>
<li>
<a href="https://pypi.org/project/torchvision/" rel="noopener nofollow ugc"><code>torchvision</code></a>: 0.9.0 and</li>
<li>
<a href="https://pypi.org/project/torchaudio/" rel="noopener nofollow ugc"><code>torchaudio</code></a>: 0.8.0</li>
</ul>
<p>releases, the Windows (CPU) builds are available on PyPI again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Autograd gamma function

username: quornmd
cooked: <p>I am running a script using a reparameterisation trick on the Dirichlet distribution sampling from a gamma distribution I am using the <code>torch.igammac(x,0)</code> function. I am getting this error message:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
the derivative for 'igammac: input' is not implemented.
</code></pre>
<p>Getting the derivative of the gamma function shouldnt be so difficult. so why is it not implemented? Is it easy to just add it myself?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It is indeed not implemented right now.<br>
If you have the formula for it, you can send a PR that adds it in our file that define derivatives here: <a href="https://github.com/pytorch/pytorch/blob/de5e3b5eb045829041144e1c4e44448d13313d74/tools/autograd/derivatives.yaml#L555" class="inline-onebox">pytorch/derivatives.yaml at de5e3b5eb045829041144e1c4e44448d13313d74 · pytorch/pytorch · GitHub</a> and we will be happy to merge that! (You can add me as a reviewer)</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="quornmd" data-post="1" data-topic="118087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/q/a3d4f5/48.png" class="avatar"> quornmd:</div>
<blockquote>
<p>Dirichlet distribution sampling</p>
</blockquote>
</aside>
<p>derivative issue aside, it is odd that you need the gamma function for that, it should be enough to calculate fractions of GammaDistribution(a,1) samples. torch._standard_gamma(a) sampling is differentiable (by using reparameterisation I assume).</p>

username: quornmd
cooked: <p>that actually seems to work. I was using a uniform proposal and the inverse cdf method to obtain may sample. I actually didn’t know, that the torch.distribution… had grad built-in.</p>
Improve UNet Recognition

username: Alex_B
cooked: <p>Hi! I am trying to implement a UNet for aerial images. I am doing a semantic segmentation and if my network has to distinguish between water(blue) and ground(green) it works fine, but when I try to recognize ground(green), forest(a darker green) and houses(grey), the network does not learn.<br>
Because the images are large, I split them into 128x128 images and I selected the images that contain the classes that I am interested in. If I want 3 classes, I select the photos that have:</p>
<ol>
<li>
<blockquote>
<p>20% of the photo with class “agriculture”</p>
</blockquote>
</li>
<li>
<blockquote>
<p>20% of the photo with class “urban”</p>
</blockquote>
</li>
<li>
<blockquote>
<p>5% of the photo unknown</p>
</blockquote>
</li>
</ol>
<p>I added multiple images, but the loss is not getting better. How can I try to improve it. Many thanks!<br>
Data set: <a href="https://www.kaggle.com/balraj98/deepglobe-land-cover-classification-dataset" class="inline-onebox" rel="noopener nofollow ugc">DeepGlobe Land Cover Classification Dataset | Kaggle</a></p>
How should I calculate accuracy of triplet neural network in pytorch?

username: S_M
cooked: <p>hello all<br>
I found the following link, a way to calculate the accuracy of the triplet network. Is that the only way to obtain the accuracy in the mentioned network?<br>
thanks<br>
<a href="https://github.com/andreasveit/triplet-network-pytorch/blob/14e64764ec33d0a6e54508cee24ef7e7fe3a8366/train.py#L251" rel="noopener nofollow ugc">https://github.com/andreasveit/triplet-network-pytorch/blob/14e64764ec33d0a6e54508cee24ef7e7fe3a8366/train.py#L251</a></p>
How to initialize weights in custom layers while keeping pretrained layers untouched

username: nusstter
cooked: <p>Hi guys. Consider the following code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()

        googlenet = torchvision.models.googlenet(pretrained=True)
        self.features= nn.Sequential(*list(googlenet.children())[0: 9])

        self.conv1 = nn.Conv2d(512, 256, 3, 1, padding=1)
        self.conv2 = nn.Conv2d(256, 256, 3, 1, padding=1)
        self.conv3 = nn.Conv2d(256, 256, 3, 1, padding=1)
        self.conv4 = nn.Conv2d(256, 256, 3, 1, padding=1)
        self.conv5 = nn.Conv2d(256, 128, 3, 1, padding=1)

        self.conv6 = nn.Conv2d(128, 512, 1, 1)
        self.conv7 = nn.Conv2d(512, 100, 1, 1)
</code></pre>
<p>I want to know how to initialize the self.conv layers from normal distribution and at the same time keep the pretrained(self.features) layers untouched. Should I do it manually?<br>
Thanks.</p>

username: ptrblck
cooked: <p>To initialize parameters you could use the <code>torch.nn.init</code> methods on the parameters and you can freeze the <code>self.features</code> module by setting the <code>.requires_grad</code> attribute to <code>False</code> for its parameters.</p>
Apply function along axis

username: MehdiZouitine
cooked: <p>Hello,</p>
<p>I have a function that work on a tensor of shape (B,1) and return (B,1).<br>
I want to apply the same function across a tensor of shape (B,S,1) along the dimension S.</p>
<p>How to do that in torch ?</p>
<p>Thanks you</p>

username: ptrblck
cooked: <p>It depends on the used function, but broadcasting might just work.</p>
Pytorch to Keras via ONNX

username: DSL
cooked: <p>Hey together,</p>
<p>i want to convert my pytorch model to Keras, use tensorrt and then run the inference with c++. I read some tutorials like <a href="https://learnopencv.com/how-to-convert-a-model-from-pytorch-to-tensorrt-" rel="noopener nofollow ugc">https://learnopencv.com/how-to-convert-a-model-from-pytorch-to-tensorrt-</a> where it looks very easy, but i have some questions. When i create an onnx file i have a binary structure of my neural network with the weights or? I learned from torchscript there are to ways to convert a model (tracing, scripting) and i read the same here <a href="https://pytorch.org/docs/stable/onnx.html#use-external-data-format" class="inline-onebox" rel="noopener nofollow ugc">torch.onnx — PyTorch 1.8.1 documentation</a>. When i try to export my model to onnx, will be it “traced” or “script”? I think as default it will be traced but then these tutorials aren’t helpful for me… Maybe someone can give me a short conclusion about my questions and an advice what  i can do next, that would be very nice.</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting">ONNX - Tracing vs. Scripting docs</a> it seems that both exports might be supported.</p>
Classification metrics can't handle a mix of multilabel-indicator and multiclass targets

username: Xubairlone90
cooked: <p>Hi, I am trying to calculate the precision and recall for two images. with one image as the output of the model (shape [3,786,1024]) and the other ground truth (shape [1, 786, 1024]). But I keep getting an error. I am new to python and I am able to understand what I am doing wrong. I have tried some things like getting the input and output to the same shape, but I am getting the error now:</p>
<p>raise ValueError("Classification metrics can’t handle a mix of {0} "<br>
ValueError: Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets<br>
If anybody can help me with this issue. i would be grateful.</p>
<p>My code is here:</p>
<pre><code>def precision(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(precision_score(la,preds, average='micro'))

def recall(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(recall_score(la,preds, average='weighted', zero_division = 1))
def f1(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(f1_score(la,preds, average='weighted'))


img50 = Image.open('0050_999.png')
numpydata_in = np.array(img50).astype('uint8')
#print("test shape",numpydata_in.shape)
numpydata_in= torch.from_numpy(numpydata_in)
print("final input shape", numpydata_in.shape)

imgT50 = Image.open('0050.jpg')
numpydata_out = np.array(imgT50).astype('uint8')
numpydata_out = transforms.ToTensor()(numpydata_out)
#print(numpydata_out.shape)
numpydata_out = torch.argmax(numpydata_out, dim=0)
#print(numpydata_out.shape)
numpydata_out = torch.squeeze(numpydata_out)
print("final shape target",numpydata_out.shape)

prec = precision(numpydata_in, numpydata_out)
print("precision is",prec)
rec = recall(numpydata_in, numpydata_out)
print("recall is",rec)
f1 = f1(numpydata_in, numpydata_out)
print("f1 score is",f1)
</code></pre>
<p>final input shape torch.Size([768, 1024])<br>
final shape target torch.Size([768, 1024])</p>
<p>raise ValueError("Classification metrics can’t handle a mix of {0} "<br>
ValueError: Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets</p>

username: ptrblck
cooked: <p>Which library are you using to calculate the <code>precision</code>, <code>recall</code>, and <code>f1</code> scores?<br>
I assume that one of these methods raises the error, so you would have to check the docs what inputs in what shapes are expected.</p>
Check dropout is deactive

username: cbd
cooked: <p>I have code below. I put the model in eval mode then also it print dropout layer when model is print. Is it common? How could i know it is deactivated once i change to evaluation mode.</p>
<h4>
<a name="drop-out-check" class="anchor" href="#drop-out-check"></a>DROP OUT CHECK</h4>
<p>import torch<br>
import torch.nn as nn<br>
class Test(nn.Module):<br>
def <strong>init</strong>(self, pool_size=(4, 4)):<br>
super(Test, self).<strong>init</strong>()<br>
self.layer = nn.Linear(1, 1, bias=False)<br>
self.dropout = nn.Dropout()</p>
<pre><code>def forward(self, x):
    out = self.dropout(self.layer(x))
    return out
</code></pre>
<p>model=Test();<br>
model.train()<br>
print("train model is : ",model)<br>
model.eval()<br>
print("eval model is : ",model)</p>

username: ptrblck
cooked: <p>Yes, the model still contains the dropout layer, which is why it’ll be printed, but would disable it.<br>
You can check its <code>training</code> attribute, which should be set to <code>False</code>, via: <code>print(model.dropout_layer.training)</code>.</p>

username: cbd
cooked: <p>Thanks a lot. Your answer cleared my doubt.</p>

username: cbd
cooked: <p>Hi, I just made a little bit change in the code and apply “DataParallel” as below. It shows the error "AttributeError: ‘DataParallel’ object has no attribute ‘layer_1’ ".</p>
<h4>
<a name="drop-out-check" class="anchor" href="#drop-out-check"></a>DROP OUT CHECK</h4>
<p>import torch<br>
import torch.nn as nn<br>
class Test(nn.Module):</p>
<pre><code>def __init__(self, pool_size=(4, 4)):
    super(Test, self).__init__()
    self.layer_1 = nn.Sequential(nn.Linear(1, 1, bias=False),
                 nn.Dropout())

  

def forward(self, x):
    out = self.dropout(self.layer_1(x))
    return out
</code></pre>
<p>model=Test();</p>
<h4>
<a name="added-extra-2-lines-as-below" class="anchor" href="#added-extra-2-lines-as-below"></a>ADDED EXTRA 2 LINES AS BELOW</h4>
<p>model.to(0)<br>
model = torch.nn.DataParallel(model, [0])</p>
<p>model.train();<br>
print("train model is : ",model)<br>
print("training ",model.layer_1[1].training)</p>
<p>model.eval();<br>
print("testing model ",model.layer_1[1].training)</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> wraps the passed model into the <code>.module</code> attribute, so you would have to access it via <code>model.module.layer1</code>.</p>

username: cbd
cooked: <p>Your prompt reply is really appreciated. Thanks.</p>
Is it better to set batch size as a integer power of 2 for torch.utils.data.DataLoader

username: songyuc
cooked: <p>Hi, guys,<br>
I have heard that it would be better to set batch size as a integer power of 2 for <code>torch.utils.data.DataLoader</code>, and I want to assure whether that is true.</p>
<p>Any answer or idea will be appreciated!</p>

username: ptrblck
cooked: <p>Powers of two might be more “friendly” regarding the input shape to specific kernels and could perform better than other shapes (internally padding could be used, if it could yield an overall speedup).<br>
However, it depends on your actual model, input shapes, etc. so you could profile different shapes and check for performance cliffs.</p>

username: songyuc
cooked: <p>Hi,<br>
specifically, for resnet, is the “kernel” mentioned denote the <code>kernel_size</code> in <code>torch.nn.Conv2d</code>?</p>

username: ptrblck
cooked: <p>Powers of two could be preferred in all dimensions, so number of channels, spatial size etc.<br>
However, as described before, internally padding could be used, so that you wouldn’t hit a performance cliff and should thus profile your workloads.</p>

username: songyuc
cooked: <p>Get it.<br>
It means internally padding is a internal mechanism of PyTorch, so that commonly, it is not that requisite for us to consider setting a 2-power batch size, am I right?</p>

username: ptrblck
cooked: <p>It can be used by cudnn and I don’t think that native PyTorch kernels use it (at least I haven’t seen it, but might be wrong).</p>

username: songyuc
cooked: <p>OK, thanks sincerely for your answer!</p>
Conv1D Input Shape and Channels

username: hachiro
cooked: <p>I have an input tensor of shape [8 , 500 , 502 ] where 8 is the batch size , 500 is the length of a bag ( i’m using multiple instance learning ) and 502 is my window size. One bag represents the concatenation of 2 histograms.</p>
<p>I want to use a feature extractor with Conv1d auto encoder-decoder.</p>
<p>Should i transpose my input to x = x.transpose(2,1).contiguous() or use something like  x = x.view(8*500, 1 , 502) . I am a bit confused with the concept of the channels to the 1D Convolution.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Each kernel in a “standard” (i.e. non-grouped) convolution will stride through the spatial dimensions and will use all input channels. The usage of the reshape operation depends on your actual use case and the logic your model should apply to the input. The <a href="https://cs231n.github.io/convolutional-networks/#conv">CS231 - Conv</a> notes explain the underlying operations of conv layers in more details.</p>
To find out input size to insert summary

username: sangheonlee
cooked: <p>Hi, today I was running GNMTv2 of PyTorch.</p>
<p>I want to know the total parameter size of GNMT, But I cannot find how to work with pytorch summary. I was looking at this, It needs “input size of GNMT”.</p>
<p>Now I set the option of GNMT, layer=16, sentence size fix to 30.</p>
<p>or I want to know how can i check the total parameter size of the model like transformer.</p>
<p>Thanks.</p>
Datasets and DataLoaders for Sequential Data

username: jagraves21
cooked: <p>I am working with sequential data, and I am trying to create a custom <code>torch.utils.data.Dataset</code> that will reshape my data as necessary.  The raw data is a collection of sequences (not all the same length) with readings taken at equally spaced time steps.  Given <code>window_len</code> contiguous readings in a row, I want to make a point forecast <code>horizon</code> steps into the future.  I’ve written the following custom Dataset to reshape my data for training:</p>
<pre><code>import numpy as np
import torch.utils.data

class TimeSeriesDataset(torch.utils.data.Dataset):
    def __init__(self, Xs, ys, window_len, horizon):
        super(torch.utils.data.Dataset,self).__init__()
        self.window_len = window_len
        self.horizon = horizon
        self.X = np.vstack([
            self.get_X(X, window_len, horizon)
            for X in Xs
        ])
        self.y = np.vstack([
            self.get_y(y, window_len, horizon)
            for y in ys
        ])

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

    @staticmethod
    def get_X(values, window_len, offset):
        return np.hstack([
            np.roll(values, -ii, axis=0)
            for ii in range(0,window_len)
        ])[:values.shape[0]-window_len-offset+1].reshape(-1,window_len,values.shape[1])

    @staticmethod
    def get_y(values, window_len, horizon):
        return values[window_len+horizon-1:].copy()
</code></pre>
<p>Example usage:</p>
<pre><code>window_len = 3
horizon = 2
n_sequences = 10
random = np.random.RandomState(0)

sequences = [
    random.randint(0, 100, size=(random.randint(5,10),2) )
    for ii in range(n_sequences)
]
targets = [
    random.randint(0, 100, size=(len(sequence),1) )
    for sequence in sequences
]

dataset = TimeSeriesDataset(sequences, targets, window_len, horizon)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=10, shuffle=True)
for X_train, y_train in dataloader:
    print(X_train.shape, y_train.shape)
</code></pre>
<p>My problem is that the original data is small enough to fit into memory, but the reshaped training data is too large. And so I have two questions:</p>
<ol>
<li>Is this the best way to go about solving the problem if the data fits into memory?</li>
<li>What is the best way to modify this so that all of the sub-sequences don’t have to be created at once?</li>
</ol>
<ul>
<li>I’d like to still be able to use slicing on the <code>dataset</code> if possible (e.g., <code>dataset[5:10]</code>).</li>
</ul>
Confused with ".act = nn.Identity()"

username: szahan
cooked: <p>I am trying to understand a model implementation where there is this following layer. I don’t understand what the last line means.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,
                 num_class,
                 num_point,
                 num_person,
                 num_gcn_scales,
                 num_g3d_scales,
                 graph,
                 in_channels=3):
        super(Model, self).__init__()

        Graph = import_class(graph)
        A_binary = Graph().A_binary
        
        self.sgcn2 = nn.Sequential(
        MS_GCN(num_gcn_scales, c1, c1, A_binary, disentangled_agg=True),
        MS_TCN(c1, c2, stride=2),
        MS_TCN(c2, c2))
        self.sgcn2[-1].act = nn.Identity()
</code></pre>
<p>It looks like the model first perform spatial graph convolution followed by two temporal convolution. But what happens with this line       <strong>self.sgcn2[-1].act = nn.Identity()</strong></p>
<p>Your suggestions will greatly help. Thanks.</p>

username: crcrpar
cooked: <p>Hi,</p>
<p><code>self.sgcn2</code> is sequential of <code>MS_GCN</code>, <code>MS_TCN</code>, and <code>MS_TCN</code>.<br>
So, <code>self.sgcn2[-1]</code> refers to the last <code>MS_TCN</code>.<br>
<code>self.sgcn2[-1].act</code> is either the attribute or member variable of an instance of <code>MS_TCN</code>. From that name, I guess by default <code>MS_TCN</code> has some activation function such as Sigmoid and ReLU. Though the author of that snippet wanted that last module to use <code>Identity</code> as its activation.</p>

username: szahan
cooked: <p>Thank you so much. That is exactly the case, there is a activation function in MS_TCN and act holds the value that determines what the activation will be.</p>
Create hdf5 file for custom dataset

username: Kapil_Rana
cooked: <p>I have very larger files in dataset.<br>
Currenty I am using a laptop gpu for my work.<br>
Now I want to use Google colab. uploading such a large number of file is not feasible.<br>
I am thinking to load images from hdf5 files stored in my google drive.</p>
<p>So currently I have custom dataset class which return two input and one label.</p>
<p>Can anyone suggest me how I can transform my dataset into hdf5 file.</p>

username: vaisakh_m
cooked: <p>Do you really need to store it as an hdf5 file?<br>
If no, my suggestion would be:</p>
<ul>
<li>
<p>Upload the dataset(files) to Drive.</p>
</li>
<li>
<p>Mount Drive in Colab.</p>
</li>
<li>
<p>Use the dataset from Drive.</p>
</li>
</ul>
Missing links to source code in docs

username: tastyminerals
cooked: <p>I like browsing PyTorch docs but it would be nice if all the methods have a link to the corresponding source code. This is not the case though. For example, I wanted to take a look at <code>torch.zeros</code> implementation and went to <a href="https://pytorch.org/docs/stable/torch.html#torch.zeros" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.zeros</a>. The source code link is missing as for many other functions, why?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately these functions are implemented in c++ and so it’s not very easy to provide the source code for them directly from the doc.<br>
Do you think cpp code would still be useful in the doc?</p>

username: PersonalFinance004
cooked: <p>I’m resurrecting this because I think the cpp code would indeed be useful in the doc.</p>
Mean is not 0 after subtracting mean

username: autoencoder
cooked: <p>I have a (178741, 304) dataset where the first dimension correponds to one sample and the second dimension are the features of a sample. The means of the features range between -0.75 and 0.99 and I want to normalize the dataset so that the mean is 0.0 afterwards.</p>
<pre><code class="lang-auto">x = the dataset
x_mean = x.mean(axis=0)
print("Before", x_mean.min(), x_mean.max())

x_centered = x - x_mean
x_centered_mean = x_centered.mean(axis=0)
print("After", x_centered_mean.min(), x_centered_mean.max())
</code></pre>
<p>However, after the normalization the mean is still not 0. It got much smaller, but is still relatively high:</p>
<blockquote>
<p>Before tensor(-0.748372793198) tensor(0.999999940395)<br>
After tensor(-0.001484304899) tensor(0.000192862455)</p>
</blockquote>
<p>The same happens when I try to normalize the std. The resulting std is not exactly 1. (All I do is divide x_centered by the std. I have no constant features that could cause std=0)</p>
<p>What could be the reason for these instabilities? I am using torch.float32 values. In the original dataset the smallest value is -0.999 and the largest is 1.0. The mean among all dimensions is 0.11</p>
<p>I read that very large values can cause bugs like this, but I have only small values. Am I not using the right data type?</p>
<p><strong>EDIT - Better results with double instead of float</strong><br>
I realized that there is also a double data type. When I convert the dataset to double before centering the mean, I get a much better result. The mean reaches exactly 0, but as soon as I divide by the STD, the mean deviates again a little bit from 0.0 (range -0.00001 to 0.000001). But still better than before.<br>
Would you recommend centering it again after dividing by the STD? I saw that sklearn is doing that (<a href="https://github.com/scikit-learn/scikit-learn/blob/7389dbac82d362f296dc2746f10e43ffa1615660/sklearn/preprocessing/data.py#L158" rel="noopener nofollow ugc">https://github.com/scikit-learn/scikit-learn/blob/7389dbac82d362f296dc2746f10e43ffa1615660/sklearn/preprocessing/data.py#L158</a>)</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid it is indeed a problem of numerical precision of floating point numbers <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  The problem here is mostly that the dimension you reduce over (178741) is big enough that all the very small errors made by each individual op build up to something that is visible at the end. Also keep in mind that floating point numbers have only 6 significant digits. So anything smaller than 1e-6 here would be “noise”. With double precision, you can expect 1e-11.<br>
If you do need more precision, double precision numbers are the way to go indeed.</p>
<p>Note though that it will be significantly slower to perform computation in double precision on GPU.</p>
Backpropagate with a given loss from another model

username: ruoyun_hu
cooked: <p>Hi all,</p>
<p>I currently face a problem of using Autograd() in a distributed learning problem.<br>
Imagine we have two identical networks separately in client and server. I calculate the loss from client side, and send it to server. Could I use the this loss to do the backpropogation in server side?</p>
<p>I know I need to build training graph in server side to do backpropogation, therefore I just use some fake data to do the forward pass to build the training graph, the pseudo code is as follow:</p>
<p>client:</p>
<p>loss_client = log(net(data_client))<br>
send loss_client to server</p>
<p>Server:</p>
<p>fake_data<br>
fake_output = log(net(fake_data))<br>
fake_output.data = loss_client.data</p>
<p>optimizer.zero_grad()<br>
fake_output.backward()<br>
optimizer.step()</p>
<p>Current idea does not work, any help will be really appreciated!</p>

username: Dwight_Foster
cooked: <p>what happens if you just use loss_client as the backward() parameter and just go</p>
<pre><code class="lang-auto">loss_client.backward()
</code></pre>

username: ruoyun_hu
cooked: <p>The training still passes, but the result is not correct comparing to train locally. I think only change the loss is not possible to change the gradient, the gradient is input-related. So until now, I didn’t find a solution for this problem</p>
How to calculate f1 score on the whole data

username: dato_nefaridze
cooked: <p>Hello, I am a training multiclass classifier. I want to calculate the F1 score on the whole dataset. earlier I calculated f1 for each batch using scikit-learn and I averaged f1 scores on batches, but it seems that it’s not the correct solution, it will be close to real f1 but it’s still not the exact value. the dataset is about 5GB to 10GB. How can I calculate the f1 score on the whole dataset?</p>

username: Sanjan_Das
cooked: <p>You technically only need the <code>y_pred</code>, <code>y_true</code> and <code>labels</code> to calculate the F1 score as per <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html" rel="noopener nofollow ugc"><code>scikit-learn</code>'s F1 score documentation</a>. You really don’t need anything else from your 5-10 GB dataset.</p>
<p>Perhaps I’ve misunderstood what you’re trying to do, in which case, could you give a bit more detail on what you’re trying to do?</p>

username: dato_nefaridze
cooked: <p>solution is to use Pytorch lightning</p>
Load and save of optimizer and scheduler

username: scarletteshu
cooked: <p>i want to resume the saved model and continue training. justing wondering what’s the exact procedure to load optimizer and scheduler, then use them on gpu.</p>
<p><strong>question 1:</strong> after loading model state dict, is my model still on gpu?<br>
here’s my code</p>
<pre><code class="lang-auto">    model = Modelclass()
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    model.to(device)  

    optimizer = optim.Adam(model.parameters(), lr = 'lr')
    scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)

    if os.path.isfile("model.pth.tar"):
        checkpoint = torch.load("model.pth.tar")
        model.load_state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
</code></pre>
<p><strong>question 2:</strong> is my optimizer is still on GPU after that?<br>
i saw some tutorials sad that the following code should be added, if you want to use optimizer on gpu:</p>
<pre><code class="lang-auto">        for state in optimizer.state.values():
            for k, v in state.items():
                if torch.is_tensor(v):
                    state[k] = v.cuda()
</code></pre>
<p>but it seems that if use Adam this is not working<br>
<strong>question3:</strong> did this scheduler need additional steps to be loaded? such as scheduler._step_count?In my case, i use ExponentialLR.</p>
<p>thanks!</p>
The performance of LibTorch which compiled by ourselves is worse than that provided by the government

username: JeremyGong
cooked: <p><strong>A. ENV</strong><br>
1. centos 7<br>
2. Linux localhost.localdomain 3.10.0-693.el7.x86_64 <span class="hashtag-raw">#1</span> SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux<br>
3. gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)<br>
4. Python 3.8.5 (default, Sep  4 2020, 07:30:14)</p>
<p><strong>B. compile method</strong><br>
1. git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
2. git checkout v1.8.1<br>
3. git submodule sync<br>
4. git submodule update --init --recursive<br>
5. cd pytorch<br>
6. mkdir pytorch-build<br>
7. cd pytorch-build<br>
8. export BUILD_TEST=0<br>
9. export USE_CUDA=0<br>
10. cmake -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=Release -DPYTHON_EXECUTABLE:PATH=<code>which python3</code> -DCMAKE_INSTALL_PREFIX:PATH=…/pytorch-install …</p>
<p><strong>C. TEST CODE</strong><br>
torch::jit::script::Module module;<br>
module = torch::jit::load(argv[1]);<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
at::Tensor input = torch::ones({1, 300, 40});<br>
inputs.push_back(input);<br>
at::Tensor output = module.forward(inputs).toTensor();</p>
<p><strong>D. performance</strong><br>
1. The performance of LibTorch which compiled by ourselves is worse than that provided by the government<br>
2. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3d517729e334a66f82c95671a5bc3a6b7322e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ce3d517729e334a66f82c95671a5bc3a6b7322e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3d517729e334a66f82c95671a5bc3a6b7322e0.png" alt="image" data-base62-sha1="tqtJq7JCQi1YTBm9Wy4ircGSgec" width="690" height="81" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×122 4.51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>E. compile libtorch with cpu options</strong><br>
1. how can i konw the official compile libtorch with cpu options ?</p>

username: ptrblck
cooked: <p>The <a href="https://github.com/pytorch/builder">pytorch/builder</a> repository should contain all build scripts, so you could compare yours to the official one.</p>

username: JeremyGong
cooked: <p>Thank u, I have solved this problem.</p>
<p>I check the cmake options and then I find that option USE_MKL is OFF .</p>
<p>there is a problem in my environmet, I do not install MKL.</p>
<p>so, I installed the MKL, and recompile the libtorch, and check the option USE_MKL=ON.</p>
<p>In the end, I made it.</p>
How to calculate the probability value or cdf of a normal distribution?

username: Shantanu_Ghosh
cooked: <p>I can sample from a normal distribution with mean 0 and sd 1 like torch.randn(size). Now how to calculate the probability value of the samples in pytorch like we do in scipy as following?<br>
----&gt; scipy.stats.norm(0, 1).cdf(0)</p>

username: KaiHoo
cooked: <p>Use torch.erf, but you need some transform.</p>
Combining two pretrained models for Unet

username: Alex_B
cooked: <p>Hi,</p>
<p>I have a UNet that does binary semantic segmentation and I wanted to extend it to detect multiple classes. The first idea that I got was to train a model for each class and then when I would do the prediction, I would use a combined model between all pretrained models. Any idea on how to approach this?</p>
<p>Also, if this is not a good idea and if you have a better approach please let me know <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p>You could try to use multiple models, where each model predicts a single class vs. others, but would then need to decide how to use the results. I.e. if you are working on a multi-label classification (zero, one, or multiple classes can be active for each sample), you could directly use the output. On the other hand, if you are working on a multi-class classification, you would have to decide, if e.g. only the highest prediction should be used as the class prediction.</p>
<p>Usually you would just use a single model and let it predict multiple classes directly.</p>

username: Alex_B
cooked: <p>Thank you for your answer! I got the part with the multi-label classification, but I am more interested in the multi-class classification and I didn’t understand it. I have to train multiple models to predict one class vs others and then choose the best predictor for every class?</p>

username: ptrblck
cooked: <p>This would be your proposed approach, if I understand it correctly, and I just tried to clarify the workflow.<br>
For a “standard” multi-class classification, you would just use an output layer which returns <code>nb_classes</code> logits (e.g. via <code>nn.Linear(in_features, nb_classes)</code>).</p>
How to select a specific batch from dataset?

username: pumplerod
cooked: <p>I have a dataset with samples by year-month.</p>
<p>I’m able to create a custom Dataset which does a wonderful job of fetching data throughout the dataset, however I now need to be able to retrieve batches from the DataLoader in year-month batches.</p>
<p>I’m not sure if I should be creating a custom Sampler (or BatchSampler), and how that will specifically select only a batch of a certain year-month to iterate over as it pulls out samples.</p>
<p>I begin with a pandas DataFrame which I use when I init my custom torch.Dataset.  I need to be able to iterate over all the year-month groups, returning each as it’s own batch of samples.</p>