akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Image classification: does softmax lose information?

FaultyBagnose: <p>Suppose you train a standard CNN classifier, where each image in the training set either shows a cat or a dog. Such a model would ideally evaluate an image with neither animal as 50-50, and the same for an image with both a cat and a dog in it.</p>
<p>Can these cases be reliably distinguished by inspecting the pre-softmax outputs? You might expect two low activations in the former case, and two high ones in the latter?</p>
<p>Sorry if this is well-known. I’m sure it must has been studied, but a quick google search didn’t help.</p>

googlebot: <p>I think this is not guaranteed: for K classes you have K-1 fractions that define output, and one dimension is essentially arbitrary (sometimes it is set to zero as “calibrating” dimension). So, while it is intuitive to think about K positive scores, sum of these scores is not enforced to be meaningful (you may contrast this with dirichlet parameters).</p>

FaultyBagnose: <p>I agree. Still, there may be something to the intuition that the raw output of the “cat node” node should correlate with the presence of cats, even if you have K-1 other animals categories. It may of course be that the “cat node” really represents the signal “lack of non-cat animal”.</p>
<p>I’m not hoping for a mathematical theorem (although that would be nice), but wonder if it has been studied empirically. Since it’s such an obvious idea, I guess something would have been published on this.</p>

googlebot: <p>I think negative numbers prevent that; in your example, “cat detector” can choose to increase cat scores or decrease all other scores. You could maybe achieve correlation if you enforce non-negative final network segment (weights &amp; features), but then softmax (exponentiation part) would be not necessary. And sum of scores would still be undetermined, and may actually grow without bound.</p>

FaultyBagnose: <p>Again, I agree, and by “lack of non-cat animal” I meant just that: instead of increasing the “cat output” on cat images, it may just as well decrease the other outputs (making them more negative). Still, it might be worth testing, and my question is if someone knows about any systematic empirical test of this (mathematically naive) hypothesis.</p>
Signal matrix Denoising

sikhar: <p>I have a signal matrix 256x256. which contains signal peaks( complex values) in some rows other are zero. I am adding noise to particular rows which contains values and try to reconstruct the same signal using Convolutional autoencoder.(not using any standard model) . I am creating my  own model with 5 layers to do but while testing with unseen data, in reconstruction  part sometimes i am getting peaks with noise or getting noise only in whole matrix. (noise vary 0DB to -10DB). can any one please suggest something how can i improve my results.</p>
Dice loss becoming negative

bluesky314: <p>Hey, I am training a simple Unet on dice and BCE loss on the Salt segmentation challenge on Kaggle. My model’s dice loss is going negative after awhile and soon after so does the BCE loss . In this example, I pick a dataset of only 5 examples and plan to overfit.</p>
<pre><code class="lang-auto">
def dice(input, taget):
    smooth=.001
    input=input.view(-1)
    target=taget.view(-1)
    
    return(1-2*(input*target).sum()/(input.sum()+taget.sum()+smooth))
        
batch_size=10
optimizer = torch.optim.Adam(net.parameters(), lr=10e-3)
criterion = nn.BCEWithLogitsLoss()


dataset=DatasetSalt(limit_paths=10)
dataloader=DataLoader(dataset,batch_size, shuffle=True, num_workers=2)
net = UNet()


def train():
    for idx, batch_data in enumerate(dataloader):
        x, target=batch_data['image'].float(),batch_data['label'].float()


        optimizer.zero_grad()
        output = net(x)
        output.squeeze_(1)


        bce_loss = criterion(output, target)


        dice_loss = dice(output, target)
        loss = bce_loss + dice_loss 
        loss.backward()
        optimizer.step()
        
        
        
        print('Epoch {}, loss {}, bce {}, dice {}'.format(
            epoch, loss.item(), bce_loss.item(), dice_loss.item()))


    

for i in range(0,1000):
    train()
</code></pre>
<p>Output: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/169082242d20b9f3e5e9a1713b013aa457858816.png" data-download-href="https://discuss.pytorch.org/uploads/default/169082242d20b9f3e5e9a1713b013aa457858816" title="42%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/169082242d20b9f3e5e9a1713b013aa457858816_2_618x500.png" alt="42%20PM" data-base62-sha1="3dC6WSSUUah29IyE6Hf8rVusYTQ" width="618" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/169082242d20b9f3e5e9a1713b013aa457858816_2_618x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/169082242d20b9f3e5e9a1713b013aa457858816_2_927x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/169082242d20b9f3e5e9a1713b013aa457858816_2_1236x1000.png 2x" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">42%20PM</span><span class="informations">1574×1272 472 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I stop the training here and the results seems quite poor.</p>
<p>Update: Only training network on BCE works great. But when dice is added, loss goes to negative and both reset to a high number and start going to negative again and cycle like this.</p>

Ashwin_Raju: <p>The <code>inputs</code> parameter in the dice function should be between 0-1 but in your case it is not.</p>

chaochon: <p>your dice function seem to handle sample ouput and target rather than batch,</p>

moreshud: <p>Can you explain this in detail? I mean batch size implementation of the dice loss.</p>
<p>Thanks</p>
Targets between 0 and 1 in torch.nn.BCELoss?

samra-irshad: <p>Hi I am using <code>torch.nn.BCELoss</code> to train my model. The targets are either 0 or 1 (segmentation application, so the targets are a matrix of 0’s and 1’s). But I saw in the description of this loss ‘Note that the targets yy should be numbers between 0 and 1’. I am curious if its necessary to scale my targets to lie between 0 and 1?</p>

KFrank: <p>Hi Samra!</p>
<aside class="quote no-group quote-modified" data-username="samra-irshad" data-post="1" data-topic="105819" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samra-irshad/48/27993_2.png" class="avatar"> samra-irshad:</div>
<blockquote>
<p>I am using <code>torch.nn.BCELoss</code> to train my model. The targets are either 0 or 1<br>
…<br>
But I saw in the description of this loss ‘Note that the targets yy should be numbers between 0 and 1’.</p>
</blockquote>
</aside>
<p>More precisely, the <code>BCELoss</code> documentation should read:</p>
<blockquote>
<p>Note that the targets y should be numbers between 0 and 1,<br>
<em>inclusive</em>.</p>
</blockquote>
<p>That is, the <code>y</code> should range over the (so-called <em>closed</em>) interval<br>
<code>[0.0, 1.0]</code>, including taking on, potentially, the values <code>0.0</code> and <code>1.0</code>.</p>
<p>The target <code>y</code> can be understood to be the “ground truth” probability of<br>
the sample being in the “1” state.  But this means that it also works as<br>
a binary label: <code>y = 1</code> means that the sample has a 100% probability<br>
of being in the “1” state, so <code>y = 1</code> means the “1” state, while <code>y = 0</code><br>
means 0% probability of the “1” state, hence 100% probability of the<br>
“0” state, so <code>y = 0</code> means the “0” state (when understood as a binary<br>
label).</p>
<p>As a side note, for numerical reasons you will be better off using<br>
<code>BCEWithLogitsLoss</code> (and using the output of the final <code>Linear</code><br>
layer of your model as the model’s output, without passing it through<br>
a <code>Sigmoid</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
How to correctly typecheck nn.Module.__call__

gchaperon: <p>Hi! I am having trouble type checking my modules. Here is an example:</p>
<pre><code class="lang-python"># example.py
import torch
import torch.nn as nn


class Model(nn.Module):
    def forward(self) -&gt; int:
        pass


model = Model()

reveal_type(model.forward)
reveal_type(model.__call__)
</code></pre>
<p>running <code>mypy example.py</code> shows the following</p>
<pre><code class="lang-auto">example.py:12: note: Revealed type is 'def () -&gt; builtins.int'
example.py:13: note: Revealed type is 'def (*Any, **Any) -&gt; Any'
</code></pre>
<p>The first line is great, the second not so much.<br>
Now if I use the model as a callable:  <code>out = model()</code>, <code>reveal_type(out)</code> gets me <code>Any</code>, which is problematic.</p>
<p>Could somebody show me how to properly typecheck my code?<br>
Thanks in advance!</p>
How to Randomly Multiply Tensor by Matrix in the Dataloader?

Stanley_C: <p>For a point cloud, I can randomly translate it by multiplying it by a 3x3 rotational matrix. Here is my get item function in my Dataloader class:</p>
<pre><code class="lang-auto">	def __getitem__(self, index):
		current = self.mesh_paths[index]
		#print("./mesh/"+current)
		low = o3d.io.read_point_cloud(self.data_low_dir +current)
		high = o3d.io.read_point_cloud(self.data_high_dir+current)
	
		input_pt = torch.from_numpy(np.asarray(low.points))
		output_pt = torch.from_numpy(np.asarray(high.points))

		return input_pt, output_pt
</code></pre>
<p>Without randomly generating my rotation matrix in the getitem function, how would I multiply a new random to a tensor in the getitem function using the torch.transpose method?</p>
Mixed precision on part of the model

Podidiving: <p>Hi, everyone. Can I do mixed precision training only on the part of the model? Suppose, I have a model, which has a backbone and a classifier. I want to train backbone with mixed precision and classifier with single precision. Is there any way to do it?</p>

ptrblck: <p>Yes, you can wrap only certain parts of the model into <code>with torch.cuda.amp.autocast()</code> or use the decorator on specific functions.</p>
How to set default cuda device in libtorch?

Anders_G: <p>There is a function in pytorch to set the default cuda device. But I cannot find something similar in libtorch. Can someone help me out here?</p>

ptrblck: <p>I’m unsure if you can set the default device globally, but you could use the <code>CUDAGuard()</code> inside your method instead:</p>
<pre><code class="lang-auto">CUDAGuard device_guard(device);
</code></pre>
<p>which makes sure to use the specified <code>device</code> id.<br>
(Thanks <a class="mention" href="/u/tom">@tom</a> for clarification <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>

Anders_G: <p>Thanks for the tip <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to install PyTorch WITHOUT CUDA

jatentaki: <p>I have a remote machine which used to have GPUs and still has part of the drivers/libs but overall is out of date in that respect. I would like to treat it as a CPU-only server and install the according version of pytorch. However when I execute</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cpuonly -c pytorch
</code></pre>
<p>and run the collect_env.py script it tells me that</p>
<pre><code class="lang-auto">Collecting environment information...
/home/tyszkiew/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() &gt; 0
PyTorch version: 1.7.0
Is debug build: Yes
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04 LTS
GCC version: (Ubuntu 7.3.0-16ubuntu3) 7.3.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: No
CUDA runtime version: 9.0.176
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
</code></pre>
<p>Is this normal? The reason it matters to me is that I want to install <code>torch_geometric</code> (the cpu-only version) but it fails with</p>
<pre><code class="lang-auto">OSError: /cvlabdata2/home/tyszkiew/miniconda3/envs/py38/lib/python3.8/site-packages/torch_sparse/_version.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationERKSs
</code></pre>
<p>and I am wondering if that could be caused by <code>torch_geometric</code> thinking that it has access to CUDA and reaching for CUDA-specific symbols where in fact it is not available. I have also tried installing the cu102 version of <code>torch_geometric</code> but it fails on import trying to search for <code>libcusparse.so.10</code>. The point of my question is to figure out whether the issue is on the side of <code>torch_geometric</code> or PyTorch. Thanks.</p>

albanD: <p>Hi,</p>
<p>I think this is fine. You can use <code>CUDA_VISIBLE_DEVICES=-1</code> to hide all the GPUs from the cuda driver (and so from pytorch as well).</p>
<p>For the symbol error, it looks like it is an issue with <code>torch_sparse</code> no?</p>
API in libtorch that achieves the same function as in python

zhangyingbit: <p>Hi, guys,<br>
I have two questions for libtorch API<br>
&lt;1&gt; In python,  we can write as<br>
boxes_filtered =  boxes[keep]<br>
the shape of boxes is [ 80, 200, 4]<br>
the shape of  keep is [80, 200], and there are 40 “True” elements in keep<br>
so, we get the shape of boxes_filtered is [40, 4]<br>
I want to ask, is there an api that implements the same funcition in libtorch？<br>
&lt;2&gt;  In python, we can write as<br>
image = image[:, :, ::-1]<br>
So, is there an api that implements the same funcition in libtorch？<br>
Thank you so much : )</p>

ptrblck: <ol>
<li>
<p>Yes, it should work as explained in the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html">Tensor Indexing API</a>.</p>
</li>
<li>
<p>Should also be possible, see 1.</p>
</li>
</ol>

zhangyingbit: <p>Thank you so much for reply~</p>
Validation loss stays the same (as well as the training)

iamneerav: <p>I am training a deep CNN based model and my validation loss is always in the same range(5.81 to 5.84). I am using Root Mean Square Loss (RMSE) as the problem is of regression and implementing the U-Net architecture. The problem is</p>
<ol>
<li>Increasing or decreasing the learning rate is doing nothing</li>
<li>Making the architecture deeper is doing nothing</li>
<li>Incresing the channels/features is doing nothing</li>
<li>Changing the training and validation splits is doing nothing.</li>
</ol>
<p>I have tried everything listed above but none of them seems to decrease the validation loss. The training loss is also in the same range and stays almost the same too (a little bit more fluctuation but almost the same). What should I do?</p>

ptrblck: <p>Try to overfit a small dataset, e.g. just 10 samples, by playing around with the hyperparameters, architecture etc.<br>
Once your model is able to do so, try to scale up the use case again by using more data.</p>
Why to use validation during training, if we are using no_grad?

Hassan_Imani: <p>I have a basic question. I want to know that when we are using torch.no_gard() during the validation process, why to do validation? How it will help our model training? Just we can do validation with testing.</p>
<p>Also,<br>
I converted the code from keras to pytorch. For the first epoch it works like keras, but from the second one, my loss is not decreasing. It fluctuates very small around the mean value. I am doing regression with 3D convs. What can I do?</p>
<p>What maybe is the cause?</p>
<p>Thank you</p>

ptrblck: <p>I’m not sure I understand the question correctly, but are you wondering why the validation step is used at all or why <code>torch.no_grad()</code> is used?</p>
<p>The validation step is applied to get a proxy signal about the model performance on the test data (which cannot be used during training as it would be a data leak) and for hyperparameter tuning.</p>
<p>It’s wrapped into a <code>torch.no_grad()</code> block to avoid storing intermediate tensors, which would be needed for the gradient calculation, and are thus unnecessary during the validation loop.<br>
This would save memory and you could potentially increase the batch size to speed up the validation step.</p>
Why does the order in the upsampling path of the U-Net developed by Monai deviate from standard u-net?

Sabien: <p>Hi guys,<br>
I have a question about the U-Net developed by <a class="mention" href="/u/monai">@MONAI</a> . They refer to this link: <a href="https://link.springer.com/chapter/10.1007/978-3-030-12029-0_40" rel="noopener nofollow ugc">https://link.springer.com/chapter/10.1007/978-3-030-12029-0_40</a>. for explanation about the residual units they use. In every layer of their upsampling path, they first apply a transposed convolution, then a ‘normal’ convolution and then they concatanate the result with the result of the same layer of the downsampling path. However, in the link they refer to, the order is transposed convolution --&gt; concatenation --&gt; normal convolution. Does anyone know why they deviate from this usual order, and does the order have any influence on the result?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Hi <a class="mention" href="/u/sabien">@Sabien</a>,</p>
<p>I would recommend to post this question in the <a class="mention" href="/u/monai">@MONAI</a> GitHub repository, as more MONAI devs would see it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Which is much better for pre-train and customization?

utkuumetin: <p>Hi! In near future i can start natural language processing project with transformer models, should i use HuggingFace Transformers library or do i write my own version.</p>
<p>It’s going to be used in several separate projects and if i write my own version i can be more free i think.</p>
<p>What is your suggestions, thanks in advance.</p>
<p>(I’m going to use HuggingFace tokenizer, it’s ridiculously fast)</p>

ptrblck: <p>I think it depends how much “flexibility” you need and if you couldn’t achieve it by manipulating Huggingface models.<br>
Rewriting the models sounds like a good way to get familiar with the implementations, but you should consider how long it might take, and check if this time would be better spent by manipulating the models and run more experiments.</p>
Why file size of difference between two tensor images is bigger than original ones?

mHr029: <p>I have two tensor activation greyscale images. I converted those image into PIL images. Then I computed difference between those two PIL images using ImageChops.  The file size of this difference is supposed to be smaller than two original ones. But in my case,  it is showing the reverse. What do you think to be the possible cause for this? Please have an answer.</p>

ptrblck: <p>Depending on the image encoding the diff image might contain more information (entropy) and would thus need more data for the encoding.<br>
A simple example would be a totally blank image which wouldn’t require a lot of information as you would be able to store the color and just repeat it for all pixels, while an image containing random noise would contain much more information and would thus also be larger in file size.<br>
The difference should not be visible, if you don’t use any image encoding/compression but store the raw pixel values.</p>

mHr029: <p>Hi, thanks for your reply.</p>
<p>You are thus saying that it can happen, right?<br>
Is there any solution to get the difference as relatively small file size?</p>

ptrblck: <p>Yes, it can happen and is quite a good description of the entropy or information of data.<br>
If your diff image “looks” like random noise you won’t be able to reduce the file size significantly without losing data. <code>PIL.Image.save</code> should have some “quality” options, which would downgrade the image while saving but should yield a smaller file size.</p>

mHr029: <p>Ok, thanks for your explanation.</p>
nn.ReLU outputs nan on forward

Averu: <p>nn.ReLU randomly outputs Nan on forward. The problem only appears on GPU and not on CPU.</p>
<p>I captured ReLU input and outputs. This happens randomly on different parts of my torchvision VGG_16bn backbone, but allways at the first half of layers.<br>
For example in one of the calculations where output contained a single Nan the input tensor was size [2, 64, 1056, 800] and value (-0.70703125000000000000, device=‘cuda:0’, dtype=torch.float16) had turned into nan. This happened when the network was in training mode. Evaluation mode not tested.</p>
<p>When I loaded the same input tensor and performed F.relu on it the output did not generate nan.</p>
<p>More project and system information:<br>
I’m performing object detection from images using torchvisions FasterRCNN. I have my custom dataset. Currently I’m replacing image data with random tensors and keeping the targets from my dataset. The image replace is done to make sure that Nan is not caused by my image data. I’m boubtfull it could be caused by target data. Both image and target data are transformed by torchvision.models.detection.transforms.GeneralizedRCNNTransform.</p>
<p>For FasterRCNN I have my own custom backbone that uses VGG_16bn as part of it. Learning rate does not seem to affect if NaNs appear. I think they appeared even with learning rate 0.0000000005 and they allways appear on relu forward.</p>
<p>Currently I’m using on GPU GradScaler() and autocast() with my model but NaNs appeared before using theese. Anomally detection does not detect anything as the error happens on forward.</p>
<p>Torch version: 1.7.0<br>
Torchvision version:0.8.1<br>
Cuda version: 11.0<br>
cudnn version: 8<br>
OS and setup: I’m using Windows 10 as my main OS. I have downloaded an Ububtu WSL where I’m running an docker container. The container has Ubuntu 18.04 as OS. I’m connecting to the container using localhost and visual studio code on windows 10. I also get the same errors when I run the code on my windows 10.</p>
<p>Code for capturing the nan in nn.relu forward:</p>
<pre><code class="lang-auto">def forward(self, input: Tensor) -&gt; Tensor:
        myinput = input.clone().detach()
        if torch.isnan(input).any():
            exit("Relu input is nan")

        out = F.relu(input, inplace=self.inplace)

        if torch.isnan(out).any():
            torch.save(out, "/workspaces/masters/code/CRAFT-pytorch/debug_relu_cpu_out")
            torch.save(myinput, "/workspaces/masters/code/CRAFT-pytorch/debug_relu_cpu")
            exit("Relu output is nan")

        return out
</code></pre>
<p>The code exits on “Relu output is nan”.</p>
<p>What could cause the Nans on relu forward?</p>

ptrblck: <p>Could you use the latest nightly, remove mixed-precision training (as it seems to also occur using <code>float32</code>), and rerun the script, please?<br>
How long does a usual run take to yield the first NaN output?</p>
One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior

11169: <blockquote>
<pre><code> loss   = F.cross_entropy(preds, y.unsqueeze(0))


 grad   = torch.autograd.grad(eps*loss, z, grad_outputs = None, only_inputs = True, retain_graph = False)[0]
 z_adv  = (z+grad).detach().clone()
 one_hot_y = torch.eye(10)[y].unsqueeze(0).detach().cuda()
 z      = z.clone().detach()
 z_adv  = (z+grad).clone().detach()
 one_hot_y.requires_grad = False
 x_adv  = generator(z_adv, one_hot_y)
 logit_eps = 10e-9 

 with torch.enable_grad():
        z_adv.requires_grad = True
        logits = model(x_adv)
       
        loss  = F.cross_entropy(logits, y.unsqueeze(0))
        grad  = torch.autograd.grad(eps*loss, z_adv, grad_outputs = None, only_inputs = False, retain_graph = False)[0]
</code></pre>
</blockquote>
<p>When I ran above code, I got</p>
<blockquote>
<p>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
</blockquote>
<p>How to solve this problem?</p>
Is there a big problem with pytoch's matrix multiplication?

Qizhou: <p>I compared the pytorch’s matrix multiplication with numpy’s, and discovered that they were very different.<br>
The experimental code is as follow：</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

a = torch.rand([1000,1000],device='cuda')
b = torch.mm(a,a)
print(b)
a = a.cpu().numpy()
c = a.dot(a)
print(c)
print(torch.sum(torch.abs((b.cpu()-c))))
</code></pre>
<p>The output results are as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75cc44df2cab62cbcab3006df7fc2f596161b9f3.png" alt="W~$3S90VA" data-base62-sha1="gO5vwn88lw4PHeVao65pGoQVYiL" width="608" height="298"><br>
We can see that the absolute error of the result of the multiplication of two matrices reaches 3000.</p>

Usama_Hasan: <p>Hy there, I guess it’s about precision, the error is due to float 32 which causes this round off error.</p>
<pre><code class="lang-auto">import torch
a = torch.rand([1000,1000],device='cuda',dtype=torch.float64)
torch.set_printoptions(7)
b = torch.mm(a,a)
print(b)
a = a.cpu().numpy()
c = a.dot(a)
print(c)
print(torch.sum(torch.abs((b.cpu()-c))))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/0/805c4ccb9f2065d440499a0e2758d7ce32baa177.png" alt="Screenshot from 2020-12-10 09-42-26" data-base62-sha1="ijwROEJQKhwfDgqfRRto0MdbS0D" width="671" height="489"></p>

Qizhou: <p>Thank you! I guess numpy might first transform the inputs to float64 before matrix multiplication.</p>
First singular vectors and singular value?

GeoffNN: <p>Is there any function in pytorch to compute the top_k singular values / vectors of a matrix ?<br>
For now I’m using the <code>torch.svd</code> function, but it’s total overkill in some situations where I only need the first singular value/vectors.</p>
<p>Note that this is to implement an Optimizer, therefore I don’t need gradients.</p>
torch.ifft(torch.fft(X)) is not X

superunification: <p>I run this:</p>
<pre><code class="lang-auto">t = torch.Tensor([-1, 1, -1, 1, -1, 1, -1, 1])

torch.irfft(torch.rfft(t, signal_dim=1), signal_ndim=1)</code></pre>
<p>This produces<br>
<code>tensor([-1.7778,  1.6706, -1.3619,  0.8889, -0.3087, -0.3087,  0.8889, -1.3619,          1.6706])</code></p>
<p>Which isn’t even a linearly dependent scaling of the input.<br>
The intermediate result from rfft seems right:</p>
<p><code>tensor([[ 0.,  0.],
        [ 0., -0.],
        [ 0.,  0.],
        [ 0.,  0.],
        [-8.,  0.]])</code></p>
<p>What am I doing wrong?</p>

superunification: <p>Ah, I was using a deprecated function. <code>torch.fft.*</code> works appropriately.</p>
Why does torch.fft.irfftn give radically different results than np.fft.irfftn?

ProGamerGov: <pre><code class="lang-auto">import torch
import torch.fft
x_start = torch.randn(1,3,224,224, 2)
x_pt = torch.view_as_complex(x_start)
x_pt = torch.fft.irfftn(x_pt, s=(224,224))

import numpy as np
x_np = x_start.numpy()
x_np = np.fft.irfftn(x_np, s=(224,224)).real

print(torch.as_tensor(x_np).float().sum(), x_pt.sum())
</code></pre>
<p>The above code shows what I mean. From my understanding both the NumPy and PyTorch code do the exact same thing, but give different results.</p>

ProGamerGov: <p>Figured it out!</p>
<p>This line causes the PyTorch version to be different:</p>
<pre><code class="lang-auto">x_pt = torch.view_as_complex(x_start)
</code></pre>
Attention without RNN

ygl: <p>I am doing a multi-modal classification task and now have 2 vectors, one of them is from texts and the other is from images.  I want to use attention for the final output. But when I look it up, looks like all of the examples use rnn/lstm/gru to do it. But here I do not think I can use it since the 2 vectors are not related. What should I do? Thanks</p>
How to Change NCCL in PyTorch

lm_modeler: <p>I ran into an issue when using 16 K80 GPUs today. It turns out that the default behavior (peer-to-peer) in PyTorch will not support more than 8 devices, probably for very good reasons that I don’t understand. A proposed solution is to set NCCL_P2P_LEVEL=1 for the environment, but I’m not sure how to actually do that because I have never had to fiddle with NVIDA environment. Is there a PyTorch command that will let me set NCCL_P2P_LEVEL? Where do I change this? I’m building models in a Jupyter Notebook on a Windows machine.</p>

ptrblck: <p>You could set the env variable directly via <code>NCCL_P2P_LEVEL=1 python script.py args</code>.<br>
However, NCCL shouldn’t be supported on Windows, so you might need to use another backend.</p>

lm_modeler: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! Is that the only way of training a model on more than 8 GPUs with PyTorch? Note that I have the model layers distributed across devices in a model parallel manner.</p>

ptrblck: <p>It might depend on your system and you can check the GPU connections via <code>nvidia-smi topo -m</code>.<br>
I’m able to train a ResNet on 16GPUs without any <code>NCCL</code> env variables.</p>
Specifying sample ratio for classes in mini-batches with Dataloader

baacus: <p>Hi there,</p>
<p>I have a very imbalanced dataset that contains 10k samples for the minority class and 1 million samples for the majority class (binary classification). What I want to do is dividing all minority samples into mini batches for one epoch equally without over-sampling them (I have already obtained 10k with oversampling).</p>
<p>I tried DataLoader with WeightedRandomSampler but it creates mini batches with equal number of samples for each class (e.g. 500 minority, 500 majority samples for batch size as 1000). But I do not want to oversample minority samples, I want to create batches like 100 minority samples and 900 majority samples without oversampling.</p>
<p>Any help would be appreciated.</p>
<p>Thanks!</p>

baacus: <p>As nobody replied my question, I wrote my sampler class from scratch. I hope it would help anyone who needs.</p>
<pre><code class="lang-auto">class ProportionalTwoClassesBatchSampler:
  """
  dataset: DataSet class that returns torch tensors
  batch_size: Size of mini-batches
  minority_size_in_batch: Number of minority class samples in each mini-batch
  majority_priority: If it is True, iterations will include all majority
  samples in the data. Otherwise, it will be completed after all minority samples are used.
  """
  def __init__(self, dataset, batch_size, minority_size_in_batch, majority_priority=True):
    self.labels = list(set(dataset.y.numpy()))
    self.minority_size = minority_size_in_batch
    self.batch = batch_size
    self.data = dataset
    self.priority = majority_priority
  def __iter__(self):
    if self.minority_size &gt; self.batch:
      raise ValueError("Number of minority samples in a batch must be lower than batch size!")
    y_indices = [np.where(self.data.y.numpy() == label)[0] for label in self.labels]
    self.indices = sorted(y_indices, key=lambda x: x.shape)
    minority_copy = self.indices[0].copy()
    while len(self.indices[1]) &gt; self.batch - self.minority_size:
      if len(self.indices[0]) &lt; self.minority_size:
        if self.priority:
          # reloading minority samples
          self.indices[0] = minority_copy.copy()
        else:
          break
      minority = np.random.choice(self.indices[0], 
                                  size = self.minority_size, 
                                  replace=False)
      majority = np.random.choice(self.indices[1], 
                                  size = (self.batch - self.minority_size), 
                                  replace=False)
      batch_inds = np.concatenate((minority, majority), axis=0)
      batch_inds = np.random.permutation(batch_inds)

      self.indices[0] = np.setdiff1d(self.indices[0], minority)
      self.indices[1] = np.setdiff1d(self.indices[1], majority)

      yield self.data.x[batch_inds], self.data.y[batch_inds]

data_tr = Data(tr_in, tr_out)
sampler =  ProportionalTwoClassesBatchSampler(data_tr, 512, 50)
ind = 0
# for x,y in sampler:
#   #dfy = pd.DataFrame(data=y.numpy())
#   #print(dfy.value_counts())
#   ind += 1

# print(ind)
</code></pre>
The right Pytorch for GeForce 3090 & CUDA 11.1

tyrgann: <p>Hello,</p>
<p>I’m having troubles installing PyTorch for the workstation we got. I understand that the full power of the GeForce 3090 needs CUDA 11.1 to unveil; are there nightly version I can get with pip or should I try to build from sources, and if yes, from which branch ?</p>
<p>Thanks.</p>

RaLo4: <p>This might help:</p>
<ul>
<li>
<p>Information on <a href="https://discuss.pytorch.org/t/pytorch-with-cuda-11-compatibility/89254/2">building from source</a> with CUDA11</p>
</li>
<li>
<p>Information on <a href="https://github.com/pytorch/pytorch/issues/45028#issuecomment-695250988" rel="noopener nofollow ugc">installing nightly version</a> with CUDA11</p>
</li>
</ul>

mikey_t: <p>I faced the same problems with the new 3090 card and cuda 11.1 trying to install pytorch. I followed the directions <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">here</a> to compile pytorch from source. I was able to do this within a conda environment also.</p>
Which layers support dynamic quantization?

ayush-1506: <p>I’m trying to quantize my model using <code>torch.quantization.quantize_dynamic(model, {nn.ConvTranspose1d}, dtype=torch.qint8)</code>, but my model size doesn’t decrease and neither does computation time.<br>
However if I add <code>nn.Linear</code> to the set of layers to quantize, it seems to have an effect. Naturally this means that ConnTranspose1d doesn’t support dynamic quantization.</p>
<p>My question is very simple: Which layers support dynamic quantization at the moment? Is a list present somewhere? I couldn’t find anything in the docs.</p>
<p>I’m running the model on i7 processor (Macbook pro), pytorch version 1.7.0 installed using pip<br>
Thanks</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/ayush-1506">@ayush-1506</a>, the list of supported dynamic quantization layers is here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/quantization_mappings.py#L76" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/quantization/quantization_mappings.py#L76</a></p>

ayush-1506: <p>Thanks for sharing this, <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> ! Are there plans for supporting other layers ?</p>

Vasiliy_Kuznetsov: <p>I’m not aware of plans to add more layers to dynamic quantization specifically.  What would be the use case?</p>

ayush-1506: <p>The currently supported layers looks like a very small subset of all possible out-of-the-box layers. Suppose I’m interested in quantizing a Convnet (Conv2d, Conv1d) layers. This won’t work at the moment, right?</p>

Vasiliy_Kuznetsov: <p><a class="mention" href="/u/ayush-1506">@ayush-1506</a>, you can check out static quantization or QAT which support convolutions.</p>

ayush-1506: <p>Thanks, will check it out. I’ve statically-quantized a model and stored the state dict locally, but loading it back gives me this error :<br>
<code>RuntimeError: Could not run 'quantized::conv_transpose1d_prepack' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::conv_transpose1d_prepack' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</code></p>
<p>Any idea what this is about ? Thanks again</p>

Vasiliy_Kuznetsov: <p>this error message means that you are trying to give a fp32 tensor to a quantized layer.  The way to fix it is to use <code>QuantStub</code> and <code>DeQuantStub</code> to control the conversions, feel free to check out the examples on <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a> .</p>
Normalized cut loss function, update network weights

kristianeschenburg: <p>Hi</p>
<p>I’m interested in implementing a normalized cut criteria as a loss function.  I want this normalized cut loss to be incorporated into the back-propagation.  For example, given the logits output of a network, pushed through a F.softmax(logits) layer, I want to be able to apply the normalized cut criteria to the predicted labels of the softmax output.  I’m using Deep Graph Library with the Pytorch backend.</p>
<p>See below for my implementation of the NCuts loss.  I want to compute the gradient of the loss with respect to the network weights and parameters. <a href="https://discuss.pytorch.org/t/softmax-to-one-hot/37302">Other forums</a> mentioned that Pytorch does not explicitly compute the one-hot encoding matrix for losses, but that the <code>scatter_</code> method might be of help?  I’m not sure how to proceed.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

def normalized_cut(graph, logits):
        
    """
    Parameters:
    - - - - - - - - -
    graph: DGL graph
        graph structure of data
    logits: torch tensor, float
        output of network

    Returns:
    - - - - 
    loss: torch float tensor
        ncut loss value 
    """

    A = graph.adjacency_matrix()
    d = torch.sparse.sum(A, dim=0)

    # compute maximum-probability class for each node
    max_idx = torch.argmax(F.softmax(logits), 1, keepdim=True)

    # initialize one-hot encoding matrix
    one_hot = torch.FloatTensor(logits.shape)
    one_hot.zero_()
    one_hot.scatter_(0, max_idx, 1)

    assoc = torch.matmul(one_hot.t(), torch.matmul(A, one_hot))
    degree = torch.matmul(d, one_hot)

    loss = torch.nansum(assoc.diag() / degree)

    return -loss
</code></pre>
<p>Any help is appreciated.  Thanks.</p>
<p>*** EDIT ***</p>
<p>One work-around would be to relax the one-hot discrete constraint as noted in this <a href="https://www1.icsi.berkeley.edu/~stellayu/publication/doc/2003kwayICCV.pdf" rel="noopener nofollow ugc">multi-class spectral clustering paper</a> – that is, rather than working with the hot encoded matrix, we work with the softmax probabilities, such that we can retain the previous gradient computation steps?</p>
Where are node values stored?

vkubicki: <p>If I create a network (for example using torch.nn.Sequential), then perform a forward pass, where are the values at the nodes stored ? They are needed for the backward operation to retropropagate the gradient of the error. Are they stored somewhere in the network during the forward pass, or does the backward pass first compute a forward pass to populate those node values in some temporary buffer ?</p>
<p>I am using Pytorch for reinforcement learning, hence I need to go back and forth through various states and I must be sure what state the network is in at any time.</p>
<p>Thank you,</p>
<p>Vincent</p>

albanD: <p>Hi,</p>
<p>These values are not stored at the <code>torch.nn</code> level. They are handled by the autograd directly that stores them along with all the informations needed to perform the backward pass.</p>

vkubicki: <p>Thank you for your reply Alban. On this simple example, can you tell me if I get it right ?</p>
<pre><code class="lang-auto">import torch

net = torch.nn.Sequential(
    torch.nn.Linear(1, 1, bias=False),
    torch.nn.Linear(1, 1, bias=False)
)

x = torch.tensor([3.])
y = torch.square(net(x))
z = 2 * y
</code></pre>
<p>Here, a single computational graph is computed “somewhere” by the autograd package. It contains <code>Function</code> objects. One of these function object can be accessed through <code>y.grad_fn</code>, and another can be accessed through <code>z.grad_fn</code>. The computational graph contains the intermediary node values obtained through the forward pass. When <code>z.backward()</code> is called, the computational graph is used to fill the <code>grad</code> attributes of the parameters in <code>net</code>. Intermediary values are then freed, and a second call to <code>z.backward()</code> will result in an error. Actually, even a call to <code>y.backward()</code> will result in an error, proving that the same computational graph is shared among those two values.</p>
<p>However, the graph is not completely destroyed after the call to backward, as <code>z.grad_fn</code> still exists and points to the <code>MulBackward0</code> object. If, however, <code>y</code> and <code>z</code> both go out of scope later, then the graph will be completely freed.</p>

albanD: <p>Hi,</p>
<p>You are correct indeed. The objects in the graph are now called <code>Node</code> though and not <code>Function</code>.</p>
<p>The reason why not all the graph is destroyed is that the graph itself is very small and on CPU memory so it should never be an issue. What gets freed is only the buffers which are the actual Tensors that use a lot of memory.</p>
Is it mandatory to flatten the embeddings?

leoentersthevoid: <p>I’m trying to replicate the results of a paper.<br>
They were using TF and they have mentioned that it is crucial to flatten the embeddings.</p>
<p>Now, given this simple implementation, does it make sense to flatten anything? If so, why?</p>
<pre><code class="lang-auto">class GMF(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.user_embedding = nn.Embedding(config['user_pool'], config['latent_dim'])
        self.item_embedding = nn.Embedding(config['item_pool'], config['latent_dim'])
        self.out = nn.Linear(config['latent_dim'], 1)

    def forward(self, users, items):
        users, items = self.user_embedding(users), self.item_embedding(items)
        haddamard = mul(users, items)
        return self.out(haddamard)
</code></pre>
Implementation of AdamW

eunseokyang: <p>I have a question about implementation of <code>torch.optim.adamw</code>.<br>
I think the weight decay should be applied at the end of the algorithm according to the <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">paper</a>, however, it applied weight decay first and execute the rest in code.</p>
<p>Can I know if it is correct?</p>
Initializing policy - reinforcment learning

youdar: <p>I am trying to build a model for a policy and have a general question regarding the initial values of the policy.<br>
It looks to me that at from the first forward step the softmax will not return equal probability.<br>
Questions:</p>
<ol>
<li>how can I force the policy to be equal probability for all the actions at the first step?</li>
<li>what would be a good method to increase exploration when training the policy?</li>
<li>what are some of the ways one can apply constraints on the actions? for example if a specific state makes one of the actions impossible, beside setting it in the rewards in a way of a penalty (restraining the actions)</li>
</ol>
<p>example policy:</p>
<pre><code class="lang-auto">class Policy(nn.Module):
    def __init__(self, state_size, action_size, fc1_size=64):
        super().__init__()
        self.fc1 = nn.Linear(state_size, fc1_size)
        self.fc2 = nn.Linear(fc1_size, action_size)

    def forward(self, state):
        x = F.relu(self.fc1(state))
        x = self.fc2(x)
        x = F.softmax(x, dim=1)
        return x

    def act(self, state):
        state = state.unsqueeze(0)
        probs = self.forward(state).cpu()
        p = Categorical(probs)
        action = p.sample()
        log_prob = p.log_prob(action)
        return action.item(), log_prob
</code></pre>

AdilZouitine: <p>Good question, I am also trying to do the same thing but I have not been able to do it.</p>
Thank you for making PyTorch awesome 💗

AladdinPerzon: <p>I feel incredibly grateful for all the great work you guys do with PyTorch and just want to express some of it. For all of you that are so helpful in the forum, you’re fucking awesome and thank you so much! <img src="https://discuss.pytorch.org/images/emoji/apple/raised_hands.png?v=9" title=":raised_hands:" class="emoji" alt=":raised_hands:"></p>

ptrblck: <p>Thank you for being part of this community and also a big thanks to all the awesome contributors!<br>
I don’t even want to start by naming these users as there are so many. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

fepegar: <p><a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a> we need more people like you.</p>