akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Future of References to Module Parameters

username: ntenenz
cooked: <p>In the following code snippet, the two assertions hold:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

conv = nn.Conv2d(8, 8, 3)
d = {k: v for k, v in conv.named_parameters()}

# torch.__future__.set_overwrite_module_params_on_conversion(True)
assert all(d[k] is v for k, v in conv.named_parameters())
conv.to('cuda:0')
assert all(d[k] is v for k, v in conv.named_parameters())
</code></pre>
<p>However, if the commented line is uncommented, they fail. Such a change would obviously have a significant impact on the future of lots of PyTorch code, e.g., any code that <a href="https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165">moves a model after creating an optimizer</a>. In general, storing references to parameters is often more efficient than recursively traversing a module, which generates one such use case.</p>
<pre><code class="lang-auto">import copy
import torch.nn as nn

conv = nn.Conv2d(8, 8, 3)
seq = nn.Sequential(*[copy.deepcopy(conv) for _ in range(5)])
for _ in range(5):
    seq = nn.Sequential(*[copy.deepcopy(seq) for _ in range(5)])

d = {k: v for k, v in seq.named_parameters()}

%timeit list(d.items())
%timeit list(seq.named_parameters())

%timeit list(d.values())
%timeit list(seq.parameters())
</code></pre>
<p>The internet doesn’t seem to offer much on the future of when this change will be rolled out despite the fact that it has been merged in master <a href="https://github.com/pytorch/pytorch/pull/21613" rel="nofollow noopener">for quite some time</a>. Can the PyTorch team provide any insights into the rollout plan?</p>

username: tom
cooked: <p>To the best of my knowledge, the advice has always been to move your model first as the fact that the parameters don’t change in this has been more an implementation detail than part of the design.</p>
<p>It would seem that in most use cases, the traversing of the modules upon setup is rather small compared to running or training the models.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ntenenz
cooked: <p>My apologies, I wasn’t entirely clear regarding my use case. I would like to be able to stably store references to parameters rather than recursively iterate over the entire set of parameters. This will be done frequently, as opposed to during the initialization of training.</p>
Multi GPU with Custom Backward and Attributes

username: maralm
cooked: <p>Hi,</p>
<p>I wrote a custom backward function for my model. I want to use the DataParallel package. However I have a problem. If I use</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model, device_ids=[0,1])
</code></pre>
<p>I get the following error:</p>
<p>“‘DataParallel’ object has no attribute ‘backward’”</p>
<p>I know this can be solved by using model.module.backward, but then it will only use one gpu. Is there a way to use the torch.nn.DataParallel with custom backward and attributes?</p>

username: ptrblck
cooked: <p>Would it be possible to return the outputs in your <code>forward</code> method and calculate the loss on the default device?<br>
This would be the vanilla use case, while it seems you’ve implemented <code>backward</code> as a class function?</p>

username: maralm
cooked: <p>Thanks for the reply. No the backward is not a separate class. It is a function inside the model class. Here is how I define it:</p>
<pre><code class="lang-auto">Class myModel():
    def __init__(self, config):
         ....
    def forward(...):
        ....
    def backward(...):
        ....
</code></pre>
<p>And I call it this way:</p>
<pre><code class="lang-auto">outputs = model(....)
loss = outputs[0]  
if args.n_gpu &gt; 1:
    loss = loss.mean()
model = model.backward(...)
</code></pre>
<p>but nn.DataParallel is not recognizing the backward and some other attributes without using module.</p>

username: ptrblck
cooked: <p>Thanks for the information.</p>
<p>What’s the design decision to put the <code>backward</code> call inside your model?<br>
Are you using some internal parameters?<br>
If so, how are these parameters updated/used inside the model?</p>

username: maralm
cooked: <p>I needed to access the activations and activation gradients in backward. I collect activations in forward pass and access to them in backward. I use autograd backward function to calculate each layer’s backward and make the changes that I want in the process.</p>

username: maralm
cooked: <p>I tried the distributed data parallel instead of data parallel and it is working.</p>
Dot product of multiple matrices

username: abhishek_karanath
cooked: <p>I have a tensor of shape 4x64x300  and i need to find the dot product of all of the four 64x300 matrices to get a 64x300 matrix. What is the most efficient way to do this?</p>

username: albanD
cooked: <p>You can use <code>torch.prod(inp, dim=0)</code> (doc <a href="https://pytorch.org/docs/stable/torch.html#torch.prod">here</a>). It is like sum but does a product instead of addition.</p>
Calling a function in 'with torch.no_grad()' block

username: TinfoilHat0
cooked: <p>Hi,</p>
<p>Suppose I’ve a code block wrapped with torch no grad and I call a function that does inference in it, i.e,</p>
<pre><code class="lang-auto">
with torch.no_grad():
    my_infer(my_model, my_data)
</code></pre>
<p>should I use with</p>
<blockquote>
<p>torch.no_grad()</p>
</blockquote>
<p>at the beginning of my_infer() too or is the outer no_grad  sufficient?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It is a context manager. So it will enable the no_grad mode when you enter it and disable it when you leave the indentation.<br>
So no need to do it again inside the function, it will still be activated.</p>
Happy to contribute my intuitive visual guide to how convolutions and transposed convolutions work

username: makeyourownneuralnet
cooked: <p>Unlike fully connected linear layers, convolutions layers need a bit of work to calculate the size of the data as it passes through them.</p>
<p>There aren’t many easy to understand guides on how convolution works. Many report that transposed convolution is particularly difficult to understand.</p>
<p>As part of my upcoming book on GANs with PyTorch I include an appendix with worked examples of convolutions and transposed convolutions, which I’ve also published a version of online for free access.</p>
<p><a href="https://makeyourownneuralnetwork.blogspot.com/2020/02/calculating-output-size-of-convolutions.html" rel="noopener nofollow ugc">https://makeyourownneuralnetwork.blogspot.com/2020/02/calculating-output-size-of-convolutions.html</a></p>
<p>I’d be happy if you thought these should be included in the Pytorch tutorials, or linked to from there.</p>
<p>Also happy to receive feedback on improving them.</p>
<p>The main point of the guide is to develop an intuitive understanding, avoiding too much mathematical jargon.</p>
<p>An example of the friendly style of diagrams used …</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/140b3dedaf189cebe8d93be4896cdffe26473c26.png" data-download-href="https://discuss.pytorch.org/uploads/default/140b3dedaf189cebe8d93be4896cdffe26473c26" title="appendix_C_eg_7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/140b3dedaf189cebe8d93be4896cdffe26473c26_2_690x275.png" alt="appendix_C_eg_7" data-base62-sha1="2RjDngJCUEsQ2q3yelIC2ryjidg" width="690" height="275" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/140b3dedaf189cebe8d93be4896cdffe26473c26_2_690x275.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/140b3dedaf189cebe8d93be4896cdffe26473c26_2_1035x412.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/140b3dedaf189cebe8d93be4896cdffe26473c26_2_1380x550.png 2x" data-dominant-color="F6F5F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">appendix_C_eg_7</span><span class="informations">2950×1179 285 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Specify retain_graph=True when calling backward the first time

username: KAISER1997
cooked: <p>Hi all, My code doesn’t work unless I specify retain_graph=True eventhough I am calling backward once every iteration. Thanks in advance</p>
<pre><code class="lang-auto">    depth_net_encoder.eval()
    depth_net_decoder.eval()
    noise=Variable(torch.tensor(img), requires_grad=True)
    while True  :
        noise=Variable(torch.tensor(noise.data), requires_grad=True)
        dummy=torch.zeros(tgt_img_var.shape).to(device)   
        true_mask=torch.zeros(tgt_img_var.shape).to(device)        
        for u in range(dummy.shape[0]):
            ex1=ex2=0 
            dummy[u,:,ry_list[u]-sz2+ex1:ry_list[u]+sz2+ex1, rx_list[u]-sz2+ex2:rx_list[u]+sz2+ex2]=F.sigmoid(noise)
            true_mask[u,:,ry_list[u]-sz2+ex1:ry_list[u]+sz2+ex1, rx_list[u]-sz2+ex2:rx_list[u]+sz2+ex2]=torch.tensor(1).to(device)
        adv_tgt_img_var = torch.mul((1-true_mask), tgt_img_var) + torch.mul(true_mask, dummy)
    enc_var=depth_net_encoder(adv_tgt_img_var)
        depth_var = depth_net_decoder(enc_var)   
        loss_data=0.0
        for i in range(1):
            o_g_x,o_g_y=torch.abs(X_grad(target_depth[("disp", i)],device)),torch.abs(Y_grad(target_depth[("disp", i)],device))
            o_g=o_g_x+o_g_y
            g_x,g_y=torch.abs(X_grad(depth_var[("disp", i)],device)),torch.abs(Y_grad(depth_var[("disp", i)],device))
            g=g_x+g_y
            if i==0:
                p_O,p_f=o_g,g
      
            gradient=torch.log(1+g_x*o_g_x).sum()+torch.log(1+g_y*o_g_y).sum()
            loss_data=gradient

        loss=loss_data*100#+loss_reg
        loss.backward()
        noise_grad = noise.grad.clone()
        noiset=noise-0.001*noise_grad
        noise=torch.tensor(noiset.cpu().detach().numpy())
        loss_scalar = loss.item()
        print(loss_scalar)```</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>What is the error you’re seeing?</p>
<p>Also you should not use <code>.data</code> and <code>Variable</code>s anymore.<br>
You can replace <code>noise=Variable(torch.tensor(img), requires_grad=True)</code> by <code>noise=torch.tensor(img, requires_grad=True)</code> for example.</p>

username: KAISER1997
cooked: <p>My issue is eventhough I am calling backward once every iteration,it asks me to set retain_graph=True . What am I doing wrong?</p>

username: albanD
cooked: <p>It is usually that the graph is reused.<br>
So either that you do some differentiable computation outside the loop, or that you re-use results from one iteration to the next.</p>
<p>I would usually try to get a repro as small as possible. That helps identify the potential faulty Tensors as mentioned above.<br>
If that doesn’t help, you can use something like <a href="https://github.com/szagoruyko/pytorchviz/">torchviz</a> to print the graph and see where the one from the second iteration is attached to the one in the first.</p>
How to find which module that a variable belongs to?

username: StoneOcean
cooked: <p>Hello, I am trying to find some modules that have certain variables contained inside. I now can find these variables using <code>model.named_parameters()</code>.<br>
For instance, I may get a parameter and its name <code>module.layer1.0.conv1.weight</code>, how can I find the corresponding <code>conv1</code> module it belongs to, in order to access other parameters inside this module? Is there any functions that I can use? Thanks in advance!</p>
<p>(I know that I can look for the module that is a instance of  <code>nn.Conv</code>  using <code>model.children()</code> or <code>model.module()</code>, but problem is that it may contain other stuff like <code>nn.Sequential</code> and other bigger blocks that contains <code>conv</code>, making the code very complicated)</p>

username: albanD
cooked: <p>I don’t think there is anything.<br>
But you can simply access the attributes by doing <code>name.split()[:-1]</code> to get all the names. And then <code>mod = getattr(mod, mod_name)</code>. for each entry in the list.</p>
Pytorch tensor.to(device) too slow?

username: josipovic
cooked: <p>I’m having an issue of slow .to(device) transfer of a single batch. If I understood correctly, dataloader should be sampled from in the main training loop and only then (when the whole batch is gathered) should be transferred to gpu with .to(device) method of the batch tensor?</p>
<p>My batch size is 32 samples x 64 features x 1000 length x 4 bytes (float32) / (1024*1024) = ~7MB and I’m using 1080Ti graphics card. 7MB tensor.to(device) should be pretty much instantaneous? Or am I missing something in my calculation?</p>
<p>bottleneck profiler data is available here <a href="https://pastebin.com/dvQe6Y2Q" rel="nofollow noopener">https://pastebin.com/dvQe6Y2Q</a> (copy raw data to gedit/notepad for easy view)</p>
<p>here is the code that was profiled:</p>
<pre><code class="lang-auto"> it = iter(dataloader)
    for i in range(5):
        time_s = time.time()
        sample = next(it)
        optimizer.zero_grad()
        data = sample[0].to(device)
        iz = model(data)
        ctc_loss = nn.CTCLoss(zero_infinity=True)
        loss = ctc_loss(iz.transpose(0, 1).transpose(0, 2), sample[1], sample[2], sample[3])
        loss.backward()
        optimizer.step()
        time_e = time.time()
        print("Time ", time_e - time_s)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The cuda api is asynchronous for most operations. So a profiler that does not do proper synchronization won’t give you accurate informations.<br>
In particular, a non-async copy to or from the GPU will force synchronization and so wait for all outstanding tasks.<br>
So this is expected. You can try to add a <code>torch.cuda.synchronize()</code> just before this line and all the time will be spent in that function instead of the copy.</p>

username: josipovic
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Thanks for response !  could you please elaborate more? I’ve seen elsewhere that pytorch just asynchronously (queues?) adds operations to gpu. If that’s the case what does torch.cuda.synchronize() do exactly (and how do I speed it up with async copy)? When I measure time with time.time() and with torch.cuda.synchronize() before t1 and t2, it says its very fast.</p>
<p>EDIT: Async copy should make it faster? I have tried pin_memory=True on the dataloader and .to(non_blocking=True), if async copy is the way, is that all ?</p>

username: albanD
cooked: <p>The cuda api is asynchronous. so yes it queues tasks for the gpu to execute. <code>torch.cuda.synchronize()</code> simply waits for all outstanding tasks to be finished before returning.</p>
<p>Async copy will be async. It depends on your workload/code whether or not it’s going to be faster. But it won’t make a huge difference for sure.</p>
Optimization Problem Implementation

username: fojjta
cooked: <p>Hi,</p>
<p>I am trying to implement a method from a paper, which requires to solve eqation <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f82d93458bf10cc583e0266e5ecf23790ceb4c8.gif" alt="CodeCogsEqn (4)" data-base62-sha1="2ddvdKTgeaida7knJyoM6WLPkN2" width="577" height="40">  . The p_\theta is a CNN, which has a sigmoid function at the end, \alpha, \beta, … are known values, \theta are parameters of the CNN, A_i and B_i are images for i-th sample. It is suggested to use weighted BCE Loss to solve this, but I am struggling to work out how to do this in PyTorch. Two of my ideas are (iterate over samples and):</p>
<ol>
<li>Use negative of the <a href="https://gifyu.com/image/78Pf" rel="noopener nofollow ugc">inner (max and sum) function</a> as a (custom) loss to backpropagate</li>
<li>Use \alpha as a weight for BCELoss function, where the target tensor is one, \beta as a weight for BCELoss, where the target tensor is zero etc.</li>
</ol>
<p>But… both of these methods should lead to the same solution, is this not a true? Is this solution correct, or am I missing something?</p>
<p>Thanks for your responses</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>would definitely work. Note that you want to keep the sum over all the samples right?</li>
<li>I am not sure you can set the weights this way with the BCELoss we have. But the formula above is a bit unclear on what are different samples in a batch and what is not. But the function we have might enforce that <code>beta = 1 - alpha</code> and <code>delta = 1-gamma</code>.</li>
</ol>
Fisher Matrix Calculation: RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior

username: Sharut_Gupta
cooked: <p>I am trying to update Fisher matrix using this code:</p>
<pre><code>def _update_fisher_params(self, current_ds, batch_size, num_batch):
	log_liklihoods = []
	i=1
	for inputs, target in tqdm(current_ds):
		inputs=inputs.cuda()
		target= target.cuda()
		if i &gt;= num_batch:
			break

		output = F.log_softmax(self.model(inputs), dim=1).detach().requires_grad_(True)
		i+=1
		log_liklihoods.append(output[:, target])
	
	log_likelihood = torch.cat(log_liklihoods).mean()
	grad_log_liklihood = autograd.grad(log_likelihood, self.model.parameters(), create_graph=True, retain_graph =True)
</code></pre>
<p>I have tried create_graph=True and retain_graph=True also. But still, it gives the error: RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
<p>Also, if I remove detach() from my code, it says Cuda out of memory at the line where I append my tensors to the list(log_liklihoods).<br>
Can someone please tell me where am I wrong? I am unable to understand anymore</p>

username: ptrblck
cooked: <p>If you are calling <code>detach()</code> on the output of <code>F.log_softmax</code>, the computation graph will be cut at this place, so that Autograd won’t be able to calculate the gradients for the former part of the graph.<br>
You might need to reduce <code>num_batch</code>, if you are running out of memory.</p>

username: Sharut_Gupta
cooked: <p>Is there any other way apart from detach() where I don’t run out of memory?<br>
Also, my batch size os 32, this loop barely runs for 5 batches and says CUDA out of memory if I don’t use detach. I want to calculate the fisher matrix with more number of samples from Dataset A, how can I still run this?</p>
<p>Or is there any efficient of calculating fisher?</p>
Pytorch tensorboard trying to use CUDA 9.0 instead of 10.1

username: Aviv_Shamsian
cooked: <p>I’m using pytorch 1.4 on Ubuntu 18.04LTS and python 3.6.8. When trying to invoke tensorboard from the terminal I always get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/bin/tensorboard", line 7, in &lt;module&gt;
    from tensorboard.main import run_main
  File "/usr/lib/python2.7/site-packages/tensorboard/main.py", line 39, in &lt;module&gt;
    from tensorboard import default
  File "/usr/lib/python2.7/site-packages/tensorboard/default.py", line 34, in &lt;module&gt;
    import tensorflow as tf
  File "/usr/lib/python2.7/site-packages/tensorflow/__init__.py", line 22, in &lt;module&gt;
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/usr/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in &lt;module&gt;
    from tensorflow.python import pywrap_tensorflow
  File "/usr/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in &lt;module&gt;
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in &lt;module&gt;
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in &lt;module&gt;
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
</code></pre>
<p>It looks like it expects to CUDA 9.0 version instead of 10.X how can this be solved?<br>
I have CUDA 10.2 installed.<br>
BTW the log dir is created and it’s not empty</p>

username: ptrblck
cooked: <p>It looks like Tensorflow is called internally and tried to open libcublas.<br>
Do you have a TF installation on your system or how did you install tensorboard?<br>
If you are using the one shipped in PyTorch, tensorflow shouldn’t be called.</p>

username: Aviv_Shamsian
cooked: <p>Problem solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I have downgraded tensorboard and installed it using conda.<br>
Thank you Patrick</p>
How to load image dataset into pytorch

username: lavender99
cooked: <p>i have a dataset ( image and mask)  i want to do preprocessing then unet but i cannot find a way to load them on pytorch most of the way online are very specific for datasets like MNSIT</p>

username: ptrblck
cooked: <p>You could write a custom <code>Dataset</code> to load the images and their corresponding masks.<br>
Have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" rel="nofollow noopener">Data loading tutorial</a> for a basic approach.<br>
<a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7?u=ptrblck">Here</a> is a dummy implementation using the functional API of <code>torchvision</code> to get identical transformations on the data and target images.</p>

username: lavender99
cooked: <p>thank you for replying , the problem with this tutorial is the data is already downloaded and available on torchvision , my data is just on my pc … maybe i am missing something here !</p>

username: ptrblck
cooked: <p>The second link shows a simple implementation of a custom <code>Dataset</code>, where your data is stored locally.<br>
You would just have to pass the paths for the images and masks.</p>

username: Scarlet
cooked: <p>Hi, I have a similar problem where I want my dataset to be containing both input and output as images. I basically want to train a pytorch model (grayscale images to colour images translation) using fastai. I would like to know how to load this format of data.</p>
Using of data.to(device ) in pytorch

username: Sanjayvarma11
cooked: <p>Hello guys. I am very interested in learning PyTorch.so I am trying to understand some basic code in PyTorch.<br>
here are some of the doubts I got while going through the code</p>
<ol>
<li>I found a piece of code like this in training function</li>
</ol>
<pre><code class="lang-auto">for batch_idx, (data, target) in enumerate(pbar):
    # get samples
    data, target = data.to(device), target.to(device)
</code></pre>
<p>Here my doubt is what does "data.to(device), target.to(device ) "do in PyTorch.</p>
<p>2)I also found line of code<br>
optimizer.zero_grad()<br>
what does this piece of code do in PyTorch? One of the comments says that it will make gradients zero.<br>
so what does the advantage if we make gradients to zero.??</p>

username: learner47
cooked: <p><code>data.to(device)</code> moves the data to cpu or GPU based on what <code>device</code> is. This is required for faster computations.</p>
<p>In PyTorch, the gradients are accumulated using <code>loss.backward()</code> and then the gradients are applied using <code>optimizer.step()</code>. The stale gradients from the previous back propagation need to be cleared before running the <code>optimizer.step()</code> again. This is achived through <code>optimzer.zero_grad()</code>.</p>

username: mmisiur
cooked: <p><code>.to(x)</code> as states in <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.to" rel="nofollow noopener">Docs</a> moves the tensor from current device to the device <code>x</code>, i.e. cpu (<code>"cpu"</code>) or gpu (<code>"cuda:0"</code>, <code>"cuda:1"</code> …). It’s useful, because you can specify <code>x</code> in the beginning of code and from there do not care whether it is cpu or gpu, and just move all your tensors, models etc. to it. The alternative is to call <code>.cpu()</code> or <code>cuda()</code>, but it lacks the flexibility of <code>.to()</code>. There is awesome explanation of this in the official PyTorch Docs <a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="nofollow noopener">here</a>.</p>
<p>about <code>.zero_grad()</code> you can find extensive discussion <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903">here</a>.</p>
Loss and accuracy

username: Mohamed_Nabih
cooked: <p>input_size = 13<br>
hidden1_size = 13<br>
hidden2_size = 128<br>
hidden3_size = 64<br>
output_size = 30</p>
<p>class DNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden1_size, hidden2_size, hidden3_size, output_size):<br>
super(DNN, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_size, hidden1_size)<br>
self.relu1 = nn.ReLU()<br>
self.fc2 = nn.Linear(hidden1_size, hidden2_size)<br>
self.relu2 = nn.ReLU()<br>
self.fc3 = nn.Linear(hidden2_size, hidden3_size)<br>
self.relu3 = nn.ReLU()<br>
self.fc4 = nn.Linear(hidden3_size, output_size)<br>
self.relu4 = nn.ReLU()</p>
<pre><code>def forward(self, x):
    out = self.fc1(x)
    out = self.relu1(out)
    out = self.fc2(out)
    out = self.relu2(out)
    out = self.fc3(out)
    out = self.relu3(out)
    out = self.fc4(out)
    out = self.relu4(out)
    return out
</code></pre>
<p>batch_size = 10<br>
n_iterations = 50<br>
no_eps = n_iterations / (13 / batch_size)<br>
no_epochs = int(no_eps)<br>
model = DNN(input_size, hidden1_size, hidden2_size, hidden3_size, output_size)</p>
<p>criterion = nn.CrossEntropyLoss()<br>
learning_rate = 0.01<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>iter = 0<br>
for epoch in range(no_epochs):<br>
for i, (X_train, y_train) in enumerate(train_loader):<br>
y_train = torch.empty(batch_size, dtype=torch.long).random_(output_size)<br>
optimizer.zero_grad()<br>
outputs = model(X_train)<br>
loss = criterion(outputs, y_train)<br>
loss.backward()<br>
optimizer.step()<br>
iter += 1<br>
if iter % 500 == 0:<br>
correct = 0<br>
total = 0<br>
for X_test, y_test in test_loader:<br>
outputs = model(X_test)<br>
pred = outputs.argmax(dim=1, keepdim=True)<br>
total += y_test.size(0)<br>
correct += pred.eq(y_test.view_as(pred)).sum().item()<br>
accuracy = 100 * correct / total<br>
print(‘Iteration: {}. Loss: {}. Accuracy: {}’.format(iter, loss.item(), accuracy))</p>
<hr>
<p>The output start with Iteration 500 not 1 why !!!<br>
Iteration: 500. Loss: 3.403779983520508. Accuracy: 0.2761323337093925<br>
Iteration: 1000. Loss: 3.4060192108154297. Accuracy: 0.3276070990724763<br>
Iteration: 1500. Loss: 3.416713237762451. Accuracy: 0.4173101012337052<br>
Iteration: 2000. Loss: 3.402294635772705. Accuracy: 0.34867708074959347<br>
Iteration: 2500. Loss: 3.3952858448028564. Accuracy: 0.2755100135754692<br>
Iteration: 3000. Loss: 3.4023067951202393. Accuracy: 0.3158719194042085</p>

username: ptrblck
cooked: <p>You are increasing <code>iter</code> directly before the if condition to 1, which result in <code>iter % 500 == 0</code> when <code>iter</code> reaches multiples of 500.<br>
If you want to print the very first iteration, increase <code>iter</code> after the condition.</p>
Is there any way to use torchvision.transforms.ToPILImage for float64 images?

username: Juna
cooked: <p>Hello, I’m trying to use torchvision.transforms.ToPILImage so that I can augment my data.<br>
My images are from MRI data so they have not the same range as normal images have and the type of them are float64 and the size of each image is 320x320x3.<br>
It does work with size 320x320 in the same rest of the setting.<br>
I don’t understand, why it doesn’t work with 3D while it works with 2D. why? why?<br>
Anyway, I hope anybody can tell me how to solve this problem.</p>

username: ptrblck
cooked: <p>It seems that FP64 images are not supported by PIL as seen <a href="https://pillow.readthedocs.io/en/5.1.x/handbook/concepts.html#concept-modes">here</a>.<br>
What kind of error message you are getting at the moment?</p>

username: Juna
cooked: <p>Thank you for replying!!</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/media/volume1/juna/.local/lib/python3.5/site-packages/torch/utils/data/_utils/worker.py”, line 99, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/media/volume1/juna/.local/lib/python3.5/site-packages/torch/utils/data/_utils/worker.py”, line 99, in <br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “…/datasets/MSDDataset.py”, line 264, in <strong>getitem</strong><br>
image = Image.fromarray(image)<br>
File “/media/volume1/juna/.local/lib/python3.5/site-packages/PIL/Image.py”, line 2647, in fromarray<br>
raise TypeError(“Cannot handle this data type”)<br>
TypeError: Cannot handle this data type</p>
</blockquote>
<p>Umm well, at the beginning I was trying to use “Image.fromarray(image)” and the error above is the error I got.<br>
So I tried to use torchvision.transforms.ToPILImage instead but people on the internet said it also doesn’t work for float images.<br>
Are there any other python packages I can use for float images?<br>
Otherwise, I’m considering making custom tranform classes.</p>

username: ptrblck
cooked: <p>I would recommend to try to use a Python package, which is specialized to read medical images, so that your data won’t be clipped or disturbed in any other way.<br>
Once you’ve found a package, you could use their load functionality inside your <code>__getitem__</code> method.</p>

username: Juna
cooked: <p>That’s great idea. Thank you for your help!</p>
Error when compiling glow on Ubuntu 18.04

username: yetanadur
cooked: <p>It seems llvm headers are not in the include path even though I provide the -DLLVM_DIR path to LLVM config i.e /lib/cmake/llvm</p>
<p>Are there other options I need to provide to enable llvm include headers are correctly sourced ?</p>
<p>When I run ninja all, I get errors like</p>
<pre><code class="lang-auto">glow/include/glow/Graph/PlaceholderBindings.h:44:36: error: no template named 'StringMap' in namespace 'llvm'
  using PlaceholderNameMap = llvm::StringMap&lt;Placeholder *&gt;;
</code></pre>
<p>I had ran cmake as follows<br>
cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug …/glow -DLLVM_DIR=&lt;path/to/llvm&gt;/local/lib/cmake/llvm</p>

username: ezyang
cooked: <p>I interpret the error differently: it looks like your version of LLVM is not the version needed by Glow (as it is complaining StringMap is missing, not that the header is missing.)</p>

username: yetanadur
cooked: <p>Sure. Basically, I had verified that the necessary header llvm/ADT/StringMap.h is present in my llvm path and the file does have StringMap symbol defined inside llvm namespace. Hence, I feel the issue is perhaps, the header file path is not correctly setup. When I look at the CC commands, I see <code>-isystem &lt;llvm/include/path&gt;</code> - is this sufficient ? I was expecting it should have been - <code>-I&lt;llvm/include/path&gt;</code> ?</p>
<p>(btw, I am using llvm 9.0)</p>

username: yetanadur
cooked: <p>Looks like glow/include/glow/Quantization/Base/Base.h is using llvm::StringMap without including header file for it.</p>
<p>When I explicitly added "<span class="hashtag">#include</span> “llvm/ADT/StringMap.h” to this file, I was able to compile. But there are few other such files as well. So I am not sure the right place to add this include to address all these files depending on it.</p>

username: jfix
cooked: <p>Thanks for catching this <a class="mention" href="/u/yetanadur">@yetanadur</a> – I’d suggest just adding it to the file if it uses <code>StringMap</code>. Better to have dependences listed explicitly anyway. Feel free to put up a PR with a fix <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to train deeplabv3 or fcn model?

username: hamzaygrtc
cooked: <p>I labeled my images with <a href="https://github.com/wkentaro/labelme" rel="nofollow noopener">labelme</a>. How could I train deeplabv3 or fcn model with my labeled dataset ? (I will apply model remove to background code that is following)</p>
<p><a href="https://github.com/spmallick/learnopencv/blob/master/app-seperation-semseg/Background-Removal.py" rel="nofollow noopener">Code </a></p>
<p>you have any train code example? or How could I learn train step?</p>
Stochastic Weight Averaging

username: justin_goodwin
cooked: <p>I was hoping to start using the SWA implementation in torchcontrib.  Two questions:</p>
<ul>
<li>Does it matter if <code>swa.bn_update</code> is before or after <code>opt.swap_swa_sgd</code>?</li>
<li>If we want to save checkpoints, do we need to execute the update functions above?</li>
</ul>

username: Mario_Parreno
cooked: <p>Hello! Have you found the answer?</p>

username: AralRalud
cooked: <p>Hello, I am dealing with the same issues here. Did you find the solution? And a related question:<br>
Does calling<code> swa.bn_update</code> affect the training when we retrain the model?</p>

username: AralRalud
cooked: <p>So, to answer my own question. According to the (pending) contribution to the pytorch library by the author <a href="https://github.com/izmailovpavel/pytorch/blob/master/torch/optim/swa.py" rel="nofollow noopener">here</a>, the documentation of the newly added class function <code>swap_swa_sgd_update_bn</code> says</p>
<pre><code class="lang-auto">    def swap_swa_sgd_update_bn(self, loader, model, device=None):
        r"""Swaps variables and swa buffers and updates BatchNorm buffers.
        This method is equivalent to subsequently calling 
        :meth:`opt.swap_swa_sgd()` and 
        :meth:`opt.bn_update(loader, model, device)`. It's meant to be called in
        the end of training to use the collected swa running averages and update
        the Batch Normalization running activation statistics in the model. It 
        can also be used to evaluate the SWA running averages during training; 
        to continue training `swap_swa_sgd` should be called again.
        """
        self.swap_swa_sgd()
        self.bn_update(loader, model, device)
</code></pre>
Chatbot tutorial help

username: Pat_Leo
cooked: <p>Hi,<br>
I’m currently following along the chatbot tutorial and i’m running the evaluation step.<br>
But I got an error:<br>
Found no NVIDIA driver on your system. Please check that you<br>
have an NVIDIA GPU and installed a driver from<br>
<a href="http://www.nvidia.com/Download/index.aspx" class="onebox" target="_blank" rel="nofollow noopener">http://www.nvidia.com/Download/index.aspx</a><br>
I’m using window 10 with intel i7 core and doesn’t come with NVIDA GPU.<br>
Is there another way I can run this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What does the <code>USE_CUDA = torch.cuda.is_available()</code> contains after running this at the beginning of the tutorial?</p>

username: Pat_Leo
cooked: <p>I have the CUDA toolkit installed on my computer but it still returns false</p>

username: albanD
cooked: <p>If you don’t have an nvidia GPU, you won’t be able to run with cuda.<br>
Even if you have the cuda toolkit.</p>
<p>It should return false, and should not use cuda.</p>
Tracing failed sanity check! CUDA out of memory on simple dot product

username: eduardo4jesus
cooked: <p>Hello community,</p>
<p>It has been a while that I am having issues with GPU memory and <a href="https://discuss.pytorch.org/t/convolution-using-fft-is-slower-than-conv2d/68552/2">performance of torch methods</a>. Recently, I’ve been trying to use <code>torch.jit</code> to speed up things, however having memory issues while having no clue if I am in the write direction.</p>
<p>Since torch doesn’t seem to have an optimized way do to element wise multiplication using complex numbers (refer to previous link). I decide to put together a small experiment for it.</p>
<p>When I get to the desirable shapes (tensor dimensions) I will have on  my application I keep getting <code>CUDA out of memory</code> exceptions.</p>
<p>My assumption is memory leak, since the size of my sample is not enough to take the <code>total capacity of 14.73 GB</code> of GPU that Colab provides.</p>
<p>This is my code snippet</p>
<pre><code class="lang-auto">def complex_dot(x, y):
  a, b = x[:, :, :, :, :, 0], x[:, :, :, :, :, 1]
  c, d = y[:, :, :, :, :, 0], y[:, :, :, :, :, 1]
  return torch.stack([a*c - b*d, a*d + b*c], dim=-1)
</code></pre>
<p><a href="https://colab.research.google.com/gist/eduardo4jesus/b1c87dc4e6b8ca96e7b5343c69efe1c6/dot-product-out-of-memory.ipynb" rel="nofollow noopener">Complete code available on Gist</a></p>
<p>PS.: This happens <a href="https://colab.research.google.com/gist/eduardo4jesus/3a4f7d375956ac36be628923bea3c201/dot-product-out-of-memory.ipynb" rel="nofollow noopener">regardless using <code>jit</code></a></p>
Is it possible to bake custom torchscript classes into scripted models?

username: Chenchao_Zhao
cooked: <p>Suppose we have a torch model like this</p>
<pre><code class="lang-auto">from script_class import CustomClass
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init___()
        self.object = CustomClass()
   def forward(self):
       print('hello')
</code></pre>
<p>Then i script the model and save it as ‘model.pt’<br>
Next when I load it without import CustomClass and run it, it will crash with unreadable c++ errors</p>
<p>How do I bake the class into the scripted model, something like torch.jit.export but for data structures?</p>
Memory reallocation when creating new torch.Tensor object

username: smikhail1
cooked: <p>AFAIK, torch.Tensor object is only a view over the corresponding torch.Storage object, which actually contains contiguous data. But why when we create a new tensor object from the existing one, by slicing it for example, another torch.Storage object is created? Does it mean that the memory, in this case, is allocated again?</p>
<p>In another word, why running the following snippet doesn’t result in AssertionError?</p>
<pre><code class="lang-auto">import torch

tensor_1 = torch.ones(5, 5)
id1 = id(tensor_1.storage())

tensor_2 = tensor[:3, :3]
id2 = id(tensor_2.storage())

assert id1 != id2
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>It will depend on the functions. You can see in the doc if a function returns a view of the original tensor or not.</p>
<p>Also <code>id()</code> tells you if the python objects are the same, but this is not the right way to check this. You can check if the location in memory of the beginning of the storage is the same:<br>
<code>tensor_1.storage().data_ptr() == tensor_2.storage().data_ptr()</code>.</p>
<p>Note that advance indexing is a bit special in this regard as depending on the arguments, you will or will not get a view.</p>
Prediction == Target but Gradient is > 0

username: maaft
cooked: <pre><code class="lang-python">import torch
import torch.nn as nn

class BinaryTverskyLossV2(nn.Module):
    def __init__(self, alpha=0.3, beta=0.7):
        super(BinaryTverskyLossV2, self).__init__()
        self.alpha = alpha
        self.beta = beta
        self.epsilon = 1e-6
        s = self.beta + self.alpha

    def forward(self, output, target):
        batch_size = output.size(0)
        output = output.view(batch_size, -1)
        target = target.view(batch_size, -1)
        P_G = torch.sum(output * target, 1)  # TP
        print("P_G: ", P_G)
        P_NG = torch.sum(output * (1. - target), 1)  # FP
        print("P_NG: ", P_NG)
        NP_G = torch.sum((1. - output) * target, 1)  # FN
        print("NP_G: ", NP_G)
        tversky_index = P_G / (P_G + self.alpha * P_NG + self.beta * NP_G)
        print("tversky_index: ", tversky_index)
        loss = 1. - tversky_index
        print("loss: ", loss)

        loss = torch.mean(loss)
        return loss

pred = nn.Parameter(torch.zeros(1,1,10,10, dtype=torch.float32), requires_grad=True)
pred.data[..., 5:8, 5:8] = 1

gt = pred.data.clone()

btl = BinaryTverskyLossV2()
loss = btl(pred, gt)

loss.backward()

print("grad mean: ", pred.grad.mean())
</code></pre>
<p>Gives:</p>
<pre><code class="lang-auto">P_G:  tensor([9.], grad_fn=&lt;SumBackward1&gt;)
P_NG:  tensor([0.], grad_fn=&lt;SumBackward1&gt;)
NP_G:  tensor([0.], grad_fn=&lt;SumBackward1&gt;)
tversky_index:  tensor([1.], grad_fn=&lt;DivBackward0&gt;)
loss:  tensor([0.], grad_fn=&lt;RsubBackward1&gt;)
grad mean:  tensor(0.0233)
</code></pre>
<p>Why is my gradient &gt; 0? Both prediction and target tensors are equal.</p>

username: albanD
cooked: <p>This is because the gradients wrt <code>P_NG</code> and <code>NP_G</code> are not 0. And these are computed in a differentiable manner wrt output. So you get some gradients for your output.</p>
Columns not contiguous after ConvTranspose2D

username: alexm1
cooked: <p>I need to build a model that takes the output of a transpose convolutional network and uses it as the mean parameter in a Normal distribution.</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.distributions import Normal

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.convT = nn.ConvTranspose2d(in_channels=3, out_channels=3, kernel_size=1)
        self.conv = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=1)

    def loss(self, x, permute=False, conv=False):
        pi_mu = self.conv(x) if conv else self.convT(x)
        pi_mu = pi_mu.permute(0, 2, 3, 1).unsqueeze(1) if permute else pi_mu
        return torch.sum(Normal(pi_mu, torch.ones_like(pi_mu)).log_prob(torch.ones_like(pi_mu)))

model = Model()
loss = model.loss(torch.randn(8, 3, 64, 64))
loss.backward()
print('ConvTranspose and no permute/unsqueeze done.')

loss = model.loss(torch.randn(8, 3, 64, 64), permute=True, conv=True)
loss.backward()
print('Conv and permute/unsqueeze done.')

loss = model.loss(torch.randn(8, 3, 64, 64), permute=True)
loss.backward()
print('ConvTranspose and permute/unsqueeze done.')
</code></pre>
<p>I’ve outlined three variations in the code, the third of which raises an error:</p>
<ol>
<li>Use a ConvTranspose2D layer but do not  permute the output. Works ok.</li>
<li>Use a Conv2D later and permute the output. Works ok.</li>
<li>Use a ConvTranspose2D and permute the output. Raises an error (see below).</li>
</ol>
<p>The output is as follows (python3.7, torch1.4.0)</p>
<pre><code class="lang-auto">ConvTranspose and no permute/unsqueeze done.
Conv and permute/unsqueeze done.

Traceback (most recent call last):
  File "/Users/alex/Developer/project/test.py", line 26, in &lt;module&gt;
    loss.backward()
  File "/Users/alex/miniconda3/envs/pytorch_env/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/Users/alex/miniconda3/envs/pytorch_env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: columns needs to be contiguous
</code></pre>
Type mismatch in optimizer?

username: 1337mathster
cooked: <pre><code class="lang-auto">import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import time
import numpy as np
import random
from matplotlib import pyplot as plt

ins = []
outs = []
for i in np.linspace(0,10,10):
    outs.append(10*(np.sin(i))+random.random())
    ins.append(i)


 
  

def criterion(out, label):
    return (label - out)**2


    
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1,5)
        self.fc2 = nn.Linear(5,5)
        self.fc3 = nn.Linear(5,5)
        self.fc4 = nn.Linear(5,5)
        self.fc5 = nn.Linear(5,1)
      
        
    def forward(self, x):
        x = self.fc1(x).relu()
        x = self.fc2(x).relu()
        x = self.fc3(x).relu()
        x = self.fc4(x).relu()
        x = self.fc5(x)
        return x


     
losses=[]
net = Net()#.cuda()
#optimizer = optim.SGD(net.parameters(), lr=(1.0e-3)*5,momentum = 0.5)
optimizer = optim.Adam(net.parameters(), lr=(1.0e-3))
#optimizer = optim.RMSprop(net.parameters(), lr=(1.0e-3),weight_decay = 0)

for epoch in range(4000):
   net.zero_grad()
   X = torch.Tensor(ins).type(torch.FloatTensor)
   Y = torch.Tensor(outs).type(torch.FloatTensor)
   out = net(X.view(-1,1)).type(torch.FloatTensor)
   #print(out,X.view(-1,1))
   print(Y)
   loss = F.nll_loss(out, Y)
   print(float(loss))
   loss.backward()
   optimizer.step()
</code></pre>
<p>The error i get is</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
</code></pre>
<p>nll_loss should be able to take float right?<br>
Any guidance is appreciated</p>

username: vainaijr
cooked: <p>maybe we need to pass index to second argument of nll loss, which must be of type Long, like,</p>
<pre><code class="lang-auto">import torch, torch.nn.functional as F
inp = torch.tensor([[0.1, 0.2, 0.3]])
target = torch.tensor([1])
x = F.nll_loss(inp, target)
x
</code></pre>
<p>tensor(-0.2000)</p>
PNSR metric for Image Super Resolution

username: bernardomig
cooked: <p>Hi.</p>
<p>I am doing some research in Image Super Resolution, and currently I am implementing some papers from the NTIRE2017 challenge (with the DIV2K dataset). I trained the models ok, but I dont quite understand a thing about the metrics, in particular, the Peak-Noise-Signal-Ratio, or PNSR.<br>
The PNSR is calculated for each image as <code>10 log10(1 / mse²)</code>, where <code>mse</code> is the mean squared error between both high-resolution and super-resolution prediction. However, this is for one image only. The way I aggregate the results for all the images in the validation dataset is where things are interesting.</p>
<ol>
<li>The first way I think doing this was by: 1) calculating the MSE for all the images, 2) average the MSE for the entire dataset; 3) apply the PNSR formula with this MSE.</li>
<li>The second way I applied was to average the PNSR for all the images.</li>
</ol>
<p>I think the correct way to do this was the 1st way, because I dont want to average log values. However, I found the model only got the correct score with the second version (31db vs 35db PNSR).</p>
<p>Saying that, what do you think about this?</p>
Saving list of models

username: stevethesteve
cooked: <p>I would like to store a list of my trained models so i can load them at any time to compare their performance.<br>
Also, i would like to append a newly trained model to the stored list.<br>
How do i do this most conveniently?</p>

username: tux
cooked: <p>Did you have a look at the doc here <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a> ?  In the ‘Saving Multiple Models in One File’ section <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: stevethesteve
cooked: <p>Thanks!<br>
I need to read manuals more carefully…<br>
I would be 100% happy if there was a way to append newly trained model to the file <em>without</em> loading the entire file from disk, like you can do with pickle. Any ideas?</p>

username: tux
cooked: <p>I am not sure to understand, can you be more explicit ? If it is a newly trained NN in your file, why would you need to load it ?</p>

username: stevethesteve
cooked: <p>Imagine i have trained a bunch of models and saved them into one file. A couple of days later, i start a new session and train a new model. This model i want to append to the file that i created previously. In order to do this, i would have to load the entire file via torch.load, then append the new model to the loaded dict, then save the file anew. This can be a problem e.g. when the total loaded dict does not fit into the working memory</p>