akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Undefined symbol: cblas_sgemm_alloc

ibacaraujo: <p>Hi, guys, after installing PyTorch 1.0, I am facing the following error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "miniconda3/envs/pytorch/lib/python3.6/site-packages/torch/__init__.py", line 84, in &lt;module&gt;
    from torch._C import *
ImportError: miniconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/libmkldnn.so.0: undefined symbol: cblas_sgemm_alloc
</code></pre>
<p>I was not able to find a solution to this error.</p>
<p>Thanks!</p>

saileshprem: <p>I had the same issue and realized I had installed mkl using</p>
<p>conda install -c anaconda mkl</p>
<p>I tried installing</p>
<p>conda install -c intel mkl</p>
<p>and import torch worked.</p>
<p>Hope this helps!</p>

yinge_zhao: <p>I had the same problem. Do you remember how the problem was solved? Installing WHL is not helpfull.</p>

yinge_zhao: <p>After installing the transformers library, the torch doesn’t work. The easiest way is to install a new torch version.</p>
Documentation for at::from_blob lifetime

_sean_silva: <p>I wasn’t able to find somewhere that documented the lifetime semantics of the buffer passed to at::from_blob.</p>
<p>From what I can tell, it’s something like “a pointer to the passed data buffer is retained for the lifetime of the created at::Tensor handle (which itself is doing some sort of refcounting under the hood); when the refcount maintained by the at::Tensor handles goes to zero, then the <code>deleter</code> is called”. Is that correct?</p>
<p>Thanks in advance.</p>

_sean_silva: <p><a href="https://pytorch.org/cppdocs/notes/tensor_basics.html#using-externally-created-data" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/notes/tensor_basics.html#using-externally-created-data</a> discusses torch::from_blob as creating a “view”, but doesn’t formally describe the lifetime (e.g. under what circumstances the deleter is invoked).</p>
What option(USE_NNAPI "Use NNAPI") is used for?

Peter_Peter: <p>Hello all,</p>
<p>there is an  <code>option(USE_NNAPI "Use NNAPI" OFF)</code>  within <a href="https://github.com/pytorch/pytorch/blob/cf48872d28f945d47793f63e19c54dd15bf580f7/CMakeLists.txt#L179" rel="noopener nofollow ugc">CMakeLists.txt#L179</a><br>
I’d like to know if this option is on, what is in this case enabled?<br>
I do have a device with NN-API driver - is this help to use this device NN-API backend?</p>
<p>Thank you.</p>
OOM Due to Pinned Parameters/Tensors

hobscrk777: <p>I’ve been iterating with a model for a while, and I’m running out of GPU memory. When I run the function below, I get a big long list of Parameter and Tensor objects. How do I tell PyTorch to release these objects? Running <code>torch.cuda.empty_cache()</code> doesn’t actually free up any memory.</p>
<pre><code class="lang-auto">def pretty_size(size):
	"""Pretty prints a torch.Size object"""
	assert(isinstance(size, torch.Size))
	return " × ".join(map(str, size))

def dump_tensors(gpu_only=True):
	"""Prints a list of the Tensors being tracked by the garbage collector."""
	import gc
	total_size = 0
	for obj in gc.get_objects():
		try:
			if torch.is_tensor(obj):
				if not gpu_only or obj.is_cuda:
					print("%s:%s%s %s" % (type(obj).__name__, 
										  " GPU" if obj.is_cuda else "",
										  " pinned" if obj.is_pinned else "",
										  pretty_size(obj.size())))
					total_size += obj.numel()
			elif hasattr(obj, "data") and torch.is_tensor(obj.data):
				if not gpu_only or obj.is_cuda:
					print("%s → %s:%s%s%s%s %s" % (type(obj).__name__, 
												   type(obj.data).__name__, 
												   " GPU" if obj.is_cuda else "",
												   " pinned" if obj.data.is_pinned else "",
												   " grad" if obj.requires_grad else "", 
												   " volatile" if obj.volatile else "",
												   pretty_size(obj.data.size())))
					total_size += obj.data.numel()
		except Exception as e:
			pass        
	print("Total size:", total_size)
    
dump_tensors()
</code></pre>

hanglearning: <p>I have the same issue.</p>
Transform Integral to Loss Function

DanielC: <p>Hi there,<br>
This question is about loss function.<br>
I have an integral equation which is shown below: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c4f35b515935c5e6b86e6f70f0cefa2af8349c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c4f35b515935c5e6b86e6f70f0cefa2af8349c2" title="Screen Shot 2020-10-05 at 12.21.53 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c4f35b515935c5e6b86e6f70f0cefa2af8349c2.png" alt="Screen Shot 2020-10-05 at 12.21.53 PM" data-base62-sha1="hJGUzNe5YYwrV5h4sNdG4D12HwS" width="690" height="112" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-10-05 at 12.21.53 PM</span><span class="informations">820×134 4.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
where y(x) is a one dimension vector. I want to use the above equation as my loss function, and here is my solution. I use matrix multiplication to rewrite it:</p>
<pre><code class="lang-auto">def my_loss (yhat, x, matrix):
    L = len(yhat)

    first_term = (2 * pi / L) * (torch.mm(yhat.T, torch.log(yhat)))
    second_term = (4 * pi ** 2/ L ** 2) * torch.mm(torch.mm(yhat.T, matrix), yhat)

    loss = first_term + second_term
    return loss
</code></pre>
<p>It works and the training result is not bad, but I find that the above code will cause a little bit of error. for example: if the y(x) = sin(x)**2, the integral vaule of first term is -1.21358, but the vector multiplication will gives me -1.2041.<br>
Does anyone know better ways to do this job?</p>
How to implement work pool in GPU cluster

Yi_Zhang: <p>Hi,</p>
<p>What is the most efficient way to implement a work pool in GPU cluster? I have tried with JoinableQueue, but it takes long time to get large item (e.g. a train batch) from the queue. Is there a better way to implement it? Is it possible to store data on GPU and shared by different processes?<br>
And I read this documentation for shared memory: <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a><br>
Is this “shared memory” in CPU or GPU? What is the structure of it?<br>
Thank you.</p>

mrshenli: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for multiprocessing questions</p>
The acc in multi gpus is lower than one gpus

sangyx: <p>Hi, my code is in <a href="https://github.com/sangyx/dgl/tree/master/examples/pytorch/GATNE-T/src" rel="nofollow noopener">https://github.com/sangyx/dgl/tree/master/examples/pytorch/GATNE-T/src</a>.</p>
<p>I run the <a href="https://github.com/sangyx/dgl/blob/master/examples/pytorch/GATNE-T/src/main_sparse.py" rel="nofollow noopener">main_sparse.py</a> will get acc 0.94. But the acc will not higher than 85% with <a href="https://github.com/sangyx/dgl/blob/master/examples/pytorch/GATNE-T/src/main_sparse_multi_gpu.py" rel="nofollow noopener">main_sparse_multi_gpu.py</a> even I set the gpu=0.<br>
Is there any error in my code?</p>
<p>My environment is Pytorch 1.6 and dgl-cu10.2 0.52. You can get the test data example in <a href="https://github.com/sangyx/dgl/tree/master/examples/pytorch/GATNE-T" rel="nofollow noopener">https://github.com/sangyx/dgl/tree/master/examples/pytorch/GATNE-T</a></p>

mrshenli: <p>Hey <a class="mention" href="/u/sangyx">@sangyx</a>, when using DDP, you might need to tune the batch size and learning rate a bit. See the discussion below:</p>
<aside class="quote quote-modified" data-post="1" data-topic="72769">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hhxx/48/21482_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769">Should we split batch_size according to ngpu_per_node when DistributedDataparallel</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Assume we have two nodes: node-A and node-B, each has 4gpus(i.e. ngpu_per_node=4). We set args.batch_size = 256 on each node, means that we want each node process 256 images in each forward. 
(1) If we use DistributedDataparallel with 1gpu-per-process mode, shall we manually divide the batchsize by ngpu_per_node in torch.utils.data.DataLoader : torch.utils.data.DataLoader(batch_size = args.batch_size / 4)(the way used in <a href="https://github.com/pytorch/examples/blob/5df464c46cf321ed1cc3df1e670358d7f5ae1887/imagenet/main.py#L42" rel="noopener nofollow ugc">pytorch-imagenet-official-example</a>). In my original opinion, I think Distrib…
  </blockquote>
</aside>
How to use tensor.item()

111137: <p>Hi,</p>
<p>My program has zero-dimension tensor like a</p>
<pre><code class="lang-auto">[1, 2, 3, ...]
</code></pre>
<p>I want to fetch one item from the tensor, so I did</p>
<pre><code class="lang-auto">scalar_val = val[index_val].item()
</code></pre>
<p>But it still makes same error of;</p>
<pre><code class="lang-auto">IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number
</code></pre>
<p>How to solve this issue?</p>
<p>Best,<br>
S.Takano</p>

albanD: <p>Are you sure that <code>val</code> is not a Tensor with a single number already?</p>

Dheerendra_Arc: <p>The dimension you defined is not correct according to the example. it must be 1-dimensional not 0-dimensional, so for accessing one dimension it is correct way to call “item” method</p>
<pre><code class="lang-auto">c = torch.tensor([1, 2, 3, 4])
print (c[0].item())
print (c.ndimension())
</code></pre>
<p>1<br>
1</p>
<p>In Case of Zero Dimension Example</p>
<pre><code class="lang-auto">c = torch.tensor(20)
print (c.ndimension()) 
print (c.item())
</code></pre>
<p>0<br>
20</p>
<p>I hope it makes clear to you</p>
Implement learning rate decay

DanielC: <p>Hi there,<br>
I wanna implement learing rate decay while useing Adam algorithm. my code is show bellow:</p>
<pre><code class="lang-auto">def lr_decay(epoch_num, init_lr, decay_rate):
    '''
    :param init_lr: initial learning rate
    :param decay_rate: if decay rate = 1, no decay
    :return: learning rate
    '''
    lr_1 = init_lr * decay_rate ** epoch_num
    return lr_1
</code></pre>
<p>and the training function is:</p>
<pre><code class="lang-auto">def fit(x, y, net, epochs, init_lr, decay_rate ):
    loss_points = []    
    for i in range(epochs):
        lr_1 = lr_decay(i, init_lr, decay_rate)
        optimizer = torch.optim.Adam(net.parameters(), lr=lr_1)
        yhat = net(x)
        loss = cross_entropy_loss(yhat, y)
        loss_points.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>However,  when I set decay rate = 1 (no decay), the output is much different with what I get before. my previous code is:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(net.parameters(), lr=constant)
def fit(x, y, net, optimizer, epochs):
    loss_points = []    
    for i in range(epochs):
        yhat = net(x)
        loss = cross_entropy_loss(yhat, y)
        loss_points.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I think the only difference between the two codes is that when I implement learning rate decay, I need to create a new optimizer for each epoch. how can I fix my code?</p>

googlebot: <p>Do you know about lr schedulers? <a href="https://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ExponentialLR" rel="noopener nofollow ugc">https://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ExponentialLR</a></p>
<p>Optimizer recreation won’t work correctly.</p>

DanielC: <p>Hi Alex,<br>
Thanks for your help, I implement my code according to the doc, and it works!</p>
Libtorch binaries built with different toolsets

melspec: <p>Is it possible to get Libtorch on Windows prebuilt binaries that are built against the v141 toolset?  I’m using the latest libtorch 1.6.0 (CUDA 10.2) with VS 2017, and the v141 toolset, and I’m having issues with the debug build.  It has a dependency on VCRUNTIME150_1D.dll, which is part of the 2019 redistributable.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea18a30a7e48b9afa9f9940af065330e17b2ab96.jpeg" alt="pytorch_deps" data-base62-sha1="xoUvLurQcXJa9ktIPgAbb5twHPw" width="368" height="376"></p>

glaringlee: <p><a class="mention" href="/u/peterjc123">@peterjc123</a> Is this possible? Any suggestions?</p>

peterjc123: <p>Sorry, but it is very unlikely for us to provide a VS2017 build for all specs. However, since the VC Debug DLLs are not redistributable, maybe we should provide binary debug builds for VS2017. By the way, would you please open a new issue at <a href="https://github.com/pytorch/pytorch" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch</a>?</p>

melspec: <p>Yes.  Find the issue <a href="https://github.com/pytorch/pytorch/issues/45843" rel="noopener nofollow ugc">here</a></p>
Help on torch::gather in C++ libtorch please

Chen0729: <p>Hi, All</p>
<p>In pytorch it is<br>
Prob_label = torch.gather(output_label, 1, y.unsqueeze(1))</p>
<p>In libtorch, I write it as<br>
torch::Tensor prob_label = torch::Tensor::gather(output_label, 1, y.unsqueeze(1),false);</p>
<p>it gives the error on “torch:Tenosr::gather”  as call to non-static member function without an object argument.</p>
<p>Any comment is appreciated.</p>
<p>Thanks.</p>

Chen0729: <p>if I use torch::gather(…) it says no matching function for torch::gather.</p>
<p>Thanks</p>

Chen0729: <p>torch::Tensor prob_label = torch::gather(torch::Tensor output_label, 1, y.unsqueeze(1));</p>
<p>I changed to the above, it says "expected ( for function-style cast or type construction.</p>
Custom optimizer - calculating n loss values

eldarab: <p>Hello everyone,</p>
<p>I am doing a research project in my undergraduate studies, and my professor requested me to implement an optimization algorithm he wrote.</p>
<p><a href="https://www.tandfonline.com/doi/pdf/10.1080/10556788.2020.1754413" rel="noopener nofollow ugc">Link to the paper</a></p>
<p>The algorithm optimizes a fully connected feed-forward neural network for classification, and in order to perform a weight update, I need to calculate the gradient w.r.t. each label - meaning that if I have n classes to classify to, I need to calculate the loss n times.</p>
<p>This is something which is strange (and for the best of my little knowledge) impossible in the current scheme of writing optimizers in PyTorch.</p>
<p>Can anyone aid? Wrote a PyTorch optimizer?</p>
<p>If a wasn’t clear in my description of the problem, please fell free to reply and ask - it can be quite confusing to fully understand whats going on here.</p>
<p>Thank you so much</p>
Questions about sequential. Plz help

sea_Fire: <p>I’m a new guy studying PyTorch and I wonder how can I get the structure with label 0 ? The following is the picture of the Network. I just want to print label 0 structure. If I use “print(net[0])” or “print(net[‘linear’][0])”, it will go wrong. I don’t know…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bcab24197e031a3725ad7a47926647870d130479.png" alt="image" data-base62-sha1="qV2qO3gpt0p6fRKCBpkkngY4RCh" width="485" height="109"></p>

Caruso: <p>Hi <a class="mention" href="/u/sea_fire">@sea_Fire</a>,</p>
<p>try <code>net.linear[0]</code>. The Sequential model is stored under the attribute <code>linear</code> in the <code>Network</code> class. Modules in a Sequential can either be selected by slicing or sometimes if there is a name given for the module by accessing the attribute via the name .</p>
<p>Here an example from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html#torch.nn.Sequential" rel="noopener nofollow ugc">docs</a>:</p>
<pre><code class="lang-auto"># Example of using Sequential
model = nn.Sequential(
          nn.Conv2d(1,20,5),
          nn.ReLU(),
          nn.Conv2d(20,64,5),
          nn.ReLU()
        )
print(model)
print(model[0])
# Example of using Sequential with OrderedDict
model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))
print(model)
print(model[0], model.conv1)
</code></pre>

sea_Fire: <p>Thank you so much. I’m glad you replied to me. <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"></p>
What does detach().clone() used for?

luckydog1996: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e221a6aaaef4973d09636cbd6c1f30f201aa77dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/e221a6aaaef4973d09636cbd6c1f30f201aa77dd" title="Screenshot 2020-10-04 at 22.56.54"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e221a6aaaef4973d09636cbd6c1f30f201aa77dd_2_690x121.png" alt="Screenshot 2020-10-04 at 22.56.54" data-base62-sha1="wgs0kyaFNYWDr4dNoUYUcXYlhsp" width="690" height="121" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e221a6aaaef4973d09636cbd6c1f30f201aa77dd_2_690x121.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e221a6aaaef4973d09636cbd6c1f30f201aa77dd_2_1035x181.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e221a6aaaef4973d09636cbd6c1f30f201aa77dd_2_1380x242.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-10-04 at 22.56.54</span><span class="informations">1818×320 47.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi I am new to pytorch, this is a piece of code. Basiclly net and fc_net is the nets we want to train. The net_input, fc_net_input are generated by other Network let call it net_pretrain. I don’t quite understand why we need to detach().clone() here. since the optimizer does not include parameter of net_pretrain</p>

googlebot: <p>detach() cuts away backprop graph for previous network (this is needed when net_input.grad_fn is not None, and is not related to optimizer)<br>
clone() duplicates storage, avoiding potential problems with inplace operations.</p>
<p>It is possible that these were added just to be safe.</p>

albanD: <p>I think that in this case, the <code>clone()</code> is used to make sure that you have different memory. And so inplace modification of one won’t change the other.<br>
The detach() is use to avoid tracking gradient information between the two so that uses of <code>net_input_saved</code> won’t affect the gradients of <code>net_input</code>.</p>
Mir_eval ported to pytorch

JuanFMontesinos: <p>Hi pytorch audio community.<br>
I’ve ported bss_eval_sources from mir_eval library to pytorch in a tiny effort to speed up everything.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/JuanFMontesinos/torch_mir_eval" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/32466310?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/JuanFMontesinos/torch_mir_eval" target="_blank" rel="noopener nofollow ugc">JuanFMontesinos/torch_mir_eval</a></h3>

<p>Pytorch implementation of https://craffel.github.io/mir_eval/. Not backpropagable. - JuanFMontesinos/torch_mir_eval</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It’s not backpropagable but it’s roughly 5 times faster if you use GPU.</p>
<p>Regaards</p>
Adding categorical features to each pixel of an image for use in CNN model

ErikJ: <p>I am building a model which generates new image from an imput image. I have a training set of input images and desired generated images and I also have some categorical data for each pixel of an image which I want to use as aditional features for the model. Currently I am jusut converting categorical features to have a float value on the interval between 0 - 1 for each category in even spaced intervals:</p>
<ul>
<li>Feature 1: 1/n</li>
<li>Feature 2: 2/n</li>
<li>Feature 3: 3/n …</li>
<li>Feature n: n/n</li>
</ul>
<p>Where  <strong>n</strong>  is the number of features. I then add this categorical feature matrix as a separate channel to the input image. I am wondering what are other aproaches for incorporating categorical data to CNN based models? How does one hot encoding work with CNNs? I have doubts because then I would essentiially create  <strong>n</strong>  sparse input channels and I have no experience with CNNs and sparse data.</p>
<p>I have asked same quyestion on CrossValidated <a href="https://stats.stackexchange.com/questions/490517/adding-categorical-features-to-each-pixel-of-an-image-for-use-in-cnn-model" rel="noopener nofollow ugc">cross validated link</a></p>
Detect patter and data prediction

TomHerror: <p>Hello everyone,</p>
<p>I have a question about my personal project.</p>
<p>I am a newbie in AI and all the neural networks.</p>
<p>my project is a graphical visualization of the passage frequency in the Paris Metro. I stock my data in a Cassandra database, with the total by station by day and an another table for the total by day on all the stations. I hope this is compressible …</p>
<p>my question is about, what PyTorch technology can help me to detect the pattern in the past passage frequencies by day (for example: on the weekend 30% less than the week) and re product this pattern for the future, (soon future)</p>
<p>the idea is to know the future frequencies and add it the data like, mobile application research to  anticipate the mass displacements for example.</p>
<p>I hope you got ideas about what technologies i can use for my project.</p>
<p>have a good day !</p>
How to manually update weights of conv2d layer in forward pass

Habibur_Rahman: <p>Is there a way I can divide by l2_norm of conv2d layer (similar as ArcFace, InsightFace but for conv layer not for fully connect layer)?<br>
My initial guess is i can add</p>
<pre><code class="lang-auto">with torch.no_grad():
     net.conv1.weight.data = custom_weight_norm_v1(net.conv1.weight)
</code></pre>
<p>Where my custom_weight_norm_v1 func is:</p>
<pre><code class="lang-auto">def custom_weight_norm_v1(a):
    x1, x2, x3, x4 = a.shape
    c = a.view(x1, -1)
    c = c/ torch.norm(c, dim= 1, keepdim=True)
    c = c.view(x1, x2, x3, x4)
    
    return c
</code></pre>
<p>I tried no_grad in forward and training loop. Is it going to work?</p>
<p>Also One more question everytime I train i get different result(I used seed though)</p>
<pre><code class="lang-auto">torch.manual_seed(3)
torch.cuda.manual_seed_all(3)
</code></pre>

Habibur_Rahman: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you help me please? :’)</p>
Semantic Segmentation from scratch

SamStick: <p>Hi People. I am new to this deep learning world and I was learning concept regarding semantic segmentation.</p>
<p>I have seen some tutorials where they are using already trained models on famous datasets. I want to learn how to train my data and test it from scratch using architectures like UNet in PyTorch.</p>
<p>It would be great if someone can suggest tutorials where they are doing it from scratch.</p>
<p>Thanks guys. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Parameters cannot update if I apply torch.cat in forward

LearnerYme: <p>In my network structure, I’d like to attach an additional neuron to the input of a fully connected layer, i.e. the output of its former layer has 128 neurons, and I want to let the model take another feature ( the number of rows of the input) into consideration, so I apply torch.cat((x, n_tensor), 1) in network’s forward method, where x is the output of former layer ( a [N, 128] tensor, N is the batch size) and n_tensor indicates the rows of input of this batch respectively ( a [N, 1] tensor). And I found that if I do so, the loss of whole training set and whole validation set will not change, in addition, I checked the weights of layers, and they did not update. I guess that’s because torch.cat() would break the gradient, and I’m confused what should I do to append this n_tensor to the network.<br>
Thanks a lot!</p>

ptrblck: <aside class="quote no-group" data-username="LearnerYme" data-post="1" data-topic="98219">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/learneryme/48/30404_2.png" class="avatar"> LearnerYme:</div>
<blockquote>
<p>I guess that’s because torch.cat() would break the gradient,</p>
</blockquote>
</aside>
<p>That’s not the case as seen in this minimal code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">x1 = torch.randn(2, 3, requires_grad=True)
x2 = torch.randn(2, 1, requires_grad=True)

y = torch.cat((x1, x2), dim=1)
y.mean().backward()

print(x1.grad)
print(x2.grad)
</code></pre>
<p>To verify it, you could also check the <code>.grad</code> attributes of your model and check, if another operation breaks the computation graph.</p>

LearnerYme: <p>Thanks a lot! I’ll check my network carefully then.  <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
FastaiLRFinder does not allow running more than 1 epoch

rafaelpadilla: <p>I have been trying to use the FastaiLRFinder to find the best learning rate for my module.</p>
<p>If we create the trainer with the function <code>create_supervised_trainer</code> as below:</p>
<pre><code class="lang-auto">trainer = create_supervised_trainer(
    model, optimizer, criterion, device, output_transform=custom_output_transform
)
</code></pre>
<p>and run it:</p>
<pre><code class="lang-auto">with lr_finder.attach(
            trainer,
            to_save=to_save,
            num_iter=50,  
            end_lr=1.,
            step_mode='exp') as lr_finder_training:
        lr_finder_training.run(train_loader)
</code></pre>
<p>A warning will come up say: “UserWarning: Desired num_iter 50 is unreachable with the current run setup of 15 iteration (1 epochs) from ignite.contrib.handlers.param_scheduler import (LRScheduler, PiecewiseLinear)”</p>
<p>My dataloader has 15 batches to iterate, which means FastaiLRFinder does not allow you to run more than 1 epoch. Why?</p>
<p>According to their source code, we can see <a href="https://github.com/pytorch/ignite/blob/14cdc046c7cac2db8a1bf838dcd09cbbe0e9c33b/ignite/contrib/handlers/lr_finder.py#L87" rel="noopener nofollow ugc">here</a> that this verification limits the user to run more iterations than the maximum of epochs in their dataloader.</p>
<p>But why? Am I missing something important here?</p>

vfdev-5: <p>Let’s discuss it here : <a href="https://github.com/pytorch/ignite/issues/1357" rel="noopener nofollow ugc">https://github.com/pytorch/ignite/issues/1357</a><br>
Most likely <a href="https://github.com/pytorch/ignite/issues/1357#issuecomment-703485006" rel="noopener nofollow ugc">https://github.com/pytorch/ignite/issues/1357#issuecomment-703485006</a> solves the issue.</p>
How to calculate probabilities of the prediction made by the model

smi222: <p>Hey everyone!</p>
<p>What is the best way to find probabilities of predictions.</p>
<p>I have custom dataset trained on ‘bert-base-german-cased’. The prediction functions look like this:<br>
def get_predictions(model, data_loader):<br>
model = model.eval()</p>
<p>passage_text = []<br>
predictions = []<br>
prediction_probs = []<br>
real_values = []</p>
<p>with torch.no_grad():<br>
for d in data_loader:</p>
<pre><code>  texts = d["passage_text"]
  input_ids = d["input_ids"].to(device)
  attention_mask = d["attention_mask"].to(device)
  targets = d["targets"].to(device)

  outputs = model(
    input_ids=input_ids,
    attention_mask=attention_mask
  )
  _, preds = torch.max(outputs, dim=1)

  probs = F.softmax(outputs, dim=1)

  passage_text.extend(texts)
  predictions.extend(preds)
  prediction_probs.extend(probs)
  real_values.extend(targets)
</code></pre>
<p>predictions = torch.stack(predictions).cpu()<br>
prediction_probs = torch.stack(prediction_probs).cpu()<br>
real_values = torch.stack(real_values).cpu()<br>
return passage_text, predictions, prediction_probs, real_values</p>
<p>y_passage_text, y_pred, y_pred_probs, y_test = get_predictions(<br>
model,<br>
test_data_loader<br>
)</p>
<p>encoded_custom_entity = tokenizer.encode_plus(<br>
passage_text,<br>
max_length=MAX_LEN,<br>
add_special_tokens=True,<br>
return_token_type_ids=False,<br>
pad_to_max_length=True,<br>
return_attention_mask=True,<br>
return_tensors=‘pt’,<br>
)</p>
<p>input_ids = encoded_custom_entity[‘input_ids’].to(device)<br>
attention_mask = encoded_custom_entity[‘attention_mask’].to(device)</p>
<p>output = model(input_ids, attention_mask)<br>
_, prediction = torch.max(output, dim=1)</p>
<p>print(f’custom_entity: {passage_text}’)<br>
print(f’entity_id  : {class_names[prediction]}’)</p>
<p>How do i find the prediction probability, given a passage?</p>
How to use fixed point numbers all the way

joepytor: <p>Hi,</p>
<p>I am a new user of pytorch. Now I have trained some network which satisfies me. Now I would like to implement the same network together with the same parameters in some resource limited hardware. I would like to use fixed point number all the time.</p>
<p>I think by default, pytorch uses 32bit floating point numbers in a cuda environment. So the best/safest way to do that is to first change every number in cuda to a fixed point 32bit number. Then run the model again to verify it. I think there is a way to force converting every 32bit floating point number to 16bit floating point number by using “half”. Is there a way to force converting every floating point number to a fixed point number?</p>
<p>Thanks, in advance!</p>

joepytor: <p>I am trying to use intigers for all the variables in the network just try to check if it works.</p>

Sumit_Diware: <p>Hi,</p>
<p>This may be helpful to you : <a href="https://qpytorch.readthedocs.io/en/latest/" rel="noopener nofollow ugc">https://qpytorch.readthedocs.io/en/latest/</a><br>
Paper for the above : <a href="https://arxiv.org/abs/1910.04540" rel="noopener nofollow ugc">https://arxiv.org/abs/1910.04540</a><br>
It works with PyTorch seamlessly and you can “simulate” fixed point behavior.</p>
<p>Regards, Sumit</p>
How to calculate epoch loss

Sumanta_Ghosh: <p>Hi everyone, if I want to print epoch loss (i.e. loss after every 300 epochs) then how should I modify this code?<br>
And what is difference between running and epoch loss?<br>
Thanks in advance</p>
<pre><code class="lang-auto">def train_model(train_dl, model):
    criterion = MSELoss()                                               #loss function
    optimizer = Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))  #optimizer should be used
    for epoch in range(300):
        running_loss = 0.0
        for i, (inputs, targets) in enumerate(train_dl):
            optimizer.zero_grad()
            pred = model(inputs)
            loss = criterion(pred, targets)
            loss.backward()
            optimizer.step()
            #print ("loss.item",loss.item())
            running_loss +=loss.item()
            epoch_loss += pred.shape[0]*loss.item()
    print('Finished Training')
</code></pre>

KarthikR: <ol>
<li>To print every 300, check for the epoch value:</li>
</ol>
<p>def train_model(train_dl, model):<br>
criterion = MSELoss()                                               <span class="hashtag">#loss</span> function<br>
optimizer = Adam(model.parameters(), lr=0.001, betas=0.9, 0.999))  <span class="hashtag">#optimizer</span> should be used</p>
<pre><code>for epoch in range(300):
    running_loss = 0.0
    for i, (inputs, targets) in enumerate(train_dl):
        optimizer.zero_grad()
        pred = model(inputs)
        loss = criterion(pred, targets)
        loss.backward()
        optimizer.step()
        if i%300 ==0:
          print ("loss.item",loss.item())
        running_loss +=loss.item()
        epoch_loss += pred.shape[0]*loss.item()
</code></pre>
<p>print(‘Finished Training’)</p>
<ol start="2">
<li>On difference between running and epoch loss, please refer this <a href="https://stackoverflow.com/questions/61092523/what-is-running-loss-in-pytorch-and-how-is-it-calculated" rel="noopener nofollow ugc">link</a>. Although they refer to the running_loss (epoch loss in your case), the concept should make things clear to you.</li>
</ol>
<p>In your example, running_loss is the aggregated loss per mini-batch, whereas, epoch_loss is to get loss undoing the reduction …</p>
How to create dataloader from a single example in a seq-to-seq model

kenenbek: <p>I am building a sequnce-to-sequence model.</p>
<p>For MWE, I have one learning example: [0, 1, 2, 3, 4] --&gt; [0, 1, 2, 3, 4]</p>
<p>How can I create a dataloader from it? I tried</p>
<pre><code class="lang-auto">x = torch.arange(5)
y = torch.arange(5)

test_dataloader = torch.utils.data.dataloader.DataLoader((x, y))
</code></pre>
<p>But it gives me:</p>
<pre><code class="lang-auto">for data in test_dataloader:
    print(data)
    print("----")

tensor([[0, 1, 2, 3, 4]])
----
tensor([[0, 1, 2, 3, 4]])
----
</code></pre>
<p>While I expected to get:</p>
<pre><code class="lang-auto">(tensor([[0, 1, 2, 3, 4]]), tensor([[0, 1, 2, 3, 4]]))
----
</code></pre>
<p>I need this because inside for loop I want to unpack learning samples like</p>
<p><code>X, y = data</code></p>

ptrblck: <p>Try to use a <code>TensorDataset</code> and pass the data in the shape <code>[batch_size, features]</code> to it.<br>
Once this dataset is created pass it to the <code>DataLoader</code>.</p>
Suggestion on Disk -IO and GPU utilisation

bing: <p>Hi guys,<br>
I am performing a multi-class image classification.<br>
Below are my Disk IO and GPU utilisation.<br>
Any suggestion on how to work towards improving them. I am using <code>torch nn.DataParallel</code> to parallelize them.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b598800e8e48941a4f3af4287848c583619437a.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b598800e8e48941a4f3af4287848c583619437a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b598800e8e48941a4f3af4287848c583619437a_2_513x500.png" alt="image" data-base62-sha1="hBcy1fM61HhfkjKiwF3NtQQpiXM" width="513" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b598800e8e48941a4f3af4287848c583619437a_2_513x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b598800e8e48941a4f3af4287848c583619437a_2_769x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b598800e8e48941a4f3af4287848c583619437a_2_1026x1000.png 2x" data-dominant-color="E4EFE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1340×1304 82.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>nn.DistributedDataParallel</code> with a single process per GPU is the recommended way as it’ll be faster than <code>nn.DataParallel</code>.<br>
To increase the data loading performance, play around with the number of workers and select the “sweet spot” for your system. Also make sure the data is stored locally (on an SSD) and not loaded from the network or a HDD.</p>

bing: <p>Will using <code>nn.DistributedDataParallel</code> will increase the GPU utilisation or IO utilisation.<br>
I varied the number of workers from 0 to 32, and surprisingly the Disk IO utilisation is the same throughout. Also, time takes to run 1 epoch is fastest, when the number of workers is 0.</p>

ptrblck: <p><code>DistributedDataParallel</code> should be more efficient than <code>DataParallel</code> and might thus increase the GPU utilization, but you would have to check it for your use case.<br>
Multiple workers should do the same and it’s unexpected that loading the data in the main thread is faster than in the background.</p>
<p>I would recommend to profile the data loading separately and check the loading times using <code>num_workers=0</code> and multiple workers to see why you are seeing a slowdown.</p>
Find indices of one tensor in another

navid_mahmoudian: <p>Hello,<br>
I have two large tensors A and B, both have type LongTensor and dimension 1. Both tensors are the results of torch.unique, so they have unique elements, and B is a subset of A (all elements of B are available in A). Is there any efficient way to find indices of tensor B elements in tensor A?</p>

ptrblck: <p>If the tensors are not large or if you have enough memory, you could probably use this broadcasting approach:</p>
<pre><code class="lang-python">a = torch.arange(10)
b = torch.arange(2, 7)[torch.randperm(5)]
print((a.unsqueeze(1) == b).nonzero())
&gt; tensor([[2, 1],
        [3, 3],
        [4, 4],
        [5, 0],
        [6, 2]])
</code></pre>
<p>Alternatively, I think you might need to use a loop (or list comprehension):</p>
<pre><code class="lang-python">for b_ in b:
    print((a == b_).nonzero())
&gt; tensor([[5]])
tensor([[2]])
tensor([[6]])
tensor([[3]])
tensor([[4]])
</code></pre>

navid_mahmoudian: <p>Thank you very much for your response. I have another question similar to this question (relaxing the assumption of bein output of unique function). Imagine I have two tensors as follows:<br>
x=torch.LongTensor([5,3,6,8])<br>
y=torch.LongTensor([2,3,4,6,8,5,7,9])</p>
<p>I want a function that returns the indices of elements in y that are available in x. So the output of this function should be:<br>
tensor([5, 1, 3, 4])<br>
I tried to use the solution in this <a href="https://discuss.pytorch.org/t/for-every-element-in-one-tensor-find-the-index-in-another-tensor/59875">link</a>, but it seems that it is not working for the case when the tensor lengths are different.</p>
<p>The broadcasting solution that you have given above, works perfectly, I am just looking for a more memory efficient approach (at the same performance).</p>

juvekaradheesh: <p>GIve me warning in pytorch 1.6.0</p>
<pre><code class="lang-auto">UserWarning: This overload of nonzero is deprecated:
        nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
        nonzero(Tensor input, *, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
</code></pre>
<p>Is there any better approach without loops?</p>

ptrblck: <p>To get rid of the warning, just pass the <code>as_tuple</code> argument as suggested:</p>
<pre><code class="lang-python">print((a.unsqueeze(1) == b).nonzero(as_tuple=False))
</code></pre>