akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Compress model size by dynamic quantization to improve the saving/loading time

mojzaar: <p>Hi,</p>
<p>I have a large network (200M parameters), and saving/loading it takes a few minutes.</p>
<p>I’m curious if we can utilize dynamic quantization to shrink the model size and then save it.</p>
<p>I attempted to quantize the linear layers, but for training, they will all remain on the CPU, and dynamic quantization does not support linear layers on the GPU.</p>
<p>Is there a way to de-quantize the loaded model for the training and only quantize it for the saving to lower the model size and save time?</p>
Vectorizing Custom Sliding Window

Kanishk_Jain: <p>Hi, I need help in vectorizing this code. Here, E is (B, 8, 256, 256) vector. K is 7 (knn’s K nearest neighbor). Basically, I want to slide a window on E, and for each pixel find the K nearest pixel vectors in the corresponding window. Right now I have two loops for doing the same procedure, but it takes a very long time.</p>
<pre><code class="lang-auto">Z = torch.zeros_like(E)
        Z = repeat(Z, 'b c h w -&gt; b (repeat c) h w', repeat=self.K)
        
        ### Vectorize this code
        for i in range(H):
            for j in range(W):
                query = E[:, :, i, j] # B, C, 1, 1
                
                tl_x = max(0, i - self.patch_size)
                tl_y = max(0, j - self.patch_size)
                br_x = min(H, i + self.patch_size)
                br_y = min(W, j + self.patch_size)
                
                neighbor_patch = E[:, :, tl_x:br_x+1, tl_y:br_y+1] # B, C, Ph, Pw
                
                query = rearrange(query, 'b c -&gt; b 1 c')
                neighbor_patch = rearrange(neighbor_patch, 'b c h w -&gt; b (h w) c')
                
                distance = -torch.cdist(query, neighbor_patch) # b 1 hw
                distance_softmax = F.softmax(distance, dim=-1)
                
                _, indices = torch.topk(distance_softmax, dim=-1, k=self.K + 1) # b 1 k
                indices = repeat(indices.squeeze(1), 'b k -&gt; b k c', c=C)
                neighbors = neighbor_patch.gather(dim=1, index=indices) # b k c
                neighbors = neighbors[:, :, 1:].reshape(B, -1)
                
                Z[:, :, i, j] = neighbors
</code></pre>
<p>I tried using <code>F.unfold</code> for doing the vectorizing, but I am having difficulty implementing it.</p>
Sliced tensor is copied as a whole when iterating over it for multiprocessing

Haydnspass: <p>Just wanted to drop a note if someone stumbles over the same problem.</p>
<p>The scenario: Applying a costly function to each of the elements along the batch dimension via multiprocessing.<br>
If I iterate over the tensor and start multiprocessing the whole tensor is being copied and not only the one element. This can lead to excessive RAM consumption which is unnecessary. Interestingly the behaviour when using numpy is different.<br>
Using <code>.clone()</code> solves the problem.</p>
<p>Please find the example below:</p>
<pre><code class="lang-auto">import numpy as np
import pathos.pools
import time
import torch

def slow_fn(x):
    time.sleep(0.1)
    return

imgs = torch.rand(300, 720, 1280)

# (a) pythonic, but RAM explosion
imgs_ = [x for x in imgs]

# (b) still RAM explosion
imgs_ = torch.unbind(imgs)

# (c) okay with pytorch
imgs_ = [x.clone() for x in imgs]

# (d) okay with numpy (which is striking)
imgs_ = [x for x in imgs.numpy()]


pool = pathos.pools.ProcessPool(8)
_ = pool.map(slow_fn, imgs_)
</code></pre>
Can we have nd node features in pytorch_geometrical.data?

mohammadreza_Aalaei: <p>Here is the documentation of pytorch_geometric.data.Data</p>
<p>PARAMETERS:</p>
<pre><code class="lang-auto">x (Tensor, optional) – Node feature matrix with shape [num_nodes, num_node_features]. 
(default: None)

edge_index (LongTensor, optional) – Graph connectivity in COO format with shape [2, 
num_edges]. (default: None)

edge_attr (Tensor, optional) – Edge feature matrix with shape [num_edges, 
num_edge_features]. (default: None)

y (Tensor, optional) – Graph-level or node-level ground-truth labels with arbitrary 
shapes. (default: None)

pos (Tensor, optional) – Node position matrix with shape [num_nodes, num_dimensions]. 
(default: None)

**kwargs (optional) – Additional attributes.
</code></pre>
<p>According to this documentation, the shape of X should be of shape [num_nodes, num_node_features]. But what should I do if my node_features is multi-dimensional, for example, images data?</p>
How to compile libtorch for mobile?

azymohliad: <p>I’m building a cross-platform (Windows, Linux, macOS, Android, iOS) C++ library which runs a couple of ML models (inference only), currently using TensorFlow Light. With mobile support in latest PyTorch release, I’m considering to switch to it. But I’m not sure if I understand things right, so here are a few questions.</p>
<ol>
<li>
<p>I would want to link libtorch statically into my lib. Is libtorch mobile a minimalistic version of libtorch (kind of like TensorFlow Lite vs TensorFlow)? Or it’s just an interface to Android/iOS build systems and it runs a regular libtorch underneath? From <a href="https://discuss.pytorch.org/t/how-to-get-the-minimal-subset-of-libtorch-for-inferencing-in-c/62091/2">this answer</a> I understand it’s a regular libtorch, but just to be sure.</p>
</li>
<li>
<p>If it is a separate minimalistic library, how to compile it with C++ interface only (minimal size possible)?</p>
</li>
</ol>

eagomez: <p>Hi,</p>
<p>I had the same question as you. Did you fin anything new? (specially regarding 1.)</p>

azymohliad: <p>Hey, I didn’t find anything back in 2019, and didn’t look into the topic since then.</p>
How to map input ids to a limited Embedding indexes

Ahmad_Pouramini: <p>I have an embedding with limited size (say 5)</p>
<pre><code class="lang-auto">        self.embedding = torch.nn.Embedding(length,embedding_dim)
</code></pre>
<p>I receive input ids like (7, 18, 6, …) as a pytorch tensor. However the embedding for 7 is in the first index of <code>embedding</code>, for 18 it is in second row, etc.</p>
<p>I want a map from these numbers to 1,2, 3… to access stored value in embedding.<br>
It seems I can’t use a dictionary as follows</p>
<pre><code class="lang-auto">   def forward(self,prompt_token_ids,pids=None):
        prompt_token_ids = [self.id_map[x] for x in prompt_token_ids]
        return self.embedding(prompt_token_ids)
</code></pre>
<p>How can I do these mappings for tensors?</p>

Jazz: <p>you can convert the embedding to a numpy array using:</p>
<p>np_array = self.embedding.weight.detach().cpu().numpy()<br>
Also, you need to create a new tensor with new indices using:<br>
indices = torch.arange( # of elements in the input ids)</p>
<p>That would give you the flexibility to iterate over the whole thing. In the end, you can convert the np_array back to a tensor and then to the embedding.</p>

Ahmad_Pouramini: <p><a class="mention" href="/u/jazz">@Jazz</a>  Thanks,<br>
With some research I reached this solution which already works for my need:</p>
<pre><code class="lang-auto">prompt_token_ids = (prompt_token_ids.view(-1,1) == self.input_ids).int().argmax(dim=1)
</code></pre>
<p>the code above finds those numbers (stored in <code>input_ids</code> ) in a series of input tokens and replace them with their indexes in <code>input_ids</code> .</p>
Model single image test

Yasar_Niyazoglu: <p>I want to predict new (single) instances with your PyTorch model what can I do</p>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/i-cant-predict-new-single-instances-with-your-pytorch-model/138143">here</a> or a different use case?</p>

Yasar_Niyazoglu: <p>same thing i want to do. but what you said in the previous one didn’t work.</p>

ptrblck: <p>Could you follow up in the other thread and explain what is not working instead of cross-posting the same question, please?</p>
Binary segmentation for dice loss and softmax output

han-yeol: <p>Hello, I have been researching medical image segmentation.</p>
<p>I have segmented tumor and non tumor(background)<br>
my model’s output(nnUNet) is 2channel and softmax<br>
my label consist of B,1,H,W,D</p>
<p>How can I calculate Dice coeffcient loss function?</p>
Module weights as function of other parameters

kswarnkar: <p>Given a base model, I want to implement another model which recomputes the weights every time <code>(W = W0 * func(W1) )</code> and uses W as the module weights for computations, but trains only the parameters W0 and W1 (instead of training W).</p>
<p>I have been trying the following, but this doesn’t seem to work:</p>
<pre><code class="lang-auto">class AdvModel(nn.Module):
    def __init__(self, model):
        super(AdvModel, self).__init__()
        self.model = model
        for param in select_params(model): # iterates on selected parameters of model
            # (W) :this need not be updated
            param.weight.requires_grad = False 
            # (W0) this will be updated by autograd
            param.register_parameter(name='W0', param=deepcopy(param.weight) ) 
            # (W1) Two cases:
            #  this is updated by autograd; or 
            #  this is updated manually after each iteration
            param.register_parameter(name='W1', 
                param=torch.nn.Parameter(torch.rand(param.weight.shape), 
                requires_grad=False))  
            
    def forward(self, x):
        for param in select_params(self.model):
            adv_W1 = func_1(param.W1) 
            # where func_1 maybe a non-differentiable/differentiable function
            param.weight = torch.nn.Parameter(param.W0 * adv_W1, 
                                             requires_grad=False)
        return self.model(x)
</code></pre>

eqy: <p>I’m about confused here, what is the error and is W1 trained in addition to W0? It seems a bit strange to say W1 is being trained if it doesn’t appear to be a leaf node of the computation graph. Additionally, how would W1’s be computed if <code>func_1</code> as described isn’t differentiable?</p>

kswarnkar: <p>The model is trained with the regular cross entropy loss.<br>
When func_1 is non-differentiable, W1 is not trainable (W1 would be set manually in this case before doing the training/inference).<br>
When func_1 is differentiable, W1 will be trainable (because then W1 would be the leaf node).</p>
Multihead attention

Rini: <p>Hello,</p>
<p>I am a bit confused on why for batch size &gt;1 in the MultiHeadAttention class, attention is calculated by passing the input as &lt;batch_size*num_heads, seq_length, embed_dim&gt; and later reshaped to &lt; batch_size, num_heads, seq_length, embed_dim&gt; to get average attention across heads. Shouldn’t the input be reshaped first to split by batch and then calculate attention?</p>
<p>Thank you!</p>

AbdulsalamBande: <p>I dont think I have understood the question. But this video might help <a href="https://www.youtube.com/watch?v=_iDanMWVj98&amp;ab_channel=PytorchModulesExplained" rel="noopener nofollow ugc">torch.nn.MultiheadAttention</a></p>

my3bikaht: <p>Purpose of multiple heads is not to parallelize computations. Multiple heads work similar to convolutional filters, each building it’s own logic. Data split you mentioned is not batch-wise split, basic attention from Attention is all you need splits Q,K,V into separate chunks feeding each chunk to different head.</p>

Rini: <p>Thank you for the explanation!</p>
How to load a pretrained model into another slitly diffrent model?

TigerVersusT: <p>supose we have a model “saved” that has been trained and saved, and we also have another model “changed” that defined completely the same with “saved” but added some extra modules, so, how can we load the “saved” model’s weights into the “changed” model.</p>
<p>For example:</p>
<p>class A(nn.Module):<br>
def <strong>init</strong>():<br>
self.stack = nn.sequantial(…Linear and ReLU )<br>
def forward(x):<br>
return self.stack(x)</p>
<p>And<br>
class B(nn.Module):<br>
self.stack = nn.sequantial(…Linear and ReLU )<br>
self.Linear = nn.Linear<br>
def forward(x):<br>
x = self.stack(x)<br>
return self.Linear(x)</p>
<p>So, how can we load a pretrained model defined by class A into class B?</p>

ptrblck: <p>The easy but potentially dangerous method would be to use <code>load_state_dict(state_dict, strict=False)</code> and let PyTorch only load the matching modules. The save approach would be to manipulate the stored <code>state_dict</code> keys to match the new naming and load explicitly the desired data via e.g. <code>modelB.stack.load_state_dict()</code>.</p>
Input for fine-tuning transformers PyTorch

BouzidiImen: <p>Hello everyone,<br>
I was hoping if anyone could help me with preparing the training data for a model composed of an encoder and decoder. I would like to know how feed the model with a text (paragraph) with the correspondent image (handwritten).<br>
First layer of encoder:</p>
<pre><code class="lang-auto">FCN_Encoder(
  (init_blocks): ModuleList(
    (0): ConvBlock(
      (activation): ReLU()
      (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm_layer): InstanceNorm2d(32, eps=0.001, momentum=0.99, affine=False, track_running_stats=False)
      (dropout): MixDropout(
        (dropout): Dropout(p=0.5, inplace=False)
        (dropout2d): Dropout2d(p=0.25, inplace=False)
      )
    )
</code></pre>
<p>Decoder:</p>
<p>Decoder(<br>
(end_conv): Conv2d(512, 101, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
)</p>
<p>I would really appreciate your help! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=10" title=":blush:" class="emoji" alt=":blush:"></p>
The results of Affinity propagation Clustering always give me a high number of clusters compared to the number of samples

A_Hassan: <p>I am looking for tips regarding my code. I am interested in clustering by affinity propagation algorithm. I have a vector whose rows correspond to  the observation (800) and columns correspond to features. Every time I change the number of max-iteration or any factors it gives me the same results; A large number of clusters as in the picture</p>
<pre><code class="lang-auto">from sklearn.cluster import AffinityPropagation
from itertools import cycle
import sklearn.metrics as metrics
from sklearn.datasets import make_blobs
import numpy as np
from sklearn.metrics.pairwise import cosine_distances

# Compute similarities
X=features_vector
X_norms = np.sum(X**2, axis=1)
S = - X_norms[:, np.newaxis] - X_norms[np.newaxis, :] + 2 * np.dot(X, X.T)
p = 10 * np.median(S)

# Compute Affinity Propagation
af = AffinityPropagation(random_state=5, convergence_iter=500, copy=True, damping=0.95, max_iter=5000,verbose=True, affinity = 'euclidean').fit(S, p)

cluster_centers_indices = af.cluster_centers_indices_
labels = af.labels_
n_clusters_ = len(cluster_centers_indices)

#D = (S / np.min(S))
#print ("Silhouette Coefficient: %0.3f" %
 #      metrics.silhouette_score(D, labels, metric='precomputed'))

##############################################################################
# Plot result
import pylab as pl
from itertools import cycle


colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    class_members = labels == k
    cluster_center = X[cluster_centers_indices[k]]
    pl.plot(X[class_members, 0], X[class_members, 1], col + '.')
    pl.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,
            markeredgecolor='k', markersize=14)
    for x in X[class_members]:
        pl.plot([cluster_center[0], x[0]], [cluster_center[1], x[1]], col)

plt.title("Estimated number of clusters: %d" % n_clusters_)

plt.show()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2170e98f85983a466520e6dca8b98b1c1169044.png" alt="1" data-base62-sha1="tYxFqHbPg4hDQ064gwEpTEtPezq" width="553" height="388"></p>
How can I define a mask function based on the values of a list in Pytorch

Ahmad_Pouramini: <p>I want to mask a tensor based on its values. In the following function if I pass a range (second part) it works, but I want to have a list with various values <code>prompt_ids</code> (3, 8, 9, 30). But it doesn’t work and throws error.</p>
<pre><code class="lang-auto">RuntimeError: Boolean value of Tensor with more than one value is ambiguous
</code></pre>
<pre><code class="lang-auto">   def get_prompt_token_fn(self):
        if self.prompt_ids:
            return lambda x: x in self.prompt_ids
        else:
            return lambda x: (x&gt;=self.id_offset)&amp;(x&lt;self.id_offset+self.length)
</code></pre>
<p>What’s the problem and how can I resolve it?</p>
<p>I found <code>isin</code> function exists in pytorch 1.10 but not lower versions.<br>
My current solution is something like this, if there are better solutions you can provide:</p>
<pre><code class="lang-auto">def isin(ar1):
    return (ar1[..., None] == self.promp_ids).any(-1)
</code></pre>
<p>Currently Have on device error (one on cuda and other on cpu)</p>
How to build network with custom connectivity in autoencoder？

Aioku_Takume: <p>For example, in the figure my input is time series values [week_index, hour_index, values]. I want my output d only get weekly information and e only contains weekly and hourly information, while in f it will have all information. But if I use the built-in fully-connected network, the output d,e,f will all get [week, hour, value] information. Is there any way to do that in PyTorch? There could be extra 1 or 2 layers between input and output.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14b133b78d1f897fb6f38b51209afb3defa0746e.png" alt="node" data-base62-sha1="2X3cAgYtRzM2EeGDyjVkAMO0lFc" width="462" height="392"></p>
Issue with device Id

Vladimir1: <p>Hello. I am getting invalid device id in<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ebe3dd8eccaefe85170a225fdefaa2b2e835e77.png" alt="image" data-base62-sha1="mEj1Vozws3vQUVJCidXAf8bwB5Z" width="476" height="51"><br>
But<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3cd85eb5b70b55602b4d22331ceb1d69ec7850f.png" alt="image" data-base62-sha1="wveIslmyzl973XUeB5rklUREBWv" width="343" height="91"><br>
Can you help me?</p>

my3bikaht: <p>You’re probably evaluating torch.device as an index but it’s an object</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.device_count()
2
&gt;&gt;&gt; device = torch.device("cuda:0")
&gt;&gt;&gt; device
device(type='cuda', index=0)
&gt;&gt;&gt; device.index
0
</code></pre>
ValueError: Expected more than 1 spatial element when training, got input size torch.Size([8, 64, 1, 1])

sonnguyen129: <p>Hi all<br>
I keep getting this error:</p>
<blockquote>
<p>ValueError: Expected more than 1 spatial element when training, got input size torch.Size([8, 64, 1, 1]).</p>
</blockquote>
<p>Here is my code:</p>
<blockquote>
<p>class DepthWiseConv2d(nn.Module):</p>
</blockquote>
<pre><code>    def __init__(self, in_channels, out_channels, kernel_size, groups, stride):
            super(DepthWiseConv2d, self).__init__()
            self.depthwise = nn.Sequential(
                       nn.Conv2d(in_channels, in_channels, kernel_size = kernel_size,
                                     groups = groups, stride = stride, padding = 1),
                       nn.InstanceNorm2d(in_channels),
                       nn.ReLU(True)
            )
            self.pointwise = nn.Sequential(
                       nn.Conv2d(in_channels, out_channels, kernel_size = kernel_size,
                               stride = stride),
                       nn.InstanceNorm2d(out_channels),
                       nn.ReLU(True)
            )
    def forward(self, x):
         out = self.depthwise(x)
         out = self.pointwise(out)
         return out
</code></pre>
<p>class VGGEncoder(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    vgg = vgg19(pretrained=True).features
    self.slice1 = vgg[: 2]
    self.slice2 = vgg[2: 7]
    self.slice3 = vgg[7: 12]
    self.slice4 = vgg[12: 21]
    for p in self.parameters():
        p.requires_grad = False

def forward(self, images, output_last_feature=False):
    h1 = self.slice1(images)
    h2 = self.slice2(h1)
    h3 = self.slice3(h2)
    h4 = self.slice4(h3)
    if output_last_feature:
        return h4
    else:
        return h1, h2, h3, h4
</code></pre>
<p>class WeightAndBias(nn.Module):<br>
“”“Weight/Bias Network”“”</p>
<pre><code>def __init__(self, in_channels = 512):
    super(WeightAndBias,self).__init__()
    self.dwconv1 = DepthWiseConv2d(in_channels, 128, 3, 128, 2)
    self.dwconv2 = DepthWiseConv2d(128, 64, 3, 64, 2)
    # self.adapool1 = nn.AdaptiveMaxPool2d()
    self.dwconv3 = DepthWiseConv2d(64, 64, 3, 64, 2)
    # self.adapool2 = nn.AdaptiveMaxPool2d()

def forward(self, x):
    out = self.dwconv1(x)
    out = self.dwconv2(out)
    print(out.shape)
    # out = self.adapool1(out)
    out = self.dwconv3(out)
    # out = self.adapool2(out)
    return out
</code></pre>
<blockquote>
<p>Here is my test case:</p>
</blockquote>
<pre><code>s = torch.rand(8,3,256,256)
out = VGGEncoder()(s, True)
out = WeightAndBias(512)(out)
print(out.shape)
</code></pre>
<p>Thank you!</p>
Using pkl file for inference

AamirM: <p>Hi,</p>
<p>I am trying to segment a CT data using a pretrained model. The paper and code I am following is this (<a href="https://github.com/ICT-MIRACLE-lab/CTPelvic1K" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ICT-MIRACLE-lab/CTPelvic1K: Resources of the paper “Deep Learning to Segment Pelvic Bones: Large-scale CT Datasets and Baseline Models”.</a>). The author provides pretrained model in the format of .pkl. They have also provided model_best.model file. I am not able to load the model in the .pkl format. Most of the documentation I went through uses .pth format.</p>
<p>Is there any method of converting .pkl file to .pth format?</p>

ptrblck: <p>What kind of error do you get when trying to load the <code>.pkl</code> file?<br>
I’m not sure if the files really differ, since only the file extension is using another naming. You could also save a <code>state_dict</code> using any custom extension as seen here, which won’t change the underlying data structure:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
torch.save(lin.state_dict(), "lin.my_custom_extension_format")

lin = nn.Linear(1, 1)
lin.load_state_dict(torch.load('lin.my_custom_extension_format'))
&gt; &lt;All keys matched successfully&gt;
</code></pre>
Error plot Convolution result CNN

randino: <p>how I display the result of the convolution as an image?</p>
<pre><code class="lang-auto">out1 = self.cnn6(out1)
        out1 = self.batchnorm6(out1)
        out1 = self.relu(out1)        
        plt.imshow(np.transpose(out1, (1,2,0)), interpolation='nearest')
</code></pre>
<p>RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.</p>

ptrblck: <p>Did you try to use the suggestion from the error message?<br>
Once you fix the <code>numpy()</code> error, you would have to check the shape of the output, since <code>imshow</code> expects an image format, while your output activation would contain the additional batch dimension. In that case, you could visualize each output separately. Also, you have to make sure the channel dimension is valid for an image format (i.e. it has to be 1 or 3). If that’s not the case, you could visualize each channel separately again.</p>
Limit DataLoader in loading in advance

Pia_Ludemann: <p>I wonder if there is a way to prevent the DataLoader from calling <strong>getitem</strong> so often, so as to say something like:<br>
Do only load 5 sequences in advance (and not plenty as it does now, which has problems with cuda out of memory).<br>
Has anyone tried this before?</p>

ptrblck: <p>You can change the <code>prefetch_factor</code>, which is defined as:</p>
<blockquote>
<p><strong>prefetch_factor</strong> (<a href="https://docs.python.org/3/library/functions.html#int"><em>int</em></a><em>,</em> <em>optional</em> <em>,</em> <em>keyword-only arg</em>) – Number of samples loaded in advance by each worker. <code>2</code> means there will be a total of 2 * num_workers samples prefetched across all workers. (default: <code>2</code>)</p>
</blockquote>
Model passes during train, and test, but when using it for inference it fails

NullPointer: <p>My model has the following structure. I am able to train, and test, but when it comes to using new data later, it fails with the error <code>Expected hidden[0] size (2, 188, 64), got (2, 150, 64)</code>.</p>
<p>Model</p>
<pre><code class="lang-auto">class BasicLSTM(nn.Module):
    def __init__(self, 
                 device,
                 input_dims: int, 
                 hidden_dims: int, 
                 output_dims: int,
                 num_layers: int = 2,
                 dropout: float = 0.5):
        super(BasicLSTM, self).__init__()
        
        self.model_type = "BasicLSTM"
        
        self.device = device
        self.input_dims = input_dims
        self.output_dims = output_dims 
        self.hidden_dims = hidden_dims
        self.num_layers = num_layers
        self.dropout = dropout
        
        self.lstm1 = nn.LSTM(input_size=input_dims, 
                             hidden_size=self.hidden_dims, 
                             num_layers=self.num_layers, 
                             dropout=self.dropout,
                             # batch_first=True
                             )
        
        self.fc = nn.Linear(self.hidden_dims, self.output_dims)
        
        hidden_state = torch.randn(self.num_layers, batch_size, self.hidden_dims).to(self.device)
        cell_state = torch.randn(self.num_layers, batch_size, self.hidden_dims).to(self.device)
        self.hidden_cell = (hidden_state, cell_state)
        
    def forward(self, x: Tensor):
        x = x.unsqueeze(0).float().cuda()
        out, self.hidden_cell = self.lstm1(x, self.hidden_cell)
        preds = self.fc(out)
        
        return preds[-1]
        
</code></pre>
<p>New predictions</p>
<pre><code class="lang-auto">def predict_future(time: int, future_data):
    model.eval()
    for i in range(time):
        seq = torch.FloatTensor(future_data)
        
        with torch.no_grad():
            np.append(future_data, model(seq).item())
</code></pre>
<p>The type of future_data is a numpy n-dimensional array.</p>

NullPointer: <p>I had more inputs than I was testing with. Changing the number of inputs fixed this but I’m not sure how it affected batch size.</p>
NaN values popping up during loss.backward()

James_Ko: <p>I’m using <code>CrossEntropyLoss</code> with a batch size of 4. These are the predicted/actual labels I’m feeding to it along with the value of the loss:</p>
<pre><code class="lang-auto">preds:
 tensor([[-0.0052,  0.2059, -0.1473],
        [-0.0250,  0.0953,  0.0047],
        [ 0.0684,  0.1638, -0.0705],
        [-0.0195,  0.0100, -0.0874]], device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)
target:
 tensor([2, 2, 2, 2], device='cuda:0')
loss: tensor(1.1942, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
</code></pre>
<p>Here is the error message I’m getting after setting <code>autograd.set_detect_anomaly(True)</code>:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "run_liunet.py", line 247, in &lt;module&gt;
    main()
  File "run_liunet.py", line 200, in main
    train(model, train_loader, optimizer, device, epoch, 'train', debug_mode)
  File "run_liunet.py", line 80, in train
    loss.backward()
  File "/home/jlko/miniconda3/envs/liuNetEnv/lib/python3.6/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/jlko/miniconda3/envs/liuNetEnv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'NativeBatchNormBackward' returned nan values in its 0th output.
</code></pre>
<p>Here is the architecture of my neural network: <a href="https://raw.githubusercontent.com/Information-Fusion-Lab-Umass/mri-features/9ccf23cd0ecb4f163c76483fed52e4709b02ea4f/liunet.py" rel="noopener nofollow ugc">https://raw.githubusercontent.com/Information-Fusion-Lab-Umass/mri-features/9ccf23cd0ecb4f163c76483fed52e4709b02ea4f/liunet.py</a>. I am only using the <code>self.conv</code> and <code>self.fc</code> modules, so you can ignore all of the stuff related to <code>self.age_encoder</code>.</p>

ptrblck: <p>Did you make sure that no inputs contain invalid values, e.g. by applying <code>torch.isfinite(input)</code>?<br>
Are you seeing an invalid output (in the model output or loss) before anomaly detection raises the error?</p>

Chloe_Su: <p>Hi Ptrblock,</p>
<p>I met a similar problem as him.<br>
I checked all the inputs are finite but the error persists.</p>
<p>Thanks</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?</p>
Compute the Jacobian wrt the input for each layer

LeanderK: <p>I am working on regularisation methods and I want do the following:</p>
<p>To compute my regularisation for each layer, i need the product of the jacobian with the weights-operation with regards to the input. In formulas:</p>
<blockquote>
<p>let l be the current layer and f be the function of the neural net so far,<br>
so that the output of layer l is l(f(x)) for the input x<br>
If l(x)=relu(Ax+b), then I need A(J_f(x)),<br>
and if l(x)=relu(convolution(x)+b), then I need convolution(xJ_f(x))</p>
</blockquote>
<p>I wonder how I can do this and share computation? I also need to be able to backgrob through the jacobian, it’s used in the loss. Since this is an early research-idea, efficiency is not super important as long as it’s somewhat manageable. I have found this gist: <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" class="inline-onebox" rel="noopener nofollow ugc">jacobian_hessian.py · GitHub</a>. I see a “<em>create_graph=False</em>”, so if I set <em>create_graph</em> to <em>True</em> (to be able to backgrob through it), does it also share computation? I think what might improve things is forward-mode differentiation wrt to the input during the forward-propagation step of reverse-mode differentiation, but I worry about memory requirements.</p>
<p>It may be easier or way more efficient, I am not sure, to compute the weight-jacobian product directly, but I have not found out how so far.</p>
"torch::cuda::is_available()" return false

zhangkun: <p>i want to apply GPU/cuda in c++.</p>
<p>“torch::cuda::is_available()” return false</p>
<p>how to do? thank you!</p>
<p>in the same computer, in python, “torch.cuda.is_available()” return true.</p>

zhangkun: <p>libtorch is libtorch-win-shared-with-deps-1.10.0+cu113</p>

zhangkun: <p>i have try “/INCLUDE:?warp_size@cuda@at@<span class="mention">@YAHXZ</span>”.</p>
<p>it does not work.</p>
Autograd for Optimal Transport distance

izzatum: <p>Hi!</p>
<p>I’m currently working on optimal transport distance for the inverse problem. I’m using PyTorch <code>loss.backward()</code> for computing the gradient of optimal transport distance. However, it doesn’t reflect any updates on my model. I would like to know, does PyTorch support the autograd of optimal transport distance?</p>
<p>Thank you.</p>
<p>Here I attached snapshots of my codes:</p>
<pre><code class="lang-auto">for b in batches:
        batch_xs = xs[b::BATCHES]
        batch_xr = xr[b::BATCHES]

        data_sim = pde_solver(batch_xs, batch_xr)
        data_true = noisy_data[b::BATCHES, ...]

        sim, data_sim_cdf = OT_data_normalization(data_sim, 6.0, 1.0)
        true, data_true_cdf = OT_data_normalization(data_true, 6.0, 1.0)

        with torch.cuda.amp.autocast():
            loss = OTloss(data_sim_cdf, data_true_cdf, sim, T)

        opt.zero_grad()       
        scaler.scale(loss).backward()
        scaler.unscale_(opt)
        nn.utils.clip_grad_value_(pde_solver.model, clip_value=1e3)
        scaler.step(opt)
        scaler.update()
        scheduler.step()
</code></pre>
<pre><code class="lang-auto"># Data normalization function
def OT_data_normalization(data, b, c):
    ns, nr, nt = data.shape
    data = data.reshape(ns*nr, nt)

    data = torch.log(torch.exp(b*data) + 1)
    data_norm = (data + c) / torch.sum(data + c, dim=-1,keepdim=True)
    data_cdf = torch.cumsum(data_norm, dim=-1)
    
    return data_norm, data_cdf
</code></pre>
<pre><code class="lang-auto"># compute OT loss
def OTloss(cdf_sim, cdf_obs, dsim, t):
    idx = torch.searchsorted(cdf_obs, cdf_sim)
    idx[idx == t.shape[-1]] = -1
    tidx = t[idx]
    
    return ((t - tidx)**2 * dsim).sum()
</code></pre>

ptrblck: <p>I don’t see any obvious issues in your code, but as it’s not executable I also cannot verify its functionality.<br>
You could check if all the expected parameters in your model receive a valid gradient by checking the <code>.grad</code> attribute before and after calling the first <code>backward()</code> operation. Once this is done you could then check their values or magnitude and see if they might be really small and thus almost no change would be visible.</p>

izzatum: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>! Great thanks for your suggestions! Sorry I couldn’t provide the complete code here because it is rather large. I followed your suggestions, and I found out that the gradient is not correct, and of course, the value is very small as you mentioned. I’m not sure how accurate is the autograd in approximating the gradient of complicated loss w.r.t PDE parameters. I need to check this out. Thanks a lot!</p>
Prevent Multithreading in PyTorch c++

Willem: <p>I have a simple (fairly small) PyTorch model (a feedforward neural network with 3 hidden layers of 64 elements) that I use a lot for inferencing. There are other processes, and I want inference to only happen on one thread. Inferencing (calling model-&gt;Forward(v)<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"> happens on CPU thread.</p>
<p>This thread somehow spawns 16 subthreads, which I don’t like. (There is lot of overhead, and cpu usage on other threads causes tasks there to slow down.) So I call:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>torch::set_num_threads(1);</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td>torch::set_num_interop_threads(1);</td>
</tr>
</tbody>
</table>
</div><p>which should prevent multithreading. Somehow this doesn’t work, and still 16 threads are created. Any idea what is causing this? Is this  a bug?</p>

Willem: <p>I did some more testing. So the program I use has inferencing on relatively small vectors on all threads. Above problems are with 1.10 (stable).</p>
<p>When I use PyTorch LTS (1.8.2), then the problems go away: When I spawn a single thread, PyTorch keeps at a single thread, i.e. it doesn’t spawn additional threads beyond the one created by me. As a consequence, with multithreading I get a huge speed-up.</p>
<p>Anyhow, for my use case it would be very helpfull if PyTorch 1.10 gains a switch to really turn off multithreading.</p>
Concept check on time series one step forecast and multiple step forecast

Chin_GM: <p>Dear all,</p>
<p>I have a question regarding the time series forecasting.</p>
<p>the model is trained in two different way:</p>
<ol>
<li>to perform one step forecasting with 60 time step of past values.</li>
<li>to perform 5 steps ahead forecasting with 60 time step of past values.</li>
</ol>
<p>My question is whether multiple step ahead forecasting will improve accuracy or not?</p>
<p>For example,<br>
I have trained one step forecasting and obtain the MSE of 0.045.<br>
I also trained 5 steps ahead forecasting, <strong>but I calculated MSE for the one step ahead forecasting and I got MSE of 0.0047.</strong><br>
I tried to compare the MSE of one step ahead forecast for both model.</p>
<p>My hypothesis is, if we train a model with multiple steps ahead forecast will improve the accuracy of one step forecast. But the MSE didn’t show any improvement.</p>
<p>May I know is my hypothesis correct? Is model trained withmultiple step ahead forecasting will improve the accuracy of the one step ahead accuracy compare to model train with one step ahead forecasting?</p>
<p>Thank you very much</p>
Expected scalar type Float but found ComplexDouble

Mam: <p>Hi,<br>
I’m trying to run a NN with complex tensors.<br>
I keep getting the following error: RuntimeError: expected scalar type Float but found ComplexDouble<br>
I suspect maybe it is because the weights are not complex?<br>
If so- how do I change it?</p>
<pre><code class="lang-auto">&gt; class NeuralNet(nn.Module):
&gt;     def __init__(self, input_nodes, hidden1_nodes, hidden2_nodes, hidden3_nodes, output_nodes):
&gt;         super(NeuralNet, self).__init__()
&gt;         self.type(torch.complex128)
&gt;         self.fc1 = nn.Linear(input_nodes, hidden1_nodes)
&gt;         self.relu = nn.ReLU()
&gt;         self.fc2 = nn.Linear(hidden1_nodes, hidden2_nodes)
&gt;         self.fc3 = nn.Linear(hidden2_nodes, hidden3_nodes)
&gt;         self.fc4 = nn.Linear(hidden3_nodes, output_nodes)
</code></pre>
<p>Thanks!</p>

tom: <aside class="quote no-group" data-username="Mam" data-post="1" data-topic="138377">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ce7236/48.png" class="avatar"> Mam:</div>
<blockquote>
<p><code>self.type(torch.complex128)</code></p>
</blockquote>
</aside>
<ol>
<li>use .to these days instead of .type (which is a very odd function and the most surprising thing is that it hasn’t been deprecated 2 years ago).</li>
<li>.to (or .type) changes the state of the NeuralNet. This also changes all submodules. But if you add submodules later, these will not be party to the change but will have the default dtype (float).</li>
<li>this suggests that moving the .to to after the submodule registration should work better.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>

Mam: <p>Thanks!<br>
I was not aware of this. But to throws me the following error:</p>
<pre><code class="lang-auto">nn.Module.to only accepts floating point dtypes

</code></pre>
<p>However, staying with type but moving it to after registration did do something, cause now I get the error:</p>
<pre><code class="lang-auto">RuntimeError: addmm does not support automatic differentiation for outputs with complex dtype.
</code></pre>
<p>Any suggestions?</p>

tom: <p>I’d probably write my own ComplexLinear class that starts out with complex parameters and uses matmul instead of addmm.</p>

Mam: <p>Got it.<br>
Thanks for your time!</p>
What happened when set `num_workers` > 0 in DataLoader?

techkang: <p>I wrote the follow code,</p>
<pre><code class="lang-python">from torch.utils.data import Dataset, DataLoader


class MyDataset(Dataset):
    def __init__(self):
        print('created！')
        self.a = None

    def __getitem__(self, item):
        if self.a is None:
            print('set a to 1')
            self.a = 1
        return item

    def __len__(self):
        return 10

    def __del__(self):
        print("deleted")


if __name__ == '__main__':
    my_dataloader = DataLoader(MyDataset(), batch_size=2, num_workers=2)
    for data in my_dataloader:
        pass
</code></pre>
<p>And the output of the script is</p>
<pre><code class="lang-auto">created！
set a to 1
set a to 1
deleted
</code></pre>
<p>There are two <code>set a to 1</code> in the output, and the <code>num_workers</code> is set to <code>2</code>, so I hypothesis that DataLoader copied one dataset instance for each process. But only one <code>created!</code> and one <code>deleted</code> are printed. This indicate that there is only one instance of <code>MyDataset</code>. So what is exactly behavior of <code>DataLoader</code> when set <code>num_workers</code> &gt; 0? How does it deal with <code>MyDataset</code> instance?</p>

ptrblck: <p>Each worker is creating a <code>Process</code> as seen <a href="https://github.com/pytorch/pytorch/blob/bfe5ad28e6a509e08c0afd785c32019f8a05a7fb/torch/utils/data/dataloader.py#L904-L910">here</a> with the <code>self._dataset</code> as one of its input arguments. My assumption is that the <code>__init__</code> of the passed <code>Dataset</code> won’t be invoked again but the object will be duplicated instead and called in each process, but I might also miss some details.</p>
How to extend out_features in last fully connected layer?

111148: <p>Hi, I have pre-trained network on my own. It has 2 out_features in last fully connected layer. I will load this network and want to add the one node in last fully connected layer. Then It will have 3 out_features in last fully connected layer. It is quite similar to transfer learning. Exactly I do not want to initialize the last layer. How can I do this one?</p>

spanev: <p>Hi,</p>
<aside class="quote no-group" data-username="111148" data-post="1" data-topic="55513">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111148/48/15767_2.png" class="avatar"> 111148:</div>
<blockquote>
<p>Exactly I do not want to initialize the last layer.</p>
</blockquote>
</aside>
<p>What do you mean by not initializing the last layer? How this would be different from transfer learning?</p>
<p>You can do like this:</p>
<pre><code class="lang-auto">class SubNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = Network()
        self.last_linear = nn.Linear(2,1)
    def forward(self, x):
        y1, y2 = self.net(x)
        return y1, y2, self.last_linear(y2)
</code></pre>

111148: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/75816a03b3b657ed560014505a202c17c0f75ae0.png" data-download-href="https://discuss.pytorch.org/uploads/default/75816a03b3b657ed560014505a202c17c0f75ae0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/75816a03b3b657ed560014505a202c17c0f75ae0.png" alt="image" data-base62-sha1="gLv8dnJhZ9pVjVOdJtzCqZExWHS" width="690" height="319" data-dominant-color="F1F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">882×409 20.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From the PyTorch transfer learning documents,  that is Imagenet 1000 case. The writer said that reset the final fully connected layer. That means 1000 class outputs change to 2 class output and initialized. Actually I want to do something similar. I want to change the last 2 class output to 3 class output layer. Not 2class to 1 class or adding the one more layer.</p>

A_Hassan: <p>please could you check my code <a href="https://stackoverflow.com/questions/70179428/extract-features-from-the-last-layer-in-alexnet-pytorch" rel="noopener nofollow ugc">here</a>. it is similar to the question above and I solved through my code</p>
Regarding M1 Processors

grndng: <p>Hey there, I just wanted to ask if there is any support planned for the M1 MacBook chips and if so, if they will utilize the GPU cores or the Neural Cores of the M1 processors. Is there any public info about this?</p>

jankelowitz: <p>As someone also interested this, I believe this long running issue is the defacto public info on M1 support: <a href="https://github.com/pytorch/pytorch/issues/47702" class="inline-onebox" rel="noopener nofollow ugc">GPU acceleration for Apple's M1 chip? · Issue #47702 · pytorch/pytorch · GitHub</a></p>
Dataloader seems to freeze

Valerio_Biscione: <p>Hello. Sometime just after loading a network my program seems to hang. I run my stuff overnight so I realize the day later, and even after 5 or 6 hours the program is still waiting for something. However, when I run CTRL+C finally I get this error trace</p>
<pre><code class="lang-auto">CException ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7f
3bb3d67dd0&gt;
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
line 961, in __del__
    self._shutdown_workers()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
line 941, in _shutdown_workers
    w.join()
  File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wai
t
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 28, in pol
l
    pid, sts = os.waitpid(self.pid, flag)
</code></pre>
<p>and, weirdly enough, the computation continues from here normally.<br>
Does anyone have any idea?<br>
I am currently running a lot of small simulations, so it’s pretty frustrating where in the morning I realized that I spent 6 hours running nothing. This doesn’t happen all the time, but aroudn once every 5 runs, and it’s pretty unpredictable.</p>
<p>Thanks</p>

Valerio_Biscione: <p>anyone has any idea about this problem?</p>

sdroh1027: <p>Not sure, but downgrading pytorch &lt;= 1.3.0 can help the problem</p>

Valerio_Biscione: <p>The reason was not due to Pytorch but to Neptune, when logging images. It seemed to interfere with multiprocessing. I reported the problem which got solved soon after that.</p>

Qirui_Sun: <p>Could you mind sharing your solutions?  I probably confront the same issue</p>
Difference between nn.BatchNorm2d.forward() and calculation using torch.Tensor

pepper8362: <p>Hi,</p>
<p>I find there is slight difference between the calculation results of nn.BatchNorm2d.forward() and torch.Tensor. Specifically, If I run the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

bn = nn.BatchNorm2d(10)

# feed data to update running_mean and running_var
for _ in range(100):
    dummy_data = torch.randn(8,10,100,100)
    _ = bn(dummy_data)

# now test
bn.eval()

#result of layer forward
data_in = torch.randn(8,10,100,100)
output_bn = bn(data_in)

#result of manual calculation
mean = bn.running_mean.view(1, 10, 1, 1)
var = bn.running_var.view(1, 10, 1, 1)
weight = bn.weight.view(1, 10, 1, 1)
bias = bn.bias.view(1, 10, 1, 1)
eps = bn.eps

data_in_norm = (data_in - mean) / torch.sqrt(var+eps)
output_mannual_calcualte = weight*data_in_norm + bias

#calculate error
err = (output_bn - output_mannual_calcualte).abs()
print(torch.max(err))
</code></pre>
<p>It will print something like:</p>
<pre><code class="lang-auto">tensor(4.7684e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Basically, it means the results are slightly different between the forward function of nn.BatchNorm2d and manually doing the same calculation using torch.Tensor. I wonder if there is something wrong in my manual calculation? Am I missing something here? Thank you very much for any suggestions.</p>

ptrblck: <p>Your calculation looks correct the the small absolute error points to the floating point precision limitation, which is expected. In case you need to get a lower error, you could use <code>float64</code> as the <code>dtype</code>.</p>

pepper8362: <p>Yes the error dropped to 1e-16 level with double precision. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=10" title=":blush:" class="emoji" alt=":blush:"></p>