akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convert CNN from Keras to Pytorch

anotherone_one: <p>Hello!<br>
Can anyone help me convert this Keras implementation to Pytorch? For an input image size of 256x256</p>
<pre><code class="lang-auto">
n_filters = 24 

def CNN(n_outputs=1):
  Conv2D = functools.partial(tf.keras.layers.Conv2D, padding='same', activation='relu')
  BatchNormalization = tf.keras.layers.BatchNormalization
  Flatten = tf.keras.layers.Flatten
  Dense = functools.partial(tf.keras.layers.Dense, activation='relu')

  model = tf.keras.Sequential([
    Conv2D(filters=1*n_filters, kernel_size=5,  strides=2),
    BatchNormalization(),
    
    Conv2D(filters=2*n_filters, kernel_size=5,  strides=2),
    BatchNormalization(),

    Conv2D(filters=4*n_filters, kernel_size=3,  strides=2),
    BatchNormalization(),

    Conv2D(filters=6*n_filters, kernel_size=3,  strides=2),
    BatchNormalization(),

    Flatten(),
    Dense(512),
    Dense(n_outputs, activation=None),
  ])
  return model
</code></pre>
<p>This was my attempt:</p>
<pre><code class="lang-auto">c = 24
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv0 = nn.Conv2d(in_channels=3, out_channels=c, kernel_size=5, stride=2, padding=1)
        self.conv1 = nn.Conv2d(in_channels=c, out_channels=2*c, kernel_size=5, stride=2, padding=1) # out: c x 14 x 14
        self.conv2 = nn.Conv2d(in_channels=2*c, out_channels=4*c, kernel_size=3, stride=2, padding=1) # out: c x 7 x 7
        self.conv3 = nn.Conv2d(in_channels=4*c, out_channels=6*c, kernel_size=3, stride=2, padding=1) # out: c x 7 x 7
        self.fc_mu = nn.Linear(in_features=36864, out_features=1)
            
    def forward(self, x):
        x = F.relu(self.conv0(x))
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        print(x.shape)
        x = x.view(x.size(0), 144*16*16) 
        x = self.fc_mu(x)
       return x
</code></pre>
<p>I think what I don’t understand here is how to write the Flatten and Linear layer properly, for that 512 input.</p>
<p>Thank you</p>

ksz: <p>Here, there should be 2 Linear layers:</p>
<pre><code class="lang-auto">self.linear1 = nn.Linear(in_features=36864, out_features=512)
self.linear2 = nn.Linear(in_features=512, out_features=1)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">x = x.view(x.size(0), 144*16*16) # Flatten(),
x = F.relu(self.linear1(x)) #Dense(512),
x = self.linear2(x) #Dense(n_outputs, activation=None)
</code></pre>
<p>Another thing is adding BatchNorm:</p>
<pre><code class="lang-auto">self.batch_norm_0 = nn.BatchNorm2d(c)
</code></pre>
<p>And in forward:</p>
<pre><code class="lang-auto">x = F.relu(self.conv0(x))
x = batch_norm_0(x)
</code></pre>
Tuning nn.Embedding weight with constraint on norm

blokhinnv: <p>Hello.</p>
<p>I have a really simple model which uses only <code>nn.Embedding</code> module. The goal is to minimize a specific loss function but with additional contraint that the L2-norm of the embeddings is 1.<br>
I found two options to normalize embeddings, specifically:</p>
<ol>
<li>Reassign weight at each <code>forward</code> call:  <code>self.embeddings.weight.data = F.normalize(self.embeddings.weight.data, p=2, dim=1)</code>
</li>
<li>Use param <em>max_norm</em> of <code>nn.Embedding</code>: <code>self.embeddings = nn.Embedding(..., max_norm=1, norm_type=2)</code>
</li>
</ol>
<p>The problem here is each of this solutions is ignored during backprop, as far as I can see. For the fist option a reassignment of the <code>weights.data</code> just doesn’t allow to track the gradients.  The second option implemented using <code>torch.no_grad()</code> context as we can see <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#embedding" rel="noopener nofollow ugc">here</a>.</p>
<p>Actually, I see many examples where people use the first option, but IMO it is not correct (if someone can explain my why it is correct, please, I would be grateful).</p>
<p>The other possible solution is to modify loss function some way, but I don’t really to do this right now…</p>
Libtorch how to add operation while slicing similar to Python?

erict: <p>Hi,</p>
<p>i’m making some test between pytorch and libtorch.<br>
In python you can make something like this: (for example tensor is size(1000,4))</p>
<p>tensor[:, :2] -= tensor[:, 2:] / 2</p>
<p>how can i achive this using libtorch.</p>
<p>I know that with .index and torch::indexing::Slice i can do slice, but how to add operation while slicing. There is a proper way of doing this in libtorch, without using a for loop on all tensor elements after slicing? Thanks</p>
BatchNorm epsilon factor tuning

Jean-Eric_Campagne: <p>Dear all,</p>
<p>I am using some model with BatchNorm2d with default eps=1e-5. But dumping the model trained parameters, it turns out that the running variances as a mean value of the order of 1e-6.</p>
<p>So the denominator of the BN sqrt(var + eps) is dominated by the eps factor.</p>
<p>What do you think ? should I put eps at 1e-7 level  for instance ?</p>
<p>Thanks</p>
How really RNNs work? and how to exploit sequential?

Laura_Montalvo: <p>Hi there! I’m trying to understand how nn.RNN(input_size, hidden_size, num_layers, batch_first=True) works for classification problems (not prediction). Let me explain:</p>
<p>I have my dataset which is compose of 1500 samples of vectors of size (1, 800). Each of the vectors of size (1,800) have one target associated, corresponding to the position were the sample was taken.<br>
Using this I created the dataset with a custom class like this:</p>
<pre><code class="lang-auto">class WifiDataset(Dataset):
    def __init__(self, data, labels=None, transforms=None):
        self.X = data
        self.y = labels
        self.transforms = transforms

    def __len__(self):
        return len(np.asarray(self.X))

    def __getitem__(self, i):
        data = self.X[i]
        if self.transforms:
            data = self.transforms(data)
        if self.y is not None:
            return data, self.y[i]
        else:
            return data
</code></pre>
<p>And then I create the dataloader as:</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True, drop_last=True) 
</code></pre>
<p>So each data, target in train_loader is data.shape = (batch_size,806) and target.shape = ([37]). Then I feed an RNN that is declared as:</p>
<pre><code class="lang-auto">class WifiRNN(nn.Module):
    def __init__(self, i_size, h_size, n_layers, num_classes):
        super(WifiRNN, self).__init__()
        self.input_size = i_size
        self.hidden_size = h_size
        self.num_layers = n_layers
        self.num_classes = num_classes

        self.wifi_rnn = nn.RNN(input_size=self.input_size, hidden_size=self.hidden_size, num_layers=self.num_layers,
                               batch_first=True)
        self.out = nn.Linear(in_features=self.hidden_size * sequence_length, out_features=self.num_classes)

    def forward(self, x_in, h_state):
        r_out, _ = self.wifi_rnn(x_in, h_state)
        r_out = r_out.reshape(r_out.shape[0], -1)
        out = self.out(r_out)
        return out

    def init_hidden_state(self, b_size):
        h0 = torch.zeros(self.num_layers, b_size, self.hidden_size).to(device)
        return h0
</code></pre>
<p>Am I taking advantage of RNNs hidden state? If the answer is yes, am i doing it between batches or how does it work?<br>
Do I need to feed the network with different timesteps to exploit the hidden state information?<br>
(Same questions asked for LSTMs, because I know RNN’s have vanishing gradient problem)</p>
<p>Thanks!</p>
Problem: merge two model parameters into other model with same network structure

zs963048949: <p>The code is flowing:</p>
<pre><code class="lang-auto">if epoch &gt; self.epoch_merge:
    if epoch == self.epoch_merge:
        for i,param in enumerate(zip(self.decoder1.parameters(),self.decoder2.parameters(),self.decoder_merge.parameters())):
            param[2].data = param[0].data + param[1].data
    out_merge = self.decoder_merge(feature_merge_list)
</code></pre>
<p>My goal is to merge two all parameters of decoders into a new decoder(named decoder_merge) with same network structure,then the new decoder will be trained continuedly.However, When the epoch equals to epoch_merge,I got a very low accuracy than previous one. After this,the model accuracy was increased normally.<br>
Therefore,I want to know what cause the cliff descent? Is it right for my merge measure?Thanks in advanced.</p>

CedricLy: <p>Maybe the optimizer use some momentum values, which are optimised for the original values?</p>

zs963048949: <p>The optimizer is adam in my code.</p>

zs963048949: <p>Do you know other methods about loading model parameters?Thanks a lot.</p>

ptrblck: <aside class="quote no-group" data-username="zs963048949" data-post="1" data-topic="112680">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zs963048949/48/33734_2.png" class="avatar"> zs963048949:</div>
<blockquote>
<p>My goal is to merge two all parameters of decoders into a new decoder(named decoder_merge) with same network structure,then the new decoder will be trained continuedly.However, When the epoch equals to epoch_merge,I got a very low accuracy than previous one. After this,the model accuracy was increased normally.</p>
</blockquote>
</aside>
<p>Was the accuracy of the two separate decoders high, while the new “merged” decoder yields a low accuracy?<br>
If so, I think this might be expected, if the parameters are not “close” to each other (whatever this means in the high dimensional space). There are methods such as stochastic weight average, which might help, but I don’t think that calculating the “mean” model of two trained models yields generally a better accuracy.</p>

zs963048949: <p>Thank you，you are right about what you said.I just want to ensure the correctness for the code.</p>
Indexing a tensor using a batch of coordinates

Omer_Cohen: <p>Hello,<br>
Given</p>
<ul>
<li>
<code>batch_maps</code>: 4D tensor of shape <code>(B, H, W, C)</code> of zeros</li>
<li>
<code>batch_label</code>: 3D tensor of shape <code>(B, N, 3)</code>
</li>
</ul>
<p>where batch_label is a batch of B sets of N points, each containing (x,y,prob) where prob is the value of an array at that point.<br>
what I want is to use batch_label to create B maps (fill batch_maps with points) , each map_i with values {prob}_i at coordinates {(x,y)}_i, where i goes from 1 to B.<br>
When I tried using x and y as indices, I got all of the points from all of the batch in all of the naps. I managed to get my goal using a  for loop on the batch dimension:<br>
for iter, (im,label) in enumerate(dataloader):</p>
<p>map = torch.zeros_like(im)<br>
label = label.unsqueeze(1)<br>
vals = label[:,:,:,2]<br>
x = label[:,:,:,0].type(torch.long)<br>
y = label[:,:,:,1].type(torch.long)<br>
for i in range(im.size()[0]):<br>
…     map[i,:,y[i,:,:],x[i,:,:]] = vals[i,:,:]</p>
<p>is there a way to do this without the for loop?</p>

Dwight_Foster: <p>You could try use torch.gather <a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc">here</a>.</p>

Omer_Cohen: <p>Not sure how gather helps me here, i can’t assign to function call</p>
Why torch.set_deterministic(True) reduce backward memory usage?

b02202050: <p>Hi all,</p>
<p>I recently tried torch.set_deterministic(True) and observe that it could reduce the GPU memory usage of backward!<br>
If I use mixed precision, the memory won’t reduce by torch.set_deterministic(True).<br>
Can anyone tell me why these happen?<br>
(My torch version: 1.7.0)</p>
<p>*** The code to reproduce my result is as the following.<br>
You can turn on/off <code>use_amp</code> and <code>deterministic</code> ***</p>
<pre><code class="lang-python">import torch
import torchvision
device = 'cuda'
use_amp = True
deterministic = False

# Set deterministic
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' # Deterministic behavior of torch.addmm. Please refer to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility
torch.set_deterministic(deterministic)

# Initialize model
model = torchvision.models.resnet50().to(device)
optim = torch.optim.Adam(model.parameters(), lr=1e-4)
scaler = torch.cuda.amp.GradScaler(enabled=use_amp)

# Forward
x = torch.rand(16, 3, 800, 800).to(device)
with torch.cuda.amp.autocast(enabled=use_amp):
    loss = model(x).sum()

# Backward
optim.zero_grad()
loss.backward()
optim.step()

print(torch.cuda.max_memory_allocated() / 1024 ** 2)
</code></pre>
<p>Thx!</p>

ptrblck: <p>The memory usage could be different based on e.g. the picked cudnn algorithms, i.e. a deterministic one could use less memory but might also be slower.</p>

b02202050: <p>Thanks!<br>
But why set_deterministic seems no effect on speed or memory if I use amp?</p>

ptrblck: <p>The kernel selection might pick the same algorithm in both cases, if you are using amp.</p>
Do we have an ROC AUC Score as an Objective function in Pytorch?

kennyseb: <p>Based on approximation of Wilcoxon-Mann-Whitney U statistic, here is the <a href="https://github.com/tflearn/tflearn/blob/c0baee9d34f41b84dbc43ea28e37baa5dbd465e4/tflearn/objectives.py#L179" rel="nofollow noopener">tflearn implementation</a></p>

phucdoitoan: <p>Any news on this? I think differentiable objective functions that directly optimize ROC-AUC and PRC-AUC scores will be useful in many scenarios.<br>
There are some paper describing such functions:<br>
ROC-AUC: <a href="https://www.aaai.org/Library/ICML/2003/icml03-110.php" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Classifier Performance via an Approximation to the Wilcoxon-Mann-Whitney Statistic</a> (ICML 2003)<br>
<a href="https://papers.nips.cc/paper/2003/hash/6ef80bb237adf4b6f77d0700e1255907-Abstract.html" class="inline-onebox" rel="noopener nofollow ugc">Paper</a> (NIPS 2003)<br>
PRC-AUC: <a href="https://arxiv.org/abs/1608.04802" class="inline-onebox" rel="noopener nofollow ugc">[1608.04802] Scalable Learning of Non-Decomposable Objectives</a> (ICML 2016)</p>
Tensor is empty after slice

caniko: <p>I have the tensor:</p>
<pre><code class="lang-auto">te = tensor([0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039])
</code></pre>
<p>Slicing it likewise returns an empty array with shape (0, 300):</p>
<pre><code class="lang-auto">roi = [0, 40, 300, 510]   # example values
te_cropped = te[roi[0] : roi[2], roi[1] : roi[3]]
</code></pre>
<p>Here is some more info:</p>
<pre><code class="lang-auto">u_mask: tensor([0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039]); te_cropped: tensor([])                                                                                                                                                                                                                                                                                              te_shape: torch.Size([1091, 300])                                                                                                                                                                                                                                                                                                                                                              te_cropped_shape: torch.Size([300, 0])
</code></pre>
<p>Any ideas what is happening here?</p>

Eta_C: <p><code>te</code> is a 1D tensor?</p>
Whereis CUDA/Cuda/cuda - Ubuntu 20.4 - cuda install issues

MANISH_GOYAL: <p>I recently switched from windows to ubuntu. I installed miniconda here, created a virtual environment with python 3.8.5, activated that environment and ran “conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch”. I ran torch.cuda.is_available() and it says True, but when I run which CUDA/Cuda/cuda or whereis CUDA/Cuda/cuda, I get nothing. nvcc is also not working. I think I might have to update bash file. Would someone please be able to guide me.</p>
<p>Also, Do I need to install cuDNN to work on my ML models using CUDA or is that not needed.</p>
<p>Thank You ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The PyTorch conda binaries and pip wheels ship with the specified CUDA (cudnn, NCCL) runtimes, not with a complete CUDA toolkit.<br>
To run PyTorch you would thus not need a local installation of these libraries and could use the binaries directly.<br>
However, if you want to build PyTorch from source, any custom PyTorch extension, or any other CUDA code, you would need to install the CUDA toolkit on your machine.</p>
Issue with different version of CUDNN

prabhuteja12: <p>Hi,</p>
<p>I’m using a pretrained model for inference that is producing wrong results when I disable CUDnn. Details below.</p>
<p>I’ve two conda environments E1, and E2.<br>
E1 - Pytorch 1.7.0 + CUDnn -8003<br>
E2 - Pytorch 1.5.1 + CUDnn -7603</p>
<p>When I’m doing inference using a pretrained model E1 and E2 give exactly the same results.</p>
<p>If I add <code>torch.backends.cudnn.enabled = False</code> to my code E1 gives completely arbitrary results (perf drop from 50% to 2%), but E2 continues to work fine. I’m at my wits end as to might even be causing this.<br>
Can someone help me with this?</p>
<p>Thanks!</p>

ptrblck: <p>This doesn’t seem to be a cudnn issue, if disabling it breaks the output or am I misunderstanding the issue?<br>
Could you update PyTorch to <code>1.7.1</code> as well as the nightly release and test your model again, as there might have been a known issue in the native conv implementation (which would be used in cudnn is disabled), which might have been fixed already?</p>
Batch matrix vector multiplication without for loop

jin8495: <p>Hi, I want to do batch matrix-vector multiplication but cannot figure out how to do that.</p>
<p>for example,<br>
input shape is (N x M x VectorSize), weight shape is (M x VectorSize x VectorSize). and I want to get an output shape (N x M x VectorSize).</p>
<p>N is batch size, M is number of vectors and VectorSize is literally size of vector. I want to compute matrix multiplication each vectors and matrices like picture below.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9870b525a097c6c3911e03f95904df4e3837fff4.png" data-download-href="https://discuss.pytorch.org/uploads/default/9870b525a097c6c3911e03f95904df4e3837fff4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9870b525a097c6c3911e03f95904df4e3837fff4_2_597x500.png" alt="image" data-base62-sha1="lKy37T3OFdkp63Skyv4so5R1xAg" width="597" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9870b525a097c6c3911e03f95904df4e3837fff4_2_597x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/8/9870b525a097c6c3911e03f95904df4e3837fff4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/8/9870b525a097c6c3911e03f95904df4e3837fff4.png 2x" data-dominant-color="FEF7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">707×592 77.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I do this without any for-loop?</p>
<p>Thank you.</p>

Naruto-Sasuke: <pre><code class="lang-python">N, M, V=2, 3, 5
a = torch.randn(N, M, V)
b = torch.randn(M, V, V)
a_expand = a.unsqueeze(-2)
b_expand = b.expand(N, -1, -1, -1)

c = torch.matmul(a_expand, b_expand).squeeze(-2)
</code></pre>

Eta_C: <p>Try <a href="https://pytorch.org/docs/stable/generated/torch.bmm.html" rel="noopener nofollow ugc">torch.bmm</a></p>
<pre><code class="lang-python">N, M, V = 2, 3, 4
X = torch.rand(N, M, V)
W = torch.rand(N, V, V)

# loop implementation
res = torch.zeros(N, M, V)
for i in range(N):
    res[i, ...] = torch.matmul(X[i, ...], W[i, ...])

print(res)

# no loop implementation
res2 = torch.bmm(X, W)
print(res2)
</code></pre>

jin8495: <p>Thank you!! It helped me a lot <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
PyTorch fully-connected network ~2x slower than tf.keras equivalent

dannyfriar: <p>I noticed that fitting a simple fully-connected network is close to 2x faster in tf.keras than in PyTorch. I have tested on an Nvidia A100 GPU and reproduced in a colab notebook.</p>
<p>This seems to be the case regardless of the input number of features, number of hidden layers and number of hidden units and appears to be more pronounced with larger batch sizes. I also noticed that using eager execution in the tf.keras model with <code>run_eagerly=True</code> seems to eliminate any performance gains.</p>
<p>Tested with pytorch 1.7.0+cu101 and Tensorflow 2.41 - please see the colab notebook below with a simple reproducer.</p>
<p>I’d be interested to know if I’m missing anything, why Tensorflow appears to be faster and if there is anything that can be done altered in the PyTorch code to match the Tensorflow performance.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210218-080408-RC00_358144746" class="site-icon" width="128" height="128">
      <a href="https://colab.research.google.com/drive/1rF49bLEMBcb3gcRhFGEUt5sMlJvG8N7W#scrollTo=rfKtgGTODD6e" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1rF49bLEMBcb3gcRhFGEUt5sMlJvG8N7W#scrollTo=rfKtgGTODD6e" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to Reshape a 2D tensor into a 4D tensor?

pauR: <p>Hi! I want to reshape a tensor of size  [batch_size, c*h*w] = [24, 1152] into one of size [batch_size, c, h, w] = [24, 128,3,3] but I can’t figure out how to do it. I’ve already tried the .view function. The 2D tensor is the output of a linear layer and I want the 4D tensor to be the input of a nn.ConvTranspose2d. This is the code, I’m trying to build a convolutional autoencoder with fully-conected layers in the middle:</p>
<pre><code class="lang-auto">class ConvAutoenc(nn.Module):
    
    def __init__(self, hidden_size):
        super().__init__() 
        
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=4, stride=2, padding=1), #out size= 14x14x32
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2, padding=1), #out size= 7x7x64
            nn.ReLU(),
            nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=0), #out size= 3x3x64
            nn.ReLU(),
            nn.Flatten()
        ) #out size= 1152 = 3*3*128
            
        self.hidden = nn.Sequential(
            nn.Linear(1152, hidden_size),
            nn.Linear(hidden_size, 1152))
        
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=2, padding=0), 
            nn.ReLU(),#out = 7x7x64
            nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=4, stride=2, padding=1),
            nn.ReLU(),#out = 14x14x32
            nn.ConvTranspose2d(in_channels=32,out_channels=1,kernel_size=4, stride=2, padding=1)
        )
        
    def forward(self, xb):
        out = self.encoder(xb) #out size= 1152 = 3*3*128
        out = self.hidden(out) #out size= 1152
        z = out.view(-1, (128, 3, 3))
        out = self.decoder(z)
        return out
</code></pre>
<p>When I run it, this error shows up:</p>
<blockquote>
<p>Blockquote<br>
“RuntimeError: Expected 4-dimensional input for 4-dimensional weight [128, 64, 3, 3], but got 2-dimensional input of size [24, 1152] instead”</p>
</blockquote>
<p>I’d appreciate any hint. Thanks in advance</p>

Dwight_Foster: <p>You can do</p>
<pre><code class="lang-auto">out.view(out.shape[0], 128, 3, 3)
</code></pre>

pauR: <p>Thanks! It works now</p>
Dataloading "freezes" when running several jobs

Knossos: <p>Hi everyone,</p>
<p>I have a problem when running several jobs on one machine (one job per GPU).</p>
<p>With only one job, the training goes well if I use the <code>pin_memory=True</code> option of the Dataloader, the batches are provided on time, the gpu utilisation is high. I use 8 workers.<br>
If <code>pin_memory=False</code>, the main process waits a bit for the dataloading on some batches, but it is not too bad.</p>
<p>However, if I start a second job, the dataloading “freezes” for every 8th batch (I have 8 workers) for as long as one minute, slowing down the training dramatically. I timed the <code>__getitem__</code> and <code>collate</code> functions and timing stays short and constant for those functions. It is therefore not a problem of disk throughput or i/o, nor processing.<br>
I found during the “freezes” that all the dataloading workers processes are dominated by Kernel Threads (all cpus are displayed in red color on htop).<br>
Turning <code>pin_memory=False</code> does not solve the issue.</p>
<p>So if it does not come from the <code>__getitem__</code> nor <code>collate</code> functions, where is all that time spent?<br>
What is going on after one batch has been processed by a worker and before it can be consumed in the main process?<br>
Could you recommend some monitoring tools to find out where the problem comes from?</p>
<p>I am working on a GCP linux virtual machine and my data is stored on a SSD drive.</p>
<p>Thanks for your help.</p>
Accessing update values

TinfoilHat0: <p>Hi,</p>
<p>Is there a way to access the update values computed by the optimizer?<br>
For example, if I’m using plain SGD with a learning rate a, then update values are simply a*gradients.</p>
<p>More concretely, these values should be what .backward() method computes, for each instance in the batch, and automatically adds to the model weights.</p>

albanD: <p>Hi,</p>
<p>You can access the gradients by accessing the <code>.grad</code> field of each parameters.<br>
The update value though will depend on the optimizer and you won’t be able to get it for the complex ones. The simplest thing I can think of here would be to save the weights before and compare them with the new value after the update.</p>

TinfoilHat0: <p>You understood me correctly. I’m not really interested in the grads, but what the optimizer computes when we call .step(). So, I guess you don’t see an efficient way of getting those values except than looking at the weights before, and after the update?</p>

albanD: <p>I’m afraid that in the general case no, there isn’t any other way to get that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
As a fun fact, some optimizers like lbfgs might actually do the step as multiple “sub-steps”, so in that case, checking before and after is really the only thing you can do.</p>
Pytorch DDP get stuck in getting free port

tengerye: <p>I try to get a free port in DDP initialization of PyTorch. However, my code get stuck. The following snippet could repeat my description:</p>
<pre><code class="lang-auto">def get_open_port():
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return s.getsockname()[1]

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    port = get_open_port()
    os.environ['MASTER_PORT'] = str(port)   # '12345'

    # Initialize the process group.
    dist.init_process_group('NCCL', rank=rank, world_size=world_size)

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 5)

    def forward(self, x):
        print(f'x device={x.device}')
        return self.net1(x)


def demo_basic(rank, world_size):
    setup(rank, world_size)

    logger = logging.getLogger('train')
    logger.setLevel(logging.DEBUG)
    logger.info(f'Running DPP on rank={rank}.')

    # Create model and move it to GPU.
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)  # optimizer takes DDP model.

    optimizer.zero_grad()
    inputs = torch.randn(20, 10)  # .to(rank)

    print(f'inputs device={inputs.device}')
    outputs = ddp_model(inputs)
    print(f'output device={outputs.device}')

    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()


def run_demo(demo_func, world_size):
    mp.spawn(
        demo_func,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )

run_demo(demo_basic, 4)
</code></pre>
<p>The function <code>get_open_port</code> is supposed to free the port after invocation. My questions are: 1. How does it happen? 2. How to fix it?</p>

pritamdamania87: <p>The problem here is that each of the processes end up with a different port due to <code>get_open_port</code>. As a result, the processes can’t rendezvous properly. The <code>MASTER_PORT</code> env variable needs to be the same on all processes and you probably need to choose a fixed port for this. The other option is to find a free port on the master, then communicate this port to all processes using some out of band mechanism to ensure all processes use the same port.</p>
<p>For example, if you’re running all processes on a single host (which is what your example code does). You can call <code>get_open_port</code> in the <code>run_demo</code> function and pass the free port to all processes as an argument to the <code>demo_func</code> method.</p>
Use non-default random generator for network initialization

vdg: <p>I would like to use a custom RNG as opposed to <code>torch::manual_seed(123)</code></p>
<p>In cpp:</p>
<pre><code class="lang-auto">    cpu_gen = at::detail::createCPUGenerator();
    std::lock_guard&lt;std::mutex&gt; lock(cpu_gen.mutex());
    cpu_gen.set_current_seed(0);
</code></pre>
<p>I can pass this generator to a random function:<br>
<code>auto random_tensor = torch::rand(100,cpu_gen);</code></p>
<p>How do I pass this generator to the construction of a module or (for example) to <code>torch::nn::Linear</code><br>
Right now the weights are initialized by the default (global) RNG.</p>
<p>(Python answers are also welcome)</p>
<p>Thanks!</p>
Is it possible to use Eigen lib manipulate the tensor inplace without detach the tensor from autograd graph?

C_Justin: <p>Hi there,</p>
<p>Is it possible to use Eigen or std libs in C++ to do some operation on a torch::Tensor without letting it detached from the autograd graph? Or I’ll need to rewrite all the operation using torch Tensor?</p>
<p>Thanks!</p>

soulitzer: <p>Hi,</p>
<p>If you want your Eigen/std lib operations recorded, you’ll want to either create a custom autograd function, or rewrite the operation using pytorch ops.</p>
<p>If you don’t want the operations recorded, but still modify the Tensor inplace, I’m curious what the use case what be since the computed gradients would be wrong.</p>

C_Justin: <p>Yeah…I want them to be recorded… so just want to confirm I’ll need to rewrite the whole thing before doing the work haha</p>

albanD: <p>Hey,</p>
<p>Unfortunately autograd is not magic. It works with all pytorch ops because we wrote manually all the formulas.<br>
So for it to work here you need to either:</p>
<ul>
<li>write things in pytorch ops for which we have the formulas</li>
<li>tell the autograd what is the formula yourself with a custom op <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox">Extending PyTorch — PyTorch 1.7.1 documentation</a> and then you can do whatever you want in the forward.</li>
</ul>
Regression model only gives a single output

brentv: <p>I’m having trouble getting my network to learn - it always seems to predict the same result regardless of input.</p>
<p>The goal of the network is to interpret 4D data and return a 2D array. The data is a simulation of a particular setup for laser-speckle imaging. It was simulated for a number of different starting positions, and 3 simulated experimental settings (in brief). This results in data with a [5, 6, 21, 22] shape. Simulations are done on 2D [21, 21] arrays, resulting in unique simulation data based on the 2D array given as input. This is the same object the network should predict.</p>
<p>My approach was to use convolutional layers followed by fully connected layers. This is the same approach I used when our problem was predicting 1D arrays from 3D data (this worked nearly perfectly). I’m assuming something went wrong when upscaling a dimension, or maybe someone has an idea of what might be going wrong.</p>
<p>My forward function:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = F.relu(self.convs(x)) # Uses convNd function from https://github.com/pvjosue/pytorch_convNd
        x = x.view(-1, x.shape[0], x.shape[1]*x.shape[2]*x.shape[3]*x.shape[4]*x.shape[5]) #flatten
        x = self.max1d(x.permute(0,1,2)).permute(0,1,2) # pooling nn.MaxPool1d(2, 2)
        x = self.max1d(x.permute(0,1,2)).permute(0,1,2) # pooling nn.MaxPool1d(2, 2)
        x = F.relu(self.fc1(x)) # nn.Linear(self._to_linear,1024), _to_linear = torch.flatten(x).shape[0] after convs
        x = F.relu(self.fc2(x)) # nn.Linear(1024,21*21)
        x = self.fc3(x) # nn.Linear(21*21,21*21)
        x = x.view(-1,21,21)
        return x
</code></pre>
<p>When I use a single data point for training, the network is able to give that output consistently. However, when I scale up to 8 data points (something which should easily be doable without any generalized learning), it is uncapable of giving different results while the loss is no longer decreasing. Images below are an example of what input/output looks like.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/508f8366cee93eadde526f129faaf042093e235a.png" alt="op3" data-base62-sha1="buFFKGIPswZ5ERfpG0e6uRxXT1E" width="590" height="369"></p>
<p>Does anyone have an idea of what might be going wrong here? If I need to give more information/code, please let me know!</p>
ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([128, 10]))

Sam3: <p>I am using bcewithlogitsloss for classification , the output should be 0 or 1 .<br>
I got this error!<br>
any idea?<br>
Thanks</p>

mmisiur: <p>It will be hard to determine what exactly is wrong without seeing the model. Could you post code of your model or specify what pytorch model are you using and what are your model’s input?</p>
Which activation function for hidden layer?

jpj: <p>I have a single hidden layer in my network, and 15 nodes in output layer (for 15 classes).<br>
After applying nn.linear to my inputs I apply sigmoid function for the hidden layer and then use nn.linear on the hidden layer output to get output layer inputs. I use those output  layer inputs in CrossEntropyLoss.<br>
What I am confused about is, which activation function should I be using from input to hidden layer? Is sigmoid fine for this?</p>

Dwight_Foster: <p>You should probably be using</p>
<pre><code class="lang-auto">torch.nn.ReLU()
</code></pre>
<p>sigmoid is not great for an inter layer activation function.</p>
Shape of check point and shape from in current modal are not same

Hamza_Ahmed: <p>I am trying to load a pre-trained modal but for some reason, the dimensions are not setting current.</p>
<ul>
<li>size mismatch for module.Prediction.weight: copying a param with shape torch.Size([50, 256]) from checkpoint, the shape in current model is torch.Size([37, 256]).</li>
<li>size mismatch for module.Prediction.bias: copying a param with shape torch.Size([50]) from checkpoint, the shape in current model is torch.Size([37]).</li>
</ul>
<p>Can someone tell me what’s happening?</p>

Dwight_Foster: <p>Can you send the code for loading the models.</p>

Hamza_Ahmed: <p>I got it the input size passing to the trained model was changed.</p>

Dwight_Foster: <p>Ok great glad you solved it. Just mark your answer as solved so people don’t think you need more help.</p>
How to create a tensor in LibTorch in C++?

arslansadiq: <p>Hi everyone,<br>
In the tutorial given here:</p>
<p><a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">LOADING A PYTORCH MODEL IN C++</a></p>
<p>Following is the code used for creating the minimal application.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }

  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(argv[1]);

  assert(module != nullptr);
  std::cout &lt;&lt; "ok\n";
  // Create a vector of inputs.
  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back(torch::ones({1, 3, 224, 224}));

  // Execute the model and turn its output into a tensor.
  at::Tensor output = module-&gt;forward(inputs).toTensor();

  std::cout &lt;&lt; output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';
}
</code></pre>
<p>Now here it is defining the vector of  <code>torch::jit::IValue</code>  (a type-erased value type  <code>script::Module</code>  methods accept and return). Upon pushing the concrete tensor values it is passing <code>(torch::ones({1, 3, 224, 224})</code>.<br>
Now my question is that, I want to pass a tensor size of <code>(1, 2)</code> which I can define my self. For changing the size, I just need to change numbers but what about defining the values explicitly, like I want to infer the output of my trained model by passing the following input: <code>torch.trensor([-0.8037, -0.2691])</code>. How exactly can I define this kind of tensor?</p>
<p>Thank you.</p>

yf225: <p>If you want to define a tensor of size (1,2) that contains [-0.8037, -0.2691], <code>torch::tensor({-0.8037, -0.2691})</code> should work.</p>

colinbroderick: <p>Could you describe roughly how one might build a tensor from some other object? e.g. I want to build a tensor of shape (1, x, y, 1) from a vector&lt;vector&gt; of shape (x,y).</p>
How to set the constraints of the parameters [weights and bias]

jackma: <p>Recently, I am reading the paper Input convex neural network whose implementation is using tensorflow. I am working to reproducing the network and do some simulation using PyTorch.</p>
<p>But how to set the constraints of the weights of the network.</p>

ksz: <p>Maybe this can help you: <a href="https://discuss.pytorch.org/t/set-constraints-on-parameters-or-layers/23620" class="inline-onebox">Set constraints on parameters or layers</a></p>
<p>If not describe your problem better. I don’t know this paper (and unfortunately I don’t have time to analyze it deeply).</p>

jackma: <p>Thanks! I will check it later.</p>
Concatenate along z axis(channel) PyTorch Dataloader

bibinwilson: <p>I have a multispectral dataset with 9 bands. Since the data is very large, I split each band to 256 x 256 samples. So I have 16 such samples for each band, and I saved them into different folders(each folder corresponds to each band - so 9 folders). Now how can I concatenate each sample of 9 bands?</p>
<p>For examples, I want to concatenate the 1st sample from the 1st band data with 1st sample of 2nd band, with 1st sample of 3rd band so on, till the 9th band. Then 2nd sample of 1st, 2nd … 9th band. so on till 16th sample.</p>
<p>Please help me to implement it.</p>
Apply something similar to Dropout2D

shivammehta007: <p>Let’s assume I have a tensor.</p>
<pre><code class="lang-auto">x  = torch.randn(64, 10, 2, 30)
# (batch, some length, some observation, some feature dimension)
</code></pre>
<p>And I want to apply dropout at the observation dimension i.e either all 30 features of 0 or either all 30 values of 1.<br>
I tried looking for it but I couldn’t find anything that could work or maybe I misunderstood some implementation.</p>
<p>Anyone has any idea how I can do it?</p>

ptrblck: <p>You could draw the mask manually using this code:</p>
<pre><code class="lang-python">x = torch.randn(64, 10, 2, 30)
N, C, _, L = x.size()

if torch.randint(0, 2, (1,)) == 0:
    mask = torch.tensor([0., 1.]).view(1, 1, -1, 1).expand(N, C, -1, L)
else:
    mask = torch.tensor([1., 0.]).view(1, 1, -1, 1).expand(N, C, -1, L)
</code></pre>
<p>and multiply it with the activation.<br>
Note that you would have to scale the activations during evaluation in case you want to disable this masking (or with the inverse during training).<br>
In that case, you could wrap this operation in a custom <code>nn.Module</code> and use the internal <code>self.training</code> flag to switch between the training and evaluation behavior.</p>

shivammehta007: <p>Thank you so much for the answer <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Also another solution I could come up was with this where I would create a dropout mask similar to yours but with one’s</p>
<p><code>dropout_mask = F.dropout(torch.ones((batch_size, length, observation , 1)), p=0.6)</code></p>
<p>and then multiply the data with it, one benefit of this I feel from the other approach would be I will not have to scale the activation, the F.dropout will do it for me.<br>
Another is that this will act like a proper dropout where I might be able to get both the features and none of them as well</p>
<p>What are your thoughts about it?</p>

ptrblck: <p>Oh, I might have misunderstood your use case and though you would like to keep one of the observations only. I.e. either zero out <code>x[:, :, 0, :]</code> completely or <code>x[:, :, 1, :]</code>.<br>
Your current approach would randomly zero out values in the <code>observations</code> dimension, wouldn’t it?</p>

shivammehta007: <p>Sorry! Maybe I was not clear in explaining, I wanted dropout but not in the last dimension (dropping some values from the 30) i.e feature dimension but rather dropping out the all 30 values or keeping all the 30 value. Sorry, I didn’t know how to name it so I called it drop out in observation dimension. where I want drop out in the dimension where my value is 2, so the case will be like <code>{ (0, 0), (1, 0), (0, 1), (1, 1) }</code> where 0 corresponds to dropping out all 30 features and 1 corresponds to keeping it.</p>
<p>I also tried dropout2D but I am unsure of its behaviour and the output had only two possibilities <code>{ (0,0), (1, 1) }</code> and not the <code>{ (0, 1), (1, 0) }</code> cases.</p>
Proportional sum of 2 outputs

FruitVinegar: <p>Hello.<br>
I’d like to train a model like below:</p>
<p>model has 2 internal submodel which are all (individually) trainable.<br>
model has proportional parameter w, <strong>which has restricted range [0, 1]</strong><br>
and w is also trainable.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/654d7a52552fe72e97d60de261c436f51722a3dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/654d7a52552fe72e97d60de261c436f51722a3dc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/654d7a52552fe72e97d60de261c436f51722a3dc.png" alt="image" data-base62-sha1="esadDAzF3Q796aHw2KR9B5jfxmI" width="342" height="500" data-dominant-color="AECF9B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">378×552 12.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I achieve this with restricted range? just clamp w for every iteration?</p>

AladdinPerzon: <p>What does <em>w</em> signify? My first idea is similar to yours where you can clamp w to be within [0,1] but perhaps it might get stuck where w is either 0 or 1. I guess one way would be to try this and see if <em>w</em> changes as you train the network. Another idea is perhaps to add a term to the loss function where you punish it for being larger than 1 and similarly if it’s below 0. Another idea I had is to use a sigmoid?</p>

FruitVinegar: <p>w means that proportional contribution of submodel 1 to total model.<br>
for example, if w=0.1, submodel 1 contributes to total model as 10% and submodel 2 contributes 90%.<br>
as you think, i’m afraid of that w stucks at 0 or 1. I’ll try your suggestion(sigmoid). is there other way to prevent stuck?</p>

shivammehta007: <p>I guess the cleanest method would be to use a sigmoid activation on w and then multiply it</p>
<pre><code class="lang-auto">w = torch.randn(3,3)
w = torch.sigmoid(w)

output = w * y_pred1 + (1-w) * pred_2
</code></pre>
<p>Sigmoid has a very smooth gradient and is differentiable everywhere.</p>
<p><a title="MartinThoma, CC0, via Wikimedia Commons" href="https://commons.wikimedia.org/wiki/File:Sigmoid-function-2.svg" rel="noopener nofollow ugc"><img width="512" alt="Sigmoid-function-2" src="https://upload.wikimedia.org/wikipedia/commons/thumb/5/53/Sigmoid-function-2.svg/512px-Sigmoid-function-2.svg.png" height="236"></a></p>