akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calculating gradients w.r.t inputs & backward

username: omers66
cooked: <p>Hi, I’m trying to implement a contractive Autoencoder with Pytorch and I’m having trouble calculating gradients w.r.t inputs and calculate backward on top of that.<br>
My output ‘hn_enc’ is a 10 dimensional tensor and my input ‘x’ is a 24 dimensional tensor.<br>
I have a model called ‘csi_enc’ used for forward pass which parameters are also trained using an additional loss, but for the moment I want to focus on gradients w.r.t ‘x’.<br>
The goal is to compute the squared sum of the gradients of ‘hn_enc’ w.r.t ‘x’ and perform backward pass on this as a loss function.<br>
Note that this is essentially computing the Jacobian matrix:<br>
[ dhn_enc(1)/dx(1), dhn_enc(1)/dx(2),…, dhn_enc(1)/dx(24) ;<br>
…<br>
dhn_enc(10)/dx(1), dhn_enc(10)/dx(2),…, dhn_enc(10)/dx(24) ]<br>
and calculating this matrix squared sum as a loss function.</p>
<p>So, I’m going in a foor loop and calculating the gradients w.r.t ‘x’ for each ‘hn_enc’ entry as follows:</p>
<pre><code class="lang-auto">enc_optimizer = optim.Adam(lr=opt_lr_adam, params=csi_enc.parameters(), eps=1e-8)
x.requires_grad = True
z_enc, hn_enc, cn_enc = csi_enc.forward(x, x.shape[0], 1, False)  # forward pass
for q in range(0,10):
      hn_enc[q].backward(retain_graph=True)
      x.grad.requires_grad = True
      contractive_loss = contractive_loss + torch.mean(x.grad**2)
      enc_optimizer.zero_grad()    # with this, all model gradients are 0 again for next x grad compute
</code></pre>
<p>Note that I’m using ‘retain_graph=True’ so the graph won’t disappear.<br>
Note that I’m calling ‘enc_optimizer.zero_grad()’ such that the model weights gradients won’t get accumulated.<br>
Now, I want to update ‘csi_enc’ parameters w.r.t ‘contractive_loss’. But calling:</p>
<pre><code class="lang-auto">contractive_loss.backward()
</code></pre>
<p>results in:</p>
<blockquote>
<p>“one of the variables needed for gradient computation has been modified by an inplace operation”</p>
</blockquote>
<p>When I’m calling ‘contractive_loss.backward()’ using only a single iteration in the loop (i.e., calculate gradients w.r.t x[0] only) then it works great. So I’m doing something wrong with the iterative summation over ‘x.grad’ but I can’t figure out how to do it properly.<br>
Any help please?</p>
Does `torch.manual_seed()` influnce GPU seed?

username: Xianyuan
cooked: <p>From <a href="https://pytorch.org/docs/stable/notes/randomness.html#pytorch-random-number-generator" rel="noopener nofollow ugc">here</a>, it shows ‘You can use <code>torch.manual_seed()</code> to seed the RNG for all devices (both CPU and CUDA)’. Does it mean <code>torch.manual_seed()</code> is also for GPU?</p>
<p>However, from <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.manual_seed" rel="noopener nofollow ugc">here</a>, it shows that <code>torch.cuda.manual_seed(seed)</code> and <code>torch.cuda.manual_seed_all(seed)</code> are needed to set seed for GPU. I feel a little confused now.</p>

username: googlebot
cooked: <p>torch.manual_seed(seed) does torch.cuda.manual_seed_all(seed) inside</p>
Using Variable to update nn.Parameter and keep gradient

username: Zeeky
cooked: <p>Hi,</p>
<p>Community, I got a problem.</p>
<p>Suppose the customized loss function has parameter <code>x</code>,  e.g.,</p>
<p><code>Loss = f(outputs, targets, x)</code></p>
<p>Now I use <code>torch.autograd.grad</code> to compute the gradient w.r.t. to model parameters (not <code>x</code>)</p>
<p>The problem is that I cannot manually update the model parameters such that <code>x</code> is contained in the graph and still backpropagable.</p>
<p>For example</p>
<p>If I do <code>param = param - lr * grad</code> OR  <code>tmp = param - lr*grad; setattr(model, name, tmp)</code>, the error is <code>cannot assign 'torch.cuda.FloatTensor' as parameter</code></p>
<p>If I convert Tensor to nn.Parameter OR using <code>param.data = ...</code>, then <code>x</code> is not in graph.<br>
(and <code>backward()</code> + <code>step()</code> is the same as updating <code>param.data</code>)</p>

username: woithook
cooked: <p>Hi Zeeky<br>
Since model parameters in PyTorch are all leaf variables, which means you can not just set the parameters to new values manually. To achieve this, you just need to wrap the update function in a torch.no_grad() block:</p>
<pre><code class="lang-auto">with torch.no_grad():
    param = param - lr * grad
    
</code></pre>
Why is it a bad idea?

username: kolorado
cooked: <p>What if we update the learning rate using backprop? Why is it a bad idea?</p>
<p>something like this:</p>
<pre><code class="lang-auto">import torch
import numpy as np
torch.manual_seed(42)
np.random.seed(42)

N = 64

alpha = 1.3
beta = np.array([[1.9],[1.5]])#

x_data = np.random.randn(N, 2)#
y_data = x_data.dot(beta) + alpha

x=torch.from_numpy(x_data).float()
y=torch.from_numpy(y_data).float()

w_beta = torch.randn((2, 1), requires_grad=True)
w_alpha = torch.randn(1, requires_grad=True)

learning_rate = torch.tensor(0.01, requires_grad=True)

torch.autograd.set_detect_anomaly(False)
for t in range(10):
    learning_rate.detach_().requires_grad_()
    w_alpha.detach_().requires_grad_()
    w_beta.detach_().requires_grad_()

    y_pred = x.mm(w_beta).add(w_alpha)
    loss = (y_pred - y).pow(2).sum()

    
    loss.backward(retain_graph=True, create_graph=True)
    print(loss.item())
    saved_w_beta_gra =  w_beta.grad.detach().clone()
    saved_w_alpha_gra =  w_alpha.grad.detach().clone()
    w_beta2 = w_beta - (w_beta.grad*learning_rate)
    w_alpha2= w_alpha - (w_alpha.grad*learning_rate)


    y_pred = x.mm(w_beta2).add(w_alpha2)
    loss = (y_pred - y).pow(2).sum()
    
    loss.backward(retain_graph=True, create_graph=True)

    print("HERE",learning_rate,learning_rate.grad)
    with torch.no_grad():
        learning_rate -= 0.0000001*learning_rate.grad
        w_beta = w_beta - (saved_w_beta_gra*learning_rate)
        w_alpha= w_alpha - (saved_w_alpha_gra *learning_rate)
</code></pre>

username: kolorado
cooked: <p>There is a paper that does a similar thing:</p><aside class="onebox pdf">
  <header class="source">
      <a href="https://assets.amazon.science/33/da/4a72eda040a5ab7ac970aa03b858/learning-the-learning-rate-for-gradient-descent-by-gradient-descent.pdf" target="_blank" rel="noopener nofollow ugc">assets.amazon.science</a>
  </header>
  <article class="onebox-body">
    <a href="https://assets.amazon.science/33/da/4a72eda040a5ab7ac970aa03b858/learning-the-learning-rate-for-gradient-descent-by-gradient-descent.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://assets.amazon.science/33/da/4a72eda040a5ab7ac970aa03b858/learning-the-learning-rate-for-gradient-descent-by-gradient-descent.pdf" target="_blank" rel="noopener nofollow ugc">learning-the-learning-rate-for-gradient-descent-by-gradient-descent.pdf</a></h3>

<p class="filesize">495.89 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Besides: It needs too much memory! and I guess the second learning-rate is really sensitive</p>
Custom Layer implementation

username: Itamar_Eliyahu
cooked: <p>Hi All<br>
I want to define a custom layer ,quite similar to MaxPool2D.<br>
The custom layer returns the left up corner value from 2x2 tile instead of the maximum value as returns by MaxPool2D.<br>
I thought to use Conv2D layer , with predefine kernel ( which is 2x2 matrix with 1 in the top left corner and the others are zeros) , stride of 2 without backprop.</p>
<p>any other suggestion?</p>
Multiple loss backward twice

username: Zecheng_Zhang
cooked: <p>Hello ,</p>
<p>My loss function contains three parts loss = p1* loss1+p2* loss2+p3* loss3<br>
I want to first get the gradient of the parameters of loss1, loss2, loss3 individually (for some purpose) and then update the net parameters by optimizing the total loss, i.e.,</p>
<p>loss1.backward()<br>
(l1 is a layer in my network)<br>
loss1.l1.weight.grad</p>
<p>loss2.backward()<br>
loss2.l1.weight.grad</p>
<p>loss3.backward()<br>
loss3.l1.weight.grad</p>
<p>no optimization involved up to now and I just want to know the gradient of each loss function. and then I want to update the parameters of the network by<br>
opt.zero_grad()<br>
loss.backward()<br>
opt.step()</p>
<p>However, it gives me the error message and ask me to use retain_graph=True when I first call backward() in each individual loss function.</p>
<p>There will be no bug if I use retain_graph=True, however, is this the correct way to solve the problem? Is there any simpler way for me to get the gradient of each loss function and then update the parameters by optimizing the total loss function? Thank you very much!</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/zecheng_zhang">@Zecheng_Zhang</a>,</p>
<aside class="quote no-group" data-username="Zecheng_Zhang" data-post="1" data-topic="107681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zecheng_zhang/48/28223_2.png" class="avatar"> Zecheng_Zhang:</div>
<blockquote>
<p>loss = p1* loss1+p2* loss2+p3* loss3</p>
</blockquote>
</aside>
<p>A better way to this would be.</p>
<pre><code class="lang-auto">loss = loss1 + loss2 + loss3
</code></pre>
<p>But I don’t think this link here makes any sense.</p>
<aside class="quote no-group" data-username="Zecheng_Zhang" data-post="1" data-topic="107681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zecheng_zhang/48/28223_2.png" class="avatar"> Zecheng_Zhang:</div>
<blockquote>
<p>loss1.l1.weight.grad</p>
</blockquote>
</aside>
<p>and the rest because loss won’t have l1 as it’s attribute.</p>

username: Zecheng_Zhang
cooked: <p>Hi, Usama,</p>
<p>The p1, p2 and p3 are the weights for three losses. I need to update the weights everytime when I calculate the gradient of three losses. Thanks</p>

username: Usama_Hasan
cooked: <p>I see so you’ve loss function defined custom with weights. In that case the above statement should help.</p>
Torch.sqrt() and other math functions requiring tensor inputs

username: marcman411
cooked: <p>I run into additional type checking frequently when I try to write a function that uses these math functions. For example: let’s say I want to have one utility function that computes Euclidean distance. Across my whole project, I might use it on tensors or Python scalars.</p>
<p>For Python scalars, I have to use something like:</p>
<pre><code class="lang-auto">import math
def euclidean_distance(a, b):
    """a and b are either Python scalars or tensors with size 1"""
    return math.sqrt(a**2 + b**2)
</code></pre>
<p>This will work for all Python scalars and all torch tensors of size 1. But for PyTorch tensors, generally, I have to use:</p>
<pre><code class="lang-auto">import torch
def euclidean_distance(a, b):
    """a and b are arbitrary tensors"""
    return torch.sqrt(a**2 + b**2)
</code></pre>
<p>Is there some way to allow for:</p>
<pre><code class="lang-auto">import torch
def euclidean_distance(a, b):
    """a and b are arbitrary tensors or Python scalars"""
    return torch.sqrt(a**2 + b**2)
</code></pre>
<p>without uses having to check like:</p>
<pre><code class="lang-auto">import torch
def euclidean_distance(a, b):
    """a and b are arbitrary tensors or Python scalars"""
    # Handle tensor vs not tensor inputs
    a_not_tensor = False
    if not torch.is_tensor(a):
        a_not_tensor = True
        a = torch.tensor([a])
    b_not_tensor = False
    if not torch.is_tensor(b):
        b_not_tensor = True
        b = torch.tensor([b])

    # Compute the distance
    dist = torch.sqrt(a**2 + b**2)

    # If both inputs are Python scalars, return a Python scalar
    if a_not_tensor and b_not_tensor:
        return dist.item()

    # If either input was a tensor, return a tensor
    return dist
</code></pre>
<p>Is there a reason this functionality doesn’t exist? It’s not limited to <code>torch.sqrt()</code>, but all PyTorch math functions. I imagine it’s easier to have <code>torch.sqrt()</code> accept Python scalars then to automatically vectorize <code>math.sqrt()</code> (a la Python build-ins like <code>abs()</code>), but I suppose either would solve the problem.</p>
<p>[An aside: this is kind of related to the lack of a <code>torch.pi</code> constant (<a href="https://discuss.pytorch.org/t/np-pi-equivalent-in-pytorch/67157">1</a>, <a href="https://discuss.pytorch.org/t/is-there-the-pi-greek-3-1415-defined-somewhere/30283">2</a>, <a href="https://github.com/pytorch/pytorch/issues/6510" rel="nofollow noopener">3</a>). Requiring <code>import math</code> for certain things is fine to remove redundant constants, etc, but it would also be useful to provide the cross-functionality when needed.]</p>

username: mishanya
cooked: <p>I’m severely missing every single point you mention. Are there any issues on github related to them? Didn’t find any.</p>
Regarding clamped learnable parameter

username: vainaijr
cooked: <p>is this a correct way to clamp a learnable parameter in a range of 0-1?</p>
<pre><code class="lang-auto">z = nn.Parameter(torch.clamp(torch.rand(1), 0, 1))
</code></pre>
<p>if I want to set a threshold as a learnable parameter, and clamp it in a range, is this correct way to do it?</p>

username: spanev
cooked: <p>Hey <a class="mention" href="/u/vainaijr">@vainaijr</a>,</p>
<p>Would gradient clipping with <a href="https://pytorch.org/docs/stable/_modules/torch/nn/utils/clip_grad.html" rel="nofollow noopener">clip_grad</a> work for you? You can call it on the model parameters before the optimizer step.</p>

username: vainaijr
cooked: <p>No, I do not think clip_grad will work, what I want to do is, zero out elements of a matrix, that fall below a threshold, but this threshold is not fixed, it is learnable, I do not set it, model needs to learn this threshold, and zero out elements of matrix that fall below this threshold.</p>

username: xpu
cooked: <p>The following code works for me:</p>
<pre><code class="lang-auto">class Clamp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return input.clamp(min=0, max=1) # the value in iterative = 2

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output.clone()

clamp_class = Clamp()
</code></pre>
<p>and in nn.Module:</p>
<pre><code class="lang-auto">self.z = nn.Parameter(torch.tensor(1.0), requires_grad=True)
clamp_class.apply(self.z)
</code></pre>
Data Reading During Hyperparameter Tuning

username: kakashi-hadakei
cooked: <p>I try to tune hyperparameters of a neural network that I set up. The plan is to start 5 independent training for each set of hyperparameters. During each training process, I will open 8~16 child processes to read data. Apparently I am opening too many child processes just in order to read the same dataset. I need a way to optimize this setup, particularly data reading. Any advice would be greatly appreciated.</p>
Accuracy and robustness trade-off

username: stases
cooked: <p>Hello!</p>
<p>I am currently working on an autoencoder dealing with sparse data. I am training AE on background noise and then using this I am looking for anomalies in other datasets. The assumption is, that if an anomaly exists, autoencoder will make a large error. However, this doesn’t work very well because some examples have bigger error than an error of the signal sample. My question is, is there a way to make a trade-off during training between average error (accuracy) and the spread of errors (reduce standard deviation of the error distribution, robustness)?</p>
<p>Thank you in advance.</p>
Training problem: cons layers weight converges to zero

username: lifeblack
cooked: <p>Hi…  after training my code, I noticed that conv layers weights converged to zero and just linear layers worked. I changed activation function, kernels but  nothing changed.<br>
how could I fix this problem</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1= nn.Sequential(
            nn.Conv2d(1,64, kernel_size=(11,11), stride=1,padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU()
        )
        self.conv2= nn.Sequential(
            nn.Conv2d(64,32, kernel_size=(1,1), stride=1,padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU()
        )
        self.conv3= nn.Sequential(
            nn.Conv2d(32,1, kernel_size=(7,7), stride=1,padding=3),
            nn.BatchNorm2d(1),
            nn.ReLU()
        )
        self.conv4= nn.Sequential(
            nn.Conv2d(1,64, kernel_size=(11,11), stride=1,padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU()
        )
        self.conv5= nn.Sequential(
            nn.Conv2d(64,32, kernel_size=(1,1), stride=1,padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU()
        )
        self.conv6= nn.Sequential(
            nn.Conv2d(32,1, kernel_size=(7,7), stride=1,padding=3)
              
        )
     
        self.fc1=nn.Linear(1600,64)
        self.fc2=nn.Linear(64,64)

    def forward(self,kr):
        out_conv1=self.conv1(kr)
        out_conv2=self.conv2(out_conv1)
        out_conv3=self.conv3(out_conv2)
        out_conv4=self.conv4(out_conv3)
        out_conv5=self.conv5(out_conv4)
        out_conv6=self.conv6(out_conv5)
        out_feedback=kr+out_conv6
        out_linear=self.fc1(out_feedback.flatten(2))
        out_linear=self.fc2(out_linear)
        x=out_linear.reshape([out_linear.shape[0],out_linear.shape[1],8,8])               
        return x
</code></pre>

username: AlperEkmekci
cooked: <p>Hello,</p>
<p>It seems you’re using a model structure like concatenating two encoder-decoder type network which each one has a 1 depth. You are probably facing <code>vanishing gradient</code> problem so for being sure about that you can control output of each layer.</p>

username: lifeblack
cooked: <p>thanks a lot <a class="mention" href="/u/alperekmekci">@AlperEkmekci</a> for your reply… yes I am facing vanishing gradient problem… Could you mind helping me solve this? what is your suggestion?</p>

username: AlperEkmekci
cooked: <p>Hello,</p>
<p>It seems you gradient value is vanishing after you layer named <code>self.conv4</code>. Your structure is approaching like Encoder-Decoder model, because of that you can use a method which I recommend to you is that implement Residual Networks (a.k.a. ResNet ) to your structure for avoid vanishing gradient.</p>
<p>Besides decreasing <code>learning rate</code> will help you to avoid that problem.</p>
Concatenate two patches

username: randinoo
cooked: <p>this code  does not work !! <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>
<pre><code class="lang-auto"> patches_image1 = image.extract_patches_2d(image1['taille'], ((10,10)))
 patches_image2 = image.extract_patches_2d(image2['taille'], ((10,10)))
 patchesconcatenate = torch.cat((patches_image1, patches_image2),1)
</code></pre>
<pre><code class="lang-auto">TypeError: expected Tensor as element 0 in argument 0, but got numpy.ndarray
</code></pre>
<p>help me plzz</p>

username: fs4ss1
cooked: <p><code>torch.cat()</code> function expects <code>torch.tensor</code> as input, not <code>np.ndarray</code>.</p>
<p>You need to cast <code>patches_image1</code> and <code>patches_image2</code> to torch.tensor.</p>
<p>The following way will work: <code>torch.tensor(patches_image1)</code> and <code>torch.tensor(patches_image1)</code>.</p>

username: randinoo
cooked: <p>thanks  fs4ss1 but  I have another problem</p>
<pre><code class="lang-auto">train_x, val_x, train_y, val_y = train_test_split(train_x, train_y, test_size=0.3)
train_x = train_x.reshape(train_x.shape[0], 1, train_x.shape[1], train_x.shape[2])
train_x = torch.from_numpy(train_x)
</code></pre>
<pre><code class="lang-auto">TypeError: expected np.ndarray (got Tensor)
</code></pre>
<p>help me please</p>
From Conv2d->MaxPool2d to Linear in_features

username: Malathy_Muthu
cooked: <p>Hi,<br>
I have an image dataset, each image size of  28 * 28 * 3. I am passing it through two Conv2d/MaxPool2d layer as follows</p>
<pre><code>    self.nn = nn.Sequential(
        nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1),#28*28*32
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2), #14*14*32

        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3), #12*12*64
        nn.BatchNorm2d(64),
        nn.ReLU(),
        nn.MaxPool2d(2), #10*10*64

        nn.Flatten(),
        nn.Linear(in_features=???) 
</code></pre>
<p>I am stuck at what value I need to pass the in_features. If I pass 6400, I get dimension error.<br>
RuntimeError: mat1 dim 1 must match mat2 dim 0</p>
<p>Can someone please help me understand where I am wrong?</p>
<p>Thank you!</p>

username: JuanFMontesinos
cooked: <p>flatten converst the whole tensor into a vector. Linear requires a batch of vectors<br>
You need to do tensor.reshape(-1,10<em>10</em>64) in the forward</p>

username: Malathy_Muthu
cooked: <p>Since I am calling it in nn.Sequential, it doesn’t allow me to add nn.reshape(-1,10<em>10</em> 64) after nn.Flatten().I get AttributeError: module ‘torch.nn’ has no attribute ‘reshape’</p>

username: JuanFMontesinos
cooked: <p>You have to call in in the forward and remove the flatten.<br>
soo if you have a sequence like:</p>
<pre><code class="lang-auto">x=self.layer(x)
x=self.layer2(x)
# add here
x = x.reshape(-1,10 * 10 * 64)
</code></pre>

username: Malathy_Muthu
cooked: <p>Thank you. What I am unable to understand is from my calculation, I get 6400 (64 * 10 * 10), for the input features for the linear call, but the number of input features that works fine is 2304, instead of 6400. In that case the Flatten and the nn.linear work perfectly fine.</p>
<p>nn.Linear(in_features=2304, out_features=512), =&gt; Can you please help me understand how the input features are 2304, (instead of 6400)?</p>
<p>Following is the steps of Conv2d and max2d done (twice) before passing to the linear.</p>
<pre><code>    nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1),#28*28*32
    nn.BatchNorm2d(32),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2), #14*14*32

    nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3), #12*12*64
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2), #10*10*64 I get 6400, but the linear works for 2304.
    nn.Linear(in_features=2304, out_features=512)</code></pre>

username: Malathy_Muthu
cooked: <p>Finally understood where I went wrong,  just declaring  nn.MaxPool2d(2) takes the kernel size as well as the stride as 2. I was expecting it to take the stride as 1 by default. So, in that case, the output size from the Max2d becomes 6<em>6. So 6</em>6*64 becomes 2304.</p>
Are VGG network trained with [0, 255] or [0.0, 1.0] tensor?

username: delcomp
cooked: <p>There are two explanations in the example of neural network style transfer.<br>
1: Now we will import the style and content images. The original PIL images have values between 0 and 255, but when transformed into torch tensors, their values are converted to be between 0 and 1. The images also need to be resized to have the same dimensions. An important detail to note is that neural networks from the torch library are trained with tensor values ranging from 0 to 1. If you try to feed the networks with 0 to 255 tensor images, then the activated feature maps will be unable to sense the intended content and style. However, pre-trained networks from the Caffe library are trained with 0 to 255 tensor images.<br>
do you need to change the tensor from [0,1] to [0，255] before feeding to VGG networks?<br>
2:Additionally, VGG networks are trained on images with each channel normalized by mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225]. We will use them to normalize the image before sending it into the network.<br>
This shows that the tensor fed into the VGG network is still [0,1].</p>

username: fs4ss1
cooked: <p>Here is your answers: <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/models.html</a></p>
<blockquote>
<p>[…] The images have to be loaded in to a range of [0, 1] […]</p>
</blockquote>
Change forward path at certain probability

username: 111429
cooked: <p>Hi,<br>
Let’s say I have a Bi-autoenocder, each stream conducts a standard AE thing.</p>
<pre><code class="lang-auto">class BiAutoencoder(nn.Module):
    def __init__(self):
        super(BiAutoencoder, self).__init__()
        self.encoder1 = nn.Sequential(
            nn.Linear(28 * 28, 128),
            nn.ReLU(True),
            nn.Linear(128, 64),
            nn.ReLU(True), nn.Linear(64, 12), nn.ReLU(True), nn.Linear(12, 3))
        self.decoder1 = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.ReLU(True), nn.Linear(128, 28 * 28), nn.Tanh())
        self.encoder2 = nn.Sequential(
            nn.Linear(28 * 28, 128),
            nn.ReLU(True),
            nn.Linear(128, 64),
            nn.ReLU(True), nn.Linear(64, 12), nn.ReLU(True), nn.Linear(12, 3))
        self.decoder2 = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.ReLU(True), nn.Linear(128, 28 * 28), nn.Tanh())

    def forward(self, x1, x2):
        x1 = self.encoder1(x1)
        x1_hat = self.decoder1(x1)
        x2 = self.encoder2(x2)
        x2_hat = self.decoder2(x2) 
       return x1_hat, x2_hat

</code></pre>
<p>Now I want to set a certain probability while training that the outputs of encoders forward through the other decoder.<br>
That is, like 50% chance, the forward pipeline goes normally.<br>
But at 50% chance, the forward would be like this.</p>
<pre><code class="lang-auto">    def forward(self, x1, x2):
        x1 = self.encoder1(x1)
        x2_hat = self.decoder2(x1)
        x2 = self.encoder2(x2)
        x1_hat = self.decoder1(x2) 
       return x1_hat, x2_hat
</code></pre>
<p>Any suggestion?Thanks.</p>

username: fs4ss1
cooked: <p>I think this should work (I don’t tested):</p>
<pre><code class="lang-auto">  import random
  def forward(self, x1, x2):
        path = random.choice([True, False])
        if path:
            x1 = self.encoder1(x1)
            x2_hat = self.decoder2(x1)
            return x2_hat
        else:
            x2 = self.encoder2(x2)
            x1_hat = self.decoder1(x2) 
            return x1_hat
</code></pre>
How to pad variable length input tensor representing text

username: neuralpat
cooked: <p>Hi,</p>
<p>This is my model:</p>
<p>ImdbReviewModel(<br>
(embed): Embedding(95423, 30)<br>
(gru): GRU(30, 128)<br>
(fc1): Linear(in_features=128, out_features=1, bias=True)<br>
)</p>
<p>I’m wanting to do sentiment analysis on the imdb dataset, but I’m having trouble batching the data.<br>
Obviously, review all have different lengths which would not be a problem for the embedding layer, but dataloader is not able to batch it.</p>
<p>Now, I think that I’m supposed to to pad my tensors but I really don’t  understand how. I’ve looked at a lot of threads in this forum and I’ve looked at the docs but I have no idea what to do.<br>
I guess I have to write my own collate_fn and pad inside it but I’m not sure which function I should use for padding and what that function has to return.</p>
<p>Also it seems that packed-padding is meant for RNNs, but my first layer is an embedding layer so maybe that’s not even the right thing?</p>
<p>Help would be very much appreciated.</p>
<p>This is my current code:</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_dataset.dataset, 10, shuffle=True, collate_fn=collate_fn_padd)
</code></pre>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    all_inputs = []
    all_labels = []

    for sample in batch:
        input, label = sample
        all_inputs.append(input)
        all_labels.append(label)
    batch = torch.nn.utils.rnn.pad_sequence(all_inputs)

    batch = list(zip(batch, all_labels))

    return batch
</code></pre>
<p>2 problems with that code. 1. the tensors don’t actually seem padded (no zeros), 2. that gives a list, not a “proper” batch, that I can feed to the network right?</p>

username: neuralpat
cooked: <p>Okay, I solved the issue. I padded my tensors with this function (inside the dataset):</p>
<pre><code class="lang-auto"> def pad_tensor(t):
     t = torch.tensor(t)
     padding = max(max_review_len) - t.size()[0]
     t = torch.nn.functional.pad(t, (0, padding))
     return t
</code></pre>
Can someone tell me what's wrong with this notebook?

username: Hacking_Pirate
cooked: <p>Hi, I am participating in a competition on hacker earth. But when I create my submission file all of the predicted classes are coming same. I would be very grateful if someone could have a look at my notebook and figure out if something is wrong?</p>
<p>Here is the link:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201222-085609-RC00_348636243" class="site-icon" width="128" height="128">
      <a href="https://colab.research.google.com/drive/1J0bJJG0y5cLm9qEr77Yty5zSAGYaYnKS?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1J0bJJG0y5cLm9qEr77Yty5zSAGYaYnKS?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Here is the competitions link:<br>
<a href="https://www.hackerearth.com/challenges/competitive/hackerearth-deep-learning-challenge-holidays/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.hackerearth.com/challenges/competitive/hackerearth-deep-learning-challenge-holidays/</a></p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Reuben_Wenisch
cooked: <p>You will have to give access to the file for the community to help.</p>
Testing my GPU with looping code example

username: NoamAS
cooked: <p>Hello everyone</p>
<p>I would like to run python code in loop to test my GPU . Can anyone share a code with me? I don’t have any knowledge in programming and couldn’t fid an example in the net yet .<br>
I’m using Pytorch 1.6 suitable for CUDA 10.2 , Python 3.7.9 and Quadro RTX 4000 . In addition I’m using Spyder 4.2.1 . Host system is based on Windows server 2019</p>
<p>Thank you</p>

username: Reuben_Wenisch
cooked: <p>Could you define what you mean by in loop? Is it a training or testing loop?<br>
Assuming you want to benchmark your GPU you can use the this repo</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/ryujaehun/pytorch-gpu-benchmark" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/22635090?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/ryujaehun/pytorch-gpu-benchmark" target="_blank" rel="noopener nofollow ugc">ryujaehun/pytorch-gpu-benchmark</a></h3>


  <p><span class="label1">Using the famous cnn model in Pytorch, we run benchmarks on various gpu.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Huge input size in RNN

username: Sine
cooked: <p>Hi everyone.<br>
My data size  is [32,10,40,300] ([batch size, context length, utterance length, embedding dimension]).<br>
However, the input size of RNN is [batch size, sequence length, embedding dimension].<br>
Thus, I reshape my data to [32,10,40 * 300] ([batch size, context length, utterance length * embedding dimension]) to learn the context representation.<br>
Is there any impact when I input this huge matrix into the RNN?</p>
<p>Another way I thought of is to use CNN + max pool  to get the feature and let input size become [32,10,200]   ([batch size, context length, feature dimension]) before inputting to the RNN.<br>
Is the second way better than the first?</p>
<p>Thanks.</p>
Cuda error `THCNumerics<AccT>::gt(sum, accZero)`

username: haorannlp
cooked: <p>I wrote a seq2seq model which requires sampling during training.<br>
I guess this error might be related to the sampling procedure but I’m not sure what this error message suggests about the multinomial distribution.<br>
<code>/opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCTensorRandom.cuh:179: void sampleMultinomialOnce(long *, long, int, T *, T *, int, int) [with T = float, AccT = float]: block: [68,0,0], thread: [0,0,0] Assertion `THCNumerics&lt;AccT&gt;::gt(sum, accZero)` failed.</code></p>
<p>Is any familar with this kind of error?</p>

username: tom
cooked: <p>This means that the categories you sample from all have probability mass 0.</p>
<p>Best regards</p>
<p>Thomas</p>
Multiprocessing with GPU less optimal?

username: hereditary
cooked: <p>Hi, I’m trying to use multiprocessing for my model processing.<br>
The reason why I feel like multiprocessing would be a good fit is that running the process, each individual task is finished extremely fast, so I’m hoping to distribute this work as there is extra bandwidth on the GPU and machine.<br>
When I try to incorporate multiprocessing, processing slows down dramatically.</p>
<p>Without multiprocessing a GPU is at 8GB/10GB of usage.<br>
When using multiprocessing, the GPU hits around 1.5GB/10GB of usage.<br>
Seems odd. Any help is appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’ve attached a skimmed version of what I am currently using.</p>
<pre><code class="lang-auto">from torch.multiprocessing import Pool
def model_process(chunk, segment):
    img = segment['image']
    img = img.to(device, dtype=torch.float)
    output = model(img)
    output = F.interpolate(output, (N, M))
    for i in range(len(output)):
        x = output[i, 0, :, :].detach().cpu().numpy()
        y = output[i, 1, :, :].detach().cpu().numpy()


def process_batch(dataloader):
    device = torch.device("cuda:0")
    model = torch.load(model_file, map_location=device)
    model.eval()
    model.share_memory()
    chunk = np.uint16(np.zeros(N, M))
    pool = Pool(8)
    pool.map(partial(model_process, chunk), dataloader)
    pool.close()
    pool.join()
    return
</code></pre>