akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compute the classification accuracy of each class?

panovr: <p>After training, one can get the average accuracy on the “val” dataset with " --evaluate" option. May I ask that how to compute the classification accuracy of each class? Thus, one can check to see which class is more difficult to be classified.</p>

smth: <p>you’ll have to manually populate this by changing the code.</p>

RajAt_SiNha: <p>Can you explain how classification accuracy is calculated for each class more briefly as I am very new to Pytorch.</p>
<p>I have two classes Positive and Negative and using Transfer Learning.</p>

shashankvkt: <p>You can do that by computing the confusion matrix.</p>
Torch.cuda.is_available() returns False, but cudnn is present

markorei94: <p>So, since recently torch.cuda.is_available() return False, but torch.backends.cudnn.enabled gives True.<br>
nvidia-smi says driver version is 417.98, cuda version is 10.0.<br>
Other version info:</p>
<ul>
<li>
<p>Windows 10</p>
</li>
<li>
<p>pytorch 1.3.1</p>
</li>
<li>
<p>cuda toolkit 10.1.243</p>
</li>
<li>
<p>cudnn 7.6.5</p>
</li>
</ul>
<p>It used to work properly till recently. I don’t see what is the problem now.</p>

ptrblck: <p>Do you know what changed in your setup, which might have broken the working version?<br>
Did you install any new drivers etc.?</p>
<p>The local CUDA installation won’t be used, if you install the binaries with <code>cudatoolkit</code>.<br>
Based on your information, it seems you have a local CUDA10.0 installation, while the PyTorch binaries ship with CUDA10.1.<br>
Based on <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">this table</a> you would need an NVIDIA driver of <code>&gt;= 418.39</code> to use CUDA10.1.<br>
Did you update PyTorch and swapped CUDA10.0 for 10.1?<br>
If so, you would have to update the driver.</p>

markorei94: <p>I updated driver to version 441.66 and now it works. It seems that some 10 days ago Windows update downgraded driver version for some reason, perhaps because of Cuda version being 10.0. Now that is also updated to 10.2, so hopefully it should stay fine.</p>
Training Fine but 0 at validation for Segmentation

Geeks_Sid: <p>Hi All,</p>
<p>I have been training this pretrained resnet50 encoder unet decoder for segmentation and for some reason, it seems to be working fine for training where the dice after first epoch is 0.6 but the validation dice for segmentation is at 1e-5 instead of being atleast 0.1.<br>
Can anyone tell me what part of code should I look at? I have already check the validation samples manually.</p>

ptrblck: <p>How does the training and validation losses look like during training?<br>
Do you see signs of overfitting or is the validation loss never going down?</p>

Geeks_Sid: <p>So the dice goes from 0.6 to 0.8 for training in the first 5 epochs but for validation, it sticks to 1e-4 for first two epochs followed by moving to 0.2 and then coming back to 0.1.</p>
<p>Although, at first, it seems like a classic case of overfitting, but I have a training set of 200000 images as compared to 30000 for validation and I have also turned on augmentation for it.</p>
<p>Thoughts?</p>
Not make a prediction

vainaijr: <p>How to make my neural network not predict anything, for example if I am training on cat dog classifier, and in the dataset, there is one image, that is neither cat nor dog, image is noise, then instead of making an incorrect prediction, I want neural network to not make any prediction, like unattempted question.</p>

ptrblck: <p>You could treat this class as a special “unknown” class (and train your model to predict it for these noise images) or alternatively you could treat your use case as a multi-label classification, i.e. each class will get a probability and your model might output zero, one or multiple classes for each sample.</p>
Customizing DataParallel

tueboesen: <p>I am running a reversible network and wish to use DataParallel on it, however my model has the following methods which I need:<br>
forward<br>
reverse<br>
backward</p>
<p>Example of my network:</p>
<pre><code class="lang-auto">class reversenet(nn.Module):
def __init__(self, args):
    super().__init__()
    self.model = model

def forward(self, X):
    Y, Yo = self.model(X)
    return Y, Yo


def backward(self, Y, Yo, dY, get_optim):
    """ Compute gradients for the model parameters using the reversible property of the network to 
    re-calculate activation. Gradients are updated in place

    Arguments:
        YN {torch.Tensor} -- Final state from the forward pass
        Yo {torch.Tensor} -- Second last state from the forward pass
        dY {torch.Tensor} -- Derivative of YN
        get_optim {function} -- Function that returns an optimizer
    """
    
    return Y, Yo


def reverse(self, YN, Yo):
    """ Use the reversible property of the network to re-calculate the input to the forward pass.

    Arguments:
        YN {torch.Tensor} -- Final state from the forward pass
        Yo {torch.Tensor} -- Second last state from the forward pass

    Returns:
        torch.Tensor -- Recovery of initial state
        torch.Tensor -- Recovery of second state
    """

    Y = Yo
    Yo = YN


    return Y, Yo
</code></pre>
<p>But when I use the standard DataParallel function on this I lose access to the reverse routine.</p>
<p>I could create a second inverse model, which reuses the same parameters as the first model, and has its forward method being the reverse operation from the original model, but that seems rather convoluted to me and not all that pretty.</p>
<p>So I was wondering whether anyone has any experience customizing DataParallel to accept more methods?</p>
TypeError: stat: path should be string, bytes, os.PathLike or integer, not cv2.VideoCapture

hunar: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/59955290/typeerror-stat-path-should-be-string-bytes-os-pathlike-or-integer-not-cv2-v" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/12652518/artiest" target="_blank" rel="nofollow noopener">
    <img alt="artiest " src="https://www.gravatar.com/avatar/77b32a239043dcb22b0c194a8ad0cd7a?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/59955290/typeerror-stat-path-should-be-string-bytes-os-pathlike-or-integer-not-cv2-v" target="_blank" rel="nofollow noopener">TypeError: stat: path should be string, bytes, os.PathLike or integer, not cv2.VideoCapture ,detect using neural network using mtcnn?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, face-detection, face-recognition</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/12652518/artiest" target="_blank" rel="nofollow noopener">
    artiest 
  </a>
  on <a href="https://stackoverflow.com/questions/59955290/typeerror-stat-path-should-be-string-bytes-os-pathlike-or-integer-not-cv2-v" target="_blank" rel="nofollow noopener">06:54PM - 28 Jan 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
My model's output suddenly flips

torch_lover: <p>During training, my model output occasionally flips and loss skyrockets.<br>
From the fact that r^2 is maintained and AUC becomes 1-AUC, I can the output have flipped.</p>
<p>Can you guess what is the problem?</p>
<p>Here is my model</p>
<pre><code class="lang-auto"> Custom_Model(
  (submodule_0): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=34.066230764894875)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.4, inplace=False)
  )
  (submodule_1): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=21.358266163124966)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.6, inplace=False)
  )
  (submodule_2): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=7.067527196888563)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.6, inplace=False)
  )
  (submodule_3): Sequential(
    (0): Linear(in_features=17741, out_features=1024, bias=True)
    (1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=1024, out_features=256, bias=True)
    (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (6): ELU(alpha=1.0)
    (7): Dropout(p=0.2, inplace=False)
    (8): Linear(in_features=256, out_features=1, bias=True)
  )
)
</code></pre>
<p>Optimizer</p>
<pre><code class="lang-auto">Adagrad (
Parameter Group 0
    eps: 1e-10
    initial_accumulator_value: 0
    lr: 0.01
    lr_decay: 0
    weight_decay: 3e-07
)
</code></pre>
<p>Log</p>
<pre><code class="lang-auto">VALID: epoch: 14-852(99.88%), loss: 0.16498, cached: 63.937
VALID: output           r2: 0.000241, auc: 0.56510
TRAIN: epoch: 15-2556(99.96%), loss: 0.00577, cached: 63.986
TRAIN: output           r2: 0.097297, auc: 0.99790
VALID: epoch: 15-852(99.88%), loss: 0.16588, cached: 63.937
VALID: output           r2: 0.000255, auc: 0.56665
TRAIN: epoch: 16-2556(99.96%), loss: 0.00535, cached: 63.986
TRAIN: output           r2: 0.092888, auc: 0.99835
VALID: epoch: 16-852(99.88%), loss: 5.11513, cached: 63.937
VALID: output           r2: 0.000259, auc: 0.56450
TRAIN: epoch: 17-2556(99.96%), loss: 0.00452, cached: 63.986
TRAIN: output           r2: 0.100936, auc: 0.99893
VALID: epoch: 17-852(99.88%), loss: 14.84378, cached: 63.937
VALID: output           r2: 0.000285, auc: 0.43133
TRAIN: epoch: 18-2556(99.96%), loss: 0.00371, cached: 63.986
TRAIN: output           r2: 0.115433, auc: 0.99923
VALID: epoch: 18-852(99.88%), loss: 0.07149, cached: 63.937
VALID: output           r2: 0.000277, auc: 0.56678
</code></pre>
How to resolve the error for module not callable?

SHASHANK_KUMAR_MISHR: <p>On executing the below code<br>
config_path=‘config/yolov3.cfg’<br>
weights_path=‘yolo.weights’<br>
class_path=‘config/coco.names’<br>
img_size=416<br>
conf_thres=0.8<br>
nms_thres=0.4</p>
<h1>Load model and weights</h1>
<p>model = Darknet(config_path, img_size=img_size)<br>
model.load_weights(weights_path)<br>
model.cuda()<br>
model.eval()<br>
classes = utils.load_classes(class_path)<br>
Tensor = torch.cuda.FloatTensor</p>
<p>Following error occurred</p>
<p>TypeError                                 Traceback (most recent call last)<br>
~/Desktop/Surya/Environments/python3/project2_inltk/lib/python3.6/site-packages/pyforest/<strong>init</strong>.py in <br>
6 nms_thres=0.4<br>
7 # Load model and weights<br>
----&gt; 8 model = Darknet(config_path, img_size=img_size)<br>
9 model.load_weights(weights_path)<br>
10 model.cuda()</p>
<p>TypeError: ‘module’ object is not callable</p>

JuanFMontesinos: <p>That’s an import error<br>
You may need to do something like<br>
<code>from Darknet import Darknet</code><br>
In short you just did<br>
<code>import Darknet</code> but Darknet is a module and the class you are looking for is inside that</p>
Sanity Check Network Initialization

gabrieldernbach: <p>When setting up new models it can be helpful to have some sanity checks. One component of a model I find particularly hard to check is the initialization.</p>
<p>As I recall from Xavier and Kaiming initialization, the common approaches to weight initialization aims at conserving the variance from a layers inputs to outs.</p>
<p>That is, if we provide at standard normal (mean=0, variance=1) to some layer, we want it’s outputs to become a standard normal, too (mean=0, variance=1). This nicely aligns with the practice of standard normalization of the inputs and targets.</p>
<p>I therefore propose the following naive sanity check for the initialized network:<br>
( a ) create a large batch of standard normal data<br>
( b ) pass the data through the model<br>
( c ) check if the outputs mean and variance (over batch dimension) resembles a standard normal</p>
<p>Performing this check with a batch of 500 standard normals with the <code>torchvision.models.resnet50(pretrained=False)</code> however yields variances of 0.0044 to 0.0084.</p>
<p>I trust the official torchvision model and believe it is an excellent and working model.<br>
However how would you explain the reduced variance?</p>
<pre><code class="lang-auto">import torchvision.models as models
net = models.resnet.resnet18(pretrained=False)
y = net(torch.randn(n_samples, 3, 224, 224))
print(y.mean(dim=0), y.var(dim=0))
print(y.mean(dim=0).min(), y.mean(dim=0).max(), y.var(dim=0).min(), y.var(dim=0).max())</code></pre>
Torch.cholesky_solve is slower than torch.solve

zbh2047: <p>For a positive definite symmetric matrix A, the linear equation</p>
<p><code>Ax=b</code></p>
<p>can be solved either by torch.cholesky_solve or by torch.solve. We might expect torch.cholesky_solve is faster because it uses the positive definite property. However, torch.cholesky_solve is slower than torch.solve. For a matrix of shape 128*128, the time consuming is:</p>
<p>torch.cholesky_solve  10.0ms<br>
torch.solve  7.03ms</p>
<p>(NVIDIA GPU Titan X performance)</p>
<p>Why? So how to achieve the best performance? Thank you!</p>
Best way to quantize Transformer architecture

Francois_Hernandez: <p>Hi there,</p>
<p>I’ve been playing around with the experimental quantization features introduced in v1.3, and trying to apply it to some Transformer model.</p>
<p>Dynamic quantization works quite well, speeding up to a factor 1.5-2.5x depending on configurations.</p>
<p>As for static quantization, it seems some parts of the model are not compatible (LayerNorm, some div/matmul in the multi-head attention setup). I’m wondering what would be the best way of tackling such a case.</p>
<ul>
<li>Can we specify some submodules to ignore with Qconfig for instance?</li>
<li>Is there a roadmap to implement the missing operators and modules (matmul, div, nn.LayerNorm, etc.)?</li>
</ul>
<p>Thanks!<br>
François</p>

lly-zero-one: <p>Yes, you can set the corresponding module’s qconfig=None to bypass the quantization. If you just want to do it for some instances, you can also do that in the parent module’s init. Also make sure the quant and dequant are set correctly on the boundary.</p>
<p>For more operators support, we are working on enhancing this part. Also feel free to contribute.</p>

snakers41: <p>I am using a standard out-of-the-box nn.transformer module (it was a choice when 1.2 was released - I did not need BERT, so I decided to opt for a standard module) like this:</p>
<pre><code class="lang-auto">layer = nn.TransformerEncoderLayer(d_model=size,
                                   nhead=8,
                                   dim_feedforward=size * decoder_girth,
                                   dropout=dropout)
self.decoder = nn.TransformerEncoder(layer, decoder_layers
</code></pre>
<p>Now I have encountered this error. It is a bit cryptic, but is my assumption correct that the <a href="https://github.com/pytorch/pytorch/blob/ef5637f85e43dff55b03276176ba2949bd83a085/torch/nn/functional.py#L1362-L1383" rel="nofollow noopener">erroneous</a> function should just be replaced with a <code>nn.module</code> version? Or am I missing sth?</p>
<p>Maybe someone had this sort of error? It is probably going to be fixed in <code>1.5.0</code>, but I would need to monkey patch it somehow now</p>
How to calculate A*A^T efficiently

zbh2047: <p>I want to calculate A * A^T where A is an n * m matrix and the result will be an n * n matrix. Here m is very large, for example 10^6. The following code can simply do this:</p>
<p><code>B = torch.mm(A, A.T)</code></p>
<p>However, the following code can’t be the fastest method to do this. First, it doesn’t take the symmetric property of the resulting matrix into consideration (it may save half of the computation). Second, it may involve extra memory accessment because the two matrix A and A^T share the same storage.</p>
<p>Are there some other functions that can be more efficient to do the multiplication? If not, I think pytorch should provide such functions because calculating  A * A^T is an important and basic operation in practice.</p>
Libtorch using up memory regardless of NoGradGuard

Mikele: <p>Hello pytorch community,</p>
<p>(Libtorch c++ 1.4 version on mac and linux):<br>
I am currently facing the issue in my code, that iterative calls from my monte carlo tree search (building alphazero structure) to a neural network’s forward function accumulates massive amounts of memory although I have put NoGradGuard there.<br>
The forward method in question looks like this:</p>
<pre><code class="lang-auto">std::tuple&lt; torch::Tensor, double &gt; NetworkWrapper::evaluate(
   const torch::Tensor &amp;board_tensor)
{
   // We dont want gradient updates here, so we need the NoGradGuard
   torch::NoGradGuard no_grad;
   m_network-&gt;eval();

   auto [pi_tensor, v_tensor] = m_network-&gt;forward(board_tensor.to(GLOBAL_DEVICE::get_device()));

   return std::make_tuple(pi_tensor.detach(), v_tensor.template item&lt; double &gt;());
}
</code></pre>
<p>This forward method will evaluate the tensor using a convolutional net and a fully connected one.<br>
The <code>board_tensor</code> also has <code>requires_grad</code> set to false. There is also no gradient on the tensor after the forward method, so I am unsure what all this extra memory is used for.</p>
<p>The tree search which calls this evaluate method all the time will eat through 32 gb memory in no time. However if I put into my main:</p>
<pre><code class="lang-auto">torch::autograd::AutoGradMode guard(false);
</code></pre>
<p>then there is absolutely no excessive memory usage anymore.</p>
<p>What exactly does AutoGradMode control, that could lead to the memory excess?</p>

ezyang: <p>It’s hard to see without seeing your whole code, but I suspect the memory usage is not in <code>evaluate</code> and actually occurring somewhere else (and doesn’t have anything to do with which guard you are using.) You said, “if I put into my main”, did you mean actually in your main() function, or in evaluate()? Do you seem the same result when you replace AutoGradMode guard(false) with NoGradGuard?</p>

Mikele: <p>Yes the codebase is relatively big and impossible to post to a forum unfortuantely.<br>
I suspect you may be right though. Replacing the global (yes in main() function) autogradmode(false) with a grad guard removes the memory accumulation as well. Do you have an idea how to best track down a seemingly unknown call to torch?</p>
<p>Edit:<br>
I figured out where the hidden call to torch actually comes from. A clear fault on my end. Thank you for your help!</p>
How do I resize ImageNet image to 224 x 224?

YeongHwa_Jin: <p>Hi, I’m gonna train VGG16 on Imagenet data.<br>
Original VGG network have 224 x 224 input size, but each original Imagenet data have different size.<br>
So, what is the standard way to resize Imagenet data to 224 x 224?</p>

LeviViana: <p>This might help -&gt;<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#load-data" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#load-data</a></p>
<pre><code class="lang-python">from torchvision import transforms
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
</code></pre>

YeongHwa_Jin: <p>Thank you for your reply!</p>
Offset values of tensor by different amounts

lucf: <p>Hi everyone,</p>
<p>I have a challenging problem. At least I find it challenging!<br>
For an ecology problem, I have a grid (matrix) representing my environment, and values on each cell representing the local population.<br>
Every now and then, the population disperses: Each individual in each cell moves left or right (not up or down at this early stage) by a distance that is sampled from a normal distribution. I would like to vectorize the dispersion operation, at least for one iteration (i.e. for one individual per occupied cell).</p>
<p>For instance, let’s assume that before the dispersion, I have the grid below:</p>
<pre><code class="lang-auto">|  6  8 10  7  5  0  0  |
|  5  7  9 11  8  7  3  |
|  4  6  6  8  5  4  1  |
</code></pre>
<p>After one iteration, let’s say that one individual in the first row moved +2 (to the right), one individual in the second row by -1 (to the left) and two individuals in the third row moved by -3 and +1 respectively. These movements could be represented by a matrix as below, where each value represents the distance travelled by the first individual at that position (obviousy this would have to be repeated for the other individuals in each position):</p>
<pre><code class="lang-auto">|  0  0  0  0  2  0  0  |
|  0 -1  0  0  0  0  0  |
|  0  0  0 -3  0  1  0  |
</code></pre>
<p>This is what the new environment could look like:</p>
<pre><code class="lang-auto">|  6  8 10  7  4  0  1  |
|  6  6  9 11  8  7  3  |
|  5  6  6  7  5  3  2  |
</code></pre>
<p>Is there a way of providing such a matrix of offsets and modifying the original matrix without going though nested for loops? Note that the distance matrix approach is what I have in mind but there might be better ways of achieving this.</p>
<p>Thanks in advance!</p>

LeviViana: <p>This way:</p>
<pre><code class="lang-python">import torch

population = torch.tensor([
                   [6, 8, 10,  7, 5, 0, 0],
                   [5, 7,  9, 11, 8, 7, 3],
                   [4, 6,  6,  8, 5, 4, 1]
             ]
)

moves = torch.tensor([
                   [0, 0,  0,  0, 2, 0, 0],
                   [0, 0, -1,  0, 0, 0, 0],
                   [0, 0,  0, -3, 0, 1, 0]
             ]
)

right = (moves &gt; 0).nonzero(as_tuple=True)
left = (moves &lt; 0).nonzero(as_tuple=True)

mask = torch.zeros_like(population)

mask[right] = -1
mask[left] = -1

mask[right[0], right[1] + moves[right]] = 1
mask[left[0], left[1] + moves[left]] = 1

population += mask

print(population)
</code></pre>

lucf: <p>Brilliant, thanks so much!</p>
Implementation of batch norm

bananacode: <p>I am looking for where batch norm is computed in the code on the pytorch github, but I cannot find it anywhere. Could anybody link me to the file its implemented?</p>

yf225: <p><a class="mention" href="/u/bananacode">@bananacode</a> Are you looking for the kernel implementation of <code>batch_norm</code>, or are you looking for the Python / C++  API entry point of <code>batch_norm</code>?</p>

ezyang: <p>cuDNN bindings are at <a href="https://github.com/pytorch/pytorch/blob/700109eb630b79fd65cb93becb7f2d14f93bdb5c/aten/src/ATen/native/cudnn/BatchNorm.cpp">https://github.com/pytorch/pytorch/blob/700109eb630b79fd65cb93becb7f2d14f93bdb5c/aten/src/ATen/native/cudnn/BatchNorm.cpp</a></p>

bananacode: <p>Sorry I should’ve specified. I am looking for the CPU implementation of batch_norm.</p>
Actor Critic Implementation Problem, Rewards

ganeshsutar: <p>Hi,<br>
I am trying to implement a A2C plain and simple. I tried but not able find why the policy network in not getting better (rewards are not getting any better).</p>
<p>Not able to figure out whether the reward functions not correct or may be I am destroying gradient somewhere so that network is not able to train. Could any one help me to find the error in my implementation?</p>
<p><a href="https://github.com/ganeshsutar/breakout-a2c/blob/master/breakout-forum.py" rel="nofollow noopener">Github Project Link</a></p>
<p>My Agent action function <a href="https://github.com/ganeshsutar/breakout-a2c/blob/master/breakout-forum.py#L95" rel="nofollow noopener">breakout-forum.py – Line 95</a></p>
<p>My Train function <a href="https://github.com/ganeshsutar/breakout-a2c/blob/master/breakout-forum.py#L156" rel="nofollow noopener">breakout-forum.py – Line 156</a></p>
<p>Any help is appreciated.</p>
<p>Thanks</p>
Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false

Zuanazzi: <p>Hello, I am working on the implementation of an adversarial training. The following code does not work:</p>
<pre><code class="lang-auto">    for i, data in tqdm(enumerate(train_loader), total=len(train_loader), smoothing=0.9):

        pc1, pc2 = data
        pc1 = pc1.to(device).transpose(2, 1).contiguous().float()
        pc2 = pc2.to(device).transpose(2, 1).contiguous().float()

        # train the discriminator
        net_gen.eval()
        net_disc.train()

        i_odds = torch.arange(start=1, end=pc1.shape[2], step=2)
        ...

        # train the flow extractor
        net_gen.train()
        net_disc.eval()

        flow_pred = net_gen(pc1, pc2)
        pc_pred = pc1 + flow_pred

        z_pred = net_disc(pc1[:, :, i_odds], pc_pred[:, :, i_odds])
        loss_flow = - z_pred.mean()

        opt_gen.zero_grad()

        loss_flow.backward()
</code></pre>
<p>I get the following error on last line: <code>Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.</code></p>
<p>What could that be? For all I could check, the model and all the tensors are all type cuda.Float.</p>
<pre><code class="lang-auto">That is the traceback:
Traceback (most recent call last):
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/pdb.py", line 1667, in main
    pdb._runscript(mainpyfile)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/bdb.py", line 434, in run
    exec(cmd, globals, locals)
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 452, in &lt;module&gt;
    main()
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 445, in main
    train(args, net_flow, net_disc, train_loader, test_loader, boardio, textio)
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 309, in train
    train_stats = train_one_epoch(args, net_flow, net_disc, train_loader, opt_flow, opt_disc, writer, epoch)
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 227, in train_one_epoch
    loss_flow.backward()
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>

shaibagon: <p>See discussion on <a href="https://stackoverflow.com/a/59944875/1714410" rel="nofollow noopener">stackoverflow</a>.</p>
Help with Entity Embeddings in Pytorch

Akshay_Tiwari: <p>I tried to build Entity Embeddings of categorical data of a dataset.<br>
I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every month.The month column has 12 categories.</p>
<p>I wanted to make 12 Entity embeddings or 12 vectors, one for each month.I trained the embeddings against the bike share/rent/sales or i should say i used bike share/rent/sales as my labels.To visualise the embeddings I plotted the learned Embeddings/vectors<br>
On a 3D graph and I expected that the months with similar sales will be close together on the 3D graph but I didn’t get the expected results.<br>
In fact when I made the same model in Keras I got the results I was expecting.For example the batch size of 8 works very well in keras but same batch size increases loss in pytorch. In pytorch big batch size(40) reduces loss.</p>
<p>Would anyone be kind enough to have a look at my code in pytorch.It’s a very short code and will only take 10 minutes.</p>
<p>Here’s the link to my Pytorch code - <a href="https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity%20Embedding%20in%20Pytorch%202.ipynb" rel="nofollow noopener">https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity%20Embedding%20in%20Pytorch%202.ipynb</a></p>
<p>and keras code - <a href="https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity_Embeddings_keras.ipynb" rel="nofollow noopener">https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity_Embeddings_keras.ipynb</a></p>
<p>Here’s the data online - <a href="https://www.kaggle.com/marklvl/bike-sharing-dataset.(very" rel="nofollow noopener">https://www.kaggle.com/marklvl/bike-sharing-dataset.(very</a> small dataset)</p>
Question regarding the Neural Network tutorial

isalirezag: <p>In this tutorial (<a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Networks — PyTorch Tutorials 2.1.1+cu121 documentation</a>), it says that:</p>
<blockquote>
<p>Note: Expected input size to this net(LeNet) is 32x32. To use this net on MNIST dataset, please resize the images from the dataset to 32x32.</p>
</blockquote>
<p>I dont understand where in the network we define it that the input should be 32x32.</p>
<p>I also try different input and it gives me error but im not sure why?</p>
<p>What should i do if i want to change the size from 32 to 120x120 for example?</p>

isalirezag: <p>Got it, nvm <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ryanjh88: <p>I have the same question. How did you end up doing this? Thank you.</p>

jpngfile: <p>What was the solution?</p>

ptrblck: <p>If the model does not use any adaptive pooling layers but linear layers at the end, the number of input features to the first linear layer basically defines the expected input shape.<br>
E.g. if the input features are set as 64, the previous activation might be e.g. <code>[batch_size, 1, 8, 8]</code> or any other valid combination which results in 64 input activations.</p>
<p>Adaptive pooling layers relax this condition, as they define the output shape and use an adaptive kernel.<br>
Also fully convolutional networks shouldn’t have any size requirements.</p>
<p>CC <a class="mention" href="/u/ryanjh88">@ryanjh88</a></p>
Autograd function for .to(device) at initialization

Sasafras: <p>Hey all!</p>
<p>This is my first post here, and I tried looking for the answer everywhere but couldn’t find any satisfying answers. However, please let me know if anything I ask has been asked and answered before.</p>
<p>I was working on an implementation of a neural network that uses prototypes. It took me countless of hours of debugging (but in the end I was able to find the culprit) to see why a certain part of the network wasn’t updating at all: it was because of the fact that I ported certain nn.parameters to GPU at initialization time. The thing that gave it away was that the specific part had a grad_fn called “copyBackwards” which seemed rather odd because I figured there was no such action related to the actual gradient I was trying to calculate on this part of the network. Could anyone explain why porting the GPU to device (which was a working GPU) at initialization may cause this behavior where the intended grad function is not calculated but a copyBackwards function is instead?</p>
<p>Thanks so much in advance!</p>

ptrblck: <p>I assume you’ve called the <code>to()</code> operation on your <code>nn.Parameter</code> not the internal tensor?<br>
If so you would create a non-leaf variable, as the result is created by the <code>to()</code> operation, which is differentiable (so that the gradients can flow between different devices).<br>
Try to call the <code>to()</code> operation on the tensor before wrapping them in an <code>nn.Parameter</code>.</p>
<p>Have a look at <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2">this post</a> for some more examples.</p>
How could I get continuous batches when sample is fewer than batch size?

coincheung: <p>Hi,</p>
<p>My dataset contains 40 samples, but I need to train my model with batchsize=64. I hope that I can fetch training sample batches continuously until certain number of iters is accomplished. I also need to shuffle the dataset if it is repeatedly used, how could I do this please ?</p>

ptrblck: <p>If you only have 40 samples, you could probably define the length of your dataset as e.g.</p>
<pre><code class="lang-python">def __len__(self):
    return 2 * len(self.data)
</code></pre>
<p>and use <code>index = index % len(self.data)</code> operation inside <code>__getitem__</code> to draw the current batch.</p>
<p>You can pass this dataset to a <code>DataLoader</code> and set <code>shuffle=True</code> to shuffle the data.</p>
Inconsistency in Network Training

kranti: <p>I notice a wired behavior of pytorch, where the network defined after the data loader has a different loss value as compared to the case where the network is defined before the dataloader. I have fixed the random seed to the same number in both cases by adding the lines below.<br>
np.random.seed(0)<br>
torch.manual_seed(0)<br>
torch.cuda.manual_seed(0)<br>
What can be the reason for this?</p>

ptrblck: <p>If you’ve added these lines at the beginning of the script, the order or calls to the pseudo random number generator might still be different in your two use cases.<br>
E.g. initializing the random indices in the <code>DataLoader</code> will change the following call to the PRNG for the parameter initialization in your model.</p>
Calculate accuracy of pretrained AlexNet with ImageFolder

Felix_Laumann: <p>I have taken five classes of the ImageNet data set, feeded them into a <strong>pretrained</strong> AlexNet and want to calculate the accuracy of correctly classified images. With my code below, the loss is decreasing, but the accuracy is always between 20 and 30% (after 50 epochs). This behaviour confuses me extremely.<br>
My five classes are in subfolders of root, as torchvision,datasets.ImageFolder wants it. Altogether are 6500 images.</p>
<p>batch_size = 32<br>
num_epochs = 100</p>
<p>train_loader = data.DataLoader(dataset=train_dataset, batch_size=batch_size)</p>
<p>val_loader = data.DataLoader(dataset=val_dataset, batch_size=batch_size)</p>
<pre><code>        for images, labels in val_loader:
            images = Variable(images)

            # Forward pass only to get logits/output
            outputs = model(images)

            # Get predictions from the maximum value
            _, predicted = torch.max(outputs.data, 1)

            # Total number of labels
            total += labels.size(0)

            correct += (predicted == labels).sum()

        accuracy = 100 * correct / total

        # Print Loss
        print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))
</code></pre>
<p>Thanks in advance!</p>

Sami_Hassan: <p>Can you please include code for your train_dataset and val_dataset as well.</p>
Transformer doesn't fit to English to English Dialogue

jafioti: <p>I’m new to transformers so maybe this is a dumb question but I’ve been trying to get a normal Seq2Seq transformer model to fit to a few examples. The examples I’m using are English to English conversations, but I don’t think that matters too much. Obviously this will be hopelessly overfit, but for some reason I can’t seem to get larger models (&gt;30 million parameters) to fit even on a few examples.</p>
<p>I’d assume that a larger model would have a better ability to fit (overfit in this scenario), so I’m not sure what’s going on. Maybe I’m thinking about this wrong, please correct me if so.</p>
<p>The expected behavior is that it will train down to a very low loss and respond to the training examples near perfect. This is not the case as the loss is stuck around 0.2 and it ends up outputting the same for any input.</p>
<p>I created the Transformer model with my own code which is 350 lines, so a bit much to post the entire code here but I believe the same effect could be seen with the standard pytorch TransformerEncoder and TransformerDecoder. Here is a link to the entire file I’ve used: <a href="https://drive.google.com/open?id=1ie-WqsOqYJ-bL6BrI88eAkh-KRsDPpvb" rel="nofollow noopener">https://drive.google.com/open?id=1ie-WqsOqYJ-bL6BrI88eAkh-KRsDPpvb</a></p>

zhangguanheng66: <aside class="quote no-group" data-username="jafioti" data-post="1" data-topic="67599">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jafioti/48/19400_2.png" class="avatar"> jafioti:</div>
<blockquote>
<p>xa</p>
</blockquote>
</aside>
<p>First, I would suggest to start a model with fewer parameters to prove the idea. It saves time to see the preliminary results before wasting time to train a complex model.</p>
<p>Second, IMO, it would be great to start with the transformer/MultiheadAttention modules in Pytorch library since they are developed and maintained by many people. This should avoid some mistakes during some proof tests.</p>

jafioti: <p>I’ve tried to scale it down, and it works. I am trying to train on only four one word examples, so I think there has to be an error in my code.</p>
<p>I’ve also switched to using the pytorch TransformerEncoder and TransformerDecoder. The same effect still happens.</p>
<p>Here is the new file. <a href="https://drive.google.com/open?id=1hiw35sL1nYZ8zh9bXNOfXjxHSIAErNL8" rel="nofollow noopener">https://drive.google.com/open?id=1hiw35sL1nYZ8zh9bXNOfXjxHSIAErNL8</a><br>
The train.from and train.to files are just text files where each line is a new example.</p>
Obtain gradients between two variables

BruceShakeham: <p>Dear PyTorch dev,</p>
<p>I am a professor in one of the US Universities working on data-driven scientific computing using PyTorch now. The ability to get gradients allows for some amazing new scientific computing algorithms. You can see from <a href="https://www.sciencedirect.com/science/article/pii/S0021999118307125?via%3Dihub" rel="nofollow noopener">this paper</a>, and this github repo <a href="https://github.com/maziarraissi/PINNs/blob/master/main/continuous_time_identification%20(Navier-Stokes)/NavierStokes.py" rel="nofollow noopener">github link</a> (e.g., starting on line 121, “u = tf.gradients(psi, y)”), the ability to get gradients between two variables is in Tensorflow and is becoming one of the major differentiator between platforms in scientific computing. This paper is published in 2019 and has gained 168 citations, very high in the realm of scientific computing. I see many people are following suite and using this algorithm. We are now using PyTorch but might be forced to migrate to TF if such functionality is not available with PyTorch. Really hoping it could be released…</p>

ezyang: <p>You can do this in PyTorch! Make sure <code>requires_grad = True</code> on x and y when you initially define them (you could directly set <code>requires_grad = True</code> on the variables) and then use <code>torch.autograd.grad</code> to compute a gradient with respect to those variables.</p>

BruceShakeham: <aside class="quote no-group" data-username="ezyang" data-post="2" data-topic="67783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ezyang/48/486_2.png" class="avatar"> ezyang:</div>
<blockquote>
<p>torch.autograd.grad</p>
</blockquote>
</aside>
<p>huh!! I didn’t know it! Gonna look into this now. Thanks!</p>
What is the difference between the following two methods, if the full connection layer is more? i want an example.Thank you

111230: <pre><code>class RNN(nn.Module):

def __init__(self):
    super(RNN, self).__init__()
    self.rnn = nn.RNN(
        input_size=INPUT_SIZE,
        hidden_size=32, 
        num_layers=1,  
    )
    self.out = nn.Linear(32, 1)

def forward(self, x, h):
    
    out, h = self.rnn(x, h)
    prediction = self.out(out)
    return prediction, h


class RNN(nn.Module):
    def __init__(self):
    super(RNN, self).__init__()

    self.rnn = nn.RNN( 
        input_size=1,
        hidden_size=32,     
        num_layers=1,      
        batch_first=True,  
    )
    self.out = nn.Linear(32, 1)

def forward(self, x, h_state):  
    r_out, h_state = self.rnn(x, h_state)  
    outs = []   
    for time_step in range(r_out.size(1)):   
        outs.append(self.out(r_out[:, time_step, :]))
    return torch.stack(outs, dim=1), h_state</code></pre>

jusjusjus: <p>Hi, your code is formatted unwell, and there are several differences, e.g. <code>batch_first=True</code>.   What exactly would you like to know?</p>
<p>(As a side: it’s customary in polite English to ask “I would like an example” instead of a harsh-sounding “I want an example”)</p>

111230: <p>Sorry, my English expression is not very good, my question is the difference between using a loop and not using a loop in the forward calculation of two pieces of code, thank you.</p>

G.M: <p>Apparently the output of your 2 <code>forward</code> is different.</p>

111230: <p>I want to know what’s the difference between these two calculations? thank you</p>

G.M: <p>For the first one, u r only applying the FC to the last hidden state. For the second one, u r applying FC to all hidden states.</p>

111230: <p>For the first, using the last state should not be <strong>prediction = self.out(out[-1,:,:])</strong>, I don’t quite understand. For the second one, if I have two layers of the full connection layer using all the hidden states, what should I write? Please give me an example, thank you</p>

G.M: <p>U should take a look at RNN doc, that tells u the shape and meaning of the outputs of RNN. <a href="https://pytorch.org/docs/1.2.0/nn.html#recurrent-layers" rel="nofollow noopener">https://pytorch.org/docs/1.2.0/nn.html#recurrent-layers</a></p>
Pytorch JIT Class

Nick_8229: <p>Is every method of Jit Class complied? If so why the JIT function is little faster than the class method?</p>
<pre><code class="lang-auto">import torch
import time

@torch.jit.script
class Test:
    def __init__(self):
        pass

    def calc(self, input: torch.Tensor) -&gt; torch.Tensor:
        return 1.0 / torch.exp(input)

@torch.jit.script
def calc(a):
  return 1.0 / torch.exp(a)

for i in range(5):
  test = Test()
  start = time.time()
  a = torch.zeros(6000, 200)
  for j in range(1000):
    a = test.calc(a)
  print(time.time() - start)

for i in range(5):
  start = time.time()
  a = torch.zeros(6000, 200)
  for j in range(1000):
    a = calc(a)
  print(time.time() - start)
</code></pre>
CUDA out of memory after error

111226: <p>Hi, I’ve been training U-Net with medical imagery dataset. I used 1 of 2 RTX 2080 ti by allocating my model only on the first one. After learning through all epochs, my code crashed because there was an error in plotting images in Matplotlib. After fixing the error, I couldn’t run my code again due to this error.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 10.76 GiB total capacity; 9.58 GiB already allocated; 61.75 MiB free; 9.74 GiB reserved in total by PyTorch)
</code></pre>
<p>But when I display <code>nvidia-smi</code>, there is no process related to PyTorch.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/248378e6ae8d8a4efc808ceca50aa9062a800218.png" alt="image" data-base62-sha1="5d0RKEvLcB8ZLIFUskHr3c5EQ64" width="639" height="364"></p>
<p>I tried adding <code>torch.cuda.empty_cache()</code> but it didn’t seem to help. Are there ways to solve this problem?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

mailcorahul: <p>Can you post the code after fixing <code>matplotlib</code> error?<br>
In general, using a larger <code>batch size</code> can cause OOM. Also the reason you were not able to see any process related to <code>PyTorch</code> is because the process actually got killed due to OOM.</p>
<p>Using <code>torch.cuda.empty_cache()</code> can free already occupied memory, but here it looks like your process never really started. Would be better if you post your full code to get more clarity <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

111226: <p>Sorry that I can’t show you the full code… But I can show you matplotlib part. It was just saving plots and I had to fix it because the output image size didn’t match.</p>
<pre><code class="lang-python">    x_temp, y = ds_test.__getitem__(10)
    x = np.expand_dims(x_temp, axis=0)
    x = torch.from_numpy(x)
    x = x.to(device)
    y = cv2.cvtColor(y, cv2.COLOR_BGR2RGB)
    y_pred = model(x)
   
    print('++==========SHAPE==========++')
    print(f'x shape: {x.shape}, y shape: {y_pred.shape}')
    
    fig = plt.figure()
    rows = 1
    cols = 2

    x = x.view(3, 400, 400)
    x = x.cpu().detach()
    x = transforms.ToPILImage()(x)
    x = np.array(x.convert('RGB'))

    y_pred = y_pred.view(400, 400)
    y_pred = y_pred.cpu().detach()
    y_pred = transforms.ToPILImage()(y_pred)
    y_pred = np.array(y_pred.convert('L'))
</code></pre>

mailcorahul: <p>Can you use the following?</p>
<pre><code class="lang-auto">with torch.no_grad():
        y_pred = model(x)
</code></pre>
<p>Gradients are generally computed by default during forward call.</p>

111226: <p>Yeah, I used it and it went very well!!! Thanks bro, helped a lot <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>