akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reproducibility over platforms

Zethson: <p>Hi,</p>
<p>I read <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a> and ran lots of tests already. Unfortunately, my model is not reproducible even when ran in Docker containers on two different platforms. It is of course reproducible on the same platform.<br>
What is the reason for the non-deterministic behavior over different platforms?<br>
How should that be dealt with?</p>
<p>Cheers</p>

tom: <p>Currently, you seem to ask “can I run my stuff anywhere and expect it to be reproducible” and the answer is just “no”.<br>
The first step to a more detailed answer would be to narrow the problem by a lot.<br>
“I want to do X and I have platforms that are exactly identical except for Y.” could be something that could be worked on. (Note, though, that the general interest seems to be mostly in having reproducibility in the sense of non-random behavior rather than implementing it.)<br>
In the end, the crux is that floating point operations don’t commute (nor does the distributive property work out exactly).Then crux is that things are not guaranteed to run in the exact same order e.g. on different hardware because the difference in hardware might actually mean that things are translated to different GPU kernels or some such.</p>
<p>Best regards</p>
<p>Thomas</p>

Zethson: <p>Thank you <a class="mention" href="/u/tom">@tom</a>.</p>
<p>Trying to explain my stuff more clearly now:</p>
<p>I am trying to reproduce a simple MNIST example on the CPU (later GPU, but that’s a different topic) with the exact same loss on two machines.<br>
The first machine is my personal laptop, the second machine is in the cloud. They do not share the same hardware. However, BOTH machines are pretty much naked and running everything inside a Docker container, which sets up the whole environment. Therefore, the software environment is absolutely the same.</p>
<p>I fixed all possible seeds and get reproducible results on the respective machines, but NOT BETWEEN the machines.</p>
<p>To summarize: the platforms share everything except for the hardware (CPU, RAM, Mainboard).</p>
<p>(I may ask the same thing again with GPUs, which make the problem more difficult, but this post focuses on the CPU).</p>
<p>If researchers want to reproduce any of my findings, they should be able to at least reproduce the CPU result on their platform, no?</p>

tom: <p>I don’t think reproducibility in terms of “exact same floating point number” is reasonably achievable, e.g. AVX vs. non-AVX will make a difference.<br>
If your reproducibility depends on floating point accuracy divergences (sadly a lot of DL stuff does), something bad is going on, anyways.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: This isn’t a criticism of your work, but my thought regarding the general situation. In numerical analysis the goal is to get stable algorithms (a nontrivial problem e.g. for stiff ODE problems) to avoid blowup of numerical errors.</p>
Unable to load dataset using ImageFolder

arjun_pukale: <p>my torch version: 1.5.0+cu92<br>
my torchvision version: 0.6.0+cu92</p>
<p>I have attached the screenshots of my folder structure and error<br>
and I have also tried this on google colab with same folder structure and it works but its not working on my local system.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb9c680b90a0a977e9007cd8b36a067931feb7be.jpeg" alt="imagefolder1" data-base62-sha1="zTQWxUy6LB6JOPeYFKIKoCZlJf0" width="676" height="478"> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/297c7ba281d651ac7f9dc95cd351fe3633b81869.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/297c7ba281d651ac7f9dc95cd351fe3633b81869" title="imagefolder2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/297c7ba281d651ac7f9dc95cd351fe3633b81869_2_632x500.jpeg" alt="imagefolder2" data-base62-sha1="5V0hlsiKQX1QDXBEWSDw3VGOkid" width="632" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/297c7ba281d651ac7f9dc95cd351fe3633b81869_2_632x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/297c7ba281d651ac7f9dc95cd351fe3633b81869_2_948x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/297c7ba281d651ac7f9dc95cd351fe3633b81869.jpeg 2x" data-dominant-color="363D4B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagefolder2</span><span class="informations">1209×955 94.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

arjun_pukale: <p>Also I have noticed a strange behaviour, when i renamed my folder from ‘train_data’ to ‘sample’, it works! but how is this possible and then again after remaning to something else, it is not working.</p>

ptrblck: <p><code>\t</code> is an escape character and will add a tab to your string.<br>
Try to use <code>./train_data</code> instead.</p>

arjun_pukale: <p>It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to downsample a time-series?

mohammadf: <p>I’m working with a sequence sampled at 2KHz, but I need to downsample it to 10Hz. I’ve reshaped the sequence to match the input shape of a GRU layer, <code>(seq_len, batch, input_size)</code> but when I try to use <code>torch.nn.functional.interpolate</code>, it seems that the function is trying to downsample the last dimension. My <code>input_size</code> is 16, corresponding to the 16 sensors the data has been collected from. I want to do a temporal interpolation on the first dimension instead.</p>

mohammadf: <p>I couldn’t find any direct solutions and ended up permuting the dimensions. For anyone that ends up here, this is how I did the downsampling:</p>
<pre><code class="lang-auto">F.interpolate(input.permute(1, 2, 0), scale_factor=..., mode=...).permute(2, 0, 1)
</code></pre>
<p>The output can now be fed into modules like GRU.</p>
Implementing LSTM using PyTorch but get an error in linear

Jinlong_Ruan: <p>The part related is as below:</p>
<p>tensor([[-0.0106,  0.8890,  0.0561,  0.9582, -0.0059,  0.7418, -0.0096,  0.9744]])<br>
this is the x in my first loop above.<br>
Sorry if it is hard to read because I can only post one image.<br>
I am wondering why I am getting this error message and what it means. If I change the shape from[1,8] to [8,1] then I get a message like dimension mismatch [8,1]  [8,5]</p>
<p>How can I fix this problem?</p>

ptrblck: <p>Could you post the code and error message directly by wrapping them into three backticks ``` please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Jinlong_Ruan: <p>Thank you for reply. The problem has been resolved. I was trying to withdraw this post.</p>
<p>ptrblck via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a>于2020年4月26日 周日04:18写道：</p>
Unnormalization

Angry_potato: <p>Hi,<br>
I  training a GAN model. So the input image is normalized to a mean say x and std say y. What is the best way to un-normalize the generated image</p>

ptrblck: <p>It depends, how you’ve applied the normalization.<br>
If you’ve used a standardization approach such as:</p>
<pre><code class="lang-python">x = torch.empty(1, 3, 224, 224).uniform_(0, 255)

mean = x.mean([0, 2, 3], keepdim=True)
std = x.std([0, 2, 3], keepdim=True)

x_norm = (x - mean) / std
</code></pre>
<p>then you could simply reverse it via:</p>
<pre><code class="lang-python">x_re = x_norm * std + mean 

print((x - x_re).abs().max())
&gt; tensor(1.5259e-05)
</code></pre>
Proper way to Compose multiple Compose transforms?

xksteven: <p>Suppose I have two Compose transforms as such</p>
<pre><code class="lang-auto">transform1 = transforms.Compose([
    transforms.RandomRotation(10),
    transforms.ColorJitter(),
])

transform2 = transforms.Compose([
     transforms.RandomGrayscale(),
     transforms.RandomHorizontalFlip(),
])
</code></pre>
<p>How could I perform the following?</p>
<pre><code class="lang-auto">final_transform = transforms.Compose([transform1, transform2)
</code></pre>

ptrblck: <p>Your code should work fine! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Are you seeing any issues?</p>
Memory Inflation Without no_grad and.eval()

Orcun_Deniz: <p>Im pretty new to PyTorch and working on my first training. I noticed that after an epoch of training when I did validation without going into eval mode and using the no_grad, GPU memory consumption goes sky high until there is no GPU memory left. It would be awesome if somebody could explain what is happening under the hood and causing such thing.</p>

ptrblck: <p>If you don’t use <code>with torch.no_grad()</code>, Autograd will create the computation graph, which is needed to calculate the gradients in the backward pass.<br>
Now if you store a tensor, which is attached to this computation graph, such as the loss or model output, in e.g. a <code>list</code>, the whole computation graph will also be stored.<br>
To avoid the increased memory usage, you could either wrap the validation code in <code>torch.no_grad()</code> or call <code>tensor.detach()</code> before storing it.</p>
Evalution coco object detection

rebeen: <p>please how to get accuracy and also save the results according to coco object detection</p>
<pre><code class="lang-auto">
[{"image_id": int, "category_id": int,"bbox": [x,y,width,height],"score": float,}]
</code></pre>
<p>here is the training</p>
<pre><code class="lang-auto"># Training
for epoch in range(config.num_epochs):
    print(f"Epoch: {epoch}/{config.num_epochs}")
    model.train()
    i = 0
    for imgs, annotations in data_loader:
        i += 1
        imgs = list(img.to(device) for img in imgs)
        annotations = [{k: v.to(device) for k, v in t.items()} for t in annotations]
        loss_dict = model(imgs, annotations)
        print(loss_dict)
        # detections = y.data

        losses = sum(loss for loss in loss_dict.values())

        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

        print(f"Iteration: {i}/{len_dataloader}, Loss: {losses}")








</code></pre>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">object detection tutorial</a> should give you a sample code for the model evaluation including the AP and AR calculation.</p>
Hi，use conda to install torchtext，failed

111289: <p>(3.6.4) lzw@resplendent-star:~$ conda install torchtext<br>
Collecting package metadata (current_repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.<br>
Collecting package metadata (repodata.json): done<br>
Solving environment: failed with initial frozen solve. Retrying with flexible solve.</p>
<p>PackagesNotFoundError: The following packages are not available from current channels:</p>
<ul>
<li>torchtext</li>
</ul>
<p>Current channels:</p>
<p>（becaue I am  new beginer to this pytorch website，so it donesnt allow me  to put websites which output by the shell console）</p>
<p>To search for alternate channels that may provide the conda package you’re<br>
looking for, navigate to</p>
<p>and use the search bar at the top of the page.</p>
<p>(3.6.4) lzw@resplendent-star:~$</p>

sai_m: <p>can you try installing using pytorch channnel, use following command:</p>
<p>conda install –c pytorch torchtext</p>

111289: <p>hi,it works ,thanks!!!</p>
Is CUDA back-compatible?

kaiseryet: <p>i.e., if I installed CUDA 10.2, can I still install pytorch via conda by</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
</code></pre>

albanD: <p>Hi,</p>
<p>In general it is not I’m afraid.<br>
But for versions that are very close, it might work <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Unfortunately, you won’t get a nice error message. So if you get weird cuda errors, this is most likely due to this mismatch.</p>
<p>Note that when you install cuda in conda, it installs a brand new cuda and won’t use your system wide one. So here you’re safe and you won’t get any issue.</p>

kaiseryet: <p>So pytorch depends on cudatoolkit that comes with it in conda environment, right?</p>

ptrblck: <p>As <a class="mention" href="/u/alband">@albanD</a> explained, if you install the PyTorch binaries with <code>cudatoolkit</code>, your local CUDA installation won’t be used, but instead the one shipped with the binaries.</p>
<p>If I understand the original question right, you would like to install PyTorch with CUDA10.1 as the binaries, while CUDA10.2 is installed locally on your machine?<br>
If that’s the case, then yes it’ll work as the binaries won’t interfere with the local installation.</p>
Training loop checking validation accuracy

No_No: <p>Hi,</p>
<p>When training my model, at the end of each epoch I check the accuracy on the validation set. To do this I use model.eval() and then set it to model.train() after checking the validation set. This leads to an accuracy of around 90%. However when I run my model without checking the validation set until after the whole training  is complete, the accuracy becomes 80%. This could be a random shift in the accuracy by the end, but I was wondering if I am doing something wrong. The code is roughly below. Check accuracy just feeds the dataloader through the model and compares the outputs to calc the accuracy</p>
<pre><code class="lang-auto">for e in range(epochs):
        for t, x in enumerate(my_dataloader):
            x_arr = np.array(x)
            x1 = x_arr[:,0]
            x1 = Batch.from_data_list(x1).to(device)
            x2 = torch.stack(x_arr[:,2].tolist(), dim=0).to(device=device, dtype=torch.long)
            x3 = x_arr[:, 3]
            x3 = Batch.from_data_list(x3).to(device)

            model.train()  # put model to training mode
            # x = x.to(device=device)  # move to device, e.g. GPU
            outputs = model(x1,x3,x2)
            # maxes = outputs.max(1)[0].reshape(-1,1)
            # print(len(maxes))
            loss = nn.CrossEntropyLoss(weight=weight)(outputs, x1.y)

            optimizer.zero_grad()

            loss.backward()

            # Update the parameters of the model using the gradients
            optimizer.step()

            if t % print_every == 0:
                print('Epoch: %d, Iteration %d, loss = %.4f' % (e, t, loss.item()))
                print()
      model.eval()
      with torch.no_grad():

        my_dataloader = DataListLoader(X_test, batch_size=128)
        acc, y_pred, y_true = check_accuracy(my_dataloader, model)
        accs_test.append(acc)
      model.train()
</code></pre>

ptrblck: <p>The code looks alright.<br>
How reproducible is this behavior? I.e. how many runs did you try with and without the eval code and what is the mean+std of the final accuracy?</p>
Multiple Matrix Multiplication (chained)

theevann: <p>Hey !</p>
<p>I would like to know if there is a way to multiply multiple matrices as a chain.<br>
Let’s say I have matrices M1, M2, … M23, and I want to do: <code>M1 @ M2 @ M3 @ ... @ M23</code>.<br>
I see only a for loop or a recursive function for that … that wouldn’t be very effective I guess.</p>
<p>Any suggestions ?<br>
Thanks !</p>

Harman_Singh: <p>Someone Please Answer this</p>

ptrblck: <p>The proposed approach seems to work:</p>
<pre><code class="lang-python">m1 = torch.randn(1, 2)
m2 = torch.randn(2, 3)
m3 = torch.randn(3, 4)
m4 = torch.randn(4, 5)

res = m1 @ m2 @ m3 @ m4
</code></pre>
<p>Are you seeing any errors using it?</p>

Harman_Singh: <p>Whats the difference between  chain_matmul and @. What is better to use?</p>
<p>Thanks</p>

ptrblck: <p>I would recommend to use <code>torch.chain_matmul</code> as it should be more efficient.<br>
From the docs:</p>
<blockquote>
<p>This product is efficiently computed using the matrix chain order algorithm which selects the order in which incurs the lowest cost in terms of arithmetic operations.</p>
</blockquote>

Harman_Singh: <p>Thanks a lot for your help</p>
Running Libtorch without CMaking it

JarvisStark: <p>Hello all. I am a newbie to this forum. I ll get straigt to my question.</p>
<p>I want to run a file called control.cpp that is a ROS program. However, this program contains of a line <span class="hashtag">#include</span> &lt;torch/torch.h&gt;, and i want to display a simple tensor in a ROS program. Can I do this without any building? Because there is a conflict between ROS CMakeLists.txt and Libtorch’s CMakeLists.txt files. I get a lot of errors, and though I have seen questions related to these on PyTorch forums, none of them have explained how to solve this issue.</p>
<p>I have also tried to build from source, but I couldn’t get it to work.</p>
<p>It would be of great help, if someone could help me out with this issue.</p>
<p>Thank you so much!</p>
Use of LossClosure in optimizers

ericrhenry: <p>Certain optimizers, particularly those which have an internal iterative structure, require a LoopClosure object to their step() method. (LBFGS and perhaps some new-ish, relatively sophisticated optimizers come to mind.) In fact, the new Pytorch 1.5 API appears to now require an object of this sort be passed to Optimizer objects; this appears to be coupled to the demise of LossClosureOptimizer as a separate base class. Many optimizers don’t actually use this, of course, in which case it can be defaulted to nullptr.</p>
<p>Instances of such objects I have seen in the frontend source code expect something like</p>
<p>LoopClosure = std::function&lt;Tensor()&gt;</p>
<p>and explicit invocations within optimizer code simply execute this without arguments, as prescribed by this signature.</p>
<p>A loss calculation requires comparing computed results with targets, so presumably these objects are expected to be something like lambdas with ‘this’ capture in order to work with this signature. I am looking for an actual example of the definition of such a LoopClosure, usable for example with the LBFGS optimizer. This is a bit beyond the scope of the tutorials I have come across, and I would appreciate any pointers (shared, unique, intrusive, whatever) to possible enlightenment.</p>
<p>Thanks,<br>
Eric</p>

ptrblck: <p>The test provide <a href="https://github.com/pytorch/pytorch/blob/856e8cf0288fe3c1701d11fae61b214c08635b9d/test/cpp/api/optim.cpp#L46-L55">this code snippet</a>. Would that help?</p>

ericrhenry: <p>Yes, that looks like an excellent model for what I’m looking for. (I’m probing the source code using KDevelop, but didn’t think to try to go down the test tree–a good lesson.) Many thanks.</p>

ptrblck: <p>Usually new code won’t land without a proper test, so it’s my first place to look for the right usage. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Tutorials usually take a bit longer and might not cover all use cases.</p>
A weird pytorch docker problem: /opt/conda/bin disappear after pip install

yisha7: <p>a very simple dockerfile:</p>
<p>FROM pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime<br>
RUN pip install request<br>
ENTRYPOINT bash</p>
<p>build :    docker build . -t test<br>
run: docker run -it --rm test bash</p>
<p>when i in the docker container, i can’t use python,  and the /opt/conda/bin dir disappear.</p>

ptrblck: <p>I just tested the container and it seems to work for me:</p>
<pre><code class="lang-python">ptrblck@...:~$ nvidia-docker run -it --ipc=host pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime
Unable to find image 'pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime' locally
1.5-cuda10.1-cudnn7-runtime: Pulling from pytorch/pytorch
...
root@c083513e9eba:/workspace# python
Python 3.7.7 (default, Mar 23 2020, 22:36:06)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(1, device='cuda')
&gt;&gt;&gt; x
tensor([-0.5422], device='cuda:0')
</code></pre>

yisha7: <p>thanks. yes, the orignal docker image works fine.</p>
<p>but when i build a new image:</p>
<p>FROM pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime<br>
RUN pip install request<br>
ENTRYPOINT bash</p>
<p>and run the new image</p>
<p>the new image can’t use python any more.</p>

ptrblck: <p>I cannot reproduce the issue.</p>
<p>My <code>Dockerfile</code>:</p>
<pre><code class="lang-python">FROM pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime
RUN pip install request
ENTRYPOINT bash
</code></pre>
<p>Building the container via:</p>
<pre><code class="lang-python">docker build -t tmp_build .
</code></pre>
<p>Running via:</p>
<pre><code class="lang-python">nvidia-docker run -it --ipc=host tmp_build
</code></pre>
<p>Inside the container:</p>
<pre><code class="lang-python">root@...:/workspace# python -c "import torch; print(torch.randn(1, device='cuda'))"
tensor([0.4954], device='cuda:0')
</code></pre>
<p>Could you check, what the difference to your workflow might be?</p>

yisha7: <p>the docker version is  18.09.8, i change to a server of docker version 19, and all things works fine now <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">.</p>
Pytorch 1.5.0 requires_grad being automatically set to false in C++ registered operators

Inspiros: <p>In <strong>pytorch 1.4.0</strong> (and probably every previous versions), I could simply rewrite my python code in C++, put it inside a vanilla function, and call it from python with Pybind11 or with torch script’s RegisterOperators. The autograd works perfectly while giving the foreign code an edge over pure python in terms of performance.</p>
<p>However, <strong>pytorch 1.5.0</strong> surprised me when it eliminate every gradient of input tensor. Now am I supposed to manually expand and write backward pass on my own whenever I touch C++ side <img src="https://discuss.pytorch.org/images/emoji/apple/angry.png?v=12" title=":angry:" class="emoji" alt=":angry:" loading="lazy" width="20" height="20">? Anyone know how to work around this please help?</p>
<p><em>The pic is an example of a very simple function from the project i’m working on, obviously there is no need for grad checking here.</em><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/750642d7360cbf786a8d2e68a24310de959cfb7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/750642d7360cbf786a8d2e68a24310de959cfb7f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/750642d7360cbf786a8d2e68a24310de959cfb7f.png" alt="image" data-base62-sha1="gHfhg5EU030R50oFDujZ1zbhxPp" width="630" height="500" data-dominant-color="1E3454"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1012×803 71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I posted an issue for this here: <a href="https://github.com/pytorch/pytorch/issues/37306" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/37306</a></p>
How to construct the Gram matrix of a gaussian RBF kernel

rzhang63: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I’m implementing a custom loss function, which has a term that involves the gram matrix of a Gaussian RBF kernel.</p>
<p>Say, for each training iteration, I get a mini-batch (batch size 128) of predicted probabilities for K=5 classes. So the predicted probability tensor has shape=(128,5). Now I wish to compute the Gram matrix (128 by 128) of the Gaussian RBF kernel exp(-||p-q||^2) where p and q are the predicted probability vectors.</p>
<p>I don’t know if there’s a way of doing this without looping through all the 128x128 possible pairs of p and q, and yet preserves the autograd compatibility so I can use it as part of the loss function.</p>
<p>Could you please help me with some code example? Thank you in advance!</p>
<p>Best</p>

ptrblck: <p>I’m not sure, but wouldn’t <code>torch.mm(mat, mat.t())</code> calculate the Gram matrix?</p>
<p>PS: as you can clearly see I’m not an expert in this topic, so tagging certain people might demotivate others to answer in your thread. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

rzhang63: <p>Thank you very much for your reply!</p>
<p>If <code>mat</code> is the predicted probability matrix with shape (128,5), then <code>torch.mm(mat,mat.t())</code> gives me the 128 by 128 matrix that contains all the inner products between pairs of rows <code>p,q</code> in <code>mat</code>.</p>
<p>But what I’m hoping is to compute a more general function <code>k(p,q)</code> between all pairs of rows <code>p,q</code> in <code>mat</code> and store it in a 128 by 128 matrix. So <code>torch.mm(mat,mat.t())</code> can be seen as a simple case of this where the function <code>k(p,q)</code> is just the inner product <code>&lt;p,q&gt;</code>.</p>

ptrblck: <p>OK, I see. Do you have a specific function in mind for <code>k</code>?</p>

rzhang63: <p>Yes, for example, <code>k(p,q)=exp(-||p-q||)</code> where the norm <code>||p-q||</code> is the L1 norm.</p>

ptrblck: <p>You could add a dummy dimension and use broadcasting for this use case:</p>
<pre><code class="lang-python">a = torch.randn(128, 2)
b = torch.randn(128, 2)
res = torch.norm(a.unsqueeze(1)-b, dim=2, p=1)

res_manual = []
for a_ in a:
    for b_ in b:
        res_manual.append(torch.norm(a_-b_, dim=0, p=1))
res_manual = torch.stack(res_manual)
res_manual = res_manual.view(128, 128)

print((res - res_manual).abs().max())
&gt; tensor(0.)
</code></pre>

rzhang63: <p>Thank you so much for the code!</p>
<p>Just to check, both methods in your code example are compatible with autograd right? coz I want to use the matrix as part of my loss function.</p>

ptrblck: <p>Yes, Autograd will be able to track these operations.</p>
If set random seed across multi-gpu is necessary in DistributedDataParallel？

hhxx: <p>Following imagenet-example: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>, It seems that  <strong>seed</strong>  is not set in default (default is None):<br>
<code>parser.add_argument('--seed', default=None, type=int, help='seed for initializing training. ')</code></p>
<p>But when we use DistributedDataParallel mode, if seed is not set, the initialized parameters across multi-gpu will be different, resulting in different model param is kept in different gpus during training process (although we only save ckpt in rank0 gpu).</p>
<p>I am not sure whether this phenomenon will cause unknown errors, or may lead to an unstable results? Is it safe for me not to set the initialization seed？</p>

mrshenli: <p>This should be fine, because <code>DistributedDataParallel</code> broadcasts model states from rank 0 to all other ranks at construction time. See the code below:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/34284c127930dc12d612c47cab44cf09b432b522/torch/nn/parallel/distributed.py#L280-L285" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/34284c127930dc12d612c47cab44cf09b432b522/torch/nn/parallel/distributed.py#L280-L285" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/34284c127930dc12d612c47cab44cf09b432b522/torch/nn/parallel/distributed.py#L280-L285</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="280" style="counter-reset: li-counter 279 ;">
<li># Sync params and buffers</li>
<li>module_states = list(self.module.state_dict().values())</li>
<li>if len(module_states) &gt; 0:</li>
<li>    self._distributed_broadcast_coalesced(</li>
<li>        module_states,</li>
<li>        self.broadcast_bucket_size)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Efficient way to crop 3d image in pytorch

DrHB: <p>I working with 3d image data and would like to implement transform which randomly crops 3d image. I have written a class that does this and was wondering if there is room for optimization.</p>
<pre><code class="lang-auto">class RandomCrop3D():
    def __init__(self, img_sz, crop_sz):
        c, h, w, d = img_sz
        assert (h, w, d) &gt; crop_sz
        self.img_sz  = tuple((h, w, d))
        self.crop_sz = tuple(crop_sz)
        
    def __call__(self, x):
        slice_hwd = [self._get_slice(i, k) for i, k in zip(self.img_sz, self.crop_sz)]
        return self._crop(x, *slice_hwd)
        
    @staticmethod
    def _get_slice(sz, crop_sz):
        try : 
            lower_bound = torch.randint(sz-crop_sz, (1,)).item()
            return lower_bound, lower_bound + crop_sz
        except: 
            return (None, None)
    
    @staticmethod
    def _crop(x, slice_h, slice_w, slice_d):
        return x[:, slice_h[0]:slice_h[1], slice_w[0]:slice_w[1], slice_d[0]:slice_d[1]]
    
</code></pre>
<p>example:</p>
<pre><code class="lang-auto">volume_3d = torch.rand(3, 100, 100, 100)
rand_crop = RandomCrop3D(volume_3d.shape, (64, 64, 64))

rand_crop(volume_3d)
</code></pre>
Object has no attribute 'weight'

J_Na: <pre><code class="lang-auto">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        torch.nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm2d') != -1:
        torch.nn.init.normal_(m.weight.data, 1.0, 0.02)
        torch.nn.init.constant_(m.bias.data, 0.0)
</code></pre>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.discriminator = nn.Sequential(
            nn.Conv2d(in_channels=256, out_channels=128, kernel_size=5, stride=1, padding=2),
            nn.ReLU(True),
            nn.Conv2d(in_channels=128, out_channels=128, kernel_size=5, stride=1, padding=2),
            nn.AvgPool2d(kernel_size=2, stride=1, padding=0),
            nn.Conv2d(in_channels=128, out_channels=1, kernel_size=1, stride=1, padding=0),
            nn.Sigmoid()
        )

    def forward(self, input_feature, alpha):
        x = self.discriminator(input_feature)
        return x

network = Network().cuda()
...
network = network.train()
network.apply(weights_init)
</code></pre>
<p>When I apply <code>weights_init()</code><br>
<code>AttributeError: 'Network' object has no attribute 'weight'</code> error occurs.`<br>
What is the wrong in my usage?</p>

ptrblck: <p>Your code seems to work on my machine using <code>1.0.0.dev20181125</code>.<br>
However, the error is most likely thrown due to a wrong if condition.<br>
If you call <code>apply</code> on your model, the function you are providing will be applied recursively on all children as well as the model itself.<br>
Even though your code works for me, I would recommend to write the checks as:</p>
<pre><code class="lang-python">if isinstance(m, nn.Conv2d):
    ...
# or
if type(m) == nn.Conv2d:
    ...
</code></pre>

J_Na: <p>It’s strange. I have another model and It works fine.<br>
Anyway, I will change the code you recommended. Thanks !</p>

tawej: <p>Hello ptrblck,<br>
I’ve just faced the same error and following your idea it was fixed, thanks.</p>
Need some help about my coding depthwise pointwise convolution

henry_Kang: <p>Hello. Nice to meet you guys.<br>
I am currently try to make the pytorch version about SDD crack segmentation network.<br>
The paper said about the pointwise first and depthwise after.<br>
I wrote the block like below.</p>
<pre><code class="lang-auto">&gt; class depthwise_separable_convs(nn.Module):
&gt;     def __init__(self, nin=64, nout=64, kernel_size, padding, bias=False):
&gt;         super(depthwise_separable_convs, self).__init__()
&gt;         d=64
&gt;         pw_filter_nums=int(d/2)
&gt;         self.pointwise = nn.Sequential(
&gt;         #pointwise
&gt;         nn.Conv2d(nin, pw_filter_nums, kernel_size=3,stride=1, padding=1, groups=pw_filter_nums, bias=bias), 
&gt;         nn.BatchNorm2d(pw_filter_nums),
&gt;         nn.ReLU(inplace=True),
&gt;         #depthwise
&gt;         nn.Conv2d(pw_filter_nums, nout, kernel_size=3, padding=padding, groups=pw_filter_nums, bias=bias),
&gt;         nn.BatchNorm2d(nout),
&gt;         nn.ReLU(inplace=True))
&gt;         
&gt;     def forward(self, x):
&gt;         out = self.pointwise(x)
&gt;         
&gt;         return out
</code></pre>
<p>I follow some inquire in this community. I am not sure that the depth wise and pointwise for groups correctly.<br>
However I am sure that there is some error about my code. The training loss is not decrease.<br>
Does anyone give some advice about my code error. I do not understand the group parts correctly. My training code is not the problem I tested this train code for unet and deeplab v3 and it works very well.<br>
Thank you for reading my question.</p>

ptrblck: <p>If I’m not mistaken, a depthwise separable convolution is applying a grouped convolution followed by a pointwise convolution as shown <a href="https://discuss.pytorch.org/t/using-optimised-depthwise-convolutions/11819/15">here</a>.<br>
Both your convolutions use a kernel size of 3 (pointwise should use a <code>1x1</code> kernel) and both are using different groups (depthwise should use <code>groups=in_channels</code>).</p>

henry_Kang: <p>Thank you for answering.<br>
I update the code. After few search also.<br>
The paper what I am trying to rewrite the code use the pointwise first instead of depthwise first.<br>
I also check that the mobilnet v2 also use this approach in some part actually.<br>
Anyway The block is not the problem about network updating look like.<br>
So I am digging the weight initialization. parts. I am not sure I am missing or something.</p>
<p>Thank you.</p>
<pre><code class="lang-auto">self.pointwise = nn.Sequential(
            nn.Conv2d(nin, pw, 1, 1, 0, 1, 1, bias=bias),
            nn.BatchNorm2d(pw),
            nn.ReLU(inplace=True),
            nn.Conv2d(pw, pw, kernel_size=3,stride=1, padding=1, groups=pw, bias=bias),
            nn.BatchNorm2d(pw),
            nn.ReLU(inplace=True))
</code></pre>
Loss not Converging for CNN Model

Tapan_Patel: <p>Image Transformation and Batch</p>
<pre><code class="lang-auto">transform = transforms.Compose([

                                transforms.Resize((100,100)),

                                transforms.ToTensor(),

                                transforms.Normalize([0.485,0.456,0.406],[0.229,0.224,0.225])

                                ])

data_set = datasets.ImageFolder(root="/content/drive/My Drive/models/pokemon/dataset",transform=transform)

train_loader = DataLoader(data_set,batch_size=10,shuffle=True,num_workers=6)
</code></pre>
<p>Below is my Model</p>
<pre><code class="lang-auto">class pokimonClassifier(nn.Module):

  def __init__(self):

    super().__init__()

    self.conv1 = nn.Conv2d(3,6,3,1)

    self.conv2 = nn.Conv2d(6,18,3,1)

    self.fc1 = nn.Linear(23*23*18,520)

    self.fc2 = nn.Linear(520,400)

    self.fc3 = nn.Linear(400,320)

    self.fc4 = nn.Linear(320,149)

  def forward(self,x):

    x = F.relu(self.conv1(x))

    x = F.max_pool2d(x,2,2)

    x = F.relu(self.conv2(x))  

    x = F.max_pool2d(x,2,2)

    x = x.view(-1,23*23*18)

    x = F.relu(self.fc1(x))

    x = F.relu(self.fc2(x))

    x = F.relu(self.fc3(x))

    x = F.log_softmax(self.fc4(x), dim=1)

    return x
</code></pre>
<p>Creating Instance of model, Use GPU, Set Criterion and optimizer<br>
Here is firsr set <code>lr = 0.001</code> then later changed to <code>0.0001</code></p>
<pre><code class="lang-auto">model = pokimonClassifier()
model.to('cuda')
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(),lr = 0.0001)
</code></pre>
<p>Training Dataset</p>
<pre><code class="lang-auto">for e in range(epochs):

  train_crt = 0

  for b,(train_x,train_y) in enumerate(train_loader):

    b+=1

    train_x, train_y = train_x.to('cuda'), train_y.to('cuda')

    # train model

    y_preds = model(train_x)

    loss = criterion(y_preds,train_y)

    # analysis model

    predicted = torch.max(y_preds,1)[1]

    correct = (predicted == train_y).sum()

    train_crt += correct

    # print loss and accuracy

    if b%50 == 0:

        print(f'Epoch {e} batch{b} loss:{loss.item()} ')

    # updating weights and bais

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

  train_loss.append(loss)

  train_correct.append(train_crt)
</code></pre>
<p>My loss value remains between 4 - 3  and its not converging to 0.<br>
I am super new to deep learning and  I don’t know much about it.</p>
<p>The dataset I am using is here: <a href="https://www.kaggle.com/thedagger/pokemon-generation-one" rel="nofollow noopener">https://www.kaggle.com/thedagger/pokemon-generation-one</a></p>
<p>A help will be much appreciated.<br>
Thank You</p>

chetan06: <p>The problem may be because of the learning rate or the optimizer you are using. The optimizer is alright but I think the learning rate is too small. Once try between 5e-3 or 1e-2 for 10 to 20 epochs and check if the loss is decreasing or not.</p>
Is normalization needed for prediction?

Wei_Wong: <p>If a model is trained by image after normalization, is normalization needed for the prediction of one single image? My intuition tells me the result will not be affected without normalization, but I am not sure.</p>

chetan06: <p>Yeah I think that you should apply normalization to your test image if your model is trained on normalized inputs because then your model may be experiencing some strange input at test time and which may not be so good for the task that you are upto.</p>
LSTM / GRU prediction with hidden state?

YuvalA: <p>Hi,<br>
I am trying to predict a value based on time series by series of 24 periods (the 25th period)</p>
<p>While training I have a validation set with I babysit the training (RMSE) and each epoch, eval the validation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f30aacf8df17bde44b965f1b70e20339c14423c.png" alt="image" data-base62-sha1="dA5CYUYGu4E3GMMZMdWgPVDKI3O" width="300" height="200"></p>
<p>Receive errors as:</p>
<pre><code class="lang-auto">Train RMSE: 3.02

Validation RMSE: 5.65
Validation r_squared: 0.75
</code></pre>
<p>However, when I evaluate the test (in the same size of the validation) I receive very bad results by:</p>
<pre><code class="lang-auto">    model.eval()
    x_test = x_test.cuda()
    y_test_pred, hidden = model(x_test)
</code></pre>
<pre><code class="lang-auto">Test RMSE: 8.18
Test r_squared: -5.62
</code></pre>
<p>I suspect the difference is that in the validation it creates while the model is training (performed model.eval()) so using the current hidden state and while test is with zero hidden state?<br>
How can the hidden state calculated for the test? The train and the test are not the same size…</p>
<p>Thanks!</p>
Jacobian of a Function of a Jacobian

aaa: <p>Hi I am calculating a Jacobian of a function of a Jacobian. I have a vector valued function f and matrix valued fucntion g. Both differentiable.</p>
<pre><code class="lang-auto">y = f(x)
nabla = jacobian(y,x)
function_nabla = g(nabla)
hessian = jacobian(function_nabla, x)
</code></pre>
<p><code>nabla</code> is calculated without problem so is <code>g(nabla).</code> Function <code>g</code> is standard pytorch function such as inverse.</p>
<p>I am using the following function for jacobian.</p>
<pre><code class="lang-auto">	def jacobian(y, x, create_graph=True):
		jac = []
		flat_y = y.reshape(-1)
		flat_y.retain_grad()
		grad_y = torch.zeros_like(flat_y)
		for i in range(len(flat_y)):
			grad_y[i] = 1.
			grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
			jac.append(grad_x.reshape(x.shape))
			grad_y[i] = 0.

		return torch.stack(jac).reshape(y.shape + x.shape)
</code></pre>
<p>I always get</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 32]] is at version 64; expected version 63 instead.</p>
</blockquote>
<p>I can’t pinpoint which operation is inplace here. Thank you.</p>
Torch.cuda.is_available() returns False only for jupyter notebook

Adam_Chrzanowski: <p>… but in console env it returns true<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de9d58381072e27e92a4fb51fe219b2480421256.png" alt="image" data-base62-sha1="vLl6G0rErEj7CGX7UhWhpXZALv8" width="246" height="41"></p>
<p>Cuda toolkit version: 10.2</p>
<p>Any ideas why?</p>

ptrblck: <p>Most likely because the Jupyter notebook isn’t using the same Python kernel.<br>
You can select the Python kernel inside the notebook to make sure you are using the same, which is called in your terminal.</p>
<p>Also, a bit unrelated, but I personally prefer to create conda environments to avoid these conflicts. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Adam_Chrzanowski: <p>I figure it out :D. You are right.</p>
<p>This command isn’t working properly with conda env.</p>
<pre><code class="lang-auto">python -m ipykernel install --user --name=notebook_env 
</code></pre>
<p>Instead, we should first install <em>nb_conda_kernels</em> in active conda env</p>
<pre><code class="lang-auto">conda install -n notebook_env nb_conda_kernels
</code></pre>
<p>and then run above command like this.</p>
<pre><code class="lang-auto">python -m ipykernel install --user --name notebook_env --display-name "Python (notebook_env)"
</code></pre>
Validation_loss_

plp161512: <p>When I train the resnet50 network, I calculate the validation set loss every time an epoch passes. Unfortunately, the verification set loss is always large, so I saved the model at this time, and then load this model to calculate the verification set loss, but It is found that the loss of the validation set is different at this time, why is this happening?<br>
Here is the code to calculate the validation set loss:</p>
<p><span class="mention">@t.no_grad</span>()<br>
def val(model, dataloader):</p>
<pre><code>model.eval()
criterion = t.nn.MSELoss()
loss_list = []

for ii, (val_img, score) in tqdm(enumerate(dataloader)):
    score = score.float()
    if opt.use_gpu and t.cuda.is_available():
        val_img = val_img.cuda()
        score = score.cuda()
    output = model(val_img).squeeze()
    loss = criterion(output, score)
    
    loss_list.append(loss.item())
loss_list = np.array(loss_list)
val_mse = np.mean(loss_list)

model.train()
return val_mse</code></pre>

ptrblck: <p>How large is the difference and are you using any random transformations in your dataset?</p>
CUDA out of memory error on batch size of 20

Arohan_Ajit: <p>I’m running my model on Google Colab and I’m faced with CUDA out of memory error. I have <code>batch_size = 20</code> and all the images are resized to <code>(224,224)</code> . I am using vgg13 model with pre-trained weights with a custom classifier:<br>
<code>classifier = nn.Sequential(OrderedDict([                           ('fc1', nn.Linear(25088, 25088)),                           ('relu', nn.ReLU()),                           ('dropout', nn.Dropout(0.5)),                           ('fc2', nn.Linear(25088, 2048)),                           ('relu', nn.ReLU()),                           ('dropout', nn.Dropout(0.5)),                           ('fc3', nn.Linear(2048, 1024)),                           ('relu', nn.ReLU()),                           ('dropout', nn.Dropout(0.5)),                           ('fc4', nn.Linear(1024, 10))                                                                              ]))</code><br>
I tried deleting images and labels of each iteration but no use. What else can I do to bring down memory usage and/or why is Cuda having an out of memory error?(I just trained a densenet 121 with no problems on same data)<br>
Notebook:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/arohanajit/disease-detector/blob/master/melanoma/torchvgg.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/arohanajit/disease-detector/blob/master/melanoma/torchvgg.ipynb" target="_blank" rel="nofollow noopener">arohanajit/disease-detector/blob/master/melanoma/torchvgg.ipynb</a></h4>
<pre><code class="lang-ipynb">{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Untitled5.ipynb",
      "provenance": [],
      "private_outputs": true,
      "authorship_tag": "ABX9TyOtIOg4+01eUxdQ/izPGWPK",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
</code></pre>

  This file has been truncated. <a href="https://github.com/arohanajit/disease-detector/blob/master/melanoma/torchvgg.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ptrblck: <p>Which GPU are you using and how much memory does it have?<br>
I just used the <code>torchvision.models.vgg13</code> model with your custom classifier and have a memory allocation of ~5.2GB after running the forward and backward pass using <code>input = torch.randn(20, 3, 224, 224, device='cuda')</code>.</p>