akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I calculate the abs of the difference between one value and its 8 surrounding values?

RoyGu: <p>To be specific, I have a tensor of shape (300,300). I want to calculate the sum of the abs of difference of the current pixel and its surrounding 8 pixels.For example, if in the tensor T, T[1,2] = 1 and its surrounding 8 pixels are 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, and the sum value should be 1.8. So there can be another (300,300) tensor according to it. For loop can be used, but it’s way too slow. Any help would be greatly appreciated!!</p>

JuanFMontesinos: <p>You can use torch fold function (<a href="https://pytorch.org/docs/stable/nn.html?highlight=fold#torch.nn.Fold" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=fold#torch.nn.Fold</a>) which allows you to slide a window with a custom function.</p>

RoyGu: <p>Thank you. I will try it!</p>

RoyGu: <p>The nn.Fold functions as conv2d, but I want the absolute difference value of each position. For example, [1, 2, 3] * [-1,2,-1].t() =0, but I want this to be 2. Is there any solution to it?</p>

phan_phan: <p>Say your tensor <code>x</code> is of shape <code>[batch, 1, 300, 300]</code><br>
You can do it by dividing the task in a few steps : <em>difference</em>, then <em>abs</em>, then <em>sum</em></p>
<ol>
<li>Compute the differences, using a custom kernel that outputs the 8 differences, on 8 channels. This kernel would be of shape <code>[8, 1, 3, 3]</code> and would look something like this:</li>
</ol>
<blockquote>
<p>w = torch.FloatTensor([<br>
[[-1, 0, 0], [0, 1, 0], [0, 0, 0]],<br>
[[0, -1, 0], [0, 1, 0], [0, 0, 0]],<br>
[[0, 0, -1], [0, 1, 0], [0, 0, 0]],<br>
[[0, 0, 0], [-1, 1, 0], [0, 0, 0]],<br>
[[0, 0, 0], [0, 1, -1], [0, 0, 0]],<br>
[[0, 0, 0], [0, 1, 0], [-1, 0, 0]],<br>
[[0, 0, 0], [0, 1, 0], [0, -1, 0]],<br>
[[0, 0, 0], [0, 1, 0], [0, 0, -1]]<br>
]).unsqueeze(1)</p>
</blockquote>
<p>You can see that each of the 8 “filters” will compute the difference with one neighbor.<br>
And with <a href="https://pytorch.org/docs/stable/nn.functional.html#conv2d" rel="noopener nofollow ugc"><strong><code>y = torch.nn.functional.conv2d(x, w)</code></strong></a>, the tensor <code>y</code> is of shape <code>[batch, 8, 298, 298]</code>.</p>
<ol start="2">
<li>
<p>You want the absolute, so <strong><code>y = y.abs()</code></strong></p>
</li>
<li>
<p>Do the sum over the channels dimension : <strong><code>y = y.sum(dim=1, keepdim=True)</code></strong></p>
</li>
</ol>
<p>You get the desired <code>y</code> with the shape <code>[batch, 1, 298, 298]</code></p>
<p>I’m not sure if using <code>Unfold</code> / <code>Fold</code> instead would be relevant.</p>

RoyGu: <p>It works! Thanks!!!</p>
Could i save weights in each layer to csv in pytorch?

astri: <p>Hello</p>
<p>I am a beginner in Deep Learning and doing research comparing keras backend tensorflow and pytorch. I just try writing model in pytorch and i had succeed print the weights. Is it possible to save those weights to csv file?</p>
<p>for reference this is my code</p>
<blockquote>
<p>class MultiLayerPerceptron(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, num_classes):<br>
super(MultiLayerPerceptron, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_size, hidden_size)<br>
self.Sigmoid = nn.Sigmoid()<br>
self.fc2 = nn.Linear(hidden_size, num_classes)<br>
def forward(self, x):<br>
out = self.fc1(x)<br>
out = self.Sigmoid(out)<br>
out = self.fc2(out)<br>
return out<br>
model = MultiLayerPerceptron(input_size, hidden_size, num_classes)<br>
weights = model.fc1.weight<br>
print(weights)</p>
</blockquote>

Nathan_Drake: <p>import numpy as np<br>
import pandas as pd</p>
<p>…<br>
weights = weights.detach().numpy()<br>
pd.DataFrame(weights).to_csv( ’ weights.csv ’ )</p>

astri: <p>thanks you it work well</p>
Basic NLLLoss Calculation Question

NestedSoftware: <p>Hi, I am confused about how to use <code>torch.nn.NLLLoss</code>. Below is a simple session in the Python REPL. I am expecting the result to be <code>0.35667494393873245</code>, but I am getting <code>-0.7000</code>. I’d greatly appreciated it if someone could steer me in the right direction on this. Thanks!</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; input = torch.tensor([[0.70, 0.26, 0.04]])
&gt;&gt;&gt; loss = nn.NLLLoss()
&gt;&gt;&gt; target = torch.tensor([0])
&gt;&gt;&gt; output = loss(input, target)
&gt;&gt;&gt; output
tensor(-0.7000)
&gt;&gt;&gt; import math
&gt;&gt;&gt; -math.log(0.70)
0.35667494393873245
</code></pre>

NestedSoftware: <p>I think I can see what’s happening now. I was a bit confused about how <code>NLLLoss</code> works. The calculation below shows that applying the negative log likelihood to an input processed through <code>softmax</code> produces the same result as running the input through <code>log_softmax</code> first, then just multiplying by -1. It also shows that applying <code>CrossEntropyLoss</code> to the <code>raw_input</code> is the same as applying <code>NLLLoss</code> to <code>log_softmax(input)</code>. I’m guessing that the <code>log_softmax</code> approach may be more numerically stable than using <code>softmax</code> first, and calculating the <code>log</code> of the result separately.</p>
<pre><code class="lang-python">&gt;&gt;&gt; raw_input = torch.tensor([[0.7, 0.26, 0.04]])
&gt;&gt;&gt; softmax_input = torch.softmax(raw_input, dim=1)
&gt;&gt;&gt; softmax_input
tensor([[0.4628, 0.2980, 0.2392]])
&gt;&gt;&gt; -torch.log(softmax_input)
tensor([[0.7705, 1.2105, 1.4305]])
&gt;&gt;&gt; log_softmax_input = torch.log_softmax(raw_input, dim=1)
&gt;&gt;&gt; log_softmax_input * -1
tensor([[0.7705, 1.2105, 1.4305]])
&gt;&gt;&gt; loss = nn.NLLLoss()
&gt;&gt;&gt; loss(log_softmax_input, torch.tensor([0]))
tensor(0.7705)
&gt;&gt;&gt; cross_entropy_loss = nn.CrossEntropyLoss()
&gt;&gt;&gt; cross_entropy_loss(raw_input, torch.tensor([0]))
tensor(0.7705)
</code></pre>

fredguth: <p>I understand from your experiment that F.NLL_Loss does not expect the likelihood as input, but the log likelihood (log softmax), do you agree with this assessment?</p>
<p>If that is so, I guess this is a source of lots of errors because I believe most people would guess (wrongly) that you should forward pass the likelihood to F.NLL_Loss.  Right?</p>

NestedSoftware: <p>Yes, indeed. The documentation does state this, though, so I either didn’t read it properly at the time or misinterpreted it. <a href="https://pytorch.org/docs/stable/nn.html#nllloss" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch 2.1 documentation</a> :</p>
<blockquote>
<p><em>The input given through a forward call is expected to contain log-probabilities of each class.</em></p>
</blockquote>
How to add extra layers to InceptionV3?

tlwill11: <p>I have been using ResNet for transfer learning, but wanted to try Inception to see if I’d get better results. My way of doing this for ResNet is as follows:</p>
<pre><code class="lang-auto">class ResNet18(nn.Module):
    def __init__(self, orig_model):
        super(ResNet18, self).__init__()
        self.drop = nn.Dropout2d(0.5).to(device)
        self.bn = nn.BatchNorm2d(512).to(device)
        self.bn2 = nn.BatchNorm1d(256)
        self.bn3 = nn.BatchNorm1d(100)
        self.orig = nn.Sequential(*(list(orig_model.children())[:-1])).to(device)
        for param in self.orig.parameters():
            param.requires_grad = True
            # Replace the last fully-connected layer
            # Parameters of newly constructed modules have requires_grad=True by default
        self.fc = nn.Linear(512, 256).to(device)
        self.fc2 = nn.Linear(256, 100).to(device)
        self.fc3 = nn.Linear(100, 2).to(device)

    def forward(self, x):
        x = self.orig(x)
        x = self.bn(x)
        x = x.view(x.size(0), -1)
        x = F.relu(self.bn2(self.fc(x)))
        x = self.drop(x)
        x = F.relu(self.bn3(self.fc2(x)))
        x = self.drop(x)
        x = F.relu(self.fc3(x))
        p = F.softmax(x, dim=1)
        return x, p
</code></pre>
<p>However, when I try this for Inception V3, I get errors. I have seen other solutions where they show how to manipulate the last FC layer. But I want to add more layers, and I haven’t seen a thread that shows a concrete example of this.</p>

ptrblck: <p>The <a href="https://github.com/pytorch/vision/blob/490966f28ea6e11421d684e94cd995d6c53d4698/torchvision/models/inception.py#L94" rel="nofollow noopener"><code>forward</code> method</a> of Inception is using some functional API calls, which will be missed, if you wrap all submodules in an <code>nn.Sequential</code> container.<br>
The better approach would be to derive your own class using Inception as the parent class and add your layers there.</p>

tlwill11: <p>Thank you. In this class, would I have to add the layers within the container, similar to <a href="https://medium.com/@14prakash/almost-any-image-classification-problem-using-pytorch-i-am-in-love-with-pytorch-26c7aa979ec4" rel="nofollow noopener">this</a>?</p>
DistributedDataParallel - need help

JohnCwok: <p>I am trying to train my network on a cluster of machines without GPU using <code>DistributedDataParallel</code> and read the tutorials associated but they are kinda terrible and many details are not explained.<br>
For instance, what do the following lines do ?</p>
<pre><code class="lang-auto">os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'  
</code></pre>
<p>Can I just copy them ? Also what is a <code>rank</code>,when do we call it and why is it even a variable ?  None of these apparently important concepts are clearly explained … Can somebody recommend me a good tutorial on how to do parallel computing with pytorch ? The lack of resources is driving crazy (I have no knowledge of parallel computing)</p>
<p>Any help will be appreciated</p>
Distributed Training with Slurm

heilaw: <p>I am trying to train a network distributively on 2 computing nodes. Our lab uses slurm for task scheduling. I am not sure if I am using the correct slurm script for running the training code.</p>
<p>I tried</p>
<blockquote>
<p>srun python train.py</p>
</blockquote>
<p>Is it the right way to launch the job? I am using <code>gloo</code> backend. How should  I verify <code>DistributedDataParallel</code> exchanges and average gradients between two nodes? I just want to make sure they are communicating correctly. Thanks!</p>

mortonjt: <p><a class="mention" href="/u/heilaw">@heilaw</a> did you have any success with this?</p>
<p>Found a link here from someone who claimed to get this working on slurm: <a href="https://www.glue.umd.edu/hpcc/help/software/pytorch.html#distrib" rel="nofollow noopener">https://www.glue.umd.edu/hpcc/help/software/pytorch.html#distrib</a></p>
GroupNorm is slower and consumes higher GPU memory than BatchNorm

zbh2047: <p>I use GroupNorm in pytorch instead of BatchNorm and keep all the others fixed. It shows that in Imagenet dataset, GroupNorm is 40% slower than BatchNorm, and consumes 33% more GPU memory than BatchNorm. I am really confused because GroupNorm shouldn’t need more calculation than BatchNorm. The details are listed below.<br>
For BatchNorm, one minibatch consumes 12.8 seconds with GPU memory 7.51GB;<br>
For GroupNorm, one minibatch consumes 17.9 seconds with GPU memory 10.02GB.</p>
Reusing a conv layer multiple times - weight doesn't change?

darrenjkt: <p>Hi I’m running the code snippet below.</p>
<pre><code class="lang-auto">class LFBlock(nn.Module):
    def __init__(self, kernel_size=(1,1,1), block_shape=(32,32,32)):
        super(LFBlock,self).__init__()
        
        self.block_shape = block_shape    
        self.conv = nn.Conv3d(1,1, kernel_size=kernel_size, stride=1, padding=0, bias=True)
                
    def forward(self, x):
        
        x1 = x[:,0][:,None]
        x2 = x[:,1][:,None]
        x3 = x[:,2][:,None]
        x4 = x[:,3][:,None]
        
        x1 = self.conv(x1)
        print('x1 | convweight=', self.conv.weight.data)
        x2 = self.conv(x2)
        print('x2 | convweight=', self.conv.weight.data)
        x3 = self.conv(x3)
        print('x3 | convweight=', self.conv.weight.data)
        x4 = self.conv(x4)
        print('x4 | convweight=', self.conv.weight.data)
</code></pre>
<p>The print outputs of this code snippet is:</p>
<p>x1 | convweight= tensor([[[[[-0.2251]]]]])<br>
x2 | convweight= tensor([[[[[-0.2251]]]]])<br>
x3 | convweight= tensor([[[[[-0.2251]]]]])<br>
x4 | convweight= tensor([[[[[-0.2251]]]]])</p>
<p>Where the weights of each conv is the same. This is a smaller scale testing of a model where I’m running lots of small convolutions ~16 for a single pass. Instead of defining 16 individual conv layers, I was thinking I could just reuse one. Where am I going wrong?</p>

albanD: <p>Hi,</p>
<p>The weight is associated to the Conv2D class instance. So if you reuse one instance, then you reuse its weights.<br>
If you want more than one instance to train different weights, you will need to create more than one instance of the Conv2D class.</p>

darrenjkt: <p>So if the weights were random e.g. 0.327 before conv, then after conv it became -0.2251. Then I do something with those weights. And I re-run the conv, wouldn’t it re-initialise with another random value then change based on what you feed it? Even if it were to randomly reinitialise to 0.327, I thought it would turn out different cause my inputs to it are different.</p>
<p>Or does it freeze the kernel weights after a single pass?</p>
<p>For my particular use case, I’m fine with overriding the conv weights cause I just want to use the weights straight after the conv for some equations</p>

albanD: <p>Hi,</p>
<p>You should check the doc about <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#nn-module" rel="nofollow noopener">nn.Module’s role</a>. That should give you a good introduction of what nn.Module (which is the parent class of Conv2D) are for.<br>
Feel free to ask more questions here afterwards.</p>
Will precision error accumulate with DistributedDataParallel?

zbh2047: <p>From Pytorch documentation, I know that when one use DistributedDataParallel, gradients on multiple GPUs will be averaged(all-reduced), and each model’s parameters will be updated based on the grapdients. So there is no need to sync model’s parameters. But we know float tensor addition might incur presicion error. With the training going on (such as millions of minibatches), will the presicion error affect the consistency of model’s parameters on miltiple GPUs? Thank you!</p>

albanD: <p>Hi,</p>
<p>I don’t think any extra step is taken on our side to avoid this. So you need to make sure that your values are well formed and can be accumulated with no problem (which should be the case if your inputs are correctly preprocessed and you use regular initialization method for your weights).</p>
Run PyTorch Programs in Background

Soniya: <p>Is there any way to run pytorch programs in background so that i can logout from terminal?</p>

trypag: <p>This question is not related to pytorch, it’s about linux in general,</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/4797050/how-to-run-node-js-as-a-background-process-and-never-die" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/196874/murvinlai" target="_blank" rel="nofollow noopener">
    <img alt="murvinlai" src="https://www.gravatar.com/avatar/3cc333b173e35a82fd27b52e71c50496?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/4797050/how-to-run-node-js-as-a-background-process-and-never-die" target="_blank" rel="nofollow noopener">How to run Node.js as a background process and never die?</a>
</h4>

<div class="tags">
  <strong>node.js, linux, background-process</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/196874/murvinlai" target="_blank" rel="nofollow noopener">
    murvinlai
  </a>
  on <a href="https://stackoverflow.com/questions/4797050/how-to-run-node-js-as-a-background-process-and-never-die" target="_blank" rel="nofollow noopener">05:53PM - 25 Jan 11 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


alexbellgrande: <p>I like tmux for this stuff. Some good guides are <a href="https://danielmiessler.com/study/tmux/" rel="nofollow noopener">here</a> and <a href="http://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/" rel="nofollow noopener">here</a>.</p>

JamesHowlettLA: <p>This is linux related issue, but it is simple, and I often forget the nohup.</p>
<pre><code>    nohup python ... &amp;</code></pre>
Data Normalization in MNIST

oboul: <p>Hi why do we need data normalization in MNIST  Data Loader example  ?</p>
<p>Thank you</p>

ptrblck: <p>Normalizing the data usually is beneficial for training the model.<br>
It’s sometimes not necessary, but might stabilize and accelerate the training.</p>

oboul: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> in order to have all the data sets ranges between 0 and 1 what parameters should I use while normalizing ? or even most of them are 0 and some of them are 1 ?</p>

ptrblck: <p>If you are loading <code>PIL.Image</code>s, <code>torchvision.transforms.ToTensor()</code> will create a normalized tensor with values in the range <code>[0, 1]</code>.<br>
<code>torchvision.transforms.Normalize()</code> will create standardized tensors with zero mean and a unit variance.<br>
Usually you would use the mean and standard deviation from the training set.<br>
However, often the ImageNet statistics are used for RGB images, especially if you are using a pretrained model (same input statistics) and if your dataset is similar to ImageNet.</p>

oboul: <p>Thankyou <a class="mention" href="/u/ptrblck">@ptrblck</a> if I understood correctly there is no need for normalization since it will be automatically between 0 and 1 ?</p>

ptrblck: <p>Normalizing the data additionally to zero mean and unit variance could further help in training.</p>

oboul: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> yes but using 0 as mean will create some negative values (since I tied to positive values only)?</p>

ptrblck: <p>Yes, this would produce positive and negative values. Have a look at <a href="https://en.m.wikipedia.org/wiki/Standard_score" rel="nofollow noopener">this wiki</a> for more information.</p>
Huge usage of virtual memory during training

Dawid_S: <p>Suddenly I noticed that the virtual memory usage is huge during my training. The size grows when the first Tensor is passed to GPU. Also, I noticed that using more GPU is much slower than training using one GPU. I am almost certain that the training time on 2 GPUs what almost 2 times faster than using one GPU.</p>
<p>I am using ubuntu, PyTorch 1.0.1 and CUDA 10.<br>
Here I can show htop output of training on one GPU</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d29d6bf9014f53fd93cb952edb1a596ff3d19741.png" data-download-href="https://discuss.pytorch.org/uploads/default/d29d6bf9014f53fd93cb952edb1a596ff3d19741" title="35"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d29d6bf9014f53fd93cb952edb1a596ff3d19741_2_690x339.png" alt="35" data-base62-sha1="u3bxGK1PDY3dr8YV8KWnfJyRpfz" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d29d6bf9014f53fd93cb952edb1a596ff3d19741_2_690x339.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d29d6bf9014f53fd93cb952edb1a596ff3d19741_2_1035x508.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d29d6bf9014f53fd93cb952edb1a596ff3d19741_2_1380x678.png 2x" data-dominant-color="414F45"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">35</span><span class="informations">2172×1068 936 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone have the idea what’s going on?</p>

Kushaj: <p>Pytorch claims all the memory of your GPU and even when it is not using that memory, if you run nvidia-smi it would show, no memory is free.<br>
Refer to memory management <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management" rel="nofollow noopener">docs</a> for more details.</p>

Dawid_S: <p>Excuse me, I am not able to understand the answer. nvidia-smi behaves normally - some of my chosen GPU’s are occupied at the same level as before in terms of RAM.</p>
<p>The simplest code that creates 3 visible lines with more than 20GB virtual memory taken in htop is here:</p>
<pre><code class="lang-auto">import torch
import time

cuda = torch.device('cuda')
A = torch.empty((100, 100), device=cuda).normal_(0.0, 1.0)

time.sleep(5)
</code></pre>

nlgranger: <p>I’d like to bump this up because it seems strange to me as well. Why so much virtual memory?</p>
Setting CUDA_VISIBLE_DEVICES after importing torch.utils.tensorboard doesn't work

cyvius96: <p>Hi,</p>
<p>I tried to use torch.utils.tensorboard (for replacing tensorboardX) and found that:</p>
<pre><code class="lang-auto">import os

import torch
import torch.nn.functional as F
from tqdm import tqdm
from torch.utils.data import DataLoader
os.environ['CUDA_VISIBLE_DEVICES'] = '4'
from torch.utils.tensorboard import SummaryWriter

(main code)
</code></pre>
<p>works normally, while</p>
<pre><code class="lang-auto">import os

import torch
import torch.nn.functional as F
from tqdm import tqdm
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
os.environ['CUDA_VISIBLE_DEVICES'] = '4'

(main code)
</code></pre>
<p>will still use gpu 0.</p>
<p>Am I using it wrong?</p>
<p>Thanks!</p>

ptrblck: <p>I would recommend to set the environment variables in the terminal before running the script, as setting it via <code>os.environ</code> might have side effects.<br>
E.g. if you are loading a library, which calls to CUDA and grabs all available GPUs, setting <code>'CUDA_VISIBLE_DEVICES'</code> won’t have any effect anymore, as it apparently the case in your code.<br>
If you really want to set it in your code, make sure <code>os.environ</code> is called first in your script.</p>

cyvius96: <p>Thank you!</p>
<p>It makes sense. However, I am wondering why replacing torch.utils.tensorboard to tensorboardX will make the code work?</p>
<p>I usually set os.environ after args are parsed since gpu is specified in args, which is typically after pakages are imported.</p>

ptrblck: <p>I’m not sure why this happens, but I would guess that some imports in the <a href="https://github.com/pytorch/pytorch/blob/98e312cf96f6a5e23933cd8794097063ee3cbc8c/torch/utils/tensorboard/__init__.py#L1" rel="nofollow noopener"><code>__init__</code> method</a> checks for GPUs and thus grabs them (which is apparently not the case in tensorboardX).<br>
However, I’m also not sure it’s worth investigating. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
pytorch fp16 inference

JensenHJS: <p>The model is trained with fp32. I try to use .half() to change layers and inputs to fp16. Actually，it indeed accelerate the inference. But its acceleration effect is far away from twice of fp32. My platform is nvidia tx2. Its compute capability is 6.2. It supports fp16 very well. So what I want to ask is whether fp16 cannot be twice as fast as fp32 in pytorch. Looking for your reply.Thank you.</p>

ptrblck: <p>The performance gains depend on the operations, shapes, and other potential bottlenecks.<br>
E.g. convolutions in cudnn7.2 and earlier needed to have input channels, output channels and batch size as multiple of 8 to be able to use TensorCores. This restriction was listed in cudnn7.3 and later.</p>
<p>GEMMs however should still use matrices with shapes of multiples of 8 to use TensorCores in cublas and cudnn.</p>
<p>Are you preloading the data or are you using e.g. a <code>DataLoader</code>?<br>
Note that you might encounter new bottlenecks in your code, if the model was the previous one and now got accelerated.</p>
Non-blocking transfer to GPU is not working

husam: <p>I am having a problem getting .to(device) to work asynchronously. The training loop in the first code snippet below takes 3X longer than the second snippet. The first snippet sets pin_memory=True, non_blocking=True and num_workers=12. The second snippet moves tensors to the GPU in <strong>getitem</strong> and uses num_workers=0. Images that are being loaded are of shape [1, 512, 512]. The target is just a single float32.<br>
Is there something I need to set in the CUDA drivers?<br>
GPU: V100<br>
PyTorch: 1.1.0<br>
Python: 3.7.4<br>
Cuda compilation tools, release 9.2, V9.2.148<br>
conda version : 4.6.14<br>
Ubuntu 16.04.5</p>
<pre><code class="lang-auto"># This is very slow. 
device = “cuda”
class MyDataset (Dataset):
        def __getitem__(self, idx):
              image = self.get_image_tensor(idx)
              target  = self.get_target(idx)
              return {"images": image, "targets": target}

train_dataset  = MyDataset()
train_loader = DataLoader(
               train_dataset,
               batch_size=16,
               shuffle=True,
               num_workers=12,
               pin_memory=True)

def train():
	for batch in train_loader:
        images = batch[“images”].to(device, non_blocking=True)
        targets = batch[“targets”].to(device, non_blocking=True)
</code></pre>
<pre><code class="lang-auto"># This is faster, but still slower than it should be.
device = “cuda”
class MyDataset (Dataset):
      def __getitem__(self, idx):
            image = self.get_image_tensor(idx).to(device)
            target  = self.get_target(idx).to(device)
            return {"images": image, "targets": target}

train_dataset  = MyDataset()
train_loader = DataLoader(
               train_dataset,
               batch_size=16,
               shuffle=True,
               num_workers=0,
               pin_memory=False)

def train():
      for batch in train_loader:
          images = batch[“images”]
          targets = batch[“targets”]
</code></pre>

albanD: <p>Hi,<br>
I think it’s expected that there is an overhead of using multiple workers if your tensors are very cheap to load. The point is to be able to load from disk ahead of time to reduce the impact of the slow hard drives.</p>
<p>Why is the second one “slower than it should be” ?</p>

husam: <p>I meant to say that it would be much faster if the multi-worker version worked properly.<br>
In the first snippet, GPU utilization is very low. The call to .to(device) starts only after the current batch being processed by the GPU is done which causes a lot of GPU idle time. The same implementation runs much faster in TensorFlow.</p>

ptrblck: <p>We had some issues using pinned memory recently (<a class="mention" href="/u/rwightman">@rwightman</a> reported it <a href="https://github.com/pytorch/pytorch/issues/25010" rel="nofollow noopener">here</a>), which were fixed recently, so you could try out the nightly build or build from source and check, if the profiling changes.<br>
For general data loading bottlenecks, have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>.</p>
Rank 0 gpu of torch.nn.parallel.DistributedDataParllel will gather gradients from other gpus?

Kyrie_Yan: <p>When I use torch.nn.parallel.DistributedDataParllel for training , I found process launched for  training on all other gpus would take up some memory at GPU 0 ,i.e worker 0. This take much more memory of GPU 0 that I can’t use bigger batch. How Can I solve this?  Is it because other process send gradients to 0 by default to calculate the average???<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8853ac5ec20d5d2022ae9b066657c641ff8f9db1.png" data-download-href="https://discuss.pytorch.org/uploads/default/8853ac5ec20d5d2022ae9b066657c641ff8f9db1" title="Screenshot_2019-04-10_18-11-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8853ac5ec20d5d2022ae9b066657c641ff8f9db1_2_690x267.png" alt="Screenshot_2019-04-10_18-11-05" data-base62-sha1="js0cIXsMaVEgp9poRUbFf2QPS93" width="690" height="267" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8853ac5ec20d5d2022ae9b066657c641ff8f9db1_2_690x267.png, https://discuss.pytorch.org/uploads/default/original/2X/8/8853ac5ec20d5d2022ae9b066657c641ff8f9db1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/8853ac5ec20d5d2022ae9b066657c641ff8f9db1.png 2x" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-04-10_18-11-05</span><span class="informations">940×364 70.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Kyrie_Yan: <p>By the way in the screenshot, the process for training step on GPU0 was killed since it crashes due to outof memory error</p>

davy: <p>hi, i meet same problem  , have you solve it? thx</p>
Autograd Functions: non-tensor arguments

AbsolutelyClueless: <p><strong>questions</strong></p>
<ol>
<li>can i pass non-tensor arguments to autograd forward/backward?</li>
<li>can i save non-tensor arguments with <code>ctx.save_for_backward</code>?</li>
</ol>
<hr>
<p><strong>background info</strong></p>
<p>In the documentation for <code>torch.autograd.Function.backward</code> it is stated that:</p>
<blockquote>
<p>It must accept a context  <code>ctx</code>  as the first argument, followed by as many outputs did <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function.forward" rel="noopener nofollow ugc"> <code>forward()</code> </a> return, and it should return as many tensors, as there were inputs to <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function.forward" rel="noopener nofollow ugc"> <code>forward()</code> </a>. Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.</p>
</blockquote>
<p>I need 2 arguments to be passed from the module variables to the autograd function and into the kernel, an integer and a boolean. From the above documentation it sounds like this will not work with <code>backward</code> as those aren’t tensors. (I don’t wanna make them tensors either)</p>
<p>Currently my forward looks like:</p>
<pre><code class="lang-auto">class modulefunction (autograd.Function):

    @staticmethod
    def forward (ctx,*args):

        output,*variables = _cuda_.module(args)
        ctx.save_for_backward(variables)

        return output

    @staticmethod
    def backward (ctx,d_output):

        return _cuda_.dmodule(d_output,*ctx.saved_variables)
    

class module (nn.Module):

    ...

    def forward (self,input):

        ...

        return modulefunction.apply(input,*other_tensors,integer,boolean)
</code></pre>
<p>Do i just have the cpp backward binding return placeholder values like <code>0</code> for those extra arguments?</p>
Returning a trained model from a class?

nile649: <p>It is more of a python question than pytorch. I have a model which I would like to pass to class A object, then the same model should be returned from class A object, and passed to class B object.</p>
<p>Class A and Class B have different data to train the model, but instead of passing the model independently to class A and then class B, I would like to pass returned model from class A to class B.</p>
<p>If anyone could help or suggest material that would be amazing.<br>
Thank you</p>

ptrblck: <p>I’m not sure, what your use case is, but would this work?</p>
<pre><code class="lang-python">class classA(object):
    def train(self, model):
        # train model
        return model

class classB(object):
    def train(self, model):
        # train model
        return model

model = nn.Linear(10, 10)
A = classA()
B = classB()
model = A.train(model)
model = B.train(model)

# alternatively
model = B.train(A.train(model))
</code></pre>
VAE KL divergence

Rojin: <p>I used to think that as a VAE model is trained, the KL gets smaller. But according to my training, it seems to be wrong. I was wondering if someone can explain why the KL term is increasing instead of decreasing? Is that what is expected? or maybe I’m doing something wrong here</p>
<p>blue = total loss<br>
green = BCE loss<br>
red = KL</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/772c3531f757a590b9fc4a89dd9a3174604a93e5.png" alt="image" data-base62-sha1="h0fx5IgZSCEs80ECFQbmvFqewn3" width="368" height="248"></p>
Find if a module is being traced

dashesy: <p>I would like to call a different function when tracing. I have registered it through<br>
<code>torch::jit::RegisterOperators()</code> and had to change it a little for that, so would like to call it only if it is tracing.<br>
Is there a way to find if the module is being traced?<br>
I currently rely on the fact that <code>tenssor.shape[0]</code> is a tensor while tracing, but an <code>int</code> otherwise. That is not the best way.</p>
<p>Something like <code>torch.jit.is_running</code></p>

wanchaol: <p>Hi thanks for posting the question. We do have a method do know if it’s in tracing state or not using <code>torch._C._get_tracing_state()</code>, it will return true if you are in tracing otherwise false. But note that this is a internal API and it might break in the future (also unlikely).</p>

dashesy: <p>Thanks! that would be very useful, a public facing API would be most very welcome. It will be better than relying on <code>if isinstance(bbs.shape[0], torch.Tensor)</code>.</p>
Visualizing feature maps - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

dib: <p>I’m trying to visualize the feature maps of my network through guided backpropagation. I’m running this through <code>model.eval()</code>.</p>
<p>My first error, upon the final stages, is <code>RuntimeError: invalid gradient at index 0 - got [1, 6] but expected shape compatible with [1, 10647, 6]</code>. It seems this stems from:</p>
<pre><code class="lang-auto">onehot_out = torch.FloatTensor(1, model_out.size()[-1]).zero_()
</code></pre>
<p>so I fix it with:</p>
<pre><code class="lang-auto">onehot_out = torch.FloatTensor(1, model_out.size()[1], model_out.size()[-1]).zero_()
</code></pre>
<p>This then seems to work, however when I then do the backward pass:</p>
<pre><code class="lang-auto">onehot_out[0][target_class] = 1
model_out.backward(gradient = onehot_out)
</code></pre>
<p>I get the error present in the title:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 3, 52, 52, 2]], which is output 0 of SigmoidBackward, is a
t version 6; expected version 5 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)
</code></pre>
<p>I’m unsure what this means or where to start looking. Would appreciate any help.</p>

albanD: <p>Hi,</p>
<p>The best way to look at these issues is to enable the <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" rel="nofollow noopener">anomaly detection</a> mode. This will give you the method in the forward that caused this issue.</p>

dib: <p>Hi,</p>
<p>Thank you yes. I enabled it shortly after posting this and solved my problem.</p>
<p>Shall I close or delete this thread?</p>

albanD: <p>You can simply accept the answer of “enable anomaly mode” and leave it like that.<br>
Nice that you found the problem !</p>
Dealing with "CUDA error: uncorrectable ECC error encountered"

Yaroslav_Bulatov: <p>I’ve recently started observing the errors of the following kind:</p>
<p>019-09-15 13:52:45,866 - log.py:60 - Failed<br>
Traceback (most recent call last):<br>
File “train.py”, line 770, in <br>
main_loop()<br>
File “train.py”, line 649, in main_loop<br>
util.record(‘params’, torch.sum(util.flat_param(model)).item())<br>
RuntimeError: CUDA error: uncorrectable ECC error encountered</p>
<p>Is it something that I could be causing, or should I blame it on cosmic rays?</p>

ptrblck: <p>How often do you see these errors?</p>

Yaroslav_Bulatov: <p>I had two such crashes in one day, but have not seen it since then after rerunning a few times, so seems hardware related.</p>
Using dataparallel on microscopic images (20000x20000)

pytorchhh: <p>Hi All,</p>
<p>I’m trying to develop an image segmentation deep learning pipeline for microscopic images at super-resolutions. Have two 2080 RTX Tis, 11gb vram each.</p>
<p>I obviously cannot get my batch size over 1, but unfortunately dataparallel only splits across the batch size. Is there an already available way to split my compute on a single image across different gpus with pytorch?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>I’m afraid these isn’t as it is not a very common use case.<br>
You might be able to split the image yourself into a batch of smaller images and then reconstruct the result?</p>
Set weights for embedding layer

Dan_Xie: <p>Hello, I tried to initialize the weights of the embedding layer with my own embedding, by methods below _create_emb_layer.</p>
<p>I am so confused why the weights changed after init the model.</p>
<p>class clf(nn.Module):</p>
<pre><code>def __init__(self, weight_matrix):        
    super(clf, self).__init__()
    self.embedding, self.vocal_size, self.embed_dim = self._create_emb_layer(weight_matrix, trainable=False)
    **print('original matrix:', weight_matrix[0])**
    **print('after init matrix', self.embedding.weight.detach().numpy()[0])**

def _create_emb_layer(self, weight_matrix, trainable=False):
    num_embeddings, embedding_dim = weight_matrix.shape
    emb_layer = nn.Embedding(num_embeddings, embedding_dim)
    emb_layer.weights = torch.nn.Parameter(torch.from_numpy(weight_matrix))
    if trainable:
        emb_layer.weight.requires_grad = True   
    else:
        emb_layer.weight.requires_grad = False
    return emb_layer, num_embeddings, embedding_dim</code></pre>

Dan_Xie: <p>Found out the problem –  I set emb_layer.weights instead of emb_layer.weight. Found out by printing out state_dict.</p>
Substitute nn.Modules in network

kcang2: <p>Hi,<br>
I am trying to prune a network. I tried getting the nn.Modules like nn.Conv2d etc. and just changed the conv.weight or other parameters etc. to a reduced tensor.</p>
<p>Eg. a Conv2d layer is pruned as below:</p>
<pre><code class="lang-auto">            for param in [prev.weight, prev.bias]:
                param.data = self.remove(param, lc)
</code></pre>
<p>My way of removing a channel:</p>
<pre><code class="lang-auto">    def remove(self, param, lc, dim=0):
        if dim == 0:  # BN params &amp; biases (any vector) OR prev conv (remove filter)
            tmp1 = param[:lc]
            tmp2 = param[lc+1:]

        elif dim == 1:  # next (remove channel)
            tmp1 = param[:, :lc, ...]  # out, in, H, W
            tmp2 = param[:, lc+1:, ...]  # out, in, H, W

        return torch.cat([tmp1, tmp2], dim=dim)
</code></pre>
<p>During the forward pass it behaves OK and the output size matches the pruned size, but during backward pass it throws this error:</p>
<p><code>RuntimeError: Function CudnnConvolutionBackward returned an invalid gradient at index 1 - got [64, 127, 3, 3] but expected shape compatible with [64, 128, 3, 3]</code></p>
<p>By printing the network, I can see that nn.Conv2d still retain the old shapes.<br>
<code>Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))</code></p>
<p>How should I change that particular nn.Conv2d? Specifically, after I created a brand-new nn.Conv2d and setting its parameters to the pruned, old-layer’s weights, how can I insert the new Conv2d at the correct part of my network? Or is there an easier way for pruning?</p>
<p>Here’s my repo and files. I mostly work in prune_utils.py and eval.py<br>
<a href="https://github.com/kcang2/Prune_U-Net" rel="nofollow noopener">https://github.com/kcang2/Prune_U-Net</a></p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<ul>
<li>You should reset the <code>.grad</code> field of your weights and biais to <code>None</code> if you change their size to avoid any size problem when accumulating gradients.</li>
<li>You should rewrap your new weight into an <code>nn.Parameter()</code> before setting it so that it will be properly catched by things like <code>model.parameters()</code>. Also note that your pruning will break the previous <code>model.parameters()</code> so you need to do it again after the pruning.</li>
<li>You can monkey patch the convolution parameters directly into the instance you have (see the names <a href="https://github.com/pytorch/pytorch/blob/a3915bdb9d542c66e857aa55ddc7eacb5612da44/torch/nn/modules/conv.py#L26-L35" rel="nofollow noopener">here</a> for what to change) or create a brand new one and replace the other nn.Module with things like <code>model.conv1 = my_new_module</code>.</li>
</ul>
Intermediate data type using model.half()

Shuan_Holmes: <p>Hello!<br>
I try to accelerate CNNs on TitanV using FP16. I use this piece of code:</p>
<pre><code class="lang-auto"># ...
model.half()
# ...
image = image.half()
#...
outputs = model(image)
</code></pre>
<p>but I wonder whether the intermediate results (i.e., accumulator) in convolution operations can be represented by FP16. Besides, several activation functions (e.g., Sigmoid, Tanh) may require high precision representations. As <a href="https://pytorch.org/docs/1.0.1/nn.html?highlight=half#torch.nn.Module.half" rel="noopener nofollow ugc">doc</a> said,</p>
<blockquote>
<p>half() Casts all floating-point parameters and buffers to  <code>half</code>  datatype.</p>
</blockquote>
<p>which means those nonlinear operations still use FP16?</p>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>It does convert everything to float16 which indeed is not optimal in most cases.<br>
The nvidia folks have done a lot of testing around this and release the <a href="https://devblogs.nvidia.com/apex-pytorch-easy-mixed-precision-training/" rel="nofollow noopener">AMP</a> library to automatically only convert the right part of the model to fp16 !</p>
How to load video data into 3D Conv

SamTX: <p>Hi,</p>
<p>I’m kind of new to PyTorch and would like to create a data loader for video inpainting. My data is currently in the following format, where I have folder for each video within which the video is broken down into frames.</p>
<pre><code class="lang-auto">-- vid 1
     -- img 0
     -- img 1
         ...
     -- img N

 -- vid 2
     -- img 0
     -- img 1
         ...
     -- img N
</code></pre>
<p>And I’d like to know how do I create a data loader to input into a 3D conv network, I believe the input tensor has to be [batch, channels, sequence, height, width],<br>
Does this mean, in a single batch, I must go through the entire video? , just a different sequence ?</p>

spanev: <p>Hi <a class="mention" href="/u/samtx">@SamTX</a>,</p>
<p>Usually, you will subdivide your videos in sequence of some length.</p>
<p>You may want to take a look at DALI and its VideoReader operator, you can find <a href="https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/examples/video/video_reader_simple_example.html" rel="nofollow noopener">here</a> a small tutorial showing how to a video data pipeline and <a href="https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/examples/pytorch/pytorch-basic_example.html" rel="nofollow noopener">here</a> how to easily plug DALI into PyTorch.</p>
Issue with nn.TransformerDecoder layer

Hasan_Sait_ARSLAN1: <p>Hi all,</p>
<p>I am getting an error with the usage of the nn.TransformerDecoder layer.</p>
<p>I initialize the layer as follows:</p>
<p>self.transformer_decoder_layer = nn.TransformerDecoderLayer(2048, 8)<br>
self.transformer_decoder = nn.TransformerDecoder(self.transformer_decoder_layer, num_layers=6)</p>
<p>However, under forward method, when I run “self.transformer_decoder” layer as following;</p>
<p>tgt_mem = self.transformer_decoder(tgt_emb, mem)</p>
<p>where tgt_emb.shape = torch.Size([8, 9, 2048])<br>
and mem.shape = torch.Size([8, 68, 2048]</p>
<p>I get the following error:</p>
<blockquote>
<p>File “/gpfs/hpc/home/hasan90/nPAPER2/IMG+TXT/model/MultiModal/Decoder.py”, line 59, in forward<br>
tgt_mem = self.transformer_decoder(tgt_emb, mem)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/transformer.py”, line 216, in forward<br>
memory_key_padding_mask=memory_key_padding_mask)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/transformer.py”, line 329, in forward<br>
key_padding_mask=memory_key_padding_mask)[0]<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/modules/activation.py”, line 783, in forward<br>
attn_mask=attn_mask)<br>
File “/gpfs/hpc/home/hasan90/build/miniconda2/envs/python3/lib/python3.6/site-packages/torch/nn/functional.py”, line 3213, in multi_head_attention_forward<br>
k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)<br>
RuntimeError: shape ‘[-1, 72, 256]’ is invalid for input of size 1114112</p>
</blockquote>
<p>How I can solve the problem?</p>
<p>Thanks,</p>

Hasan_Sait_ARSLAN1: <p>ohh my bad,</p>
<p>I just read the docs of transformer class, I give the tensors wrongly. it must be in (seq_len, B, feat_size) format.</p>
ConvTranspose2d VS Bilinear upsample

Mandy: <p>Hi,</p>
<p>I was wondering if someone could tell me what’re the differences between</p>
<pre><code class="lang-python">ConvTranspose2d(group=in_channel) and Upsample(mode='bilinear')
</code></pre>
<p>Thanks</p>

ptrblck: <p><code>Upsample</code> will use the <code>mode</code> to “mathematically” upsample the activation (no training), while <code>ConvTranspose2d</code> will use trainable filter kernels.</p>

Mandy: <p>That make sense. Is there any possible to convert <code>ConvTranspose2d</code> to <code>Upsample</code>?</p>

ptrblck: <p>You could replace the layer and try to make sure to keep the same output shape.</p>
I have 8 GPUs ,buyt why my torch.cuda.device_count() return only 1?

Steve_Hu: <p>i use ‘nvidia-smi -L’ to list my avaiable gpus,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/991c104a79063a9194142085a549d0dd5dc0799c.png" alt="image" data-base62-sha1="lQtb6eNquEzYBe39Mf4Cad6k7k8" width="620" height="190"><br>
.<br>
i don’t set the CUDA_VISIBLE_DEVICES and run my script ‘print(torch.cuda.device_count())’ and it always return 1, anyone can tell me what’s happen? thx!</p>

ptrblck: <p>Could you check, if this environment variable is set somewhere in your code?<br>
The line should e.g. look like this:</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"]="0"
</code></pre>