akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convolution with a stack of images

Shubham_Jain: <p>Hi everyone!<br>
I have a stack of medical images (CT scans) that are labeled as a bunch (malignant or benign). I would like to train a model to predict whether a patient’s tumor is malignant or benign. Here is a list of issues I can’t find solutions for:</p>
<ol>
<li>
<p><strong>Loading the data</strong>: Is it necessary to use a dataloader? All the tutorials I’ve seen seem to be using them. Is an iterable list of the stack of images and their labels sufficient?</p>
</li>
<li>
<p><strong>Variable number of images</strong>: As I mentioned, the number of images in each stack is different. How do I overcome this? Is there a way to create a model to work around this issue without grouping/padding?</p>
</li>
<li>
<p><strong>conv2d or conv3d</strong>: I can’t seem to find any information online about conv3d being used for anything other than colour images with 3 channels. My images are grayscale so do I use conv2d or treat the number of images in the stack as the number of channels? I’m also struggling with finding good resources to understand how conv3d works so any links would be great.</p>
</li>
</ol>
<p>Any and all help is much appreciated! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=6" title=":grinning:" class="emoji" alt=":grinning:"></p>
<p><em>I’m sorry if I’ve asked for too much in a single post. New to this branch of computer science so please go easy</em> <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

Rahimi_Phz: <p>Hi, did you managed to solve this problem? I have the same questions.</p>
<p>Hope to hear from you soon.</p>
How to achieve high precision?

lephesien: <p>Hi all,</p>
<p>In an image binary classification problem, I’ve got an unbalanced dataset (10 000 positives for 200 000 negatives).</p>
<p>The metric I want to optimize is Precision : I need positive to be mostly true positive when an image is predicted as positive.</p>
<p>So as expected If I equally randomly split my train dataset (5000 POS / 5000 NEG) I got acceptable precision on an equally splitted test dataset (70%) but very poor on a REAL unbalanced dataset (24%).</p>
<p>How do i train my model for high precision with pytorch ?</p>
<p>Today i am using this optimizer :<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)</p>
<p>Thanks all for your help,</p>
<p>Nicolas</p>

arinaruck: <p>Hey! Did you solve the issue after all? I’m struggling with the same thing at the moment</p>

Vrushank98: <p>You have data imbalance problem, to alleviate that, You can,</p>
<ol>
<li>Use weighted Cross Entropy loss.  Or</li>
<li>Use WeightedRandomSampler() , see <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this post</a> for more clarification.</li>
</ol>
<p>After that you can experiment with different optimizers, and learning rates by using learning rate schedulers.</p>

arinaruck: <p>Thank you for your reply!<br>
Would you mind clarifying the difference between WeightedRandomSampling and over-/under-sampling in training when it comes to imbalanced inference?<br>
As for the Focal Loss suggestion, I am not sure why it helps mitigate the issue when transferring from train to test, but I am definitely going to try it since I have not yet experimented with it. Thanks!</p>

Vrushank98: <p>WeightedRandomSampler(WRS) is used when you don’t want to or can’t under-/over-sample your dataset. <a href="https://androidkt.com/deal-with-an-imbalanced-dataset-using-weightedrandomsampler-in-pytorch/" rel="noopener nofollow ugc">This blog</a> explains WRS really well, would recommend checking it out.</p>
<p>When you assign class-weights to Cross Entropy Loss / BCELoss, every time your model makes an incorrect prediction, the loss is multiplied with the given class-weight. So ultimately, the model will push itself to learn meaningful representations of the minority class to minimize the loss. This works adequately 90% of the time.</p>
<p>For example: If you have two classes: Dog(2000 images), and Cat(100 images). If you define your loss as:</p>
<p><code>BCELoss(weights = Tensor([1.0 , 20.0]))</code></p>
<p>every time your model predicts cat image as a dog image while training, the loss will be multipled by 20.0, so to minimize the loss, model has to learn the meaningful representations of cat. Hence, you’ll get a better accuracy and precision.</p>
<p>But, still, over-sampling your dataset properly is the best option if you want better overall performance.</p>
<p>Hope it helps!</p>
Loading the pretrained network on different number of labels

bing: <p>Hi,<br>
I trained a network using transfer learning to classify 2 labels.<br>
Then I tried to use the trained weights from the above case to retrain another problem to classify 3 labels.<br><br>
Now, I am getting a <code>shape mismatch</code> error, obviously, because the endpoint classes are different.<br>
Is it possible to load the weights in the way I am thinking, <br>How to resolve the error?</p>
<pre><code class="lang-auto">def load_checkpoint(checkpoint_path, model):
    """To Load model."""
    checkpoint = torch.load(checkpoint_path, map_location='cuda:0' if torch.cuda.is_available() else 'cpu') # noqa
    model.load_state_dict(checkpoint['state_dict'])
    return model

# To load the architecture of the model to be loaded
model_transfer1 = EfficientNet.from_pretrained('efficientnet-b2',weights_path='/home/rxs1576/latest_scripts/Project_QA/EfficientNetPytorch/efficientnet-b2-8bb594d6.pth') # noqa
n_inputs = model_transfer1._fc.in_features
model_transfer1._fc = nn.Linear(n_inputs, 3)

# Loading my retrained model
model_transfer = load_checkpoint(checkpoint_path='My retrained model',model = model_transfer1)

# freezing the entire network
for name, parameter in model_transfer.named_parameters():
    parameter.requires_grad = False


# %%
# Defreezing the end points of the network so I can train them again.
update_params_name = ['_fc.weight', '_fc.bias', '_conv_head.weight']
for name, parameter in model_transfer.named_parameters():
    if name in update_params_name:
        parameter.requires_grad = True
</code></pre>

eqy: <p>If the goal is to finetune the network after changing the last layer, can you do the modification after loading the weights? For example, can you move the line</p>
<pre><code class="lang-auto">model_transfer1._fc = nn.Linear(n_inputs, 3)
</code></pre>
<p>to after</p>
<pre><code class="lang-auto">model_transfer = load_checkpoint(checkpoint_path='My retrained model',model = model_transfer1)
</code></pre>
<p>?</p>

bing: <aside class="quote no-group" data-username="bing" data-post="1" data-topic="127241">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">n_inputs = model_transfer1._fc.in_features
model_transfer1._fc = nn.Linear(n_inputs, 3)
</code></pre>
</blockquote>
</aside>
<p>Thanks, I have to do the step twice, first to remove the 1000 labels(ImageNet dataset) to 2(My trained network) and then to 3 (my desired finetuning network).</p>
How to read just one pic?

ForeverZH0204: <p>I finished my training about a classify network with the help of DataLoader.That’s awesome.</p>
<p>When i use just one pic to test my model, i found that the image read with the help of <code>opencv</code> in the format of array is different from that get from <code>DataLoader</code>.<br>
Is that really different from the result of <code>DataLoader</code> if i read with <code>cv2.imread</code> or did i make some mistake?</p>
<p>If i want to read one pic with the <strong>same result</strong> as that getting by <code>DataLoader</code>,what can i do ?</p>

ptrblck: <p>It depends how you’ve read the images in your <code>Dataset</code>. The <code>DataLoader</code> just calls <code>Dataset</code>'s <code>__getitem__</code> method.<br>
Did you use <a href="https://pytorch.org/docs/master/torchvision/datasets.html#torchvision.datasets.ImageFolder">torchvision.datasets.ImageFolder</a>?</p>
<p>If so, the images were probably loaded with PIL, if you didn’t install <code>accimage</code>.<br>
You can find the line of code <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L126">here</a>.</p>
<p>However, you need to provide the batch dimension at <code>dim0</code>, so your code would look like this:</p>
<pre><code class="lang-auto">from PIL import Image
import torchvision.transforms.functional as TF

image = Image.open('YOUR_PATH')
x = TF.to_tensor(image)
x.unsqueeze_(0)
print(x.shape)

output = model(X)
</code></pre>

ForeverZH0204: <p>Thanks a lot!<br>
I found another question:</p>
<blockquote>
<p>when all my pics (n-class)are stored like this:<br>
train/ant/***.jpg <em><strong>.jpg<br>
train/snake/</strong></em>.jpg <em><strong>.jpg<br>
train/bear/</strong></em>.jpg ***.jpg</p>
</blockquote>
<p>when i use DataLoader,the label is <code>0</code>–<code>n-1</code>,how do pytorch decide that which class is class 0,which is class 1 etc. According to the alphabet order?</p>

ptrblck: <p>Yes, the folders are found using</p>
<pre><code>classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
classes.sort()
</code></pre>
<p>The <code>.sort()</code> call sorts the folders in alphabetic order.<br>
You can find the code <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L22">here</a>.</p>

ForeverZH0204: <p>Thank U！<br>
Wish U a good day!</p>

andrejankas: <p>In newer versions ImageFolder (alias <em>DatasetFolder</em> now) doesn’t read any more images, since logic is given by user-defined callable, right?</p>

ptrblck: <p><code>ImageFolder</code> is not an alias for <code>DatasetFolder</code>, but uses is as its base class (which was this case for a while now). You are correct: if you need more flexibility in e.g. the image loading etc. you can derive your custom dataset from <code>DatasetFolder</code>.</p>
How to manually set gradients for the entire network?

kyy: <p>Hi, I am trying to manually set the gradients of the entire network during training.  Specifically I am interested to do the following:</p>
<pre data-code-wrap="Python"><code class="lang-Python">optimizer.zero_grad()
# set gradients for all components of the network
optimizer.step()
</code></pre>
<p>I know that for a single module we can use following:</p>
<aside class="quote quote-modified" data-post="2" data-topic="17428">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/48/4165_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-set-gradients-manually-and-update-weights-using-that/17428/2">How to set gradients manually and update weights using that?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    1.Are you 100% sure that there is no difference in param after the step?, maybe you use an adaptive optimizer and the learning rate is almost zero? 
2.Are you 100% sure that the optimizer is responsible for this param update? 
I wrote this code to test this: (I am using v0.4.0): 
import torch

target = torch.randn(1, 10)
input = torch.randn(1, 20)
network = torch.nn.Linear(20, 10)
optimizer = torch.optim.SGD(network.parameters(), lr=0.01)
optimizer.zero_grad()
loss = ((network(input)-target)**2)…
  </blockquote>
</aside>

<p>But it would be complicated to set each module of the entire network in this way since the network may contain dozens of different modules.  I am wondering if there is a better way to do so?  I am thinking of possibilities of using something similar to <code>state_dict</code> when copying weights for the entire network, just wonder if there are similar things for gradients?</p>
<p>Thank you!</p>
Copying nn.Modules without shared memory

lenny: <p>You can use the <code>share_memory()</code> function on an <code>nn.Module</code> so that the same parameters can be accessed from multiple processes (using the multiprocessing module). If I do not want these parameters to be shared, and I want each subprocess to get an independent copy of the parameters to work with, will simply not calling <code>share_memory()</code> provide this behavior?</p>

smth: <p>no, what you need to do is to send them model to the new process, and then do:</p>
<pre><code class="lang-auto">import copy
model = copy.deepcopy(model)
</code></pre>
<p>share_memory() only shares the memory ahead of time (in case you want to reuse the shared memory for example)</p>

apaszke: <p>Or, assuming your model is a module called <code>MyModel</code>, you can create a separate instance in each process, send a <code>state_dict</code> from a single one to everyone else, and have them <code>load_state_dict</code>. The parameters in the <code>state_dict</code> will be shared among the processes, but <code>load_state_dict</code> only copies the content, without reassigning tensors, so your model won’t be shared.</p>
<p>However, if you want to duplicate models within a single process, I definitely recommend the way proposed by <a class="mention" href="/u/smth">@smth</a>.</p>

Juna: <p>Then could you give me an example of using load_state_dict() to copy a module?</p>

JosueCom: <p>He is referring to saving the state_dict (for instance into a file), sending it over to another process, then loading it with load_state_dict</p>
How transform outputs of maxpooling module to tensor?

students: <p>Hi, I divide a learned network in submodule. output of  maxpooling module is tuple and I need transform to tensor that to give next one module that have conv2d and relu. what am I doing?</p>
<pre><code class="lang-auto">torch.save(model, 'model1.pth')
model1 = torch.load('model1.pth')
#%%

l1 = nn.Sequential(*list(model1.children())[:2]).cuda()
print('l1=',l1)

l2 = nn.Sequential(list(model1.children())[2]).cuda()
print('l2=',l2)

l3 = nn.Sequential(*list(model1.children())[3:5]).cuda()
print('l3=',l3)


l4 = nn.Sequential(*list(model1.children())[5:7]).cuda()
print('l4=',l4)

data1 = next(iter(testloader))

img2,_ = data1
img2[:,:, 45:55, 40:50]=0
 
y1=l1(img2.cuda()).cuda()

y2=l2(y1)

y2 = torch.Tensor(y2)
</code></pre>
<p>this part have error:</p>
<pre><code class="lang-auto">   y2 = torch.Tensor(y2)

ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>l1,l2,l3,l4:</p>
<pre><code class="lang-auto">l1= Sequential(
  (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU()
)
l2= Sequential(
  (0): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
)
l3= Sequential(
  (0): Conv2d(16, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU()
)
l4= Sequential(
  (0): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1))
  (1): ReLU()
)</code></pre>

eqy: <p>I don’t think the error is due to MaxPooling because the output should be a tensor as shown in the docs: <a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html" class="inline-onebox" rel="noopener nofollow ugc">MaxPool2d — PyTorch 1.9.0 documentation</a></p>
<p>Can you post a minimal code snippet that reproduces this error or provide some details like the type of <code>y2</code>?</p>

students: <p>thank for your help. type is tuple. because I extract indices of maxpoolings the type was tuple.</p>
Why my network converges slowly

S_M: <p>Hello, I use a simple CNN with 70000 parameters and I use the Tiny-Imagenet dataset<br>
my preprocessing is like below</p>
<pre><code class="lang-auto">       train_batch_size = 128
        test_batch_size = 128
        lr = 0.001
        momentum = 0.9
        weight_decay = 0
        seed = 7
        margin = 1.0
        log_interval = 10
        resume = '-r'
        epochs = 10
        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])
        transform_train = transforms.Compose([
                transforms.Resize(32),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize])
                    
        transform_test = transforms.Compose([
                transforms.Resize(32),
                transforms.ToTensor(),
                normalize])
        train_dataset = datasets.ImageFolder('./tiny-imagenet-200/train', transform=transform_train)
        test_dataset = datasets.ImageFolder('./tiny-imagenet-200/val', transform=transform_test)
        train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = train_batch_size, shuffle = True, num_workers=0)    
        test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = test_batch_size, shuffle = False, num_workers=0)
</code></pre>
<p>after 100 epochs accuracy is equal to 23% and loss is 3% my optimizer is SGD. I used a deeper network like Alesxnet but my result didn’t change.<br>
what’s wrong with my training. Any help would be appreciated.</p>
Fine tuning the model

Myailab: <p>Hi</p>
<p>I am in search of appropriate loss function and corresponding learning rate for my problem.<br>
I am new to deep learning and pytorch, so just want to confirm if my approach is correct ?</p>
<p>Model is trained for each loss function in different attempts with different learning rates, such as 0.01,0.001,0.0001.<br>
I observed that at first epoch only, there is a considerable difference in the metric at different learning rates and similar difference is more or less maintained upto 5 epochs.<br>
May i know if observing just 5 epochs is sensible approach or should I wait for the model to converge to make decision regarding learning rate?<br>
Thanks</p>
How to get the features from the Resnet model

Myailab: <p>Hi, Can I please get some help here.<br>
I have been using the following code to obtain:<br>
output = model(images)</p>
<p>where model = get_resnet34()</p>
<pre><code class="lang-auto">def get_resnet34(num_classes=6, **_):
    model_name = 'resnet34'
    model = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
    arc_margin_product=ArcMarginProduct(512, num_classes)
    conv1 = model.conv1
    model.conv1 = nn.Conv2d(in_channels=4,
                            out_channels=conv1.out_channels,
                            kernel_size=conv1.kernel_size,
                            stride=conv1.stride,
                            padding=conv1.padding,
                            bias=conv1.bias)
    model.conv1.weight.data[:,:3,:,:] = conv1.weight.data
    model.conv1.weight.data[:,3:,:,:] = conv1.weight.data[:,:1,:,:]

    model.avgpool = nn.AdaptiveAvgPool2d(1)
    in_features = model.last_linear.in_features   
    model.last_linear = nn.Linear(in_features, num_classes)
    #cosine=arc_margin_product(in_features)
    
    return model
</code></pre>
<p>Now I want to use Resnet34 output as features to be an input into the following code</p>
<pre><code class="lang-auto">def forward(self, features):
        cosine = F.linear(F.normalize(features), F.normalize(self.weight.cuda()))
        return cosine
</code></pre>
<p>i am getting this error 'torch.nn.modules.module.ModuleAttributeError: ‘ResNet’ object has no attribute ‘norm’ '.</p>
<p>May I know how do i get to ‘return features’ from ‘get_resnet34(num_classes=6, **_)’, instead of ‘returning model’.</p>
<p>Thanks</p>

ptrblck: <p>The error doesn’t match the posted code, as you are not using <code>model.norm()</code> anywhere.</p>
<aside class="quote no-group" data-username="Myailab" data-post="1" data-topic="127297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/9de053/48.png" class="avatar"> Myailab:</div>
<blockquote>
<p>May I know how do i get to ‘return features’ from ‘get_resnet34(num_classes=6, **_)’, instead of ‘returning model’.</p>
</blockquote>
</aside>
<p>After creating the model, you could replace the last linear layer with an <code>nn.Identity</code> module to get the features instead of the logits from the last layer.</p>

Myailab: <p>Thank you very much.</p>
DDP batch size division

char-t: <p>Hi!</p>
<p>I am implementing a model using DDP (1 node 2 GPU) and am confused about the batch size. I am using a distributed data sampler with a dataloader to load my data. When initialising the dataloader I specify batch_size = 16. In the training loop each process then receives a batch of 16 making a total batch size of 32.</p>
<p>Does this behaviour sound correct? In the below text, it seems to me that the batch size could be automatically scaled by the DDP module.</p>
<p>"This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension. The module is replicated on each machine and each device, and each such replica handles a portion of the input. During the backwards pass, gradients from each node are averaged.</p>
<p>The batch size should be larger than the number of GPUs used locally."</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html</a></p>
<p>Any help much appreciated!</p>
Why FaceNet uses un-normalized feature embedding for training?

xinqi: <p>Follow the FaceNet code <a href="https://github.com/bubbliiiing/facenet-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bubbliiiing/facenet-pytorch: 这是一个facenet-pytorch的库，可以用于训练自己的人脸识别模型。</a>. During training, we can find that it used unnormalized feature embedding, and added a classifier. However, during inference, it used normalized feature embedding.</p>
<p>My question is that, why not directly use the normalized feature embedding for training? What is the thoery behind using unormalized feature embedding for trainning, while normalized one for inference?</p>
<p>BTW, I understand normalization can put all features into the same scale, and will work better for inference. Please make sure to answer why not use normalized feature embedding for trainning？</p>
Exact pybind11 commit PyTorch 1.9.0 was built against

csdw: <p>Hi Everyone,</p>
<p>I am getting a runtime error with my JIT/ATen code that seems to indicate that the pybind11 version that I am building against is not the correct one that PyTorch 1.9 was built against.<br>
I am struggling to find any information on where to find the pybind11 version used, can anyone help please?</p>
<p>Many thanks and with best wishes</p>
<p>Christian</p>

ptrblck: <p>This is the currently tagged <code>pybind11</code> version:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pybind/pybind11/tree/8de7772cc72daca8e947b79b83fea46214931604">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pybind/pybind11/tree/8de7772cc72daca8e947b79b83fea46214931604" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pybind/pybind11/tree/8de7772cc72daca8e947b79b83fea46214931604" target="_blank" rel="noopener">GitHub - pybind/pybind11 at 8de7772cc72daca8e947b79b83fea46214931604</a></h3>

  <p><a href="https://github.com/pybind/pybind11/tree/8de7772cc72daca8e947b79b83fea46214931604" target="_blank" rel="noopener">8de7772cc72daca8e947b79b83fea46214931604</a></p>

  <p><span class="label1">Seamless operability between C++11 and Python. Contribute to pybind/pybind11 development by creating an account on GitHub.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>(you can find it <a href="https://github.com/pytorch/pytorch/tree/master/third_party">here</a>)</p>
Libtorch torch::save() seem to have a problem

Chaser-wind: <p>I use Libtorch 1.8.1(LTS)+cpu version and try to run the simplest example in Visual Studio 2019:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;
int main() {
    auto tensor = torch::ones({3, 4});
    torch::save(tensor, "ones.pt");
    return 0;
}
</code></pre>
<p>Then it runs into error…</p>
<pre><code class="lang-auto">Uncaught exception: Microsoft C++ exception std::bad_alloc
</code></pre>
<p>and  it points to libtorch\include\torch\csrc\api\include\torch\serialize\torch.h line 10</p>
<pre><code class="lang-auto">inline serialize::OutputArchive&amp; operator&lt;&lt;(
    serialize::OutputArchive&amp; archive,
    const Tensor&amp; tensor) {
    archive.write("0", tensor); // line 10
    return archive;
}
</code></pre>
Converting HDF file to Pytorch torch geometric data format

csl: <p>I am trying to convert a hdf5 file to a torch geometric format by wrapping the loaded input with the tg.data.data function but I am getting the following error. Has anyone happened to run into this before?</p>
<pre><code class="lang-auto">class HDFDataset(tg.data.Dataset):

  def __init__(self, path, **kwargs):
    print('Path: ', path)
    '''Initialiser for HDFDataset class'''
    self.files = h5py.File(path, 'r')
    self.data = list((self.files).keys())

  def __len__(self):
    return len(self.data)

  def __getitem__(self, idx):
    '''Return training information at provided index'''
    if not 0 &lt;= idx &lt; len(self):
      raise Exception(f'Event number {idx} invalid – must be in range 0 -&gt; {len(self)-1}.')
    # Retrieve datasets from graph at index
    f = self.files
    ret = { dataset: f[str(idx)][dataset][:] for dataset in f[str(idx)].keys() }
    ret['edge_index'] = torch.tensor(ret['edge_index'][:])
    return tg.data.Data(**ret)
</code></pre>
<p>Traceback (most recent call last):<br>
File “model_shuffle.py”, line 80, in <br>
print(‘HDF DataLoader ‘, next(iter(h_loader)))<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 517, in <strong>next</strong><br>
data = self._next_data()<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 557, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch_geometric/data/dataloader.py”, line 36, in <strong>call</strong><br>
return self.collate(batch)<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch_geometric/data/dataloader.py”, line 16, in collate<br>
return Batch.from_data_list(batch, self.follow_batch,<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch_geometric/data/batch.py”, line 105, in from_data_list<br>
inc = data.<strong>inc</strong>(key, item)<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch_geometric/data/data.py”, line 190, in <strong>inc</strong><br>
return self.num_nodes if bool(re.search(’(index|face)’, key)) else 0<br>
File “/Users/songhyunlee/venv/lib/python3.8/site-packages/torch_geometric/data/data.py”, line 215, in num_nodes<br>
return item.size(self.<strong>cat_dim</strong>(key, item))<br>
TypeError: ‘int’ object is not callable</p>

ptrblck: <p>I guess you are using a numpy array, which returns an <code>int</code> when accessing the <code>size</code> attribute, while PyTorch tensors provide a <code>size()</code> method, which would return the size of the tensor:</p>
<pre><code class="lang-python">a = np.random.randn(3, 2)
print(a.size)
&gt; 6
print(a.size())
&gt; TypeError: 'int' object is not callable
</code></pre>
<p>so you could transform the numpy array to a tensor.</p>
How do detect if a layer is an output layer?

Samue1: <p>Typically I use</p>
<pre><code class="lang-auto">for name, module in self.model.named_children():
    # do stuff here with layers
</code></pre>
<p>to iterate through my model to modify layers. Is there a way where I can detect if the current <code>module</code> is the last in my network, so that I can treat this layer differently?</p>

ptrblck: <p>The proper way would be to check the <code>forward</code> method of your model, as it would define which layer is used last.<br>
Since you could use arbitrary conditions inside the <code>forward</code>, you would have to check the actual logic.<br>
In case you are using an <code>nn.Sequential</code> module, you could directly use the last layer.</p>
Multiprocessing is much slower than for loop in forward

huangdi: <p>For some reason, I need to use a <code>for loop</code> to process each batch differently in training. To make it faster, I want to use <code>multiprocessing</code> to deploy different batch on different process. However, I found that using multiprocessing is even slower than a simple for loop. What’s wrong with my code?</p>
<p>Here’s my example code:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Pool, Process, set_start_method
try:
    set_start_method('spawn', force=True)
except RuntimeError:
    pass
from torch import nn
import torch
from tqdm import tqdm

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.model = nn.Linear(128, 128)

    def forward(self, x, pool):
        output = self.model(x)
        ### select multiprocess or for loop here ###
        self.multiprocess(x, pool)
#        self.singleprocess(x, pool)

        return output 

    def f(self, x):
        return x ** 2

    def multiprocess(self, x, pool):
        res = list(pool.imap(self.f, x))

    def singleprocess(self, x, pool):
        for idx in range(len(x)):
            x[idx] = x[idx] ** 2

if __name__ == '__main__':
    model = Model()
    pool = Pool(processes=8)
    for iter in tqdm(range(1000)):
        ### fake data ###
        x = torch.ones(100, 128)
        output = model(x, pool)
</code></pre>
<p>The results shows that <code>multiprocessing</code> is at  least 36 times slower than <code>for loop</code>.</p>
How to automatically free intermediate tensors in memory?

kyy: <p>Hi, I notice my jupyter notebook is using super large memory (~40GB) when it is running, and after using the tool here: <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/2" class="inline-onebox">How to debug causes of GPU memory leaks? - #2 by SpandanMadan</a>, I found most memory is used by some intermediate tensor variables.  For instance, if I have a large numpy array <code>X</code>, and pass it to a network <code>net</code> as <code>net.cpu()(torch.tensor(X))</code>, then this intermediate variable <code>torch.tensor(X)</code> stays in memory forever even after this function call finished.</p>
<p>I am wondering if there is a way to keep only “named” variables/tensors in memory, while automatically free anything that are intermediate results?</p>
<p>Thank you!</p>

Shanbhag: <p>Hey! I have this query too. I am looking forward to a reply.<br>
Thanks.</p>

ptrblck: <p>Intermediate tensors will be freed, once <code>backward()</code> (without <code>retain_graph=True</code>) is used and these intermediates are not needed for the gradient calculation anymore.<br>
If you don’t need to compute the gradients and call <code>backward()</code>, you can wrap the forward pass into a <code>with torch.no_grad()</code> block, which would not store intermediate activations.</p>
<p>CC <a class="mention" href="/u/shanbhag">@Shanbhag</a></p>
Issue in allocating a large tensor to GPU

Peishi: <p>Hello,</p>
<p>I’m new to pytorch, and have an issue with assigning a large tensor to GPU. The memory of the tensor is larger than the GPU 16GB memory (and I have multiple such tensors for training/val/test). I suppose it should be a common issue that has been well resolved by many others. Any suggestion?</p>
<p>I read a related post <a href="https://discuss.pytorch.org/t/is-it-possible-to-load-a-tensor-larger-than-the-capacity-of-a-single-gpu-memory/22547">here</a>, which hasn’t been answered.</p>
<p>Thanks in advance!</p>

ptrblck: <p>You could try to use a model parallel approach (i.e. splitting the operation into separate parts) to save memory, as you won’t be able to allocate more device memory than is available.<br>
The implementation depends on the actual operation.</p>

Peishi: <p>Thank you for the reply! The problem is solved by simply feeding the GPU with each batch during training, instead of sending all data together. But the model parallel approach looks also interesting. Is there any pytorch tutorial for that?</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html">Here</a> is a tutorial for a simple model parallelism.</p>
Simple VGG16 on MNIST (10 classes)

CasellaJr: <p>I have this notebook, where there is a simple VGG16 used to do classification on MNIST:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1IN0HD7-ljlPFtsbstfxLSKWvg2y2ndmO?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210716-060119-RC00_385112924" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1IN0HD7-ljlPFtsbstfxLSKWvg2y2ndmO?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1IN0HD7-ljlPFtsbstfxLSKWvg2y2ndmO?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
These are the results of a training of 10 epochs:</p>
<pre><code class="lang-auto">Epoch 1: TrL=0.3749, TrA=0.1150, VL=0.1300, VA=0.1121, TeL=0.1198, TeA=0.1120,
Epoch 2: TrL=0.0821, TrA=0.1123, VL=0.0889, VA=0.1115, TeL=0.0556, TeA=0.1131,
Epoch 3: TrL=0.0577, TrA=0.1122, VL=0.0677, VA=0.1130, TeL=0.0566, TeA=0.1137,
Epoch 4: TrL=0.0428, TrA=0.1122, VL=0.0648, VA=0.1130, TeL=0.0461, TeA=0.1139,
Epoch 5: TrL=0.0343, TrA=0.1122, VL=0.0611, VA=0.1130, TeL=0.0412, TeA=0.1138,
Epoch 6: TrL=0.0302, TrA=0.1120, VL=0.0581, VA=0.1128, TeL=0.0401, TeA=0.1144,
Epoch 7: TrL=0.0257, TrA=0.1122, VL=0.0474, VA=0.1129, TeL=0.0384, TeA=0.1138,
Epoch 8: TrL=0.0224, TrA=0.1122, VL=0.0465, VA=0.1124, TeL=0.0394, TeA=0.1136,
Epoch 9: TrL=0.0179, TrA=0.1122, VL=0.0460, VA=0.1130, TeL=0.0374, TeA=0.1138,
Epoch 10: TrL=0.0162, TrA=0.1122, VL=0.0435, VA=0.1126, TeL=0.0348, TeA=0.1136,
</code></pre>
<p>How is it possible that accuracy is so low? Do you think there is an error in the training function when I define the accuracy?</p>

eqy: <p>I think there is an error in the accuracy calculation, as the equality comparison is done twice.</p>
<pre><code class="lang-auto">                    pred_labels = (pred_label == labels).float()

                    batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
</code></pre>
<p>should probably be</p>
<pre><code class="lang-auto">                    pred_labels = (pred_label == labels).float()

                    batch_accuracy = pred_labels.sum().item()/input.size(0)
</code></pre>
<p>The previous code will compare the labels with the already compared result, so if most of the predictions are correct (the result is 1), you will get a low accuracy value reported as the second comparison will effectively count how many 1s there are in the current batch.</p>

CasellaJr: <p>Thank you, it’s correct</p>
Weights loaded through torch.hub.load give 0% performance

439327bcc62cd8e45e96: <p>I download imagenet data through Kaggle and loaded the swav pretrained weights using the torch.hub.load method.</p>
<p>After that, for evaluation, we tried to evaluate the imagenet val set as a swav model. But the performance is 10% Is it possible to receive data from Kaggle?(<a href="https://www.kaggle.com/c/imagenet-object-localization-challenge" class="inline-onebox" rel="noopener nofollow ugc">ImageNet Object Localization Challenge | Kaggle</a>)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff67d6e319a424f616abdf7bbbc15629adb4c766.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff67d6e319a424f616abdf7bbbc15629adb4c766" title="Screen Shot 2021-07-20 at 8.44.38 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff67d6e319a424f616abdf7bbbc15629adb4c766_2_550x500.png" alt="Screen Shot 2021-07-20 at 8.44.38 PM" data-base62-sha1="ArqefzOrNderTbQ7fN8SG6eyExU" width="550" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff67d6e319a424f616abdf7bbbc15629adb4c766_2_550x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff67d6e319a424f616abdf7bbbc15629adb4c766_2_825x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff67d6e319a424f616abdf7bbbc15629adb4c766_2_1100x1000.png 2x" data-dominant-color="3B3F49"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-07-20 at 8.44.38 PM</span><span class="informations">1386×1258 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If anyone knows please help me.</p>

ptrblck: <p>The repository contains an <a href="https://github.com/facebookresearch/swav/blob/5e073db0cc69dea22aa75e92bfdd75011e888f28/eval_linear.py">evaluation script</a>, which I would assume should be able to reproduce the reported performance.<br>
Are you able to run it and verify the perf.?</p>
Iterating over a 1d tensor starting from the end

fudu: <p>trying to make this work:</p>
<pre><code class="lang-auto">for (i,idx) in enumerate(ids[::-1]):
  ...
</code></pre>

ptrblck: <p>The <code>.flip</code> operation should work:</p>
<pre><code class="lang-python">ids = torch.arange(10)
for (i,idx) in enumerate(ids.flip(0)):
  print(idx)
</code></pre>
Having trouble importing PyTorch

Telespielstube: <p>Hello,</p>
<p>I’m having a rather unusual problem importing PyTorch. I use VirtualBox 6.1 and Ubuntu 20.4 to run my ROS application with PyTorch as backend. Since yesterday I have the following error when I want to train my neural network. It worked a couple of weeks ago and I haven’t changed the code sind then. the only thing changed was the PyTorch 1.9.0 release.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/marta/catkin_ws/src/imagineer/src/ai_service/ai_service.py", line 2, in &lt;module&gt;
    import torch, os, numpy
ImportError: No module named torch
</code></pre>
<p>But <code>PyTorch</code> is definitely installed I tested it with:</p>
<pre><code class="lang-auto">Python 3.8.5 (default, May 27 2021, 13:30:53) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
1.8.0+cpu
&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt; 
</code></pre>
<p>I already deleted my <code>catkin_ws</code> Workspace, deleted and reinstalled ROS, reinstalled PyTorch 1.9.0 and 1.8.0 (current version now is 1.8.0)  but the error message remains. Any idea where the problem could be?</p>

ptrblck: <p>Are you using a different conda/pip environment or a different <code>python</code> (e.g. launched via <code>sudo python</code>)?</p>

Telespielstube: <p>No, I haven’t touched my VirtualBox since I had a working application.</p>
Quantized Functional Embedding

pintonos: <p>Hello!</p>
<p>Since it seems there is no implementation for quantized embeddings with the Functional interace available, I want to ask if there is a workaround to this?</p>
<p>To be specific: How can I use a Functional embedding with quantization?</p>
<p>Thanks!</p>

HDCharles: <p>The only thing we support for quantized embeddings are modules. I think the only real workaround is converting to embedding modules unfortunately.</p>
Torch.linalg.solve large matrix cuda error

fswerty: <p>Hi, I am trying to execute the following commands on cuda but facing the error. The code is running fine on CPU and with smaller matrix size (like 400 instead of 4096 with cuda). I am using Volta100 GPU with pytorch 1.9 and cuda 11.1. Can anyone recommend any workaround? Thanks.</p>
<p>A = torch.randn(2, 3, 4096).cuda()<br>
B = torch.randn(2, 3, 3).cuda()<br>
X = torch.linalg.solve( B,A)</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
/tmp/ipykernel_29550/983601019.py in <br>
1 A = torch.randn(2, 3, 4096).cuda()<br>
2 B = torch.randn(2, 3, 3).cuda()<br>
----&gt; 3 X = torch.linalg.solve( B,A)<br>
RuntimeError: CUDA error: invalid configuration argument<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track and fix it, please?</p>

fswerty: <p>Thanks for the response. I have posted in on <a href="https://github.com/pytorch/pytorch/issues/61929" rel="noopener nofollow ugc">github</a>.</p>
Weights loaded through torch.hub.load give 0% performance.

111546: <p>I download imagenet data through Kaggle and loaded the swav pretrained weights using the torch.hub.load method.</p>
<p>After that, for evaluation, we tried to evaluate the imagenet val set as a swav model. But the performance is 10% Is it possible to receive data from Kaggle?(<a href="https://www.kaggle.com/c/imagenet-object-localization-challenge" class="inline-onebox" rel="noopener nofollow ugc">ImageNet Object Localization Challenge | Kaggle</a>)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a.png" data-download-href="https://discuss.pytorch.org/uploads/default/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a" title="Screen Shot 2021-07-20 at 8.25.32 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a_2_575x500.png" alt="Screen Shot 2021-07-20 at 8.25.32 PM" data-base62-sha1="cr4aLGPpQRKWTZ90UwXaUsV7spk" width="575" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a_2_575x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a_2_862x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57295e98401e0a3dca18b6b7c65f7bcee1aa7b6a_2_1150x1000.png 2x" data-dominant-color="3B3F49"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-07-20 at 8.25.32 PM</span><span class="informations">1420×1234 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If anyone knows please help me.</p>

eqy: <p>If I understand correctly, the expected ~75% accuracy of the swav model is on the ImageNet 2012 validation set. However, it looks like the linked data is for a newer, different ImageNet competition.</p>
<p>10% accuracy seems to suggest that the data is from a totally different distribution and that the object categories are different. However, 10% seems low if the object categories are the same. Could you check the directory organization of the validation set and see if the class labels match with the classes in the 2012 dataset? You might be able to compare the directory names to check this (e.g., if the class directory names match those in this shell script: <a href="https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh" rel="noopener nofollow ugc">https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh</a>).</p>
Scatter_ for computing histogram: problems with dimensions

vadimkantorov: <p>I’d like to compute a flat 5-bin histogram counts using <code>scatter_(..., reduce = 'add')</code>. I expect the following to produce <code>[0.0, 16.0, 0.0, 0.0, 0.0]</code>. What am I doing wrong dimension-wise? Also, the error message is hard to parse (especially the latter part).</p>
<p>Thanks!</p>
<pre><code class="lang-python">import torch
histsize = 5
Z = torch.ones(16, dtype = torch.int64)
print(torch.zeros((histsize,), dtype = torch.float32).scatter_(-1, Z, 1.0, reduce = 'add'))

# torch.zeros((histsize,), dtype = torch.float32).scatter_(-1, Z, 1.0, reduce = 'add')
RuntimeError: Expected index [16] to be smaller than self [5] apart from dimension 0 and to be smaller size than src [5]
</code></pre>
<p>This works:</p>
<pre><code class="lang-python">import torch
histsize = 5
Z = torch.ones(16, dtype = torch.int64)
torch.zeros((histsize,), dtype = torch.int64).scatter_add_(-1, Z, torch.ones_like(Z))
</code></pre>

vadimkantorov: <p>Seems the same problem as in <a href="https://github.com/pytorch/pytorch/issues/61854" class="inline-onebox" rel="noopener nofollow ugc">Inconsistent behavior of `torch.Tensor.scatter_` for float `src` · Issue #61854 · pytorch/pytorch · GitHub</a> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Understanding use of xm.mark_step() in torch_xla

thepurpleowl: <p>In documentation, <a href="http://pytorch.org/xla/release/1.9/index.html" rel="noopener nofollow ugc">here</a>, when <a href="http://pytorch.org/xla/release/1.9/index.html#running-on-a-single-xla-device" rel="noopener nofollow ugc">running on single XLA device</a> requires <code>xm.mark_step()</code> and not <code>xm.optimizer_step(optimizer)</code>. But when <a href="http://pytorch.org/xla/release/1.9/index.html#running-on-multiple-xla-devices-with-multi-processing" rel="noopener nofollow ugc">running on Multiple XLA Devices</a> requires <code>xm.optimizer_step(optimizer)</code> and not <code>xm.mark_step()</code>.</p>
<p>I couldn’t understand this.</p>

Jack_Cao: <p>If you check the code in <a href="https://github.com/pytorch/xla/blob/815197139b94e5655ed6b347f48864e73dc73011/torch_xla/core/xla_model.py#L795" class="inline-onebox" rel="noopener nofollow ugc">xla/xla_model.py at 815197139b94e5655ed6b347f48864e73dc73011 · pytorch/xla · GitHub</a>, you will find that <code>optimizer_step</code> will call <code>mark_step</code> after the <code>optimizer.step</code>. This means if you call <code>xm.optimizer_step</code> at the step end, you don’t need another call to <code>xm.mark_step()</code>.</p>
Torch scatter non-deterministic behavior with not unique indices

Maciek_Gruszczynski: <p>Hi,<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html#torch.Tensor.scatter_" rel="noopener nofollow ugc">scatter documentation</a> warns that “When indices are not unique, the behavior is non-deterministic (one of the values from <code>src</code> will be picked arbitrarily) and the gradient will be incorrect (it will be propagated to all locations in the source that correspond to the same index)!”</p>
<p>Given <em>src</em> tensor and <em>index</em> tensor with not unique indices, is there a way to make scatter deterministic? I don’t care about gradients.<br>
I tried to make values at duplicated indices the same (I need max) but failed with it after a few hours…</p>
<p>Do you know any solution? I know how to do it with for loops only.</p>
<p>Cheers</p>