akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is Mean Average Precision the same for online action/event detection

Sazid_Ali: <p>hello,</p>
<p>This is my first time writing something. So, if any mistakes are there do let me know.</p>
<p>I was recently working on an online action/event detection where they have used to metrics namely:</p>
<ol>
<li>mAP( Mean Average Precision)</li>
<li>mcAP( Mean Average Calibrated Precision).</li>
</ol>
<p>Can someone tell me if the mAP used here and the one used at Object detection are both the same or not?<br>
If not can anyone kindly explain what’s the difference between them? Or If anyone has used them can explain them in brief ( Talking about mAP and mcAP).</p>
<p>I have attached the link where I came across these two metrics. You can read about them at Page no 9 <strong>" Proposed evaluation protocol "</strong>:</p>
<p><a href="http://isis-data.science.uva.nl/cgmsnoek/pub/geest-online-action-eccv2016.pdf" rel="nofollow noopener">http://isis-data.science.uva.nl/cgmsnoek/pub/geest-online-action-eccv2016.pdf</a></p>
Unexpected nan gradient

gebbissimo: <p>Hello everybody,</p>
<p>in the very simple example below, pytorch produces a nan-gradient. However, I would have expected a finite gradient. Is my math wrong or something strange happening inside pytorch?</p>
<pre><code class="lang-auto">import torch
def run_simple():
    #  y = s*[x1,x2] = 3 * [1,NaN]
    x = torch.tensor([1, float('NaN')]).float()
    scalor = torch.tensor([3]).float()
    scalor.requires_grad = True
    y = x * scalor

    # loss = y[mask_valid].mean() = s*x1 = 3
    mask_valid = torch.isfinite(x)
    loss = y[mask_valid].mean()  # = 3
    loss.backward()
    print(scalor.grad)  # [nan] ?! Would have expected dloss/ds = d(s*x1)/ds = x1 = 1
</code></pre>
<p>P.s: Tested with version 0.4.1 and 1.4.0</p>
Sigmoid for regression

arjun_pukale: <p>Can we use sigmoid activation function for regression task where i have to predict a value between 0 and 1?</p>

googlebot: <p>sure, that’s called logistic regression</p>

Julien_Despois: <p>Just make sure that you’re using a Binary Cross-Entropy loss, as the MSE loss can have some issues with the Sigmoid (<a href="https://www.noobtonoob.com/post/mse-after-sigmoid#:~:text=The%20use%20of%20MSE%20after,instead%20of%20MSE%20loss%20function." rel="nofollow noopener">source</a>)</p>
densenet/Alexnet finetuning

Sayeed_Shaiban: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7917e9f18c814a8af597d5018d50900edb84d06a.png" data-download-href="https://discuss.pytorch.org/uploads/default/7917e9f18c814a8af597d5018d50900edb84d06a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7917e9f18c814a8af597d5018d50900edb84d06a.png" alt="image" data-base62-sha1="hhf0yXtyfYqek0dqiiNCSEK9iFs" width="690" height="184" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">807×216 10.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the last layers of densenet from the pytorch hub models. I added an extra linear layer after the given linear layer in the following way.</p>
<pre><code class="lang-auto">import torch.nn as nn
model = torch.hub.load('pytorch/vision:v0.6.0', 'densenet121', pretrained=True)
modules = list(model.children()) 
new = nn.Sequential(*modules,
                    nn.Linear(model.classifier.out_features, 100))
</code></pre>
<p>But when I use</p>
<pre><code class="lang-auto">x = torch.ones((1,3, 224, 224))

out = new(x)

out
</code></pre>
<p>I get the followin error-<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c53547e76cb6f773d0584c07396bb96c43a1bf2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c53547e76cb6f773d0584c07396bb96c43a1bf2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c53547e76cb6f773d0584c07396bb96c43a1bf2.png" alt="image" data-base62-sha1="1L2gkKT57txTaW17GXckTP977pg" width="690" height="255" data-dominant-color="F8F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">947×350 16.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Why so? Aren’t the dimensions right here?<br>
Similar things happen when adding  a new layer to Alexnet too. but it works if I edit the dimension of the existing layers.</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>The problem comes with the way of constructing the new model. Please use the following code snippet. It will work.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

model = torch.hub.load('pytorch/vision:v0.6.0', 'densenet121', pretrained=True)
n_model = nn.Sequential(model,
                    nn.Linear(model.classifier.out_features, 100))
x = torch.ones((1,3, 224, 224))
out = n_model(x)
print(out.shape)
</code></pre>
<p>Thanks</p>
Can't concatenate tensors

Vitsuji_Aura: <p>Hi,</p>
<p>I am trying to append some more data points to my batch but I can’t get it to work.<br>
I have one tensor of [64, 40] and the other a [64, 10].</p>
<p>Whenever I do torch.cat or torch.stack it gives me a dimension error.</p>
<p>Anyone know how to append the latter [64, 10] to the [64, 40] tensor?</p>

pfloat: <p>hello<br>
try this:</p>
<pre><code class="lang-auto">b = torch.rand(64, 10)
a = torch.rand(64, 40) 
torch.cat([a, b], dim=1) # shape is torch.Size([64, 50])
</code></pre>
<p>you can have more details in these posts:<br>
<a href="https://discuss.pytorch.org/t/how-to-concatenate-list-of-pytorch-tensors/1350/5">how-to-concatenate-list-of-pytorch-tensors</a><br>
<a href="https://discuss.pytorch.org/t/tensor-stack-or-concatenate/34331">tensor-stack-or-concatenate</a></p>

Vitsuji_Aura: <aside class="quote no-group" data-username="pfloat" data-post="2" data-topic="95903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pfloat/48/28529_2.png" class="avatar"> pfloat:</div>
<blockquote>
<p><code>torch.cat([a, b], dim=1)</code></p>
</blockquote>
</aside>
<p>Nice it worked. Thank you.</p>
Detecting a torch.jit.script function

mfkasim: <p>Is there any easy way to reliably detect if a function is a JIT ScriptFunction?<br>
For example:</p>
<pre><code class="lang-auto">@torch.jit.script
def func1(x):
    return x.sum()

def func2(x):
    return x.sum()

def detect_jit_script(func):
    pass # should return True for func1 and False for func2
</code></pre>
<p>I realize that <code>func1</code> is compiled to a method (so it’s no longer a function).<br>
However, using <code>inspect.ismethod</code> doesn’t work because the input <code>func</code> can be a method of a class.<br>
And apparently the compiled <code>func1</code> does not have any attributes other than <code>__call__</code> which makes it harder to get and check the object.</p>

tom: <p>Is <code>isinstance(func1, torch.jit.ScriptFunction)</code> working for you?</p>
<p>Best regards</p>
<p>Thomas</p>

mfkasim: <p>Ah, yes, it actually works!<br>
I tried it before and it returns False, I didn’t realize my code put <code>func.__call__</code> into my function that’s why it fails.</p>
Conversion of Speech model with PyTorch as a backend into ONNX

Neha_Soni: <p>Hi,<br>
I am working with ESPNet. My task is to convert ESPNet model(PyTorch banckend) into TFLite for edge deployment.</p>
<p>But, ESPNet is for ASR. I do find a solution which is – convert ESPNet model -&gt; ONNX model -&gt; Tensorflow model -&gt; TFite.</p>
<p>But, I am not succeed. Looking for help.</p>
<p>Please do needful.</p>

Neha_Soni: <p>looking for response.</p>

ptrblck: <p>Could you explain where you are stuck at the moment and what is not working? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Neha_Soni: <p>Hi</p>
<p>I am trying to convert ESPNet (PyTorch as a backend) model into ONNX, but I’m getting error i.e.<br>
<code>TypeError: forward() missing 2 required positional arguments: 'ilens' and 'ys_pad'</code></p>
<p>I am using this code for conversion.<br>
<code># Export the trained model to ONNX dummy_input = Variable(torch.randn(83,320)) # [83,320] will be the input to the model torch.onnx.export(model, dummy_input, "model.onnx")</code></p>
<p>Thanks</p>

ptrblck: <p>Based on the error message it seems that your model expects more inputs to its <code>forward</code> method than the provided tensor.<br>
Could you check, where <code>ilens</code> and <code>ys_pad</code> are used in your model?</p>
<p>Also, <code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors now. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Neha_Soni: <p>Thanks for response.<br>
But this model uses <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noopener">Transformer</a> predefined model.<br>
I am trying to convert this Transformer model into ONNX.</p>
<p>Thanks!</p>
Cannot get the value of hidden weights of RNN

magnusderrote: <p>I declare my RNN as</p>
<pre><code class="lang-auto">self.rnn = torch.nn.RNN(input_size=encoding_dim, hidden_size=1, num_layers=1, nonlinearity='relu')
</code></pre>
<p>Later</p>
<pre><code class="lang-auto">self.rnn.all_weights
# [[Parameter containing:
tensor([[-0.8099, -0.9543,  0.1117,  0.6221,  0.5034, -0.6766, -0.3360, -0.1700,
         -0.9361, -0.3428]], requires_grad=True), Parameter containing:
tensor([[-0.1929]], requires_grad=True), Parameter containing:
tensor([0.7881], requires_grad=True), Parameter containing:
tensor([0.4320], requires_grad=True)]]

self.rnn.all_weights[0][0][0].values
# {RuntimeError}Could not run 'aten::values' with arguments from the 'CPU' backend. 'aten::values' is only available for these backends: [SparseCPU, Autograd, Profiler, Tracer].
</code></pre>
<p>Clearly I see the value of the weights, but cannot access to it. <a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">Documentation</a> says I need to specify <code>requires_grad=True</code>, but that does not work.</p>
<p>Is there a more elegant and usable way than <code>self.rnn.all_weights[0][0][0]</code>?</p>
Translating @once_differentiable to C++

mclaeys: <p>Hi all,</p>
<p>I’m currently converting an autograd Function from Python to C++. In the python implementation, the backward method is annotated with <code>@once_differentiable</code>. I’m wondering what steps I should take to translate this to C++.</p>
<p>Many thanks in advance for the help!</p>
<p>With best regards,<br>
Maxim</p>
Data normalization, learning rate and loss relationship

Henry_Chibueze: <p>I’m currently working on a regression model with input data of range(1, 26) and target data of range(1,  42) and I discovered that when I normalize the data to be within range(0, 1) and set learning rate to 0.001 the loss reduces by a huge factor after one epoch it’s so unbelievable than when I don’t normalize the training data (for eg: without normalization, loss can 5.0 and with normalization, loss can be 0.01 all after 1st epoch).<br>
So this leaves me wondering could it be due to a relationship between the normalized input and learning rate or loss and learning rate tho, I don’t think the loss should change whether normalized or not coz the target data too was normalized accordingly<br>
So what could be the cause?</p>

ptrblck: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="1" data-topic="95491">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>I don’t think the loss should change whether normalized or not coz the target data too was normalized accordingly</p>
</blockquote>
</aside>
<p>I’m not sure, if I understand this sentence correctly.<br>
Are you comparing these use cases:</p>
<pre data-code-wrap="python"><code class="lang-python"># case0
output = model(unnormalized_input)
loss0 = criterion(output, unnormalized_target)

# case1
output = model(normalized_input)
loss1 = criterion(output, normalized_target)
</code></pre>
<p>If so, then I would expect that <code>loss1</code> would be potentially smaller than <code>loss0</code>, as the magnitude of the output and target is most likely smaller (assuming you are using e.g. <code>nn.MSELoss</code>).</p>

Henry_Chibueze: <p>Ok I think I understand now<br>
I’ll try using the smoothL1loss instead coz the MSEloss is leading to exploding and vanishing gradients coz of the way it penalize the algorithm based on losses greater than 1 and losses less than zero</p>
Specifying ports to be used in Pytorch multi-node distributed training

sarthak_garg: <p>Hello,</p>
<p>I am facing problems with using torch distributed training in a multi-node setup. Apart from the specified master port, looks like Pytorch tries to open random ports for inter-node communication. In my setup I get a limited number of specified open ports. Is there some way I can force Pytorch to use only the given ports for internode communication?</p>
<p>Thanks!</p>

vadimkantorov: <p>We got the same problem…</p>
Is there any approach in pytorch to achieve distributed network connection between CPU prediction and GPU training?

bqdqj: <p>Out team is planning to use CPUs from multiple computers to do network prediction and data production, and then use a single GPU server to do network training. Is there any method we can use in torch.distributed package that can help us with this situation?</p>

mrshenli: <p><a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener"><code>torch.distributed.rpc</code></a> should be able to help. <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#general-distributed-training" rel="nofollow noopener">Here</a> is a list of tutorials.</p>
<p>The use case looks similar to the following two examples:</p>
<ol>
<li><a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref</a></li>
<li><a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-processing-cartpole-solver" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-processing-cartpole-solver</a></li>
</ol>

bqdqj: <p>Thank you very much, I’ll look into them carefully!</p>
[Pytorch Beginner] What's the difference between these two lines?

jennykim1016: <pre><code class="lang-auto">channel_1, _, KH1, KW1 = conv_w1.shape
conv_1 = F.conv2d(x, conv_w1, bias=conv_b1, padding=2)
relu_1 = F.relu(conv_1)
</code></pre>
<p>was the model implementation where x was an input vector.</p>
<p>I instead did this:</p>
<pre><code class="lang-auto">first_conv = torch.nn.Conv2d(in_channels = input_channel, out_channels = channel_1, padding = 2, kernel_size = (KH1, KW1))
first_conv.weight = torch.nn.Parameter(conv_w1)
first_conv.bias = torch.nn.Parameter(conv_b1)
conv_1 = first_conv(x)
</code></pre>
<p>I think these two are equivalent; however, when I am running the block of the training code, only the second one complains that the gradient for conv_w1 vector is None. Can someone please explain? Thanks!</p>

pfloat: <p>Hello,</p>
<p>I think these posts can help you answer your questions, just in case you haven’t already checked them out:<br>
<a href="https://discuss.pytorch.org/t/how-to-choose-between-torch-nn-functional-and-torch-nn-module/2800">how-to-choose-between-torch-nn-functional-and-torch-nn-module</a><br>
<a href="https://discuss.pytorch.org/t/what-is-the-difference-between-torch-nn-and-torch-nn-functional/33597">what-is-the-difference-between-torch-nn-and-torch-nn-functional</a></p>
<p>Besides I think more details are needed to solve the error. Can you post a complete example of code that we can run ?</p>
Expected value of nll_loss

copah: <p>From the official documentation <a href="https://pytorch.org/docs/stable/nn.functional.html#nll-loss" rel="nofollow noopener">here</a>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # input is of size N x C = 3 x 5
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; # each element in target has to have 0 &lt;= value &lt; C
&gt;&gt;&gt; target = torch.tensor([1, 0, 4])
&gt;&gt;&gt; output = F.nll_loss(F.log_softmax(input), target)
</code></pre>
<p>I would expect the <code>output</code> to be <code>output == torch.log(torch.tensor(1 / C))</code>, since we would expect a random activation tensor to produce softmax inputs with random probability and hence the expected output from the negative log likelihood will be <code>-log(1/num_classes)</code>. Where is my logic flawed?</p>

ptrblck: <p>I think you might not be considering the “confidence” of the predictions.<br>
I.e. even if your model is predicting random classes, the loss would be much higher, if you scale the logits as:</p>
<pre><code class="lang-python">input = torch.randn(3, 5) * 100
</code></pre>
<p>However, you should get approx. the same loss if you keep the logits close to the random output:</p>
<pre><code class="lang-python">input = torch.zeros(3, 5) + torch.randn(3, 5) * 1e-3
</code></pre>
Strange values confusion matrix evaluation

ptorres: <p>Hello, i’m doing the evaluation of and custom model in the following way:</p>
<pre><code class="lang-auto">BATCH_SIZE = 20
image_dataset = datasets.ImageFolder(data_dir, image_transform)
dataloader = torch.utils.data.DataLoader(image_dataset, batch_size=BATCH_SIZE, shuffle=True)
print('classes_idx: {}'.format(image_dataset.class_to_idx))

confusion_matrix = torch.zeros(NUM_CLASSES, NUM_CLASSES)

with torch.no_grad():
	for idx, (inputs, classes) in enumerate(tqdm(dataloader)):
		inputs = inputs.to(DEVICE)
		classes = classes.to(DEVICE)
		outputs, _, _, _ = model(inputs)
		_, preds = torch.max(outputs, 1)
		
		for t, p in zip(classes.view(-1), preds.view(-1)):
			confusion_matrix[t.long(), p.long()] += 1

		print(confusion_matrix)
</code></pre>
<p>But after some batches this happens (im printing confusion matrix for each batch):</p>
<pre><code class="lang-auto">tensor([[ 278.,  125.,   36.], # (CM BATCH_N = N)
        [   0.,  728.,   36.],
        [ 107.,  134., 1476.]])
 20%|█████████████████████████▉                                                                                                          | 146/743 [09:00&lt;34:20,  3.45s/it]
tensor([[2.7900e+02, 1.2800e+02, 3.6000e+01], # (CM BATCH_N = N + 1)
        [1.0000e+00, 7.3100e+02, 3.6000e+01],
        [1.0900e+02, 1.3400e+02, 1.4860e+03]])
</code></pre>
<p>Anyone knows why the values are changing in this way?</p>

ptrblck: <p>The integer values are just being displayed in the exponential form (scientific notation).<br>
You can disable it as shown here:</p>
<pre><code class="lang-python">x = torch.tensor([[2.7900e+02, 1.2800e+02, 3.6000e+01], 
                  [1.0000e+00, 7.3100e+02, 3.6000e+01],
                  [1.0900e+02, 1.3400e+02, 1.4860e+03]])

print(x)
torch.set_printoptions(sci_mode=False)
print(x)
</code></pre>
BCEWithLogitsLoss is accepting labels of list

Jani_Basha_S: <p>I have a dataset where each image can have 14 labels. So i have converted labels of each image to format [0,1,0,0,1,0,0,1,0,0,0,1,1,0] and trying to apply BCEWithLogitsLoss.<br>
inputs should come from NN and target  is 14 label list.</p>
<p>like below</p>
<blockquote>
<p>def <strong>getitem</strong>(self, idx):<br>
if torch.is_tensor(idx):<br>
idx = idx.tolist()<br>
img_path = self.img_list[idx][0]<br>
image = Image.open(img_path).convert(‘RGB’)<br>
if self.transform:<br>
image = self.transform(image)<br>
<span class="hashtag-raw">#sample</span> = {‘img’: image,<br>
#          ‘label’: self.img_list[idx][1]}<br>
<span class="hashtag-raw">#return</span> sample<br>
label = self.img_list[idx][1]<br>
return image,label</p>
</blockquote>
<p>##############  Training function</p>
<blockquote>
<p>outputs = model(inputs)<br>
loss = criterion(outputs, labels)   ----&gt; error line</p>
</blockquote>
<p><strong>Kindly suggest how to train multi label classifier if i am doing wrong</strong></p>

Jani_Basha_S: <h2>Error Message is like below</h2>
<p>51                         outputs = model(inputs)<br>
—&gt; 52                         <strong>loss = criterion(outputs, labels)</strong><br>
53<br>
54                     _, preds = torch.max(outputs, 1)</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
720             result = self._slow_forward(*input, **kwargs)<br>
721         else:<br>
–&gt; 722             result = self.forward(*input, **kwargs)<br>
723         for hook in itertools.chain(<br>
724                 _global_forward_hooks.values(),</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)<br>
629                                                   self.weight,<br>
630                                                   pos_weight=self.pos_weight,<br>
–&gt; 631                                                   reduction=self.reduction)<br>
632<br>
633</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)<br>
2535         reduction_enum = _Reduction.get_enum(reduction)<br>
2536<br>
-&gt; 2537     if not (target.size() == input.size()):<br>
2538         raise ValueError(“Target size ({}) must be the same as input size ({})”.format(target.size(), input.size()))<br>
2539</p>
<p><strong>AttributeError: ‘list’ object has no attribute 'size’</strong></p>

Jani_Basha_S: <p>I have solved it by converting the list of labels to tensor.</p>
<p>instead of this:</p>
<blockquote>
<p>label = self.img_list[idx][1]</p>
</blockquote>
<p>Used the below line</p>
<blockquote>
<p>label = torch.tensor(self.img_list[idx][1] , dtype=torch.float32)</p>
</blockquote>
How to custom operations in the forward function

copyrightly: <p>I am doing an experiment on images of small size as follows:</p>
<p>The input image has only one channel, then I apply 2 customed filters (denoted by A and B) to do the convolution with the image. The filter has the same size as the image and each of A and B have 10 channels respectively. So after the convolution we have two 1 * 10 tensors. Denote them as (a1, a2, … a10) and (b1, b2, … b10). Let w = (w1, w2, …, w10) be a trainable 1 *10 tensor. Then I want to compute the following fraction: (the numerator is the norm of weighted sum of vectors (ai, bi) and the denominator is the sum of weighted norm. The norm here is the ordinary Euclidean norm.)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ef04efe9473d686b9c32e066ec1eb4690528b20.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ef04efe9473d686b9c32e066ec1eb4690528b20" title="Screen Shot 2020-09-07 at 6.46.40 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ef04efe9473d686b9c32e066ec1eb4690528b20.png" alt="Screen Shot 2020-09-07 at 6.46.40 PM" data-base62-sha1="bgk6hV5t0WmHorrvfiOy5IOm2Na" width="690" height="89" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-09-07 at 6.46.40 PM</span><span class="informations">1060×138 4.88 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am wondering how to do this computation in the forward function and where I should claim w as trainable tensor? The template of my code is as follows:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.filter_A = ...
        self.filter_B = ...

    def forward(self, x):
        x_A = F.conv2d(x, self.filter_A)
        x_B = F.conv2d(x, self.filter_B)
        ...
</code></pre>

ptrblck: <p>I don’t completely understand the shape information and what e.g. “we have two 1 * 10 tensors” means in this case.<br>
However, based on your description and code snippet it seems you are not using any padding and since the filters have the same shape as the input, the result would be an elementwise multiplication and summation, wouldn’t it?<br>
The trainable filters should be defined in the <code>__init__</code>, which seems to be the case in your code snippet (assuming the <code>...</code> refer to `nn.Parameter(torch.tensor(…)) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
<p>Let me know, if I misunderstood the question.</p>
Profiler count cuda time or cpu time?

yigu: <p>Hi, I am using profiler to count cuda kernels time, but I met a very strange question.<br>
codes is below:</p>
<pre><code class="lang-auto">x = torch.randn((10, 3, 224, 224)).cuda()
a = torch.randn((5, 6, 7, 8))

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    y = x ** 2
    y = y + 1.5
    y = y * 2.4
    y = torch.relu(y)
    b = a + 1.1
    y = y * 1/4
print("prof is:\n{}".format(prof))
</code></pre>
<p>and I find the kernel a is placed on cpu, but in profiler table, its time also shown, and its cpu time are neally nearly to cuda time. I am confused.</p>

yigu: <p>Here is image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/530a65f051fd7fcc6ec45eee1256199c0d4d44f7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/530a65f051fd7fcc6ec45eee1256199c0d4d44f7" title="9_11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/530a65f051fd7fcc6ec45eee1256199c0d4d44f7_2_690x210.jpeg" alt="9_11" data-base62-sha1="bQBUiFUh5wWyRVfSKcOzAEnHlNJ" width="690" height="210" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/530a65f051fd7fcc6ec45eee1256199c0d4d44f7_2_690x210.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/530a65f051fd7fcc6ec45eee1256199c0d4d44f7_2_1035x315.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/530a65f051fd7fcc6ec45eee1256199c0d4d44f7_2_1380x420.jpeg 2x" data-dominant-color="29292E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">9_11</span><span class="informations">4032×1231 334 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I know the cuda time is get by cudaEvent series kernels, I think it is hardware time, but a is cpu kernel, its cpu time is shown in column of CUDA time.<br>
I want to know, profiler used to count time,  is the time including hardware time and cpu time which cpu kernel is inserted in cuda kernels?<br>
Thank you very much!</p>
Cons of using named tensors

David_Alford: <p>I have not seen named tensors in any project I have viewed but they seem almost as good of a practice as writing tests. What is the reason they have not caught on? Are there any cons to using them besides taking the time to write them?</p>

googlebot: <ol>
<li>Implementation is incomplete, and I don’t think anyone works on it</li>
<li>It is oddly non-polymorphic, i.e. lots of operations are just blocked when names are present. Not only ops with dim=int arguments, but also things like unsqueeze(). If these could re-synchronize names instead, it would be much easier to include code with named tensors into existing codebases.</li>
<li>Related to above, they’re not compatible with many standard routines (first of all, stuff from nn namespace). For example, while F.linear/mm works, convolutions or rnns fail.</li>
<li>Most of ML code doesn’t require dimension permutations, I guess people working with BCHW tensors don’t care about dimension names that much.</li>
</ol>
What is the input format for Mesh RCNN?

Kiran_Ikram: <p><a href="https://pytorch3d.readthedocs.io/en/latest/modules/structures.html" rel="nofollow noopener">https://pytorch3d.readthedocs.io/en/latest/modules/structures.html</a></p>
<p>I would like to implement MeshRCNN but it states Pytorch 3D as a dependency. Does that mean that the input has to be obj files ? Or can they be images ?</p>
<p>Thanks!</p>

albertotono: <p>The imput format has to be an image, MeshRCNN does several operations (with detectron2 and MaskRCNN segments the image and then take the highest priority object in the picture and it transform to a mesh) but the final output will be an .obj file .<br>
I did it in colab if you need help <a href="https://youtu.be/YFSJgF39G7c" rel="nofollow noopener">https://youtu.be/YFSJgF39G7c</a> <a class="mention" href="/u/kiran_ikram">@Kiran_Ikram</a></p>
Difference between pytorch geometric, pytorch3D and Kaolin?

sriharsha0806: <p>What is the difference between pytorch geometric, pytorch 3D and Kaolin? Which one is best for what task?</p>

albertotono: <p>Dear <a class="mention" href="/u/sriharsha0806">@sriharsha0806</a>,</p>
<p>I think is a good question but a bit broad. I will try to answer based on my knowledge ( I haven’t used Kaolin yet)</p>
<ul>
<li>
<p>but <a href="https://pytorch-geometric.readthedocs.io/en/latest/" rel="nofollow noopener">Pytorch geometric</a> and <a href="https://github.com/rusty1s/pytorch_geometric" rel="nofollow noopener">github</a> has different methods implemented that you can see there and it is completely in Python (around 100 contributors),</p>
</li>
<li>
<p><a href="https://github.com/NVIDIAGameWorks/kaolin" rel="nofollow noopener">Kaolin</a> in C++ and Python (of course Pytorch) with only 13 contributors</p>
</li>
<li>
<p><a href="https://github.com/facebookresearch/pytorch3d" rel="nofollow noopener">Pytorch3D</a> with around 40 contributors</p>
</li>
</ul>
<p>They all have different targets and applications, I would consider what is your goal and try to find people who are working on similar problems and ask what they are using and why.</p>
My pytorch.mobile model can not be loaded! APP CLOSED!

SungmanHong: <p>Hi, i made my pytorch.mobile(.pt file) by convert code</p>
<p>and in JAVA I loaded</p>
<p>import org.pytorch.Module;<br>
module = Module.load(assetFilePath(this, “model_OCR.pt”));</p>
<p>but app is just closed when launched… (the demo model.pt offered by pytorch team is working well.<br>
why it is crushed…??? I don know why…<br>
(APP is build successfully)</p>
In an RNN, why don't we average the loss before backward?

BramVanroy: <p>Looking at the tutorial <a href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html#training-the-network" rel="noopener nofollow ugc">here</a>, we see</p>
<blockquote>
<p>The magic of autograd allows you to simply sum these losses at each step and call backward at the end.</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">    loss = 0

    for i in range(input_line_tensor.size(0)):
        output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)
        l = criterion(output, target_line_tensor[i])
        loss += l

    loss.backward()
</code></pre>
<p>So we just sum all errors and then do a backward pass based on the gradients that are calculated on this summed error. My question is why, for an RNN, we do not average over the loss of all predicted tokens. It seems more natural to me to take the average rather than the sum to get the average error of the input sequence.</p>

vdw: <p>I don’t think it would affect the gradients anyway, which are basically determined by <code>l = citerion(...)</code>. Summing them up just makes it convenient since we can only call <code>loss.backward()</code> once.</p>
<p>If I understand correctly, we could in principle do something like this:</p>
<pre><code class="lang-auto">losses = []

for i in range(input_line_tensor.size(0)):
    output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)
    l = criterion(output, target_line_tensor[i])
    losses.append(l)

for l in losses:
    l.backward()
</code></pre>
How to change the input size for pretrained resnet model?

Zhiqiu_Lin: <p>I am new to Pytorch and I am following the transfer learning tutorial to build my own classifier. One thing I would like to know is how do I change the input range of resnet? Right now it is just taking images of size (224,224), but I would like it to work on images of size (512,512).</p>
<p>What is the best way to achieve this? Is it better to create my own network? And in specific, what layers do I need to change in order to make the network work?</p>
<p>I am a college undergrad using Pytorch to do my research, but I am new to deep learning and I just know some basic facts about resnet. If the input size is not possible change unless I modify the entire network, let me know!</p>

tjoseph: <p>Look at what convolutional layers and pooling layers do. They work on any input size, so your network will work on any input size, too.<br>
You only have to change the fully connected layers such as nn.Linear in VGG. Newer nets use a GlobalPooling layer before the fully connected layers and you do not even have to change the last linear layer.</p>

Zhiqiu_Lin: <p>But when I push images of size (512,512) other than (224,224) into my Resnet, it will have 'size mismatch" error at the AvgPooling layer. I am not sure whether that is the case because from documentation these layers seem to work for any shape.</p>
<p>If I change that AvgPooling layer to AdaptivePooling(1) then it will work, but it doesn’t look very efficient.</p>

tjoseph: <p>I don’t think there is much of a difference between AdaptiveAveragePooling and AveragePooling.<br>
Just change the AveragePooling size from 7 to 16 and it should work, too.</p>

chenwu: <p>Use PIL or similar libraries to resize the images to 224 x 224, then feed them to the pre-trained model should be OK.</p>

Aitor_Arronte: <p>You can just use torchvision.transforms.Resize ( <em>size</em> ,  <em>interpolation=2</em> ) to get the desired dimensions.</p>

Kristof_Horvath: <p>Hey tsjoeph!<br>
I arrived at this topic while I was working with FasterRCNN. I tried to reduce the input shape from 1024x1024 to 512x512 and it worked. Now I’m trying to understand why. Can you explain it a bit deeper, if you have some time?<br>
Thank you!</p>

Shubhankar: <p>Adaptive pool layers take care of the different image sizes.</p>
Where is `torch::lstm` defined?

ndronen: <p>I see in <a href="https://discuss.pytorch.org/t/vf-lstm-implementation/35536">this post</a> where <code>_VF.lstm</code> is defined (presumably as <code>LSTMImpl</code>). Since it calls <code>torch::lstm</code> (<a href="https://github.com/pytorch/pytorch/blob/49777e67303f608987ec0948c7fd8f46f6d3ca83/torch/csrc/api/src/nn/modules/rnn.cpp#L275" rel="nofollow noopener">this line</a>), I’m trying to track down <code>torch::lstm</code>. Any pointers would be much appreciated!</p>

albanD: <p>Hi,</p>
<p>The <code>torch::XXX</code> binding is generated based on the info from <a href="https://github.com/pytorch/pytorch/blob/4bead6438a4271f154980dc242199c2f1b897bc1/aten/src/ATen/native/native_functions.yaml#L4269-L4273">here</a> which then hits the native code <a href="https://github.com/pytorch/pytorch/blob/4bead6438a4271f154980dc242199c2f1b897bc1/aten/src/ATen/native/RNN.cpp#L1396-L1399">here</a>.<br>
Hope this helps.</p>
Fairseq use with Torchscript

mestto: <p>Hi! I’m doing the capstone project from Udacity C++ Nano &amp; want to use the fn “translate” from <a href="https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md" rel="nofollow noopener">Fairseq</a> into the project. Is it exportable to torchscript? If so, how would one use it? (some guidance of ways of achieving the same thing with other libraries would be helpful, too)</p>
Issues with implementation of Riemannian Adam on a hyperboloid with trainable curvature

arijitthegame: <p>Hi everyone,</p>
<p>I am trying to construct an Adam optimizer for the hyperboloid manifold. I am using the implementation from <a href="https://github.com/HazyResearch/hgcn/blob/master/optimizers/radam.py" rel="nofollow noopener">https://github.com/HazyResearch/hgcn/blob/master/optimizers/radam.py</a>. And I am basically rewriting their Hyperbolic Graph Neural Networks so that I gain some understanding. I have basically managed to implement the models but I am stuck with implementing the optimizer. This is my implementation here <a href="https://github.com/arijitthegame/Geometric-Deep-Learning/blob/master/Hyperbolic_GCN%20.ipynb" rel="nofollow noopener">https://github.com/arijitthegame/Geometric-Deep-Learning/blob/master/Hyperbolic_GCN%20.ipynb</a></p>
<p>But unlike the hgcn repo, I do not construct a manifold class and my trainable parameter curvature ‘c’ is a different parameter for each layer of the model. So the problem I am facing is to project the tensors back on the manifold during the optimization step, I need to know the curvature of that layer. And how to grab that parameter is not really clear to me.</p>
<p>I am sorry about this jumbled mess of thoughts in my head and would really appreciate any help.</p>
<p>Best,</p>
<p>Arijit.</p>
Correct way to use custom weight maps in unet architecture

Rishav_Sapahia: <p>Hi Guys,<br>
There is a famous trick in u-net architecture to use custom weight maps to increase accuracy.Below are the details of it-</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fd59e49947fae82f2d65e10a478384bdc9d6dd0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8fd59e49947fae82f2d65e10a478384bdc9d6dd0" title="25%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd59e49947fae82f2d65e10a478384bdc9d6dd0_2_690x264.jpeg" alt="25%20PM" data-base62-sha1="kwpX6QhiAJFIMVC4BkoktuyJ9D2" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd59e49947fae82f2d65e10a478384bdc9d6dd0_2_690x264.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd59e49947fae82f2d65e10a478384bdc9d6dd0_2_1035x396.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd59e49947fae82f2d65e10a478384bdc9d6dd0_2_1380x528.jpeg 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">25%20PM</span><span class="informations">1726×662 338 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now, by asking here and at multiple other place, I get to know about 2 approaches.I want to know which one is correct or is there any other right approach which is more correct?</p>
<p>1)First is to use <code>torch.nn.Functional</code> method in the training loop-</p>
<p><code>loss = torch.nn.functional.cross_entropy(output, target, w)</code><br>
where w will be the calculated custom weight.</p>
<p>2)Second is to use <code>reduction='none'</code> in the calling of loss function outside the training loop<br>
<code>criterion = torch.nn.CrossEntropy(reduction='none')</code></p>
<p>and then in the training loop multiplying with the custom weight-</p>
<pre><code class="lang-auto">gt # Ground truth, format torch.long
pd # Network output
W # per-element weighting based on the distance map from UNet
loss = criterion(pd, gt)
loss = W*loss # Ensure that weights are scaled appropriately
loss = torch.sum(loss.flatten(start_dim=1), axis=0) # Sums the loss per image
loss = torch.mean(loss) # Average across a batch

</code></pre>
<p>What’s the right way to do things in PyTorch in this case</p>

rgonzalez60: <p>Hello, I am trying to implement the correct way of weight maps from U-net architecture as well. Did you come to the conclusion from the 2 options you brought us in this question?</p>
<p>I am reading the documentation for the first option: <code>torch.nn.functional.cross_entropy</code>, and for the input parameter <code>weight</code>: a <code>number_of_classes</code> tensor size is needed, so I think this option is not what you were looking for.</p>
RuntimeError: bool value of Variable objects containing non-empty torch.LongTensor is ambiguous

vishal: <p>Consider the following code which works as expected:<br>
<code><br>
In [1]: import torch<br>
In [2]: var = torch.autograd.Variable(torch.LongTensor(0))<br>
In [3]: bool(var)<br>
Out[3]: False<br>
</code><br>
However, if I do the same with a non-empty variable:<br>
<code><br>
In [4]: var = torch.autograd.Variable(torch.LongTensor(1))<br>
In [5]: bool(var)<br>
RuntimeError: bool value of Variable objects containing non-empty torch.LongTensor is ambiguous<br>
</code><br>
It shows an error!<br>
How to handle such scenarios when using an if/else condition etc?</p>

smth: <p>I guess a more appropriate question is: why are you doing <code>bool(var)</code>?</p>

vishal: <p>I am not exactly using bool. I am trying to run facebook’s end to end negotiator code, where the following snippet produces a problem:<br>
<code></code></p>
<pre><code>    if resume:
        inpt = None
    else:
        inpt = Variable(torch.LongTensor(1))
        inpt.data.fill_(self.word_dict.get_idx('YOU:'))
        inpt = self.to_device(inpt)

    for _ in range(max_words):
            if inpt:   #this condition produces the error when inpt is not None
                inpt_emb = torch.cat([self.word_encoder(inpt), ctx_h], 1)
                lang_h = self.writer(inpt_emb, lang_h)
                lang_hs.append(lang_h)
</code></pre>


smth: <p>the <code>if inpt</code> should be: <code>if inpt is not None</code>. I think it’s a bug in their code.</p>

pinocchio: <p>I found some related posts:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/52946920/bool-value-of-tensor-with-more-than-one-value-is-ambiguous-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/52946920/bool-value-of-tensor-with-more-than-one-value-is-ambiguous-in-pytorch</a></li>
<li><a href="https://discuss.pytorch.org/t/runtimeerror-bool-value-of-variable-objects-containing-non-empty-torch-longtensor-is-ambiguous/7310" class="inline-onebox">RuntimeError: bool value of Variable objects containing non-empty torch.LongTensor is ambiguous</a></li>
<li><a href="https://discuss.pytorch.org/t/runtimeerror-bool-value-of-tensor-with-more-than-one-value-is-ambiguous/43800" class="inline-onebox">RuntimeError: bool value of Tensor with more than one value is ambiguous</a></li>
<li><a href="https://discuss.pytorch.org/t/why-cant-one-pass-data-through-a-torch-relu-module-directly">https://discuss.pytorch.org/t/why-cant-one-pass-data-through-a-torch-relu-module-directly</a></li>
</ul>

Brando_Miranda: <p>if you are here and you are not constructing an layer or loss before using it that’s why your getting the error. Do:</p>
<pre><code class="lang-auto">loss = nn.MSELoss()(y, y_pred)
</code></pre>
<p>not</p>
<pre><code class="lang-auto">loss = nn.MSELoss(y, y_pred)
</code></pre>
<p>very different it seems.</p>
How does one start using double without unexpected bugs?

Brando_Miranda: <p>On a related issue to <a href="https://discuss.pytorch.org/t/what-is-the-machine-precision-of-pytorch-with-cpus-or-gpus/9384">this</a>, I get errors in my regression that are close to machine precision (<code>10^-7</code>). As I tried things out machine precision things might be a lot more subtle than I expected, so I want to make sure. Is changing all the parameters to <code>double</code> in my model sufficient so that EVERYTHING is now computed using double to avoid errors?</p>
<p>(btw read that post for more detail comments on issues if your curious)</p>
<p>But what I want is:</p>
<ul>
<li>change code so everything uses double</li>
<li>hope that there are no unexpected (silent) bugs (e.g. I know if cpu tensors are used when gpus are expected then pytorch throws errors but I want to make sure there isn’t a bug I could be missing that occurs if I only change the model to double)</li>
</ul>
Altering tensor hook invocation order

rGure: <p>Hey everyone,</p>
<p>I am in the process of extending captum’s lrp implementation by some of the rules that are not in the current pull request as I need those for a project at work. Due to that I have some questions on tensor hooks:<br>
1.) It seems the invocation order of hooks on tensors is determined by the order of calling regsiter_hook(), which seems reasonable, but I could not find a definitive statement on this in the documentation. Can anyone clarify whether this is in fact the case?<br>
2.) Is there a way to alter the invocation order after registration? I tried to alter the OrderedDict stored in the _backward_hooks attribute, but this does not work. So I assume the order is somehow baked into the computational graph? Is there any way to alter this apart from removing hooks and reattaching them in the desired order?</p>
<p>Best regards and thanks</p>

albanD: <blockquote>
<p>It seems the invocation order of hooks on tensors is determined by the order of calling regsiter_hook()</p>
</blockquote>
<p>Yes they are called in the order they were installed.</p>
<blockquote>
<p>Is there a way to alter the invocation order after registration?</p>
</blockquote>
<p>There is no official way to do this no, you will need to uninstall and reinstall them.<br>
The way these hooks are implemented is a bit complex as it depends on where the Tensor is in the graph so I don’t think you can hack your way around that right now. Or you will have to read the code related to hooks in the engine but I don’t think we expose the internal structures to python anyways.</p>
Torch.distributed.launch vs torch.multiprocessing.spawn

briankosw: <p>What is the implementation and performance differences between <code>torch.distributed.launch</code> and <code>torch.multiprocessing.spawn</code>?</p>

mrshenli: <p><code>torch.distributed.launch</code> uses <code>subprocess.Popen</code>. The perf differences between these two are typical <a href="https://stackoverflow.com/questions/47689297/speed-comparison-using-multiprocessing-process-versus-subprocess-popen." rel="nofollow noopener"><code>multiprocessing</code> vs <code>subprocess</code></a></p>
<p>Besides that, <code>torch.distributed.launch</code> also tries to configure several env vars and pass command line arguments for distributed training script, e.g., <code>RANK</code>, <code>LOCAL_RANK</code>, WORLD_SIZE etc. On the other hand, <code>torch.multiprocessing.spawn</code> is general multi-processing, not specifically tailored for <code>torch.distributed</code>.</p>
<p>If you need multi-server distributed data parallel training, it might be more convenient to use <code>torch.distributed.launch</code> as it automatically calculates ranks for you, through <code>--nnode</code>, <code>--node_rank</code>, and <code>--nproc_per_node</code>.If you need single-server multi-gpu data parallel training, both should work the same.</p>
Imagenet validation ground truth file

Cagri_Kaplan: <p>Hello,</p>
<p>I am deploying pretrained vgg16 of Pytorch as my model and I need to work on validation data of Imagenet. However, I believe the order of classes in ground truth that model weights are optimized with are not same with the one I found in web. Where can I find validation ground truth file for vgg16 and resnet models which are trained on Imagenet and with Pytorch?</p>

Cagri_Kaplan: <p>I fixed my problem. You can close this thread if you wish.</p>

gakadam: <p>Hi, how did you fix your issue? I have same problem. Thanks.</p>