akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
'torch._C' has no attribute '_scatter'

username: jli
cooked: <p>Hi! I have been searching this error endlessly but no one seems to have gotten it.</p>
<p>File “/usr/local/lib/python3.6/site-packages/torch/cuda/comm.py”, line 148, in scatter<br>
return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))<br>
AttributeError: module ‘torch._C’ has no attribute ‘_scatter’</p>
<p>It comes from some code that tries to distribute tensors across multiple processors. I am running on my personal machine (mac) and specifying device_id=[-1] (which means just run on one cpu), but am still having issues. I dug through the torch code and could not find where _scatter is in torch._C. Could anyone shed some light on this? Thanks, much appreciated!!</p>

username: richard
cooked: <p>It’s here: <a href="https://github.com/pytorch/pytorch/blob/acc83ad54ef85eacd3c7ebfe20ee8225164e5bd1/torch/csrc/cuda/python_comm.cpp#L35-L50" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/acc83ad54ef85eacd3c7ebfe20ee8225164e5bd1/torch/csrc/cuda/python_comm.cpp#L35-L50</a></p>

username: jli
cooked: <p>Oh okay thank you! Hm I have the latest version of torch, do you know why it thinks _scatter is not there?</p>

username: gauone
cooked: <p><a class="mention" href="/u/jli">@jli</a> did you find a fix by now? I encounter the same problem on with the Nightly version of pytorch (pip installed on a mac with MPS acceleration)</p>
Tensorboard: graph view: how to show Op attributes?

username: cee1
cooked: <p>Hi all,</p>
<p>With torch.utils.tensorboard, a model can be viewed in browser via SummaryWriter.add_graph()</p>
<p>Though, I’m unable to view the attributes of an Op</p>
<ul>
<li>e.g. kernel size, stride of a Conv2D:</li>
<li><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86c11117cf0d603b9448b05ec74c5917d6d1fa7e.png" data-download-href="https://discuss.pytorch.org/uploads/default/86c11117cf0d603b9448b05ec74c5917d6d1fa7e" title="conv2d-tensorboard"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c11117cf0d603b9448b05ec74c5917d6d1fa7e_2_690x399.png" alt="conv2d-tensorboard" data-base62-sha1="je5CJidq2c9ym7GEUkuwxCp6vO6" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c11117cf0d603b9448b05ec74c5917d6d1fa7e_2_690x399.png, https://discuss.pytorch.org/uploads/default/original/3X/8/6/86c11117cf0d603b9448b05ec74c5917d6d1fa7e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/6/86c11117cf0d603b9448b05ec74c5917d6d1fa7e.png 2x" data-dominant-color="E6E5E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">conv2d-tensorboard</span><span class="informations">1004×581 222 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<p>I guess these attributes are passed to Op as normal parameters …?</p>
<ul>
<li>Is there anyway to show them in tensorboard?</li>
</ul>
Nan training and validation losses of unet model

username: alyeko
cooked: <p>Hello all, I am using a simple UNET model which I adapted from <a href="https://medium.com/analytics-vidhya/creating-a-very-simple-u-net-model-with-pytorch-for-semantic-segmentation-of-satellite-images-223aa216e705" rel="noopener nofollow ugc">here</a>. The model works well with the Pytorch dataset class of the author.</p>
<p>However, when I try to use this model on a new Pytorch dataset (which I created), it returns nan training loss and nan validation losses. I am using nn.CrossEntropyLoss(). I believe my Pytorch dataset class works fine but I am missing something.</p>
<p>I have made my notebook openly available and can be accessed <a href="https://www.kaggle.com/code/alyeko/segmentation-of-floodwater-s2data" rel="noopener nofollow ugc">here</a>. I am obtaining sentinel-2 optical imagery and trying to segment floodwater from the imagery, with the unet model. So basically binary segmentation (floodwater or not floodwater).</p>
<p>Any help would be greatly appreciated. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: azhanmohammed
cooked: <p>Hey I went through your implementation. Quite impressive work on using optical and SAR images together. What I would suggest is, you can check the following:</p>
<ol>
<li>Check if the loss is nan even when using simple cross entropy loss, try removing the dice loss.</li>
<li>If the loss is still nan when using cross entropy loss, change datatype to long for calculating cross entropy loss, and float when calculating dice loss.</li>
</ol>
<p>Hope this helps.</p>

username: alyeko
cooked: <p>Thank you very much for your reply! I tried your suggestion, but later found that I used a different way of normalizing the input images from the author which caused the model not to work as expected.</p>
CNN model Training Loss does not decrease and Validation Accuracy stuck

username: MayThet
cooked: <p>My CNN-based deep learning model is fluctuating in validation accuracy at certain epochs. I learned the deep learning model from frequency domain images. Do I need to normalize or not these iift2 frequency images ? And then, the training loss does not decrease and is stable. My training dataset is 500 images.<br>
and I tried 80 epochs. The training loss and validation accuracy are still stuck in epoch 20. There are no improvements. I have already tried tuning hyperparameters such as learning rate, batch size. I also added regularization parameters such as weight decay.</p>

username: jondapper
cooked: <p>Do you have a specific question? It’s normal for any metric to fluctuate from epoch to epoch, and the extent of the fluctuation is much greater when the validation set is small (as is yours).</p>
<p>It’s entirely possible, given such a small dataset, that validation loss plateaus after 20 epochs.<br>
If you have 500 images only, try fine-tuning a pre-trained model. Learning a model from scratch is a bad idea.</p>

username: May_Thet
cooked: <p>I am testing Image denoising research. My training dataset is 500 images and testing dataset is 12 images. I think, it is normal to test image denoising research. My training loss does not decrease.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f61ce08cf3838b709ae61e4e9a1617eb14843604.png" alt="loss" data-base62-sha1="z7dkfZqUuMOoKpoTm1y2pn1TMLG" width="640" height="480"></p>

username: May_Thet
cooked: <p>I also train 2/3 training and 1/3 validation set. However, the training loss does not decrease. <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/baa762590113fb2120d96a0c94e1a7c74db050aa.png" alt="28gray" data-base62-sha1="qDdqyU4U1DicsHthKeXloC6ngVQ" width="640" height="480"></p>
Mini-batch gradient decent bad accuracy and loss

username: IoannisTselekoglou
cooked: <p>Hey, I’m trying mini-batch gradient descent on the popular iris dataset, but somehow I don’t manage to get the accuracy of the model above 75-80%. Also, I’m not certain if I’m calculating the loss as well as the accuracy correctly.  Any suggestions on how to improve my code or mistakes I’m doing are appreciated.</p>
<pre><code class="lang-auto">batch_size = 10
train_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True) 
test_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>Training loop:</p>
<pre><code class="lang-auto">n_iters = 1000
steps = n_iters/10
LOSS = []
for epochs in range(n_iters):  
    for i,(inputs, labels) in enumerate(train_loader):
        out = model(inputs)
        train_labels = transform_label(labels)
        l = loss(out, train_labels)
        l.backward()
        #update weights
        optim.step()
        optim.zero_grad()
    LOSS.append(l.item())
    if epochs%steps == 0:
        print(f"\n epoch: {int(epochs+steps)}/{n_iters}, loss: {sum(LOSS)/len(LOSS)}")
        #if i % 1 == 0:
            #print(f" steps: {i+1}, loss : {l.item()}")
</code></pre>
<p>claculate accuracy:</p>
<pre><code class="lang-auto">    def accuracy(model,test_loader):
        sum_acc= 0
        #map labels with 0,1,2
        def transform_label(label_data):
            data = []
            for i in label_data:
                if i == "Iris-setosa":
                    data.append(torch.tensor([0]))
                if i == "Iris-versicolor":
                    data.append(torch.tensor([1]))
                if i == "Iris-virginica":
                    data.append(torch.tensor([2]))
            return torch.stack(data)

        for i,(X_test, test_labels) in enumerate(test_loader):
            test_labels = transform_label(test_labels)
            x_label_pre = model(X_test)
            _, x_label_pre_hat = torch.max(x_label_pre, 1)
            idx = 0
            number_pred = 0
            while idx &lt; len(X_test):
                if x_label_pre_hat[idx].item() == test_labels[idx].item():
                    number_correct += 1
                idx +=1
            accuracy_per_epoch = (number_correct/len(X_test))*100
            print(f"accuracy of batch {i}:\n{accuracy_per_epoch}%")
            sum_acc += accuracy_per_epoch
        return print(f"\ntotal accuracy of model {(sum_acc/len(test_loader)):.2f}%")

</code></pre>
Why we fit the model (DQN) after each step?

username: laro
cooked: <p>I looked on the code of DQN solution for  CartPole:<br>
<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a><br>
And there are some issues I don’t understand:</p>
<ol>
<li>
<p>It seems that at each episode and at <em>each step</em> the <strong>optimize_model</strong> is called.<br>
i.e on each step the model is fitted with the next values (GT).<br>
Why is this right ?<br>
let’s say in the first X steps, the random steps give us the <strong>right</strong> move action and in this case we are too close that pole will fall or 2.4 units away from the center, so we trained the model with X  rewards (for each step) but it can direct us to the wrong solution, am I right ?</p>
</li>
<li>
<p>I saw other implementations of DQN (CartPole) which used one model (and not 2 as we saw in the example above): <a href="https://towardsdatascience.com/deep-q-networks-theory-and-implementation-37543f60dd67" rel="noopener nofollow ugc">https://towardsdatascience.com/deep-q-networks-theory-and-implementation-37543f60dd67</a><br>
What is the benefit to use 2 models and not one model ?</p>
</li>
<li>
<p>We get to <strong>done</strong>  state at each epoch.<br>
So we can get to <strong>done</strong> after small number of steps (the pole fell too early) and in other epochs we can get to reward with higher number of steps (and rewards) (we saved the pole not to fall for too long).<br>
In both cases, the model is fitted (and wights are updated).<br>
Why we fit the model for those 2 cases and not just for epochs which the reward is higher than the previous epoch (or best epoch) rewards?</p>
</li>
</ol>
Assertion failed: Runtime canonicalization must simplify reduction axes to minor 4 dimensions

username: abiantonio
cooked: <p>Hello,</p>
<p>I’m trying to train a model with the AlexNet architecture. I’m able to train it on a cuda device using google colab but when I try to train it on my mac using mps, I get the following error:</p>
<p><code>Assertion failed: (0 &lt;= mpsAxis &amp;&amp; mpsAxis &lt; 4 &amp;&amp; "Runtime canonicalization must simplify reduction axes to minor 4 dimensions."), function encodeNDArrayOp, file GPUReductionOps.mm, line 76.</code></p>
<p>Python would crash and my kernel would die, so I’m at a loss on how to debug this. I would greatly appreciate any help or hints to point me in the right direction for fixing this.</p>

username: Radu
cooked: <p>Hello,</p>
<p>I have the same problem when calling the backpropagation function on a custom GAN (<code>self.lossD.backward()</code>) or (<code>self.lossG.backward()</code>). This only happens when using MPS, as it works when using the CPU.</p>
<pre><code class="lang-auto">Assertion failed: (0 &lt;= mpsAxis &amp;&amp; mpsAxis &lt; 4 &amp;&amp; "Runtime canonicalization must simplify reduction axes to minor 4 dimensions."), function encodeNDArrayOp, file GPUReductionOps.mm, line 76.
</code></pre>
<p>As well, the kernel just stops and could not debug further more than which line is causing this error.</p>
Split train and test set

username: John4
cooked: <p>Hi everyone! I have a dataset (input) that is a matrix H * W(ndarray) and a ndarray (target) of dimension H. I tried to split the dataset in train and test set:</p>
<pre><code>    X_train, X_test, y_train, y_test = train_test_split(M, L, test_size=0.3)
</code></pre>
<p>where M is the matrix and L is the array containing the labels. The code works but the most of the rows of X_train and y_train are zero. Where am I wrong?</p>

username: vdw
cooked: <p>Are you sure that W and L are non-zero? <code>train_test_split()</code> does not change the data.</p>

username: John4
cooked: <p>I used a function to create the matrix. I also did the debug and M has many rows(H=100000) and 25 columns (W) and it has not zero elements. L is H dimensional array.</p>
Loading pretrained model from .npz

username: learning2torch
cooked: <p>Hello all!</p>
<p>I am fairly new to PyTorch and I am struggling with the following problem:</p>
<p>I have a pretrained model saved as .npz and I want to load it as a torch model.<br>
(The trained model was created with objax).</p>
<p>With np.load(path) I get a numpy.lib.npyio.NpzFile.</p>
<p>Some additional info:<br>
npzfile = np.load(pretrained_weights_path)<br>
print(list(npzfile.keys()))</p>
<p>results in:<br>
[‘names’, ‘0’, ‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘8’, ‘9’, ‘10’, ‘11’, ‘12’, ‘13’, ‘14’, ‘15’, ‘16’, ‘17’, ‘18’, ‘19’, ‘20’, ‘21’, ‘22’, ‘23’, ‘24’, ‘25’, ‘26’, ‘27’, ‘28’, ‘29’, ‘30’, ‘31’, ‘32’, ‘33’, ‘34’, ‘35’, ‘36’, ‘37’, ‘38’, ‘39’, ‘40’, ‘41’]</p>
<ul>
<li>In ‘names’ I have my model “structure”, i.e. a ResNet.</li>
<li>In ‘0’, ‘1’,… ‘41’ I have the weights as numpy.ndarray.</li>
</ul>
<p>How can I load this as a torch model? Or is that even possible?</p>
<p>Please let me know if you need additional information. I refrained from sharing my code because the weights are confidential.</p>

username: ptrblck
cooked: <p>Assuming each integer key corresponds to a parameter you could try to map them to the actual layer names and restore the <code>state_dict</code>. Take a look at the original <code>model.state_dict</code> and check the key value pairs to get an idea how your custom object can be mapped to it.</p>
No-Grad Scope and Detach

username: goktug_guvercin
cooked: <p>Hello Torch Community,</p>
<p>I have a question about the usage of <code>detach()</code> during validation of the model. As we know, <code>detach()</code> creates a new tensor that shares storage with its original one, but does not requires grad. If the validation of the model is done inside <code>with torch.no_grad()</code> scope, do we need to use <code>detach()</code> in that case ?</p>
<p>Thanks in advance.</p>

username: srishti-git1110
cooked: <p>Inside the no_grad context manager, the autograd essentially “looks away” so no expensive gradient computation is performed and you should be fine using it mostly.</p>
<p>That said, there are certain cases in which tensors even when created within no_grad require gradients. (requires_grad isn’t False)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d70a23f8ca88b98206ef973a92803428db52e058.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d70a23f8ca88b98206ef973a92803428db52e058" title="20221126_182758"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d70a23f8ca88b98206ef973a92803428db52e058_2_690x130.jpeg" alt="20221126_182758" data-base62-sha1="uGknGN47rFIexYoGXcCIMubZoow" width="690" height="130" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d70a23f8ca88b98206ef973a92803428db52e058_2_690x130.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d70a23f8ca88b98206ef973a92803428db52e058.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d70a23f8ca88b98206ef973a92803428db52e058.jpeg 2x" data-dominant-color="C1BDB8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20221126_182758</span><span class="informations">1024×193 38.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
An excerpt from the Deep Learning with PyTorch book.</p>
<p>That said, you might want to dig into these cases more and choose whether to use detach or not. Otherwise, no_grad works for evaluating.</p>
<p>PS. Also don’t forget to call <code>model.eval()</code></p>
Same network two losses

username: shashank_gupta1
cooked: <p>I have one network and two different loss functions. One of the loss functions, a kind of regularizer,  is a weighted loss function, where the weight decays with number of data points. I want the effect of the other loss function to decay as num. data points increase. However, as num points increase it still has a very strong effect.</p>
<blockquote>
<p>model = Model()<br>
opt1 = optimizer.Adam(model.parameters())<br>
opt2 = optimizer.Adam(model.parameters())</p>
<p>loss1 = loss_function1(model, inp, out)<br>
opt1.zero_grad()<br>
loss1.backward()<br>
opt1.step()</p>
<p>loss2 = (10/(num_samples) ) * loss_func2(model, inp, out)<br>
opt2.zero_grad()<br>
loss2.backward()<br>
opt2.step()</p>
</blockquote>
<p>When “num_samples” is very large (10^9), since the weight is so small, loss_func2’s effect should be negligible (weight=10^-8). However, it still has a strong effect. At smaller data points, it is working as expected, and it has a strong effect. I am not sure what’s wrong.</p>
RuntimeError: Storage size calculation overflowed with sizes=[-96]

username: Siladittya_Manna
cooked: <p>This error is arising when I am trying to run “gpytorch.utils.lanczos.lanczos_tridiag” on the GPU.</p>
<p>The error is occurring in this line of code</p>
<p>sim_i_j = torch.diag(sim, self.batch_size)</p>
<p>where sim is a NxN array of cosine similarities between two batches of samples, (N = 2* self.batch_size). So, the (i,j)-th sample of “sim” array is the cosine similarity value between i-th feature vector in the first batch and the j-th feature vector in the second batch.</p>
<p>This line is part of my custom loss function where I am calculating the loss for SimCLR algorithm.</p>
<p>However when I use “device = ‘cpu’” I do not get this error “RuntimeError: Storage size calculation overflowed with sizes=[-96]”</p>
<p>But, the model.forward() is being executed on GPU regardless of what device I use for lanczos_tridiag function.</p>
<p>I set the self.batch_size to 16 down from 128 in this run. The number in in “[ ]” changes when using different batch_sizes.</p>
<p>I do not understand the reason for this error. What is the reason behind this error?</p>
Detaching predicted output from history

username: Messiah
cooked: <p>Hello,</p>
<p>In the context of learning with an RNN using a teacher forcing option, that is, to decide whether to condition the next timestep prediction on either the current ground-truth or model predicted state, should we <code>.detach()</code> the predicted output in the case when we do not use teacher forcing?</p>
<p>The source of my confusion arises from an apparent discrepency in two PyTorch tutorials using a similar encoder-decoder architecture:<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<pre><code class="lang-auto">decoder_input = topi.squeeze().detach()  # detach from history as input
</code></pre>
<p><a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html</a></p>
<pre><code class="lang-auto">decoder_input = torch.LongTensor([[topi[i][0] for i in range(batch_size)]])
</code></pre>
<p>Is this context-dependent or is there a canonical practice for handling the output of a model with teacher forcing optionality? Is there a difference between the two approaches in terms of performance?</p>
How to fuse layers of pretrained model?

username: MD_DIDARUL_ISLAM
cooked: <p>Hi,<br>
I am using the DeepLabV3 with MobileNetV3Large backbone. What could be the good way to fuse the layers ? All layer fusing examples I see is mostly for user defined models.<br>
Thanks in avdance.</p>
<p>My model is as follows:</p>
<p>from torchvision import models<br>
model = models.segmentation.deeplabv3_mobilenet_v3_large(weights=‘COCO_WITH_VOC_LABELS_V1’,progress=True)</p>
How to calculate gradient of the upper part of the network

username: CC_Doomer
cooked: <p>I have a simple network and the grad of the final FC layer<br>
How can I calculate the rest grad</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">This tutorial</a> might be a good starter as it introduces you to the general neural network training.</p>

username: CC_Doomer
cooked: <p>For example, I have a simple network  like:<br>
conv→conv→relu→linear<br>
I only have the gradient of the linear layer.<br>
I don’t have access to the loss or something like smashed data in split learning.<br>
How can I calculate the gradient of the two conv layers or where should I use backward ?</p>

username: ptrblck
cooked: <p>Assuming you have the output gradient of your linear layer, something like this would work:</p>
<pre><code class="lang-python">out = model(input) # linear output
out.backward(gradient=output_gradient)
</code></pre>

username: CC_Doomer
cooked: <p>So does it mean I still need both the output and the gradient of one layer to backward</p>
Use running_mean instead of batch mean when calculating variance in BatchNorm2d at train time

username: Tycho_van_der_Oudera
cooked: <p>Is there an easy way to let BatchNorm2d at train time calculate the variance of a batch w.r.t. the running_mean instead of the mean of that batch? That is, to use variance = mean(x - running_mean) instead of mean(x - mean(x)^2).</p>
<p>I need this feature for batch_size&gt;1. However, I expect this to be generally very useful since it would allow variance estimates in BatchNorm even when training with batch_size=1.</p>
<p>Many thanks in advance!<br>
Tycho</p>

username: ptrblck
cooked: <p>You could take a look at my <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">manual batchnorm implementation</a> and manipulate it as you wish.<br>
I would also recommend to try scripting your module afterwards to allow e.g. nvFuser to code-gen the (fused) kernels to speed it up again.</p>

username: Tycho_van_der_Oudera
cooked: <p>Thanks, Piotr. That was exactly what I was looking for!</p>
Weird debug messages in log when using resnet50?

username: LukasKarner
cooked: <p>I just successfully ran Pytorch’s implementation of ResNet50 with pre-trained weights on ImageNet.<br>
However something dumped thousands of strange debug messages in my log file.<br>
They appear to come in groups as there are recurring patterns in the tags, here is an exemplary collection of them:</p>
<pre><code class="lang-auto">2022-11-25 21:44:12,649 DEBUG: tag: Orientation (274) - type: short (3) - value: b'\x00\x01'
2022-11-25 21:44:12,649 DEBUG: tag: XResolution (282) - type: rational (5) Tag Location: 34 - Data Location: 98 - value: b'\x00\x00\x00\x1c\x00\x00\x00\x01'
2022-11-25 21:44:12,649 DEBUG: tag: YResolution (283) - type: rational (5) Tag Location: 46 - Data Location: 106 - value: b'\x00\x00\x00\x1c\x00\x00\x00\x01'
2022-11-25 21:44:12,650 DEBUG: tag: ResolutionUnit (296) - type: short (3) - value: b'\x00\x03'
2022-11-25 21:44:12,650 DEBUG: tag: Software (305) - type: string (2) Tag Location: 70 - Data Location: 114 - value: b'Adobe Photoshop 7.0\x00'
2022-11-25 21:44:12,650 DEBUG: tag: DateTime (306) - type: string (2) Tag Location: 82 - Data Location: 134 - value: b'2004:01:20 13:12:15\x00'
2022-11-25 21:44:12,650 DEBUG: tag: ExifIFD (34665) - type: long (4) - value: b'\x00\x00\x00\x9c'
2022-11-25 21:44:15,997 DEBUG: tag: Software (305) - type: string (2) Tag Location: 22 - Data Location: 62 - value: b'EXIFutils V2.7.4'
2022-11-25 21:44:15,998 DEBUG: tag: Artist (315) - type: string (2) Tag Location: 34 - Data Location: 78 - value: b'Wendy Knipe'
2022-11-25 21:44:15,998 DEBUG: tag: Copyright (33432) - type: string (2) Tag Location: 46 - Data Location: 90 - value: &lt;table: 59 bytes&gt;
2022-11-25 21:44:15,998 DEBUG: tag: ExifIFD (34665) - type: long (4) - value: b'\x00\x00\x00\x96'

</code></pre>
<p>Can someone tell me where these come from and/or how I can suppress them?</p>
<p>best<br>
Lukas</p>

username: ptrblck
cooked: <p>I think these debug logs are raised by <code>PIL</code> if it cannot detect the image file format.<br>
You might try to set the logging level a bit higher via:</p>
<pre><code class="lang-python">logging.getLogger("PIL.PngImagePlugin").setLevel(logging.CRITICAL + 1)
</code></pre>
<p>and see if it helps.<br>
Also, I’m unsure if the <code>PIL.PngImagePlugin</code> is raising these logs, so you might need to change it.</p>

username: LukasKarner
cooked: <p>Almost correct! The suggested fix didn’t work, but I then changed the log format to display the path of the file where the message was logged, and it turns out <code>PIL.TiffImagePlugin</code> causes the trouble.<br>
So the solution is:</p>
<pre><code class="lang-auto">logging.getLogger("PIL.TiffImagePlugin").setLevel(51)
</code></pre>
<p>.</p>
Strange behavior in constraint optimization

username: shashank_gupta1
cooked: <p>I am trying to solve a constraint optimization problem in offline RL/bandit (in a recommendation context).<br>
The policy is h(y | x), where y is an action, x is the current input. The policy is parameterized by simple MLP (x → h1 → h2). I have logged data from a logging policy h_0(y | x).  The optimization formulation looks like:</p>
<blockquote>
<p>max Utility(h(y | x))</p>
<p>such that, KL-divergence(h || h_0) &lt; eps.</p>
</blockquote>
<p>Where the constraint is helpful in the low-data setting by falling back to the logging policy.</p>
<p>I am implementing it in an alternate optimization formulation with the following pseudo-code:</p>
<pre><code class="lang-auto">optimizer = Adam(lr, weight_decay=1e-5)
for i, batch in dataloader1:
     action, context = batch['action'], batch['context']
     utility_loss = get_utility(action, context)
     optimizer.zero_grad()
     obj.backward()
     optimizer.step()
     for j, batch in dataloader2:
          action, context,  = batch['action'], batch['context']
          kld_loss = 10/np.sqrt(total_data) * get_kld(policy, action, context, logging_policy)
          optimizer.zero_grad()
         kld_loss.backward()
         optimizer.step()    
         if j &gt; 1:
            break
</code></pre>
<p>I am multiplying the kld-loss with 10/sqrt(total_points), so that when total number of logged actions are very large, the kld-loss decreases, since we can have high confidence on the utility loss.</p>
<p>The strange behavior I am observing is, even when the number of datapoints are 10^9, the performance with kld-loss is better than w/o kld-loss, even though the weight is 0.00031. I have set the weight to 0 also manually, but same behavior. Any idea why it is happening?</p>
Unreadable letters in Pandas DF

username: Tornike
cooked: <p>Hi, i have CSV File where data is written in the Georgian Language. but once i open that as Pandas Dataframe it writes some unknown  letters like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f08d8b7277804e9976794b187145c264abf5c15e.png" alt="image" data-base62-sha1="yk1QLtRSgRRk6o4cyITCiEDLp26" width="385" height="107"></p>
<p>i think it should be done using utf-8 decoding, but returns this error which i can’t solve.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91939b2e32bbaf697129b14acf802fd0eded01df.png" data-download-href="https://discuss.pytorch.org/uploads/default/91939b2e32bbaf697129b14acf802fd0eded01df" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91939b2e32bbaf697129b14acf802fd0eded01df_2_690x56.png" alt="image" data-base62-sha1="kLPtDmTWGqopzKcPRZ46oQDvqaj" width="690" height="56" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91939b2e32bbaf697129b14acf802fd0eded01df_2_690x56.png, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91939b2e32bbaf697129b14acf802fd0eded01df.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91939b2e32bbaf697129b14acf802fd0eded01df.png 2x" data-dominant-color="CFE4FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">935×76 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i tried different decoding types such as: latin-1, ISO-8859-1 but these don’t work. no error but the output is the same.</p>
<p>what should i do?</p>
Combine sparse features with pre-trained word embeddings

username: caesarsalad
cooked: <p>Hi, i’m trying to create a working example of RASA’s Dual Intent &amp; Entity Transformer (<a href="https://arxiv.org/pdf/2004.09936.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2004.09936.pdf</a>) model in pytorch and am having trouble with combining sparse features with dense features.</p>
<p>Basically, the paper uses sparse features and embeds them, and then combines them with BERT’s output.</p>
<p>My nn.Module has the following layers</p>
<pre><code class="lang-auto">        self.entities_list = ["O"] + config.entities
        self.num_entities = len(self.entities_list)
        self.intents_list = config.intents
        self.num_intents = len(self.intents_list)

        self.ffnn = nn.Linear(config.hidden_size, self.config.embedding_dim)
        self.transformer = TransformerModel(
            d_model=self.config.embedding_dim,
            nhead=8,
            nlayers=2,
            d_hid=self.config.max_token_len)

        tag_to_idx = prepare_tag_to_idx(self.entities_list)
        self.crf = BiLSTM_CRF(len(self.entities_list)+3, 
emb_dim=self.config.embedding_dim)
        self.intents_classifier = nn.Linear(
            self.config.embedding_dim,
            self.num_intents)
</code></pre>
<p>and my forward method accepts both features &amp; dense (BERT) features. My question is, how do i combine them?</p>
<p>thank you, i would love if you could direct me somewhere as well that could help me fully assemble the model as I’m having trouble finishing it (it has horrible results).</p>

username: monherry9
cooked: <p>How to Implement BERT</p>
<p>1.Getting the BERT model from the TensorFlow hub.<br>
2. Build a Model according to our use <a href="https://seotoolsystem.com/word-combiner" rel="noopener nofollow ugc">word combining generator</a> case using BERT pre-trained layers.<br>
3. Setting the tokenizer.<br>
4. Loading the dataset and preprocessing it.<br>
5. Model Evaluation.</p>
Gradient becomes nan with random RuntimeError: Function 'ExpBackward0' returned nan values in its 0th output

username: Laya1
cooked: <p>I am training a classification model, and getting <code>nan</code> loss values after some time. I used <code>torch.autograd.set_detect_anomaly(True)</code> to trace the error and I get this as output:</p>
<p><code>RuntimeError: Function 'ExpBackward0' returned nan values in its 0th output.</code></p>
<p>I then used the following checks for values of variables and gradients in my training routine to find out where the problem arises first:</p>
<pre><code class="lang-auto">batch_predictions = model(batch_inputs)
batch_loss = loss(batch_predictions, batch_targets)
if torch.isfinite(batch_predictions).all() and torch.isfinite(batch_loss.item()):
    print("outputs and loss ok")
else:
    print("outputs and loss not ok")
batch_loss.backward()
grads = [p.grad for p in model.parameters() if p.requires_grad is True]
for grads_ in grads:
    if torch.isnan(grads_).any():
        print("grads nan")
    else:
        print("grads ok")
optimizer.step()
</code></pre>
<p>After some time I get <code>grads nan</code> as output, but the immediately preceding <code>outputs and loss ok</code> is also printed. This means that the outputs are ok, the loss is ok but the gradient calculations with <code>batch_loss.backward()</code> leads to <code>nan</code> gradients being calculated. I have tried changing the optimizer and reducing the learning rate, but nothing works. I am not sure why this is happening, and how to probe further and correct this. Thanks in advance for any help.</p>

username: srishti-git1110
cooked: <p>Hi,<br>
Non nan losses and nan gradients are mostly a result of some absurd (undefined) mathematical operation like 0⁰, dividing by 0 and so on.</p>
<p>The hint provided by anomaly detection probably hints at the step in the computational graph where such an operation is occurring leading to nan gradients.</p>
<p>I would also think limited precision can be a potential reason but less likely as compared to the former.</p>

username: ConvolutionalAtom
cooked: <p>Srishti is quite right. I would also note that runtimeerror is due to <code>ExpBackward</code>, meaning backprop of exponential function. This implies somewhere in the network exponential function is used.<br>
Since exponential function blows too fast (exponentially fast :)) → overflow<br>
or can return too small value → underflow. I would take a look at this part of the network.</p>
<p>Maybe, apply some cutofff to output of exponential function, just an idea.</p>
Pytorch to Verilog-A

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>There is any library or anyone have done any script capable to transform the PyTorch neural network into the Hardware Description Language Verilog-A?</p>
<p>Regards,<br>
André</p>

username: xinlnix
cooked: <p>Do you find it? Welcom to comunicate with me: xinlnix[at]<a href="http://qq.com" rel="noopener nofollow ugc">qq.com</a></p>
Trying to restore training state but checkpoint contains only the model. This is probably due to `ModelCheckpoint.save_weights_only` being set to `True`

username: Imanuel_Roz
cooked: <p>HI, I am using Pytorch Lightning, trying to restore a model, I have de model_epoch=15.ckpt file and would like to restore from here, so I introduced the resume_from_checkpoint in the trainer, but I get the following error:<br>
Trying to restore training state but checkpoint contains only the model. This is probably due to <code>ModelCheckpoint.save_weights_only</code> being set to <code>True</code>.</p>
<p>This is because I put save_weights_only=True in the ModelCheckpoint.<br>
What should I do to restore the model?</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">def main():
    exp_folder = get_experiment_folder()
    train_dataset = Hotelimages(split='train')
    valid_dataset = Hotelimages(split='valid')
    batch_size = 32 
    train_dl = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    valid_dl = DataLoader(valid_dataset, batch_size=batch_size, shuffle=True)
    model = SwinTransformerFineTuning()
    model_checkpoint = ModelCheckpoint(exp_folder, save_last=True, filename='model_{epoch}',
                                       save_weights_only=True, every_n_epochs=5)
    trainer = pl.Trainer(precision=16, default_root_dir=exp_folder, callbacks=[model_checkpoint], max_epochs=100, accelerator='gpu', devices=1,
                         resume_from_checkpoint=run_folder/'1/model_epoch=14.ckpt')
    trainer.fit(model, train_dl, valid_dl)
</code></pre>

username: pascal_notsawo
cooked: <p>I already had this problem. You can solved it this way.</p>
<pre><code class="lang-python">model = SwinTransformerFineTuning.load_from_checkpoint(
    # hparams = ..., # if needed 
    checkpoint_path = run_folder/'1/model_epoch=14.ckpt'
)
model.train()
# ...

trainer = pl.Trainer(resume_from_checkpoint=None)
trainer.fit(model, train_dl, valid_dl)
</code></pre>
AttributeError: 'LSTMModel' object has no attribute 'fc'

username: mhof
cooked: <p>Hi,</p>
<p>I am trying to train a LSTM Network for a Text Classification task. I have tried  instantiating the model through pytorch</p>
<pre><code class="lang-auto">model = LSTMModel(input_dim=input_dim, hidden_dim=hidden_dim,layer_dim=layer_dim, output_dim=output_dim)
</code></pre>
<p>and through skorch using</p>
<pre><code class="lang-auto">classifier = NeuralNetClassifier(
    module=LSTMModel,
    module__input_dim=input_dim,
    module__hidden_dim=hidden_dim,
    module__layer_dim=layer_dim,
    module__output_dim=output_dim,
    batch_size=batch_size,
    max_epochs= num_epochs,
    criterion=nn.CrossEntropyLoss,
    optimizer=torch.optim.Adam,
    lr=0.1,
    train_split=None,
    callbacks=callbacks,
    verbose=1,
    device=device)
</code></pre>
<p>in both cases I get AttributeError: ‘LSTMModel’ object has no attribute ‘fc’. The model is defined as follows:</p>
<pre><code class="lang-auto">from torch import nn

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc == nn.Linear(hidden_dim, output_dim)
    
    def foward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc(out[:,-1, :])
        return out

input_dim = train_features.shape[2] 
hidden_dim = 32 
layer_dim = 1 
output_dim = len(df.unique())
</code></pre>
<p>What am I missing? Pytorch version is torch==1.13.0.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>You have a typo in your code and are trying to compare <code>self.fc</code> to an <code>nn.Linear</code> layer:</p>
<pre><code class="lang-python">self.fc == nn.Linear(hidden_dim, output_dim)
</code></pre>
<p>instead of assigning it:</p>
<pre><code class="lang-python">self.fc = nn.Linear(hidden_dim, output_dim)
</code></pre>

username: mhof
cooked: <p>Thank you so much! I have been staring at this code for way to long to ever figure this out on my own.</p>
Visualize Areas of Interest

username: fabianslife
cooked: <p>Hello Everyone,</p>
<p>I have recently created a PyTorch model, which is able to detect brain tumours from CT Images. However, I want to highlight the tumor or what the model perceives as a tumor in a color or perhaps just a frame around it. Is this possible using torch vision?</p>
<p>Possibly looking like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e7ec3db3db309c7eddf2aaf90ff3d73d612495b.png" alt="download" data-base62-sha1="4lLUQZkMiZvcAk8RNjFlMzJLGht" width="427" height="412"></p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: azhanmohammed
cooked: <p>You can use the generated binary map, overlay it using alpha blending in opencv and get the closest highlight possible. Let me know if you need the code for it.</p>
Variance caluculation for an image patch after converting to patch tensor and normal patch

username: jyothi_sri
cooked: <p>i have calculated variance of an image patch after converting it to tensor and normal patch ,variance for both the same patch is differing Can anyone help me out .</p>

username: srishti-git1110
cooked: <p>Hi, could you please try posting some code to demonstrate what exactly you mean by normal?</p>
<p>A minimum executable snippet should be good, please.</p>

username: jyothi_sri
cooked: <p>def NonOverlappingCropPatches(img, patch_size=128, stride=128):<br>
w, h = img.size<br>
patches = ()<br>
for i in range(0, h - stride, stride):<br>
for j in range(0, w - stride, stride):<br>
patch = img.crop((j, i, j + patch_size, i + patch_size))<br>
patches = patches + (patch,)<br>
return patches</p>
<p>patches = []<br>
img = Image.open(r"D:\GITHUB\CNNIQA-NEW\bikes.bmp").convert(‘L’)<br>
patches = NonOverlappingCropPatches(img,8,8)<br>
print(patches[0])<br>
im=[99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 98, 99, 99, 100, 100, 100, 93, 96, 95, 99, 100, 104, 112, 118, 96, 99, 102, 100, 100, 106, 112, 119, 102, 102, 103, 103, 99, 104, 110, 114, 103, 106, 106, 103, 102, 106, 104, 107, 109, 109, 106, 103, 104, 103, 100, 103]<br>
var1=statistics.variance(im)<br>
print(var1)</p>
Memory management - cuda programming

username: Sumesh_Sankar
cooked: <p>look this training loop shown, i have moved my inputs and labels to GPU by .to(device) command, as the inputs and labels needed only to the point of loss calculation, i need to simply erase the memory<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad847b41c94394f6d59ffee6c21a077d8422b940.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad847b41c94394f6d59ffee6c21a077d8422b940" title="Screenshot from 2022-11-23 10-13-16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad847b41c94394f6d59ffee6c21a077d8422b940_2_690x388.png" alt="Screenshot from 2022-11-23 10-13-16" data-base62-sha1="oL0rrtYS3MQJuCiEZWxCdP3SIlq" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad847b41c94394f6d59ffee6c21a077d8422b940_2_690x388.png, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad847b41c94394f6d59ffee6c21a077d8422b940.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad847b41c94394f6d59ffee6c21a077d8422b940.png 2x" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-11-23 10-13-16</span><span class="informations">937×528 71.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
allocated in the GPU for inputs and labels. so that i can utilise the same memory or without loss of memory for the next batch</p>
Transforms on subset

username: mmg
cooked: <p>I created a test_train split using <code>torch.utils.data.Subset</code>. How do I apply different train/test transforms on these before passing them as an argument to the Dataloader?</p>

username: ptrblck
cooked: <p>One approach would be to write a simple custom <code>Dataset</code>, which just applies your transformation to the internal subset as described <a href="https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/4">here</a>.</p>

username: mmg
cooked: <p>Thanks. It works like a charm.</p>
Split loss across two different models and calculate gradients

username: avyavkumar
cooked: <p>Hi,</p>
<p>I have two separate models, each of which outputs a probability distribution. I weigh the probability distribution of each model by a dynamic scaling factor and calculate the outputs. I compare the outputs against the true labels and generate the overall loss.</p>
<p>However, I want to split the loss across both models in the same scaling factor, essentially a weighted average. Once each model receives the scaled loss, it backpropagates and computes the gradients. The learning factors and optimisers can be different for both models.</p>
<p>At a high level, the (pseudo)code looks like this -</p>
<pre><code class="lang-auto">import torch

modelA = Model(paramsA)
modelB = Model(paramsB)

optimiserA = getOptimiser(modelA.parameters(), lr=lr_A)
optimiserB = getOptimiser(modelB.parameters(), lr=lr_B)

outputs = modelA(inputs)*scalingFactor_A + modelB(inputs)*scalingFactor_B
loss = criterion(outputs, labels)

# unsure how exactly loss.backward() and optimiserA.step() and optimiserB.step() should work
</code></pre>
<p>Any help is appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Your pseudo code looks alright and <code>loss.backward()</code> will calculate the gradients for both outputs.<br>
For Autograd it doesn’t matter if the scaling and addition is performed “outside of the model” or in the <code>forward</code> pass and it’s just a mathematical operation which <code>loss.backward()</code> will backpropagate through.</p>
<p><code>optimizerX.step()</code> will update the passed parameters using their <code>.grad</code> attribute.</p>
Is torch.cuda.Stream().cuda_stream equivalent to cudaStream_t

username: Rhett_Ying
cooked: <p>Hi,</p>
<p>I’d like to obtain the underlying cudaStream_t pointer from torch.cuda.Stream().cuda_stream in python, then pass such obtained pointer into my cpp code which does not depend on torch. I’d like to use such pointer to call cudaStreamSynchronize(ptr).</p>
<p>Is this possible and safe to use like this? just reinterpret_cast&lt;cudaStream_t&gt;(torch.cuda.Stream().cuda_stream).</p>

username: ptrblck
cooked: <p>Yes, I believe you can create the pointer via:</p>
<pre><code class="lang-python">cuda_stream = torch.cuda.Stream().cuda_stream
ctypes.c_void_p(cuda_stream)
</code></pre>

username: Rhett_Ying
cooked: <p>Thanks for your reply.</p>

username: Rhett_Ying
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
As stream, I think the same equivalence  is applied to torch.cuda.Event().cuda_event and cudaEvent_t as well. Right? just create pointer via <code>cuda_event =  torch.cuda.Event()._cuda_event   ctypes.c_void_p(cuda_event)</code>.</p>

username: ptrblck
cooked: <p>Yes, this should also work for events.</p>

username: QiuJingkai
cooked: <p>hi,<br>
I want to pass torch.cuda.Stream() to my cpp code and use it as c10::cuda::CUDAStream but failed.<br>
Whether there are some APIs that can do this? Hope for your reply.</p>

username: ptrblck
cooked: <p>Could you describe what in <a class="mention" href="/u/rhett_ying">@Rhett_Ying</a>’s suggestion is failing?</p>
After adding a custom parametrization, when does it get called?

username: vi.le
cooked: <p>Hello,</p>
<p>After adding adding a parametrization module with:</p>
<pre><code class="lang-python">torch.nn.utils.parametrize.register_parametrization(
    self.conv, "weight", _SumOne()
)
</code></pre>
<p>, when does the <code>forward()</code> function of  <code>_SumOne()</code> get called? The <code>forward</code> function makes is so that the sum of the weights is 1:</p>
<pre><code class="lang-python">def forward(self, weight: torch.Tensor) -&gt; torch.Tensor:
    return torch.divide(weight, weight.sum())
</code></pre>
<p>Is it called when the weights are accessed (in the forward and backward passes) or only when the weights are modified (in the backward pass)?</p>
<p>Note: <code>self.conv</code> is a <code>torch.nn.Conv2d</code></p>
<p>Thanks.</p>