akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Broadcast a tensor

username: lengocduc195khtn
cooked: <p>Hello everyone, I have a question about how I can broadcast my tensor. For example, I a tensor X = [1,0,0,1,0] and an embedding as follow:</p>
<pre><code class="lang-auto">Y = [[0.1992, 0.5196, 0.4954],
          [0.1224, 0.0704, 0.6504],
          [0.5171, 0.8013, 0.1156],
          [0.8703, 0.2654, 0.6088],
          [0.1722, 0.2609, 0.0734]]
</code></pre>
<p>And then, I want to convert X from [1,0,0,1,0] to</p>
<pre><code class="lang-auto">X = [[0.1992, 0.5196, 0.4954],
          [0.8703, 0.2654, 0.6088]]
</code></pre>
<p>This is the first time I post a topic, so if there are any problem, please remind me. Thanks</p>

username: Srinjoy_Mukherjee
cooked: <p>I did not get your question. I think your understanding of embedding is wrong. For the desired output , The values of X should be</p>
<pre><code class="lang-auto">X = [0,3]
</code></pre>
<p>The embedding layer is basically a dictionary lookup. So, new_X[i] = Y[X[i]].</p>

username: zero1
cooked: <p><a class="mention" href="/u/lengocduc195khtn">@lengocduc195khtn</a> I guess you might want to know about the process of forward propagation of neural networks.</p>
<p>it depends. For example, whether it is a convolutional layer or a linear layer. Take convolution as an example.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a376e002d488d8a5e107d65206ee5fee43fa6020.png" data-download-href="https://discuss.pytorch.org/uploads/default/a376e002d488d8a5e107d65206ee5fee43fa6020" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a376e002d488d8a5e107d65206ee5fee43fa6020.png" alt="image" data-base62-sha1="nk4w0C1Y1bH1UaF4bRdNJK3gpRS" width="690" height="350" data-dominant-color="EFF0EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">716×364 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>below is a pytorch example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; m = torch.nn.Conv2d(1, 1, 2)
&gt;&gt;&gt; input = torch.randn(1, 3, 3)
&gt;&gt;&gt; input
tensor([[[ 0.4180, -0.0296, -1.3577],
         [ 0.0083, -0.0097,  0.7227],
         [ 0.8191, -0.1416,  0.4224]]])
&gt;&gt;&gt; output = m(input)
&gt;&gt;&gt; output
tensor([[[0.0241, 0.1335],
         [0.0582, 0.2718]]], grad_fn=&lt;SqueezeBackward1&gt;)
</code></pre>
Torch.sparse.max()?

username: Thibgolds
cooked: <p>I’m looking for a torch.sparse function that could take the max (and ideally argmax) of torch.sparse_coo_tensor, along a specific sparse dimension. I’ve seen in the docs that there is no such function (despite there being a torch.sparse.sum() function). I was wondering if there was an obvious workaround that I’m missing, or if it’s too difficult to implement.</p>
<pre><code class="lang-auto">i = torch.tensor([[0, 1, 1],
                  [2, 0, 2]])
v = torch.tensor([3, 4, 5], dtype=torch.float32)

coo_tensor=torch.sparse_coo_tensor(i, v, [2, 4])

max_coo_tensor = coo_tensor.to_dense().max(0)[0] #Works

max_coo_tensor = coo_tensor.max(0)[0] #Error
</code></pre>
Mysterious link error seen by multiple people while building cpp/cuda extensions

username: Victory
cooked: <p>I am trying to build two different PyTorch cpp/cuda extensions on Windows. One is Facebook provided “SparseConvNet” and another named “pointgroup_ops” by a research group somewhere else. Seems like they both have generally been built and tested on Linux machines. I need to build them on Windows. After fixing a few compilation issues, I have reached a stage where both of them have a single mysterious looking link error.<br>
For the first library, the error is:</p>
<pre><code class="lang-auto">sparseconvnet_cuda.obj : error LNK2001: unresolved external symbol "public: long * __cdecl at::Tensor::data_ptr&lt;long&gt;(void)const " (??$data_ptr@J@Tensor@at@@QEBAPEAJXZ)
build\lib.win-amd64-3.7\sparseconvnet\SCN.cp37-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
</code></pre>
<p>For the second library, the error is:</p>
<pre><code class="lang-auto">pointgroup_ops.obj : error LNK2001: unresolved external symbol "public: long * __cdecl at::Tensor::data_ptr&lt;long&gt;(void)const " (??$data_ptr@J@Tensor@at@@QEBAPEAJXZ)
build\lib.win-amd64-3.7\PG_OP.cp37-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
</code></pre>
<p>They both look very similar. Any insights about the cause/fix? Of course, the code is same as the one that builds on Linux; so, it can’t be the case where something is genuinely not defined. ‘QEBAPEAJXZ’ not a string that occurs in code.</p>
<p>I did an online search for the strange ‘QEBAPEAJXZ’ string and seems like people have encountered this in the context of building other libraries as well.<br>
Here are two for example: <a href="https://github.com/sshaoshuai/PointRCNN/issues/75" rel="nofollow noopener">https://github.com/sshaoshuai/PointRCNN/issues/75</a><br>
<a href="https://zhuanlan.zhihu.com/p/142676847" rel="nofollow noopener">https://zhuanlan.zhihu.com/p/142676847</a></p>
<p>But, I did not find a solution that has worked. Any help in resolving this would be appreciated by me, and also probably by other people who have encountered this as well.<br>
Thanks.</p>

username: Victory
cooked: <p>For the benefit of others running into this: Thanks to a poster on NVidia CUDA developers forum, I was able to solve the mysterious link error. The cause of the link error is the usage of “long” type which happens to be taken to be 32-bit long on Windows 64 bit machines while it is taken to be 64 bit long on Linux 64 bit machines. Replacing all occurrences of “long” in the code by “int64_t” got rid of the link error. As mentioned by that knowledgeable poster on the CUDA forum, it is a really bad idea to use “long” in code that is meant to be cross-platform.</p>

username: limaodaxia.ontherise
cooked: <p>hello, Victory, why replace “long” in the code by “int64_t”?<br>
int64_t seems not to be a legal datatype. I think “long long” is suitable, which is also 64-bit long.</p>
DataParallel vs DistributedDataParallel

username: Deeply
cooked: <p>Hi,</p>
<p>what is the difference between</p>
<p><code>model = nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])</code></p>
<p>and</p>
<p><code>model = nn.DataParallel(model, device_ids=[args.gpu])</code></p>
<p>?</p>

username: mrshenli
cooked: <p><code>DistributedDataParallel</code> is multi-process parallelism, where those processes can live on different machines. So, for <code>model = nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])</code>, this creates one DDP instance on one process, there could be other DDP instances from other processes in the same group working together with this DDP instance.  Check out this <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/ddp.html</a></p>
<p><code>DataParallel</code> is single-process multi-thread parallelism. It’s basically a wrapper of scatter + paralllel_apply + gather. For <code>model = nn.DataParallel(model, device_ids=[args.gpu])</code>, since it only works on a single device, it’s the same as just using the original model on GPU with id <code>args.gpu</code>. See <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L153" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L153</a></p>
<p><code>DataParallel</code> is easier to use, as you don’t need additional code to setup process groups, and a one-line change should be sufficient to enable it.</p>
<p><code>DistributedDataParallel</code> is faster and scalable. If you have multiple GPUs or machines and care about training speed, <code>DistributedDataParallel</code> should be the way to go.</p>

username: Deeply
cooked: <p>But <code>DataParallel</code> also enable multiple GPUs, in one node/machine, right?</p>

username: mrshenli
cooked: <p>Yes, but <code>DataParallel</code> cannot scale beyond one machine. It is slower than <code>DistributedDataParallel</code> even in a single machine with multiple GPUs due to GIL contention across multiple threads and the extra overhead introduced by scatter and gather and per-iteration model replication.</p>

username: Denuwan_Weerarathne
cooked: <p>Could you please elaborate on this “scatter and gather and per-iteration” model replication process of DP over DDP?</p>
Performance degradation on Android after quantization

username: 165749
cooked: <p>Hi everyone,</p>
<p>I am following the official mobile performance recipe to deploy a bunch of models on Android devices with PyTorch v2.0.0, including the steps:</p>
<ol>
<li>Fuse conv and bn operations</li>
<li>Conduct post training static quantization</li>
<li>Apply mobile_optimizer on the scripted model</li>
</ol>
<p>During the experiments, I observed performance degradation for some models after quantization. For example, the latency measurement of ResNet-18 on Pixel 5 (limited to one thread on the 2.4GHz core) is 132ms without quantization while 213ms with quantization.</p>
<p>One observation from the trace is that the convolutions in fp32 always select the xnnpack backend (i.e., conv2d_clamp_run), while the quantized models utilize the qnnpack backend (e.g., <a href="https://github.com/pytorch/pytorch/blob/a29b9101fa68f0043f2b142cbffd649435a196a2/aten/src/ATen/native/quantized/cpu/qnnpack/src/conv-run.cc#L332" rel="noopener nofollow ugc">qnnpackConv</a>) - I guess this can be one explanation of the distinct performance (due to different backend implementations), but I am surprised that the quantized models can be slower.</p>
<p>Also, I noticed that the quantized convolution (qconv) would actually check the applicability of xnnpack (i.e., <a href="https://github.com/pytorch/pytorch/blob/a29b9101fa68f0043f2b142cbffd649435a196a2/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1014" rel="noopener nofollow ugc">can_use_xnnp</a>), but I found the type of qconv is <strong>kQUInt8</strong>, which fails in the following condition:</p>
<pre><code class="lang-auto">bool supported_dtypes = dtype == c10::kQInt8;
</code></pre>
<p>I am curious it is possible to make use of <strong>xnnpack</strong> for quantized model to see if we can achieve any potential performance improvement?</p>
<p>I would really appreciate it if someone could share some insights.</p>

username: jerryzh168
cooked: <p>can you add a <code>mobile</code> tag for this?</p>

username: 165749
cooked: <p>Thanks for the suggestions. Unfortunately I joined the forum recently and haven’t got enough permission (of trust level) to edit my post. Would you recommend to create another post with <em>mobile</em> tag?</p>

username: jerryzh168
cooked: <p>yeah that sounds good</p>

username: ptrblck
cooked: <p>I’ll move it for you to <code>mobile</code>.</p>
1 GPU with nn.DataParallel vs 1 GPU with simple net

username: gracious-patience
cooked: <p>Hi there! I have net for regression which I would like to train on multiple GPUs. The thing is that the Spearman correlation which I get on 1 GPU simple net is different from the results I obtained with torch.nn.DataParallel(<a href="http://self.net" rel="noopener nofollow ugc">self.net</a>, device_ids=[0]) with the same batch size= b and lr=lr. Moreover, when I take batch size k*b on a simple net and scale lr linearly, I got the same correlation, so the linear scale law holds; but it doesn’t take place in the torch.nn.DataParallel(<a href="http://self.net" rel="noopener nofollow ugc">self.net</a>, device_ids=[0]) setup. Here’s the figures:<br>
setup | batch size | lr | correlation<br>
simple | 53 | 2e-5 | 0.9190<br>
dataparallel | 53 | 2e-5 | 0.9064<br>
simple | 206 | 7.7735e-05 | 0.9151<br>
dataparallel | 206 | 7.7735e-05 | 0.8690</p>
<p>It is logically to expect the same behaviour for both setups. What can be the possible problem?</p>
C++ simple neural network code loss not changing

username: Ramansh_Sharma
cooked: <p>Hi, I am trying to use the C++ frontend to train two neural network together with one optimizer. But my loss function value is not changing throughout the training process. Can someone please help me by taking a look at the following code? I suspect the problem is the way I am giving the parameters to the optimizer. I have defined a NN class with ModuleList, that given a certain list of layer sizes creates a vanilla neural network with tanh activation (making sure to call <code>register_module</code> on each linear module), and setting <code>requires_grad</code> to be true, and given dtype and device for all parameters. My training function has the following contents.</p>
<pre><code class="lang-cpp">    std::vector&lt;torch::Tensor&gt; param_list, net1_params, net2_params;
    net1_params = network1-&gt;parameters();
    param_list = net1_params;
    net2_params = network2-&gt;parameters();
    param_list.insert(param_list.end(), net2_params.begin(), net2_params.end());
    param_list.push_back(bias_param);

    torch::optim::AdamOptions options(initial_lr);
    torch::optim::Adam optimizer(param_list, options);

    auto dummy_y = torch::tensor({4.});
    auto x = torch::linspace(1, 100, 500).reshape({1, -1});
    auto y = torch::tensor({{3.}}).reshape({-1, 1});

    for (int epoch = 0; epoch &lt; epochs; ++epoch) {
        optimizer.step([&amp;]() {
            optimizer.zero_grad();

            auto net1_output = network1-&gt;forward(x); // outputs a (1,50) vector
            auto net2_output = network2-&gt;forward(y); // outputs a (1,50) vector

            auto total_loss = torch::mse_loss(torch::matmul(net1_output, torch::transpose(net2_output, 1, 0)), dummy_y);

            total_loss.backward();

            if (epoch % 100 == 0) {
                std::cout &lt;&lt; "Loss for epoch " &lt;&lt; epoch &lt;&lt; ": " &lt;&lt; total_loss.item() &lt;&lt; "\n";
            }

            return total_loss;
        });
</code></pre>
<p>The loss output I see with this is</p>
<pre><code class="lang-auto">Loss for epoch 0: 19.8229
Loss for epoch 100: 19.8229
Loss for epoch 200: 19.8229
Loss for epoch 300: 19.8229
Loss for epoch 400: 19.8229
Loss for epoch 500: 19.8229
Loss for epoch 600: 19.8229
Loss for epoch 700: 19.8229
Loss for epoch 800: 19.8229
Loss for epoch 900: 19.8229
</code></pre>
<p>Could someone please point me to how I can properly give an optimizer parameters from multiple neural networks in C++ PyTorch?</p>

username: Ramansh_Sharma
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I apologize for once again tagging you promptly in my question post. I am having trouble with C++ PyTorch providing parameters from multiple neural networks to an optimizer. Do you know what might be going wrong here?</p>
<p>Thank you.</p>

username: Ramansh_Sharma
cooked: <p>Turns out the learning rate I passed in was 0. The code works fine otherwise.</p>
Expected sparse functionality

username: alexj
cooked: <p>Hi Everyone,</p>
<p>I really love sparse tensors, however there seem not much one can do with them, a lot of functions don’t work even such simple as Softmax. Will it change soon? Is there more sparse functionality in the worls?</p>

username: SuzanneIllac
cooked: <p>Sparse tensors have immense potential, but their functionality can be limited in certain operations like Softmax due to computational challenges. Progress in the field of sparse tensor processing is ongoing, and we can expect more advancements and expanded functionality in the future<a href="https://raiderlinkttu.us/" rel="noopener nofollow ugc">.</a></p>
F.batch_norm give nan value

username: Gears_Gears
cooked: <p>Hi! Could someone please help me with the following code? I can’t figure out why I’m getting nan values in my batch normalization (I found it with autograd anomaly detection)</p>
<pre><code class="lang-auto">def ConvolutionBlock_F(x : torch.Tensor, 
  weight : torch.Tensor, bias : torch.Tensor) -&gt; torch.Tensor:
  # input dimension: (number of images, channels, height, width)
  x = F.conv2d(x, weight, bias, stride=1, padding='same')
  x = F.batch_norm(x, None, None, training=True) 
  x = F.relu(x)
  return x
</code></pre>
<p>The error message is:<br>
Function ‘NativeBatchNormBackwardBackward0’ returned nan values in its 0th output.</p>
How does changing the value of tensor[:, mask] affect the original tensor?

username: WAN_NIE
cooked: <p>Hi all. It seems that <code>neg_edge[:, mask]</code> will create a new storage with a dynamic address, and single value indexing (<code>neg_edge[0,0]</code>) will not.<br>
I am wondering how does changing the values of <code>neg_edge[:, mask]</code> affect the corresponding values in the original <code>neg_edge.</code> Any help would be appreciated.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f46bf0eb0605e42a02586775054bae93d1a07b45.png" data-download-href="https://discuss.pytorch.org/uploads/default/f46bf0eb0605e42a02586775054bae93d1a07b45" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f46bf0eb0605e42a02586775054bae93d1a07b45_2_408x500.png" alt="image" data-base62-sha1="ySfLsA73Or5qZbNu4Kuyhqh3aBL" width="408" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f46bf0eb0605e42a02586775054bae93d1a07b45_2_408x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f46bf0eb0605e42a02586775054bae93d1a07b45_2_612x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f46bf0eb0605e42a02586775054bae93d1a07b45.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">728×890 40.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to quantize my model? Using quant dequant?

username: Hamza_Khalil
cooked: <p>class ISNetDIS(nn.Module):</p>
<pre><code>def __init__(self,in_ch=3,out_ch=1):
    super(ISNetDIS,self).__init__()

    self.conv_in = nn.Conv2d(in_ch,64,3,stride=2,padding=1)
    self.pool_in = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage1 = RSU7(64,32,64)
    self.pool12 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage2 = RSU6(64,32,128)
    self.pool23 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage3 = RSU5(128,64,256)
    self.pool34 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage4 = RSU4(256,128,512)
    self.pool45 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage5 = RSU4F(512,256,512)
    self.pool56 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

    self.stage6 = RSU4F(512,256,512)

    # decoder
    self.stage5d = RSU4F(1024,256,512)
    self.stage4d = RSU4(1024,128,256)
    self.stage3d = RSU5(512,64,128)
    self.stage2d = RSU6(256,32,64)
    self.stage1d = RSU7(128,16,64)

    self.side1 = nn.Conv2d(64,out_ch,3,padding=1)
    self.side2 = nn.Conv2d(64,out_ch,3,padding=1)
    self.side3 = nn.Conv2d(128,out_ch,3,padding=1)
    self.side4 = nn.Conv2d(256,out_ch,3,padding=1)
    self.side5 = nn.Conv2d(512,out_ch,3,padding=1)
    self.side6 = nn.Conv2d(512,out_ch,3,padding=1)

    # self.outconv = nn.Conv2d(6*out_ch,out_ch,1)

def compute_loss_kl(self, preds, targets, dfs, fs, mode='MSE'):

    # return muti_loss_fusion(preds,targets)
    return muti_loss_fusion_kl(preds, targets, dfs, fs, mode=mode)

def compute_loss(self, preds, targets):

    # return muti_loss_fusion(preds,targets)
    return muti_loss_fusion(preds, targets)

def forward(self,x):

    hx = x

    hxin = self.conv_in(hx)
    #hx = self.pool_in(hxin)

    #stage 1
    hx1 = self.stage1(hxin)
    hx = self.pool12(hx1)

    #stage 2
    hx2 = self.stage2(hx)
    hx = self.pool23(hx2)

    #stage 3
    hx3 = self.stage3(hx)
    hx = self.pool34(hx3)

    #stage 4
    hx4 = self.stage4(hx)
    hx = self.pool45(hx4)

    #stage 5
    hx5 = self.stage5(hx)
    hx = self.pool56(hx5)

    #stage 6
    hx6 = self.stage6(hx)
    hx6up = _upsample_like(hx6,hx5)

    #-------------------- decoder --------------------
    hx5d = self.stage5d(torch.cat((hx6up,hx5),1))
    hx5dup = _upsample_like(hx5d,hx4)

    hx4d = self.stage4d(torch.cat((hx5dup,hx4),1))
    hx4dup = _upsample_like(hx4d,hx3)

    hx3d = self.stage3d(torch.cat((hx4dup,hx3),1))
    hx3dup = _upsample_like(hx3d,hx2)

    hx2d = self.stage2d(torch.cat((hx3dup,hx2),1))
    hx2dup = _upsample_like(hx2d,hx1)

    hx1d = self.stage1d(torch.cat((hx2dup,hx1),1))


    #side output
    d1 = self.side1(hx1d)
    d1 = _upsample_like(d1,x)

    d2 = self.side2(hx2d)
    d2 = _upsample_like(d2,x)

    d3 = self.side3(hx3d)
    d3 = _upsample_like(d3,x)

    d4 = self.side4(hx4d)
    d4 = _upsample_like(d4,x)

    d5 = self.side5(hx5d)
    d5 = _upsample_like(d5,x)

    d6 = self.side6(hx6)
    d6 = _upsample_like(d6,x)

    # d0 = self.outconv(torch.cat((d1,d2,d3,d4,d5,d6),1))

    return F.sigmoid(d1)
</code></pre>
Torch elastic scaling down can't work

username: louis101
cooked: <p>I start the training using the following command:</p>
<pre><code class="lang-auto">torchrun \
    --nnodes=1:3\
    --nproc_per_node=2\
    --max_restarts=3\
    --rdzv_id=1\
    --rdzv_backend=c10d\
    --rdzv_endpoint="$endpoint_ip:1234"\
    train_elastic.py
</code></pre>
<p>I can start one node first and then start another, and it can scale up normally.<br>
When I end the worker node process with ctrl c, the node where the master is located will get stuck, instead of a new rendezvous.</p>
<p><strong>I want to know what happens when the elastic agent is killed, and how to make other agents aware and initiate a new round of rendezvous.</strong></p>
<p>(btw, I read the Kafka streaming data, but I guess the problem is not here.)</p>
CPU usage in GPU based PyTorch code

username: Tomojit
cooked: <p>Hi - I have been using PyTorch for a long time. I use Torch’s dataLoader() functionality to mini batch my dataset. After that I loop through each mini batch of training and test sample to put them in GPU and call my model on each mini batch to run forward and backward pass as shown below:<br>
for i, (trInput, trOutput) in enumerate(dataLoader):<br>
# Move tensors to the configured device<br>
trInput = trInput.to(self.device)<br>
trOutput = trOutput.to(self.device)</p>
<p>I was under impression that dataLoader is taking the whole dataset in the GPU and then split the data into mini batches, so that there won’t be any transfer of data between CPU and GPU. But it looks like this is not the case. When I run my GPU code I can see 100% CPU utilization which indicates lot of I/O operations.</p>
<p>Having said that, how should I change my code to:</p>
<ol>
<li>first take the whole training set in GPU memory</li>
<li>Minibatch data in GPU</li>
<li>Train model</li>
</ol>
<p>I’m looking to avoid any data transfer from CPU to GPU during the training process. Should I use different type of data loader? Any help will be appreciated.</p>
<p>Thank you,<br>
Tomojit</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Tomojit" data-post="1" data-topic="188677">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tomojit/48/29601_2.png" class="avatar"> Tomojit:</div>
<blockquote>
<p>I was under impression that dataLoader is taking the whole dataset in the GPU and then split the data into mini batches, so that there won’t be any transfer of data between CPU and GPU.</p>
</blockquote>
</aside>
<p>In your <code>DataLoader</code> loop you are explicitly moving the tensors to <code>self.device</code> so I assume you are sticking to the standard workflow of loading the data to the host in order to avoid wasting GPU memory for the dataset (if it can even fit onto the GPU).</p>
<p>You can load the entire dataset in the <code>Dataset.__init__</code> onto the GPU and then just index each sample in the <code>__getitem__</code>.</p>
Single Random Operation different across DDP

username: quornmd
cooked: <p>I am using DDP and working with stoachstic models. I wish to add noise as part of my forward pass. I have each process seeded properly, as I generally wish the randomness to be the same. However, for <em>one, single</em> operation, I wish each process would result in a <em>different</em> random outcome. I imagine something like this:</p>
<pre><code class="lang-python">seed_everything(0)
a = torch.randn((1024,10), global_rank) # should return the exact same number across all GPUs
b = torch.randn((1024, 10), global_rank, decouple_ddp_randomness=True) # should return different numbers per GPU
c = torch.randn((1024,10), global_rank) # should return the exact same number across all GPUs
</code></pre>
<p>I dont want <code>b</code> to be the same across GPUs in DDP, because I am scared of it biasing my results.</p>
<p>Is that possible?</p>

username: yiftach
cooked: <p>What I would do in this case is have all of the GPUs generate an int tensor of length world_size (the number of GPUs) and then call <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.scatter" rel="noopener nofollow ugc">dist.scatter()</a> to send the one that rank 0 (the main GPU) generated to the other ones, each one getting one int. And then each one can initialize a <a href="https://pytorch.org/docs/stable/generated/torch.Generator.html" rel="noopener nofollow ugc">Generator</a> and use the int it received as this Generator’s seed (<code>generator.set_manual_seed()</code>), and pass the generator to the random operation if it can be expressed as one of <a href="https://pytorch.org/docs/stable/torch.html#inplace-random-sampling" rel="noopener nofollow ugc">these</a> that get a Generator argument. If it’s not one of these (e.g. you prefer to stick to <code>torch.randn()</code>) then see the Generator link for how to save a seed (the original one), switch to a new one (for one time for your operation) and then restore the original one.</p>
Shape inference in FX Graphs

username: trusira
cooked: <p>Hi, I’m studying the torch dynamo graph export workflow and realized FX graphs have symbolic function arguments. For instance,</p>
<pre><code class="lang-auto">class MatMulNet(nn.Module):
    def __init__(self):
        super().__init__()

    @torch.no_grad()
    def forward(self, x, y):
        return torch.matmul(x, y)

model = MatMulNet()

torch._dynamo.export(model, aten_graph=True)(
    torch.rand(5,5), torch.rand(5,5))
</code></pre>
<p>produces the following Aten graph.</p>
<pre><code class="lang-auto">GraphModule()
def forward(self, x, y):
    arg0, arg1, = fx_pytree.tree_flatten_spec(([x, y], {}), self._in_spec)
    mm_default = torch.ops.aten.mm.default(arg0, arg1);  arg0 = arg1 = None
    return pytree.tree_unflatten([mm_default], self._out_spec)
</code></pre>
<p>As you can see, even though I’m passing initialized arguments to the export function, the traced FX graph has symbolic args x, y.<br>
I’m curious as to where shape inference happens and placeholder nodes get their specific shapes.</p>
<p>When I generate debug traces with <code>TORCH_COMPILE_DEBUG=1</code>, I see concrete type information in <code>fx_graph_readable.py</code></p>
<pre><code class="lang-auto">class &lt;lambda&gt;(torch.nn.Module):
    def forward(self, arg0_1: f32[5, 5], arg1_1: f32[5, 5]):
        # File: /home/pytorch-cuda/my_scripts/mytorch.py:35, code: return torch.matmul(x, y)
        mm: f32[5, 5] = torch.ops.aten.mm.default(arg0_1, arg1_1);  arg0_1 = arg1_1 = None
        return (mm,)
</code></pre>
<p><strong>Update:</strong> I see that <code>meta</code> attribute carries this information for a <code>torch.fx.Node</code> object using a <code>FakeTensor</code>.</p>
Help with the grouped convolutions using the local binary neural networks

username: Foram
cooked: <p>I am trying implement the effecientetv2_s model by substituting the 3x3 convolution blocks with local binary convolution blocks. However, I am getting confused on how to implement the depth wise separable scheme with them.</p>
<p>A local binary convolutional block (LBCB): The weights of the 3x3 convolutions are replaced with random designation of ‘0’,‘1’ and ‘-1’. The maps obtained from there are called, difference maps, which are then passed through Relu to generate the binary maps, and finally, use learnable 1x1 convolutions to generate feature maps.</p>
<p>Should I just perform the group convolution while computing the difference maps and let the 1x1 be the pointwise computation? or perform the whole LBCB in place of depth-wise convolution and again use 1x1 point wise convolution?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1aef5c33c9941da5fbba79161c5eeaf42377bf7.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1aef5c33c9941da5fbba79161c5eeaf42377bf7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1aef5c33c9941da5fbba79161c5eeaf42377bf7_2_690x268.png" alt="image" data-base62-sha1="wcuhpWUpez8POyMBiHXIz5QNEer" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1aef5c33c9941da5fbba79161c5eeaf42377bf7_2_690x268.png, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1aef5c33c9941da5fbba79161c5eeaf42377bf7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1aef5c33c9941da5fbba79161c5eeaf42377bf7.png 2x" data-dominant-color="D5DFC7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">874×340 78.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For reference, I am attaching the link of the local binary neural network: <a href="https://openaccess.thecvf.com/content_cvpr_2017/papers/Juefei-Xu_Local_Binary_Convolutional_CVPR_2017_paper.pdf" rel="noopener nofollow ugc">https://openaccess.thecvf.com/content_cvpr_2017/papers/Juefei-Xu_Local_Binary_Convolutional_CVPR_2017_paper.pdf</a></p>
Is autocast expected to reflect changes to weights?

username: KellerJordan
cooked: <p>I’m wondering if the following is expected behavior:</p>
<pre><code class="lang-auto">import torch
torch.set_grad_enabled(False)

model = torch.nn.Linear(5, 1, bias=False).cuda()
model.weight.data[:] = 1

inputs = torch.ones(2, 5).cuda()

with torch.autocast(device_type='cuda'):
    out = model(inputs)
    print(out.sum().cpu()) # prints tensor(10.)
    model.weight.data[:] = 0
    out = model(inputs)
    print(out.sum().cpu()) # prints tensor(10.), but this should be tensor(0.)
</code></pre>
<p>It seems that changes to weights which occur in the autocast context aren’t reflected in outputs.</p>
<p>I ran into this issue because I had code like this</p>
<pre><code class="lang-auto">with torch.no_grad(), autocast():
    results = []
    for ckpt in ckpts_list:
        model.load_state_dict(ckpt)
        results.append(model(images))
</code></pre>
<p>which was giving nonsense results – I believe because my model’s BatchNorm weights were being updated with the new checkpoints in the autocast cache, but the conv/linear layers were not.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="KellerJordan" data-post="1" data-topic="188651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e480ec/48.png" class="avatar"> KellerJordan:</div>
<blockquote>
<p>It seems that changes to weights which occur in the autocast context aren’t reflected in outputs.</p>
</blockquote>
</aside>
<p>That’s expected since <code>autocast</code> caches the parameters to avoid casting them repeatedly inside its context. If you want to manipulate internal data you might want to disable caching via <code>cache_enabled=False</code>.</p>
Customize state_dict() and load_state_dict() pytorch

username: NagabhushanSN95
cooked: <p>I have a nested set of classes (each of type <code>torch.nn.module</code>). I need to do some preprocessing before saving the weights of one of the nested classes. Is it possible to override the <code>state_dict()</code> function so that I can insert the preprocessing in my custom implementation?</p>
<p>Sample code:</p>
<pre><code>Class A(torch.nn.module):
    def __init__(self):
        super().__init__()
        self.b1 = B1()
        self.b2 = B2()

Class B1(torch.nn.module):
    def __init__(self):
        super().__init__()
        self.var = torch.nn.Parameter(torch.Tensor((3, 5), dtype=float))

Class B2(torch.nn.module):
    def __init__(self):
        super().__init__()
        self.var = torch.nn.Parameter(torch.Tensor((3, 5), dtype=float))

    def state_dict():
        # override the default state_dict
        bool_var = self.var.bool().cpu().numpy()
        state_dict1 = super.state_dict()
        state_dict1.update({'var': bool_var})
        return state_dict1

    def load_state_dict(state_dict):
        state_dict['var'] = state_dict['var'].float()
        super.load_state_dict(state_dict)
        return
</code></pre>
<p>Specifically, for one of the classes, I want to convert the variable to <code>bool</code> before saving it and convert it back to <code>float</code> while loading it. I can’t make the tensor as <code>bool</code> by default since it needs to be learned as a float value.</p>
<p>The code I am working with is <a href="https://github.com/apchenstu/TensoRF/blob/main/models/tensorBase.py#L248--L263" rel="noopener nofollow ugc">this</a>. Here, they’ve hard-coded saving of such variable. I don’t want to do that. I want <code>state_dict()</code> and <code>load_state_dict()</code> to automatically take care of the conversion.</p>
<p>PS: While <a href="https://discuss.pytorch.org/t/how-to-customize-the-module-state-dict-load-state-dict/139604/4">this discussion</a> indicates that customizing is not possible, I am hoping PyTorch would have added it in the recent versions or there are some other ways of achieving this.</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/75287">this feature request</a> the hooks, mentioned already in the linked post, are still internal and can break at any point. You could of course use these but might need to adapt your code since these internal APIs do not have any guarantee to be stable.</p>
How edge_index have to be built in order to fed the Dataset class for GCNcon net?

username: Filippo_Portera
cooked: <p>Hello!<br>
I’ve implemented a Neural Net that uses 2 GCNConv layers, besides other 3 Linear layers.<br>
The problem is that I don’t understand how the Dataset object should be fed with edge_index’s for several graphs, since I’m considering a graphs binary classification task.<br>
Since the graphs have different numbers of edges, the edge_index cannot be an array of fixed dimensions. So I don’t understand how to build the batch diagonal matrix described in the official PyG docs: <a href="https://pytorch-geometric.readthedocs.io/en/latest/advanced/batching.html" rel="noopener nofollow ugc">Batching</a> .<br>
Is it possible to have a numerical example?<br>
Thanks for any help!</p>
Confusion about memory allocation mechanism

username: aonrdbit
cooked: <p>I saw a rule about the memory allocation mechanism from line 66 of  <a href="https://github.com/pytorch/pytorch/blob/main/c10/cuda/CUDACachingAllocator.cpp" rel="noopener nofollow ugc">this file</a>: <strong>To further reduce fragmentation, blocks &gt;= 200MB are not allowed to be split. These oversize cached blocks will still satisfy requests within 20MB of the oversize cached block size.</strong></p>
<p>My understanding of this is that blocks larger than or equal to 200MB will not be split. So, I did a test.</p>
<pre><code class="lang-auto">import torch
import torch.cuda
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
temp1 = torch.tensor([1.0]*1024*1024*100).to(device)
del temp1
temp2 = torch.tensor([1.0]*1024*1024*6).to(device)
print(torch.cuda.memory_reserved()/1024/1024)
</code></pre>
<p>The output of this program is 400. It looks like temp2 split 400MB of temp1’s memory.</p>
<p>I’m very confused about this. Can anyone give me some advice? Thanks!</p>

username: yiftach
cooked: <p>I’m very possibly wrong here, but is temp1 not 400MB itself (100M * 4 bytes of 32float), and then you delete it (but pytorch still keeps that memory reserved, so you’ll still see 400MB), and then since it’s free the next smaller allocation (6MB*4=24MB) uses up that some of that freed space, and the rest remains reserved?</p>

username: aonrdbit
cooked: <p>Can blocks larger than or equal to 200MB be split?</p>
Regression heads in Centernet network models

username: YYL-SCAR
cooked: <p>I’m working on the centernet code and found a query. In the code below, why is the code for predicting the width and height the same as the code for predicting the offset of center point? Does the difference between the two only appear when doing the loss function?</p>
<p>class resnet50_Head(nn.Module):<br>
def <strong>init</strong>(self, num_classes=80, channel=64, bn_momentum=0.1):<br>
super(resnet50_Head, self).<strong>init</strong>()<br>
<span class="hashtag-raw">#-----------------------------------------------------------------</span>#<br>
#<br>
#   128, 128, 64 → 128, 128, 64 → 128, 128, num_classes<br>
#                → 128, 128, 64 → 128, 128, 2<br>
#                → 128, 128, 64 → 128, 128, 2<br>
<span class="hashtag-raw">#-----------------------------------------------------------------</span>#<br>
# predict heatmap<br>
self.cls_head = nn.Sequential(<br>
nn.Conv2d(64, channel,<br>
kernel_size=3, padding=1, bias=False),<br>
nn.BatchNorm2d(64, momentum=bn_momentum),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(channel, num_classes,<br>
kernel_size=1, stride=1, padding=0))<br>
# predict  w，h<br>
self.wh_head = nn.Sequential(<br>
nn.Conv2d(64, channel,<br>
kernel_size=3, padding=1, bias=False),<br>
nn.BatchNorm2d(64, momentum=bn_momentum),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(channel, 2,<br>
kernel_size=1, stride=1, padding=0))</p>
<pre><code>    # predict offset of center point
    self.reg_head = nn.Sequential(
        nn.Conv2d(64, channel,
                  kernel_size=3, padding=1, bias=False),
        nn.BatchNorm2d(64, momentum=bn_momentum),
        nn.ReLU(inplace=True),
        nn.Conv2d(channel, 2,
                  kernel_size=1, stride=1, padding=0))
  
def forward(self, x):
    hm = self.cls_head(x).sigmoid_()
    wh = self.wh_head(x)
    offset = self.reg_head(x)
    return hm, wh, offset
</code></pre>
<p>So all I have to do to predict anything is to change the number of output channels (like 2 in the code below) in the following code?<br>
nn.Conv2d(channel, 2.<br>
kernel_size=1, stride=1, padding=0))</p>
<p>For example, if I want to predict the positions of the four vertices, I change the code to:</p>
<p>self.bbox_head = nn.Sequential(<br>
nn.Conv2d(64, channel.<br>
kernel_size=3, padding=1, bias=False),<br>
nn.BatchNorm2d(64, momentum=bn_momentum),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(channel, 8，<br>
kernel_size=1, stride=1, padding=0))</p>
Extract the values of a tensor along an arbitrary line

username: Georges_Leukic
cooked: <p>Hello,</p>
<p>I’m looking for a pytorch tool for interpolating tensor along a given line (batch of lines)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81409839fe9bd1a6521d880d02095bc113fc3f02.png" data-download-href="https://discuss.pytorch.org/uploads/default/81409839fe9bd1a6521d880d02095bc113fc3f02" title="T9MgR"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81409839fe9bd1a6521d880d02095bc113fc3f02_2_666x500.png" alt="T9MgR" data-base62-sha1="irpZceQg61J6O1gg2KkKRtDpXV0" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81409839fe9bd1a6521d880d02095bc113fc3f02_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81409839fe9bd1a6521d880d02095bc113fc3f02.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81409839fe9bd1a6521d880d02095bc113fc3f02.png 2x" data-dominant-color="ECF4ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">T9MgR</span><span class="informations">800×600 87.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.</p>
<p>I suppose grid_sample could do the work.</p>
RFC-0019 Extending PyTorch Quantization to Custom Backends

username: April_Hu
cooked: <p>Hi, just wondering, has <a href="https://github.com/pytorch/rfcs/blob/master/RFC-0019-Extending-PyTorch-Quantization-to-Custom-Backends.md" rel="noopener nofollow ugc">RFC-0019</a> been accepted? What described in the documentation seems really helpful for implementing a custom backend for PyTorch Quantization. So kindly wanted to know if there is an issue/a page that tracks the progress of this RFC.<br>
<a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

username: jcaip
cooked: <p>Hi <a class="mention" href="/u/april_hu">@April_Hu</a> I believe that this work is currently in progress, but I’m not aware of a tracker. Jerry might have more information.</p>

username: jerryzh168
cooked: <p>yeah this is accepted and implemented in the fx graph mode quantization flow (convert_to_reference_fx), but we have a new flow coming up, so if you want start integration it might be better to go with the new flow (that also implements this design). Here are some docs for the new flow:</p>
<ul>
<li><a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></li>
<li><a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2.0 Export Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></li>
</ul>
No module named  'torch.C'

username: nickums
cooked: <p>On windows10  Python38 and torch-2.0.0, suddenly all my working torch-related programs have started failing thus:<br>
import torch<br>
File “C:\Python\Python38\lib\site-packages\torch_<em>init</em>_.py”, line 229, in <br>
from torch._C import *  # noqa: F403<br>
ModuleNotFoundError: No module named ‘torch._C’</p>
<p>What changed in my system? maybe a transformer install, i;m not sure,</p>
<p>\site-packages\torch\      contains a folder _c   (lower[case c),  not ._C<br>
so I tried altering ._C to -_c in <strong>init</strong>.py and got this behaviour instead:<br>
from torch._c import *  # noqa: F403<br>
ImportError: dynamic module does not define module export function (PyInit__c)</p>
<p>Existing posts o this topic do=;t help.  I am baffled!</p>
<p>Guess I’ll have to figure out how to re-install torch with my cuda…</p>

username: ptrblck
cooked: <p>Yes, a 3rd part library might have created conflicts in your environment and reinstalling PyTorch in a new and clean virtual environment sounds like a valid idea.</p>
Many small device transfers vs one large one

username: yiftach
cooked: <p>Hi,<br>
I’ve tested to see if moving one large tensor from cpu to cuda take less time than copying the same amount of data split to chunks, as follows:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from time import time

arr = [torch.randn(1000) for _ in range(1000)]
large = torch.randn(1000 * 1000)

start = time()
large.cuda()
torch.cuda.synchronize()
print(time() - start) # 0.0015168190002441406

start = time()
for x in arr:
    x.cuda()
torch.cuda.synchronize()
print(time() - start) # 0.023027658462524414
</code></pre>
<p>The numbers do vary from run to run but the difference in scales remains.<br>
What motivated my checking was seeing that calling <code>model.cuda()</code> is actually a recursive call to <code>param.cuda()</code>.</p>
<p>My question - would it be more efficient to replace the recursive call with something that first chunks all the data to a single tensor on CPU, moves it once then unpacks it back? I would think it is and the reason it’s not done is that the code is ugly (though it would have to be written once?). My code also didn’t simulate the CPU to CPU copies but I assume that takes less time than the CPU to GPU copy.</p>
<p>Any thoughts?</p>
Use vmap and grad to calculate gradients for one layer independently for each input in batch

username: jdesmarais
cooked: <p>Hi,</p>
<p>I am working on a project where I need to calculate gradients for a particular model layer independently for each input in the batch. I have it working well unvectorized, but when I try to vectorize to speed things up I get the following error: “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”</p>
<p>Has anyone see this issue before?</p>
<p>Some relevant code:</p>
<pre><code class="lang-auto">from torch.autograd import grad

def point_gradients(loss, params=None):
    gradients = grad(loss, params, retain_graph = True,)
    gradients = np.concatenate([g.cpu().numpy().flatten() for g in gradients])
    return(gradients)

if vecorize:
    grad_fn = vmap(point_gradients, in_dims=0)
    batch_gradients = grad_fn(losses, params=layer_params)
else:
    batch_gradients = [point_gradients(loss, params=layer_params) for loss in losses]
</code></pre>
<p>If I set vectorize to false, so point_gradients is called on individual losses everything works as expected<br>
but if I set vectorize to True so the tensor of losses is passed to the vmap-ed function, I get the error:<br>
“RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”</p>
<p>the shape of losses is torch.Size([256, 1]) if that is important.</p>
<p>Thanks for the help!</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/jdesmarais">@jdesmarais</a>,</p>
<p>The reason why you don’t have a <code>.grad</code> attribute is that you’re mixing <code>torch.func</code> with <code>torch.autograd.grad</code>,  which must be done with care.</p>
<p>If you want to vectorize your gradient calculation you’ll need to calculate it entirely within the <code>torch.func</code> namespace, so something like this,</p>
<pre><code class="lang-auto">def calc_loss(params, x): #and other args if need be
  return loss #insert loss func here

from torch.func import vmap
gradients = vmap(torch.func.grad(calc_loss, argnums=(0)), in_dims=(None,0))(params, x)
</code></pre>
<p>then you can concatenate them like you do above</p>

username: jdesmarais
cooked: <p>Is there a way to use torch.func.grad to calculate gradients for a specific model layer instead of the entire model? Thanks!</p>