akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error with convert CUDA tensor to numpy

long_ng: <p>Hi guys,</p>
<p>I am beginner with Pytorch and Object Detection,<br>
I am running the code on Google Colab with example code</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7ce5f28975499e954dcaa2fc5c7371bde4275978.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ce5f28975499e954dcaa2fc5c7371bde4275978" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce5f28975499e954dcaa2fc5c7371bde4275978_2_523x500.png" alt="image" data-base62-sha1="hOTRIoQnoM14cdSTh1NVmDs9haM" width="523" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce5f28975499e954dcaa2fc5c7371bde4275978_2_523x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce5f28975499e954dcaa2fc5c7371bde4275978_2_784x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7ce5f28975499e954dcaa2fc5c7371bde4275978.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">853×815 94.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.</p>
<p>Does anyone have idea about this problem?</p>
<p>Thank you so much</p>

ptrblck: <p>You cannot directly convert a tensor stored on the GPU to a numpy array, which would be stored on the CPU. As the error message describes, transfer the tensor to the CPU first and then convert it to a numpy array:</p>
<pre><code class="lang-python">cpu_arr = gpu_tensor.cpu().numpy()
</code></pre>

long_ng: <p>Thank you for your response,</p>
<p>I have tried</p>
<pre><code class="lang-auto">nms_prediction = apply_nms(prediction, iou_thresh=0.01)
prediction = nms_prediction.gpu_tensor.cpu().numpy()
plot_img_bbox(torch_to_pil(img), nms_prediction)
</code></pre>
<p>However, it is not working because of nms_prediction is ‘dict’ and has on attribute.</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-64-09fd68999e27&gt; in &lt;module&gt;
----&gt; 1 nms_prediction.values.gpu_tensor.cpu().numpy()

AttributeError: 'builtin_function_or_method' object has no attribute 'gpu_tensor'
</code></pre>
<p>Do you have any solution for this?</p>

ptrblck: <p>Sorry for the confusion. <code>gpu_tensor</code> is just a placeholder and example tor any tensor stored on the GPU.<br>
I assume <code>nms_prediction</code> is already a <code>CUDATensor</code>, so something like:</p>
<pre><code class="lang-python">nms_array = nms_prediction.cpu().numpy()
# or 
nms_array = nms_prediction.detach().cpu().numpy()

plot_img_bbox(torch_to_pil(img), nms_array)
</code></pre>
<p>might work.</p>
What happens if I transfer the Python code "p.grad=none" to the c++ code?

abloom5448: <p>Hi</p>
<p>I tried to transfer python to c++.</p>
<p>in python code<br>
p.grad = none</p>
<p>in c++ code<br>
i don’t know what i have to use c++ function<br>
p.grad() = ???;<br>
p.grad().???;</p>
<p>vs2022<br>
libtorch cpu release</p>
<p>please help me</p>

tom: <p>Does <code>p.mutuable_grad().reset()</code> work?</p>
<p>Best regards</p>
<p>Thomas</p>

abloom5448: <p>thanks for reply</p>
<p>“p.mutuable_grad().reset();” Build and run is done.</p>
<p>The verification remains.</p>
<p>I’ll verify it and share the results with you.</p>
<p>thank you.</p>
Torch.nn.functional.grid_sample: does input scale matter?

Abhijay: <p>Will it matter if the input image is scaled between 0…1 or -1…1? Ideally, it shouldn’t.</p>

ptrblck: <p>No, I don’t think the input image value range should matter as <code>grid_sample</code> would just sample and interpolate the input to create the output grid. Are you seeing any issues using these scales?</p>
Creating complex tensor from two real part and imaginary part

minzhao.liu: <p>Hi,</p>
<p>I want to do something in C++ that would look like this in Python:</p>
<pre><code class="lang-python">c = a + 1j*b
</code></pre>
<p>where <code>a</code> and <code>b</code> are real-valued tensors. In C++, I have two <code>torch::Tensor</code>’s of type <code>torch::kFloat32</code> for <code>a</code> and <code>b</code>.</p>

minzhao.liu: <p>Oops, seems like this is it: <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a0b6372c72f78a80f1277d3b5d9a6aa84.html?highlight=complex" class="inline-onebox" rel="noopener nofollow ugc">Function at::complex — PyTorch master documentation</a></p>
2 loss functions combination

philipp0827: <p>Hi i’m trying to train a SSD-model<br>
For that i define 2 loss functions,<br>
One for location loss and another for class loss(cross entropy loss)</p>
<pre><code class="lang-auto">loss1 = SSD_box_loss(labels_for_batch, gt_with_max_overlap, boxPrediction.cuda())
loss2 = SSD_cls_loss(labels_for_batch, positive_predicted_Boxes, negative_predicted_Boxes, labelPrediction.cuda())
          
loss = loss1 * loss2
loss.backward
optimizer.step() 
</code></pre>
<p>and for optimizer im using</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
</code></pre>
<p>but the loss never decrease,<br>
if i try with only one loss, each losses decreased…<br>
how can i deal with it?</p>

KFrank: <p>Hi Philipp!</p>
<aside class="quote no-group quote-modified" data-username="philipp0827" data-post="1" data-topic="160050" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/philipp0827/48/52719_2.png" class="avatar"> philipp0827:</div>
<blockquote>
<pre><code class="lang-auto">loss = loss1 * loss2
</code></pre>
<p>if i try with only one loss, each losses decreased…</p>
</blockquote>
</aside>
<p>The conventional approach would be to <em>sum</em> the two losses:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss1 + loss2
</code></pre>
<p>It’s hard to know exactly what is going on without knowing what<br>
<code>SSD_box_loss()</code> and <code>SSD_cls_loss()</code> are, but try summing the<br>
losses and see if that fixes your problem.</p>
<p>Best.</p>
<p>K. Frank</p>
How to batch multiply a 3D matrix?

arjunsankarlal: <p>I am looking to batch multiply two matrices of dimensions <code>torch.Size([7, 265, 768])</code>.</p>
<p>When I tried with <code>torch.bmm(x,y)</code> I got the error like this,</p>
<p><code>RuntimeError: invalid argument 2: wrong matrix size, batch1: 265x768, batch2: 265x768 at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:2312</code></p>
<p>Any idea what might be causing this?</p>

JuanFMontesinos: <p>I might be because u have to multiply a 256x768 by another matrix 768x N to get a 256xN matrix.</p>

arjunsankarlal: <p>Actually both the matrices have same dimensions only. I checked their sizes by printing them.</p>
<p>torch.Size([7, 265, 768])<br>
torch.Size([7, 265, 768])</p>

JuanFMontesinos: <p>That’s the problem… you cannot multiply those matrices</p>
<p>If you multiply a matrix you need a matrix<br>
A: NxM<br>
B: MxS</p>
<p>then A*B --&gt; NxS</p>

arjunsankarlal: <p>Thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>. I thought it would work like the same way in maths. So even for a 2D matrix of same dimensions, does it has to be squeezed / changed in dimension so that it can form the structure,</p>
<p>A: NxM<br>
B: MxS<br>
then A*B --&gt;; NxS</p>
<p>or I am understanding it wrong.</p>

JuanFMontesinos: <p>I’m sorry but the way I cited is the way it works on math. The only change is that you are adding a 3rd dimension corresponding to the batch.</p>
<pre><code class="lang-auto">import torch

a = torch.rand(7,265,768)

b= torch.rand(7,768,500)

c=torch.bmm(a,b)

c.size()
Out[5]: torch.Size([7, 265, 500])

</code></pre>

Alan_Yu: <p>You need to take the transpose of the second matrix to make the dimensions match.</p>
<pre><code class="lang-auto">import torch

a = torch.rand(7,265,768)
b= torch.rand(7,265,768)
c = torch.matmul(a, b.transpose(-2, -1))
c.size()
torch.Size([7, 265, 265])
</code></pre>
Pickling ScriptModule

vhollav: <p>Hello everyone.</p>
<h3>
<a name="issue-1" class="anchor" href="#issue-1"></a>Issue</h3>
<p>I’ve been wondering whether it’s possible to somehow pickle/unpickle ScriptModule rather than use <code>torch.jit.save</code>/<code>torch.jit.load</code>?</p>
<pre><code class="lang-python">import io

import torch
import torch.jit

m = nn.Linear(1, 10).eval()
torch.save(m, io.BytesIO())  # OK

m_  = torch.jit.script(m)
torch.save(m, io.BytesIO())  # Not OK, compiled module does not have a __getstate__ method defined!
</code></pre>
<p>Is there any workaround to pickle/ monkey-patch to make picklable a scipted module?</p>
<h3>
<a name="pre-history-2" class="anchor" href="#pre-history-2"></a>Pre-history</h3>
<p>Original question arose from the following situation.</p>
<p>I have a scripted model of a 3D classification network weighted 500 MB, and it takes <code>torch.jit.load</code> 0.8 second to complete.</p>
<p>If to remove (substitute with empty <code>torch.Tensor()</code>) all parameters and buffers of the scripted module and save it again — new file would weight 0.2 MB, but <code>torch.jit.load</code> would take 0.45 seconds to complete.</p>
<p>I’m not a sophisticated person in terms of how <code>torch.jit.load</code> works internally, but 0.45 seems rather slow.</p>
<p>And I thought, if to make it picklable I could store in shared memory and load it faster.</p>
<h3>
<a name="system-version-3" class="anchor" href="#system-version-3"></a>System version</h3>
<p>Here’s Python and PyTorch versions.</p>
<pre><code class="lang-auto">sys.version_info(major=3, minor=8, micro=13, releaselevel='final', serial=0)
1.10.1
</code></pre>

tom: <p>I don’t think you can fully avoid torch.jit.load - or at least the things it uses - if you want to get a ScriptModule from it.</p>
<p>Best regards</p>
<p>Thomas</p>

vhollav: <p>That’s a pity. Maybe there are any hacks to somehow pickle <code>ScriptModule</code> once it’s obtained after <code>torch.jit.load</code> you know of?</p>

tom: <p>So my reading of this is:</p>
<ul>
<li>
<code>torch.jit.load</code> needs to do a lot of work to create the <code>ScriptModule</code>, quite likely a lot of that is to get to a usable state, (but I did not benchmark it, so there could be inefficiencies), let’s imagine it has two parts “read from disk” and “create objects and reach state”, I would venture that likely, the second part is where the time is spent,</li>
<li>pickle is a protocol. It will rely on object creation and such to create the objects it is deserializing,</li>
<li>so if there were a way to pickle / unpickle, it would need the second part of the <code>torch.jit.load</code>.</li>
</ul>
<p>This is roughly why I don’t expect there to be a solution faster than torch.jit.load.</p>
<p>Best regards</p>
<p>Thomas</p>

vhollav: <p>Yes, this seems very logical. Thanks!</p>
How can I use attn_output_weights for next attention blocks?

jBloodless: <p><code>nn.MultiheadAttention</code> able to return mysterious thing called <code>attn_output_weights</code>. My problem is to transfer results of MultiHeadAttention block to next iterations to keep attention results through them (frame by frame evaluation of audio, let’s say). How can I use those weights as input for <code>nn.MultiheadAttention</code>? Or I should use something different for this task?</p>
Why does quantized model parameters return an empty list?

kohenchia: <p>I’m trying to inspect the parameters of a quantized model to see how they differ from a regular, non-quantized model. However, the <code>.parameters()</code> call on a quantized model returns an empty list:</p>
<pre><code class="lang-auto">qmodel = torch.quantization.quantize_dynamic(MyCell(), {torch.nn.Linear}, dtype=torch.qint8)

list(qmodel.parameters())
</code></pre>
<p>returns:</p>
<pre><code class="lang-auto">[]
</code></pre>
<p>Is this expected? Is this because they don’t expect the weights to be trained anymore?</p>
<p>How can I see what the weight / bias values are?</p>

jerryzh168: <p>yeah the weights are not trainable anymore, you can print the qmodel and look at the structure of the model and print the weight/bias for linear layers I think</p>
Heatmap loss function?

MohamedAliRashad: <p>So i used the MobileNetV3 architecture to train an object detector with heat maps instead of bounding boxes.<br>
I didn’t change anything in the MobileNetV3 arch except raising the number of output neurons to 2704 so it can be resized to 52x52 image with a sigmoid activation function , this should enable me to subtract the heat maps from the output with MSE loss function and get a good results from the model.</p>
<p>The problem that this doesn’t happen and after some research i found out that i need to give weights for the background pixels less than the foreground ones and my question how i can do something like this ?</p>
<p>And if someone tried training with heatmaps, is my configuration good or there is something better ?</p>

mailcorahul: <p>I haven’t studied the architecture of MobileNetv3, but to generate heatmaps(or in general activation maps as such), a fully convolutional network should suffice where there is a feature extraction stem followed by a 1x1 conv to bring down the number of channels to 1 keeping the spatial dims as it is.</p>
<p>Coming to the problem of class imbalance, you can counter it using <code>Balanced/Weighted Cross Entropy</code> or <code>Dice loss</code>. These loss functions are generally used to tackle class imbalance in detection/segmentation tasks.</p>

MohamedAliRashad: <p>yeah, i made it this way but through a Linear layer and resizing it.</p>
<p>i thought about cross entropy but the <a href="https://datascience.stackexchange.com/questions/51048/mse-vs-cross-entropy-for-training-with-facial-landmark-pose-heatmaps" rel="nofollow noopener">link</a> here didn’t advice it.</p>

mailcorahul: <p>Are you sure that the link says <code>Cross Entropy</code> is not a good choice? Nevertheless,  <code>Balanced Cross Entropy/Dice loss</code>  has shown good results for problems like these and it is always better to try and see before arriving at a conclusion.<br>
Check out EAST paper: <a href="https://arxiv.org/abs/1704.03155" rel="nofollow noopener">https://arxiv.org/abs/1704.03155</a></p>

MohamedAliRashad: <p>Got this error, has any clues for a solution ?</p>
<p><code> Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss2d_forward</code></p>

mailcorahul: <p>Passing ground truth tensor of type <code>float</code> to loss function?</p>

MohamedAliRashad: <p>i solved it, it seems Cross Entropy only accepts <code>Long</code> as data type</p>

test_server: <p><a class="mention" href="/u/mohamedalirashad">@MohamedAliRashad</a> I want to do same. Can you share your code?  It will help a lot</p>
Low GPU Utilization on RTX3090Ti

Muhammad_Umar1: <p>Hi, Thanks for amazing work. I am using RTX3090Ti GPU, and I am training on 1M images. My system GPU utilization is fluctuating and it’s not very fast as it should be. We trained with a small dataset with 8k images, the GPU utilization was stable, however, it is not stable when trained with 1M images. It is constantly fluctuating between 0 - 98% and slow. We varied the batch size and number of workers. My system has the following specs:<br>
GPU: RTX3090TI<br>
CUDA: 11.6<br>
pytorch: 1.12.1 stable<br>
YOLOv5: yolov5m<br>
nproc: 16</p>
<p>could you please suggest some promising directions to fix this issue.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/muhammad_umar1">@Muhammad_Umar1</a>,</p>
<p>Make sure to share a minimal reproducible script so people can debug your error! (As opposed to describing your script)</p>

Muhammad_Umar1: <p>I am using YOLOV5, the code is available in the yolov5 repository.  <a href="https://github.com/ultralytics/yolov5/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">yolov5/train.py at master · ultralytics/yolov5 · GitHub</a></p>

Lelouch1: <p>“it is constantly fluctuating between 0 - 98% and slow” maybe you are reading data too slowly and thus your GPU waiting for new data most of the time ?</p>
<p>Try to see the time needed by your GPU to process a batch and the time needed  by your dataloader to give a new batch ?</p>
<p>If you have 1M images maybe you should use a format adapted for big image datatasets like h5 files</p>
Implementation of delated convolution

111577: <p>I am interested in time of inference of delated convolution. Is it the same time like usual convolution?<br>
I know that Flops depend on output size, kernel size and amount of in-/out- channels.<br>
I can’t find an implementation of delated convolution. Does kernel size change in delated convolution?</p>
<p>I did a test.  It turned out that usual convolution faster, then delated. Why does it happen?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bdbb71d245e61d0c59181ad46b163b4de5f6d6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5bdbb71d245e61d0c59181ad46b163b4de5f6d6d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bdbb71d245e61d0c59181ad46b163b4de5f6d6d.png" alt="image" data-base62-sha1="d6CbDj7QEFWfvaJVwzvDLXlrFXD" width="690" height="211" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">913×280 7.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41dc17f2430132c1eaaa423d2658c2b8e5a18aac.png" alt="image" data-base62-sha1="9oCAcbnO9cBYqDmkSorIPIrxbn6" width="612" height="469"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80a6683fe84ad26d2b5fba9a73e2b99149801fb1.png" data-download-href="https://discuss.pytorch.org/uploads/default/80a6683fe84ad26d2b5fba9a73e2b99149801fb1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80a6683fe84ad26d2b5fba9a73e2b99149801fb1.png" alt="image" data-base62-sha1="im5DO3e4bwTqIJNx4Kys10YZnSF" width="690" height="68" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1376×136 6.57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><span class="mention">@ptrblck_de</span>, please, answer the question)</p>

tom: <p>The flops will be nominally the same, but</p>
<ul>
<li>The memory accesses will be less “compact”,</li>
<li>they are less often used and likely less well-tuned - convolutions are among the crazily well-tuned and computationally very demanding functions.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

111577: <p>Thank you, Thomas.</p>
<p>Is there something better then dilated convolution, if I want to have a big receptive field?<br>
I want to replace dilated convolution, because to count it longer then usual convolution.</p>

tom: <p>The most standard way is pooling, probably. I’m not sure there are many much better ways.</p>
<p>Best regards</p>
<p>Thomas</p>

111577: <p>Thank you very much, Thomas</p>
Reverse Vanishing Gradient - CNN

Vishak_Raj: <p>Hello,</p>
<p>In my classification project, I followed to check the gradient flow with help of answers from <a href="https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063/7" class="inline-onebox">Check gradient flow in network - #7 by RoshanRane</a></p>
<p>the network structure is -</p>
<pre><code>CNN layers - [c1-c7]
batch-normalization layer [ b1-b7]
the relu activation function is in between batch-normalization and cnn layers
for analysing the gradient flow, I plotted this layers only,

c1
b1
c2
b2
c3
b3
c4
b4
c5
b5
c6
b6
c7
b7

then one output_layer[linear layer] which is not in the gradient flow graph
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68eae2c2eeee5720c224dba5e5e8295a73197be1.png" alt="Screenshot from 2022-08-25 14-56-28" data-base62-sha1="eY8TDCzFLIto5Dny5uDDaaHadLH" width="380" height="280"></p>
<p>there is no much update in average gradient in the cnn layers [ c5-c7 ] in all the last epochs</p>
<p>and this is the loss graph for 50 epoch</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d418c00cbc51094df42720631be2b5241d6ea65.png" alt="Screenshot from 2022-08-25 16-09-28" data-base62-sha1="1TgDIKwr69isSfX03gwCZaTMCsl" width="426" height="284"></p>
<p>If the gradient is very small or near to <code>0</code> then it means vanishing gradient,</p>
<p>but the vanishing gradient actual definition - The gradient may be so small by the time it reaches layers close to the input of the model that it may have very little effect</p>
<p>In this situation, the gradients are almost zero in the layers[layers near to output layer], and gradient changes in the layers which are near to input layer.</p>
<p>how is it possible and how to solve the vanishing gradient in the layers near to output layer,</p>
<p>thanks</p>
SSD Multi head implementation

philipp0827: <p>Hi im trying to train a SSD With mobilenet backbone</p>
<p>I want to delete the output layer of backbone and put 2 convolutional output layers one for classification and another for location</p>
<pre><code class="lang-auto">class ssdLightModel(torch.nn.Module):
  def __init__(self):
      super(ssdLightModel, self).__init__()
      self.backbone = models.mobilenet_v2(weights='MobileNet_V2_Weights.IMAGENET1K_V1')

      infeatures = self.backbone._modules['classifier'][-1].in_features
      labelOutfeatures = len(SCALES) * len(ASPECT_RATIOS) * 2
      boxOutfeatures = len(SCALES) * len(ASPECT_RATIOS) * 4

      self.backbone = self.backbone.features
      self.fc1 = nn.Conv2d(in_channels=infeatures, out_channels=labelOutfeatures, kernel_size=1, bias=True)
      self.fc2 = nn.Conv2d(in_channels=infeatures, out_channels=boxOutfeatures, kernel_size=1, bias=True)
      # kernel_size=(FMAP_ROW, FMAP_COL)

  def forward(self, x):
      x = self.backbone(x)
      x1 = self.fc1(x)
      x2 = self.fc2(x)
      return x1, x2 
</code></pre>
<p>This is my model definition, Did i do something wrong? Because my model learned nothing….</p>
Tooling for checkpointing and restart multi model training?

turian: <p>Can we discuss tooling btw? (If there is a better channel for this, let me know, or maybe even we should create one?)</p>
<p>The general question:<br>
I wrote a pure pytorch prototype last night using wandb logging, and saved JUST the model checkpoint as artifacts. It crashed in the middle of the night and I am manually restarting from that checkpoint.</p>
<p>Should I adopt pytorch-lightning? I’ve used it in the past but I used to run into complications with it when using stranger models like GANs etc. But it also was nice for DDP on multi-GPUs on a single node. Should I use replicate.ai? Are there other tools people prefer for this sort of thing?</p>
<p>The more specific setting I’m working in:<br>
I adopted facebook vicreg code to PRETRAIN a noncontrastive vicreg model that torchsynth synth1b1 audio synthesis PARAMETERS embeddings to match orchsynth synth1b1 audio synthesis AUDIO embeddings. It has some weird facebook pure LARS optimizer I couldn’t get working so I used good old SGD instead. I am currently training it on a single GPU. It crashed after 60K batches for reasons I’m still investigating. The training loss is still going down.</p>
<p>Next, I am going to write a DOWNSTREAM model that uses held out dev and test batches that take the vicreg model and learns a parameter embedding to parameter model on dev, either freeze or tuning the vicreg model, and evaluate on train.  I will do QUALITATIVE evaluation by listening to the torchsynth synth of torchsynth audio from predicted parameters versus torchsynth audio from true parameters. I will do QUANTITATIVE evaluation I think using l1 melspectrogram distance of torchsynth audio from predicted parameters versus torchsynth audio from true parameters. (I want to use MSSTFT distance using auraloss but lambdalabs has a funny numba version and auraloss dependency on librosa and thus numba is a huge pain. So instead I code my own audio distance using torchaudio I think.) I first will pretrain vicreg then learn downstream later. Later I might interleave the training (even tho they both have separate optimizers etc.)</p>
<p>I’ll share code and models alter when it’s slightly more mature.</p>

scottire: <p>Hi,<br>
You may want to narrow down your question to make it more specific and easier to answer.</p>
<p>I work for W&amp;B and using it for saving logs and models, and resuming crashed runs is very common.<br>
Here is some documentation: <a href="https://docs.wandb.ai/guides/track/advanced/resuming" class="inline-onebox" rel="noopener nofollow ugc">Resume Runs - Documentation</a><br>
Let me know if you have any other issues.</p>
<p>As for using pytorch-lightning, it is a pretty mature library and it really depends on your preference whether you like using their abstractions to make your code cleaner.<br>
Both wandb and pytorch-lightning (including when using the WandbLogger) will work fine for training GANs and resuming crashed runs.</p>
How to understand the 768 of vit?

x_u: <ul>
<li>why is the vector length of vit patch embedding is 768, rather than other values ?</li>
<li>If we thought 768 = <code> 16 x 16</code> with 3 channels, which is correct ?</li>
</ul>
<blockquote>
<p>16 x 16x3  or   x3  x  16 x 16</p>
</blockquote>

tom: <p>There are two parts to it. Yes, 768 happens to be the number of inputs per patch in the 16x16 transformers, but it is not necessary to align these numbers. In fact, the 8x8 / 32x32 patch base VIT variants use 768 as an embedding dimension as well and the VIT Large will use 1024…<br>
In practice, there is a linear layer (after you flatten the patch) or equivalent conv layer (with stride/kernel shape = patch size) to bring whatever the number of input values to the embedding dimension.</p>
<p>Best regards</p>
<p>Thomas</p>
How do I choose the correct n-gram for BLEU-score?

maxrivera: <p>How do I choose the correct n-gram for my BLEU-score calculation? Clearly, the below translations are very similar, but if I chose BLEU-4 (the default), then the score would be close to 0, which isn’t very representative. I could just choose BLEU-1, since it’s the highest. But is that like cheating? I don’t know which one is most appropriate.</p>
<pre><code class="lang-auto">reference:  [[['i', 'want', 'some', 'ice', 'cream', '.']]]
hypothesis:  [['i', 'want', 'an', 'ice', 'cream', '.']]
BLEU-1:  0.8333333333333334
BLEU-2:  0.7071067811865476
BLEU-3:  0.5034777750283593
BLEU-4:  7.262123179505913e-78
</code></pre>

srishti-git1110: <p>Hi,<br>
In your case (according to what you’ve demonstrated), you are using BLEU to evaluate individual sentences.</p>
<p>In its original form, BLEU score’s formula uses multiplication of precision scores of i grams ( i = 1, 2, 3, 4). Since the two sentences you show here do not have any 4-grams in common, it’s why BLEU-4 is close to 0.<br>
The reason why BLEU-4 isn’t very representative as you noted is because BLEU isn’t designed to evaluate individual sentences. It’s a corpus based metric and is only good for its purpose when statistics used to calculate it are accumulated over an entire corpus.</p>
<p>As you mention, BLEU uses n-grams – these n-grams are such statistics that do not much make sense (or are less meaningful) when calculated for individual sentences. Moreover, the BLEU score cannot be factorized for individual sentences.</p>
<p>Apart from this, BLEU as a metric suffers from problems even when used to evaluate large corpuses, an example of which is its inability to distinguish between content words and function words (a, the etc.)</p>
<p>So, I’d say along with BLEU, you should evaluate your model translations using other metrics/ways as well.<br>
Hope this helps,<br>
Srishti</p>
How to convert checkpoint file (.ckpt) to state dict file (.pth)?

davic11915: <p>Hi! New PyTorch user here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I am trained my model using Pytorch Lighting and ModelCheckpoint with parameter <code>save_top_k=1</code>, so only the best checkpoint is saved. . After the training is finished I saved the model as usual with <code>torch.save(model.state_dict())</code>.<br>
Now I want to deploy my model for inference. My <code>epoch=42.ckpt</code> file contains a model with better performance than the final model, so I want to use this checkpoint file. But the checkpoint file is three times larger than the normal model file (.pth).<br>
How do I convert a .ckpt file to a .pth file?</p>

ptrblck: <p>CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a> and <a class="mention" href="/u/justusschock">@justusschock</a> for Lightning questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

nswarnkar: <p>what was the reply over converting checkpoint file to .pth file? Or we are going in wrong direction at all?</p>

ptrblck: <p>I’m not familiar with Lightning, but PyTorch shouldn’t change the file format based on the file extension.<br>
Both files should contain the same data if you save them as <code>.ckpt</code> or <code>.pth</code>.</p>

oublalkhalid: <p>For a quick explanation, which can help other Lightning users! After a quite long experience with Lightning (which I enjoy), the hyper parameters are in <code>hprams.yaml</code> and the .ckpt (generated to the log_lightning folder) is exactly pth (simply the name differs) torch lightning keeps in the same file a little more information about hyperprams! you can upload your model and then take only <strong>state_dict</strong> of your model in the hyperprams that you do not want!</p>
Setting up a pytorch-forecasting TimeSeriesDataSet for a sequence-to-sequence DataLoader

krishnab: <p>Hey Folks. I just discovered the <code>pytorch-forecasting</code> package’s <code>TimeSeriesDataSet</code> class, and how it helps with taking data from a pandas dataframe and creating a pytorch DataLoader. They show one example of creating a TimeSeries Dataset, but don’t but don’t have much in the way of a tutorial etc. I was wondering if I can create a sequence-to-sequence dataset using this class, meaning I would need the TimeSeriesDataSet class to cut the time series data into windows, where say the first 10 days are the data and the subsequent 10 days are the labels.</p>
<p>I have code to do this manually right now, and it works. But for the future it would be nice to have a tool like this to setup new datasets faster. So was just wondering if anyone know whether I can setup a Sequence-to-Sequence set of (data, labels) for the DataLoader, and if anyone knows of a good example of this.</p>
<p>Thanks.</p>

Jose_Peeterson: <p>The <code>TimeSeriesDataSet</code> does exactly what you are describing. first 10 days are in x attribute encoder_target and the next 10 days in x attribute decorder_target.  the next 10 days is also in y attribute target. read about this in to_dataloader method in timeseries.py.</p>
<p>I believe the encoder_cont and  encoder_real are fed into encoder input.  attribute target of y is used as label.</p>
Jit.trace affects the accuracy of the model on GPU

daniellevi: <p>I tried to export my torch model into a pt file(while my model is located on the gpu), using this code-</p>
<pre><code class="lang-auto">traced_script_module = torch.jit.trace(self.model, gray_img_tens, strict=False)
traced_script_module.save("model.pt")
</code></pre>
<p>But when I try to activate this model on the same input, using-</p>
<pre><code class="lang-auto">tracedPT_outputs = traced_script_module.forward(gray_img_tens)
</code></pre>
<p>I recieve different output. The output is usually close to the original, but I can’t rely on that. The mispredictions are accumulated during the code, and th final result is far than expected.</p>
<ul>
<li>The problem occures just on the gpu!<br>
When I try to do the same on the cpu, the output I recieve is the same.<br>
In this case, the latency is very high, and I can’t rely on that.</li>
</ul>

eqy: <p>It might be difficult, but could you check if it would be possible to isolate where the differences first start appearing between the traced/non-traced versions of the model? For example, one approach would be to remove layers until the model is very small yet still showing differences.</p>
<p>Alternatively, depending on the size of the model and the numerics of the traced ops, small numerical differences might be expected. A litmus test for this could be to run the model in higher (e.g., double) precision and to see if the traced model’s output is further away than the standard eager mode model output.</p>
How to implement Long range Connections in Linear Layers in pt

mmv: <p>Please help me to implement Long range Connections in Linear layers (pytorch)</p>
<blockquote><a href="//imgur.com/a/5eB4BCE" rel="noopener nofollow ugc"></a></blockquote>
<p>My Current Implementation without long range connections. I want to implement a random long range connections from one layer to another layer. How can i accomplish that.</p>
<pre><code class="lang-auto">def get_weights_tensor(nodes):
    weights = torch.ones(nodes, nodes)
    for i in range(nodes):
        for j in range(nodes):
            if i != j and i != j+1 and j != i+1:
                weights[i, j] = 0
    weight_tensor = weights.type(torch.float32)
    return weight_tensor

class NN(nn.Module):
    def __init__(self, input_size, num_classes, hidden_nodes=50):
        super(NN, self).__init__()
        self.hidden_nodes = hidden_nodes

        self.input_layer = nn.Linear(input_size, hidden_nodes)
        
        self.hl_1 = nn.Linear(hidden_nodes, hidden_nodes)
        self.hl_1.weight = torch.nn.Parameter(get_weights_tensor(hidden_nodes))

        self.hl_2 = nn.Linear(hidden_nodes, hidden_nodes)
        self.hl_2.weight = torch.nn.Parameter(get_weights_tensor(hidden_nodes))

        self.hl_3 = nn.Linear(hidden_nodes, hidden_nodes)
        self.hl_3.weight = torch.nn.Parameter(get_weights_tensor(hidden_nodes))

        self.hl_4 = nn.Linear(hidden_nodes, hidden_nodes)
        self.hl_4.weight = torch.nn.Parameter(get_weights_tensor(hidden_nodes))

        self.hl_5 = nn.Linear(hidden_nodes, hidden_nodes)
        self.hl_5.weight = torch.nn.Parameter(get_weights_tensor(hidden_nodes))

        self.output_layer = nn.Linear(hidden_nodes, num_classes)

    def forward(self, x):
        x = F.relu(self.input_layer(x))
        x = F.relu(self.hl_1(x))
        x = F.relu(self.hl_2(x))
        x = F.relu(self.hl_3(x))
        x = F.relu(self.hl_4(x))
        x = F.relu(self.hl_5(x))
        x = self.output_layer(x)
        return x

</code></pre>
Increasing accuracy

A_Ebrahimi: <p>hi i’m training a model unet++ with resnet18 as encoder that segments satellite images into 6 classes .after 200 epochs i get 82% accuracy without over-fitting but accuracy doesn’t increase . is there any way to increase accuracy for example using other encoder</p>

azhanmohammed: <p>Try using a deeper backbone like ResNet 32 or something. Also does the loss stagnate after a few epochs, or is the loss still decreasing?</p>

A_Ebrahimi: <p>thanks for your reply it has been 30 epoch that loss and accuracy dont change</p>

azhanmohammed: <p>If loss and accuracy both do not change, I guess that you should use a deeper backbone network. Try using deeper ResNet models or Inception ResNet models, they might help.</p>
Why does the gradient of the first element of all_loss accumulate, which was originally 0.7215, how to solve it?

xyangyan: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af9ccddffde3f31cb2bea77a012d8c2504aab27f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/af9ccddffde3f31cb2bea77a012d8c2504aab27f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af9ccddffde3f31cb2bea77a012d8c2504aab27f.jpeg" alt="image" data-base62-sha1="p3xvxJvqjpGnl2DXOLdNWXUoKL5" width="690" height="408" data-dominant-color="3D3C3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1091×646 68.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

srishti-git1110: <p>Can you please post some code that reproduces the error?</p>
<p>From what I can understand as of now, <code>logits</code> contains sets of predictions as its elements (essentially arrays of set of predictions - like prediction sets from various inference runs).<br>
And you are trying to record the loss from each set of predictions in <code>all_loss</code>. That to say, for 5 inferential runs, all_loss will have 5 values of losses. Is that it?</p>
<p>Generally, a common reason for grad accumulation is when one misses <code>torch.no_grad</code>, but here it’s happening just for the first element so not sure, I’ll have a look at more code first.</p>

xyangyan: <p>Thank you. I have solved this problem yesterday. The above screenshot just obscures the key place, that is, total_loss += loss ( as you can see in the images below) makes the all_loss and total_loss point to the loss, and there is a value binding, so that the sum of the total_loss acts on the first element of the all_loss.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ec93479f4c90aa49a212a9bcc98ca87e7399672.png" alt="image" data-base62-sha1="26NLnTqZ5Qy6ksR8xaXgGheWwDg" width="301" height="134"></p>

srishti-git1110: <p>I see.<br>
According to this code, it’s happening because of the inplace operation <code>total_loss += loss</code>.<br>
Changing it to <code>total_loss = total_loss + loss</code> should solve the accumulation error.</p>
<p>Did you try anything else to solve this?</p>

xyangyan: <p>You’re great！My solution is to sum up outside the for loop, but your solution is more elegant.</p>

srishti-git1110: <p>Ok, nice to know it works. you could mark it as the solution that’ll help anyone reading this thread in future.</p>
Running pytorch with rocm on RDNA2

flolf: <p>Hello there. I wanted to running the recently released Stable Diffusion locally and tried to get pytorch running on my rx6600 to that end. I already have all the rocm 5.1.3 stuff installed from an attempt to get blender ruining earlier. I installed pytorch according to this (with pip in user mode)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b0475851c8dca362c7b9624b4cb0df96639c2ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b0475851c8dca362c7b9624b4cb0df96639c2ad" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b0475851c8dca362c7b9624b4cb0df96639c2ad_2_690x249.png" alt="grafik" data-base62-sha1="68y5fGFrdfVHq90Yvh4F8nj7IL3" width="690" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b0475851c8dca362c7b9624b4cb0df96639c2ad_2_690x249.png, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b0475851c8dca362c7b9624b4cb0df96639c2ad.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b0475851c8dca362c7b9624b4cb0df96639c2ad.png 2x" data-dominant-color="F3D7D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">806×291 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>rocminfo output: <a href="https://pastebin.com/6sBdc9WR" class="inline-onebox" rel="noopener nofollow ugc">OCk module is loaded===================== HSA System Attributes ==== - Pastebin.com</a></p>
<p>when i try to run the verify step i get this error:</p>
<pre><code class="lang-auto">Python 3.9.12 (main, May 16 2022, 02:30:54) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"
[1]    25715 IOT instruction  python3
</code></pre>
<p>searching online, this seems to indecate a gpu unsuported in the version of rocm used, which shouldnt be the case for me.</p>
<p>Any ideas what im doing wrong here?</p>
Restructuring repeated tensor data using as_strided

William_Ravenscroft: <p>I’m trying to transform a tensor from this structure:</p>
<pre><code class="lang-auto">[[[
      [1,2,3,4,5,6,7,8,9,10,11,12],
      [99,98,97,96,95,94,93,92,91,90,98,97]
]]]
</code></pre>
<p>(shape 1 x 1 x 2 x 12)<br>
to this structure</p>
<pre><code class="lang-auto">[[[
      [[1,2,3,4,5],
      [3,4,5,6,7],
      [5,6,7,8,9],
      [7,8,9,10,11]],
      [[99,98,97,96,95],
      [97,96,95,94,93],
      [95,94,93,92,91],
      [93,92,91,90,98]]
]]]
</code></pre>
<p>(shape 1 x 1 x 2 x 4 x 5)<br>
I’ve been trying to use <code>torch.as_strided()</code> for this but I am not getting my desired output data the way I am using it. Here is the code I am using for figuring this out</p>
<pre><code class="lang-auto">import torch
from torch import tensor

stride=2
kernel_size=5
output_length=4
num_channels=2

input=[[[
      [1,2,3,4,5,6,7,8,9,10,11,12],
      [99,98,97,96,95,94,93,92,91,90,98,97]
    ]]]

t = tensor(input)

print(t.shape)

output = torch.as_strided(t,(1,1,num_channels,output_length,kernel_size),(1,1,1,2,1))

desired_output = tensor([[[
      [[1,2,3,4,5],
      [3,4,5,6,7],
      [5,6,7,8,9],
      [7,8,9,10,11]],
      [[99,98,97,96,95],
      [97,96,95,94,93],
      [95,94,93,92,91],
      [93,92,91,90,98]]
    ]]])

print(output.shape)
print(desired_output.shape)
print(output)
</code></pre>
<p>My output looks like</p>
<pre><code class="lang-auto">[[[[[ 1,  2,  3,  4,  5],
           [ 3,  4,  5,  6,  7],
           [ 5,  6,  7,  8,  9],
           [ 7,  8,  9, 10, 11]],
          [[ 2,  3,  4,  5,  6],
           [ 4,  5,  6,  7,  8],
           [ 6,  7,  8,  9, 10],
           [ 8,  9, 10, 11, 12]]
]]]
</code></pre>
<p>So the shape is correct but the numbers are not. Can anyone offer any hints as to how I can get the output I am looking for</p>

ptrblck: <p><code>unfold</code> should work:</p>
<pre><code class="lang-python">out = t.unfold(dimension=3, size=5, step=2)
print(out)
# tensor([[[[[ 1,  2,  3,  4,  5],
#            [ 3,  4,  5,  6,  7],
#            [ 5,  6,  7,  8,  9],
#            [ 7,  8,  9, 10, 11]],

#           [[99, 98, 97, 96, 95],
#            [97, 96, 95, 94, 93],
#            [95, 94, 93, 92, 91],
#            [93, 92, 91, 90, 98]]]]])
</code></pre>
Update timeout for pytorch ligthning ddp

kaipakiran: <p>I am trying to update the default distributed task timeout from 30 mins to 3 hours using<br>
ce = pl.plugins.environments.lightning_environment.LightningEnvironment()<br>
pl.utilities.distributed.init_dist_connection(cluster_environment=ce,torch_distributed_backend=‘nccl’,timeout=dt.timedelta(seconds=3<em>60</em>60))<br>
passing timeout as kwargs for torch.distributed.init_process_group but I am getting the below error -<br>
pl.utilities.distributed.init_dist_connection(cluster_environment=ce,torch_distributed_backend=‘nccl’,timeout=dt.timedelta(seconds=3<em>60</em>60))<br>
File “/mnt/task_runtime/py36/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py”, line 388, in init_dist_connection<br>
torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs<br>
File “/mnt/task_runtime/py36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 500, in init_process_group<br>
store, rank, world_size = next(rendezvous_iterator)<br>
File “/mnt/task_runtime/py36/lib/python3.6/site-packages/torch/distributed/rendezvous.py”, line 190, in _env_rendezvous_handler<br>
store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)<br>
RuntimeError: Address already in use<br>
How do I set the timeout?</p>

ptrblck: <aside class="quote no-group" data-username="kaipakiran" data-post="1" data-topic="159972">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaipakiran/48/34174_2.png" class="avatar"> kaipakiran:</div>
<blockquote>
<p>RuntimeError: Address already in use</p>
</blockquote>
</aside>
<p>This error is raised if the network address is already used by another process and unrelated to setting the <code>timeout</code> value, which looks correct.<br>
Btw. you can also use <code>timedelta(hours=3)</code>, which sounds quite excessive. Would you mind explaining why you are expecting such long timeouts in your training?</p>
Survival analysism

Filippo: <p>Hi All,</p>
<p>I have to deploy a model to estimate the probability distribution of trains belonging to a “train slot made of 10 trains” to arrive in the next hour/2 hour/3 hour/more then 4 hour at the next train station.<br>
For training, as regressors, the departure time from previous station, number of passengers, the status ( train slot has/has not reached the intermediate touchpoint between two consecutive stations) are available.<br>
If the trains slot has left at 1p.m station A and 3 trains of the slot has reached station B at 2 p.m and the remaining 7 at 3 p.m and the model run at 2 p.m than the model output should be ([“next one hour”, 3 trains],[next two hours, 7trains],[next 3 hours, 0 trains],[more then 4 hours, 0 trains]) if we wanna model it as classification. The model will keep to generate inferences till all train arrive to the station.<br>
The train_slot_Id (and the number of trains we are still waiting) appears multiple time in the dataset per different time stamp (hours) till all the trains in the train slot reach the station. Train_id is not available, data are aggregated at [train_slot_id, time of sample]<br>
I was thinking about survival models and I found a library, namely pycox, with deep survival models. Though the library is not mentioned in the hub neither in the official PyTorch doc so I am not sure I can trust it as state of the art.<br>
Do you have any recommendation on how I should treat the problem and which model to use? Do we have NLP predicting the time between two words?</p>
Pickling tensors is much much slower than numpy arrays

ecolss: <p>I tested <code>pickle</code> on list of pytorch tensors, and found it was 20~30x slower than numpy arrays, what could be the reason? And how to make it faster for tensors?</p>
<p>Benchmark code goes as follows</p>
<pre><code class="lang-auto">import numpy as np
import torch as th
import pickle

# create same size data (2d list, inner list is a list of tensors or arrays)
numpy_array = [[np.random.rand(20) for _ in range(100)] for _ in range(100)]
torch_array = [[th.rand(20) for _ in range(100)] for _ in range(100)]

# measure pickle time cost
def time_pickle(array):
    t = time.time()
    with open("array.pkl", "wb") as fp:
        pickle.dump(array, fp)
    print(f"dump time cost is {time.time() - t}")

    t = time.time()
    with open("array.pkl", "rb") as fp:
        array = pickle.load(fp)
    print(f"load time cost is {time.time() - t}")

&gt;&gt;&gt; time_pickle(torch_array)
dump time cost is 0.35193419456481934
load time cost is 0.3934769630432129

&gt;&gt;&gt; time_pickle(numpy_array)
dump time cost is 0.039617061614990234
load time cost is 0.017045259475708008
</code></pre>