akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Even with setting `allow_unused = True` in `torch.autograd.grad()`, I get an error

gebrahimi: <p>Hi, Running the following code, I get an error like this:</p>
<p><code>torch.autograd.grad(cross_entropy_loss, model.omninet_first.parameters(), allow_unused=True)</code></p>
<p>The error:<br>
`<br>
File “/Users/ghaniebrahimi/opt/anaconda3/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 190, in grad<br>
return Variable._execution_engine.run_backward(</p>
<p>RuntimeError: One of the differentiated Tensors does not require grad`</p>
<p>Don’t we do <code>allow_unused = True</code>, to bypass this error? What is the problem here?</p>
How to speed up using DataLoader

Ahmad_Pouramini: <p>I had a dataset including about a million of rows. Before,  I read the rows, preprocessed data and created a list of rows to be trained. Then I defined a Dataloader over this data like:</p>
<pre><code class="lang-auto">train_dataloader = torch.utils.data.DataLoader(mydata['train'],
        batch_size=node_batch_size,shuffle=shuffle,collate_fn=data_collator)
</code></pre>
<p>Preprocessing could be time consuming, so I thought to define an <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">IterableDataSet</a> with <code>__iter__</code> function. Then I could define my Dataloader like:</p>
<pre><code class="lang-auto">train_dataloader = torch.utils.data.DataLoader(myds['train'],
        batch_size=node_batch_size,shuffle=shuffle,collate_fn=data_collator)
</code></pre>
<p>However, still to begin training it seems that it calls my preprocessing function and creates an Iteration over it. So, I wonder what is the use of this dataloader. it seems I didn’t gain much speed up.</p>
<p>Please guide me how could I use speed up in this case?</p>
Pytorch autograd hook in Megatron distributed data parallel

Nguyen_Anh: <p>Hi everyone, just wondering why do we need to expand the tensor to get access to grad_fn as described <a href="https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/model/distributed.py#L161" rel="noopener nofollow ugc">here</a>? Can we replace the expand_as with a view operation instead? Thanks in advance!</p>
Reshape to use every second item in tensor

HallerPatrick: <p>Hello,</p>
<p>my goal is the following:</p>
<pre><code class="lang-python">t = torch.tensor([1, 2, 3, 4])

# Every second element now in own tensor
expected = torch.tensor([[1, 3], [2, 4]])

assert expected == t
</code></pre>
<p>What operations do I have to perform to reshape the tensor, such that we use every second element as a new dimension.</p>
<p>With best regards,<br>
Patrick</p>

mMagmer: <p>the following code works for your example but i don’t know if that’s what you want.</p>
<pre><code class="lang-auto">import torch
t = torch.tensor([1, 2, 3, 4])

# Every second element now in own tensor
expected = torch.tensor([[1, 3], [2, 4]])
t.view(-1,2).t()

assert (expected == t.view(-1,2).t()).all()
</code></pre>
One loss not converge out of Two loss

cbd: <p>I have two losses L1 and L2. I train model by giving single image at a time. For some image, both the losses converge towards 0 but for some image L1 converge to significant value but L2 loss not decreasing after some epochs.</p>
<p>After increasing dataset, problem remains to those images for which L2 loss not decreasing. I work around for different learning rate and dense layer but problem still persist.</p>
<p>Which parameters should i need to try to reduce both the losses?</p>

tom: <p>One aspect is the balancing (weighting) of the two losses: giving l2 a higher weight will prefer reducing it over achieving the same reduction of l1.</p>
<p>Another thing to check is whether l2 has actually achieved a minimum, egif the gradients of l2are (almost) zero. In that case you would want to revisit the modelling. As always, overfitting a single batch is a good sanity check here.</p>
<p>Best regards</p>
<p>Thomas</p>

cbd: <p>Giving higher weights to l2 loss doesn’t improve the result. I need to check the gradients of different layer. Could you please give reference code or link to plot and visualize  the gradients of different layer?</p>
F.interpolate 's misleading error messages and documentation

zetyquickly: <p>Hello,</p>
<p>I think I found some inconsistency between documentation, behavior and error messages while using F.interpolate.</p>
<p>Regarding <code>mode=linear</code> documentation says that input should be a 3D tensor. That’s why we see expected error when passing the 4D tensor.</p>
<p>Code</p>
<pre><code class="lang-auto">x = torch.rand(1,2,100,100)
F.interpolate(x, size=(64,64), mode="linear")
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">NotImplementedError: Got 4D input, but linear mode needs 3D input
</code></pre>
<p>But at the same time when we pass 3D tensor to this very function error states that it’s 1D.</p>
<p>Code</p>
<pre><code class="lang-auto">x = torch.rand(1,100,100)
F.interpolate(x, size=(64,64), mode="linear")
</code></pre>
<p>Error</p>
<pre><code class="lang-auto">ValueError: size shape must match input shape. Input is 1D, size is 2
</code></pre>
<p>Why is this way? What do I miss?</p>

mMagmer: <p><code>mode='linear'</code> is for data with only one <em>spatial</em> dimension. Image data has 2 spatial dim.<br>
for image like data, you should use <code>mode='bilinear'</code>.<br>
The following code will work for 1d data. 100 is <span class="hashtag">#numchannel</span> for 1d input.</p>
<pre><code class="lang-auto">x = torch.rand(1,100,100)
F.interpolate(x, size=(64), mode="linear")
</code></pre>
<p>or for 2d,</p>
<pre><code class="lang-auto">x = torch.rand(1,1,100,100)
F.interpolate(x, size=(64,64), mode="bilinear")
</code></pre>

zetyquickly: <aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="139971">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<pre><code class="lang-auto">x = torch.rand(1,1,100,100)
F.interpolate(x, size=(64,64), mode="bilinear")
</code></pre>
</blockquote>
</aside>
<p>Yes, thank you it’s a good explanation. Documentation says</p>
<blockquote>
<p>“size”` — output spatial size</p>
</blockquote>
<p>so it’s needed to understand what is “spatial” dimensions of the input tensor.</p>
Ensemble two or three models in pytorch

Sang_Dinh: <p>I want to study about ensemble in pytorch for my project which using efficient and resnet. But I dont know how to start with it? I also search for information on google. Probably my keywords are not good to have good document. Anyone know? Thank you so much.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/75">This post</a> and the entire thread might be helpful for you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Sang_Dinh: <p>Sorry, I’m late. I just have read the post. Amazing! The stacking ensemble improved my model very much. Thank you.<br>
Sorry, 1 question. If I have 2 different inputs, what should I do?</p>

ptrblck: <p>You can pass both inputs to the <code>forward</code> method of the main model and use the different inputs where needed.</p>

Sang_Dinh: <p>Thank you so much, will try. Hopefully, you will support me next problem</p>
Equivalent of tf.nn.dilation2d

cthnguyen: <p>Hello,</p>
<p>I’m new to PyTorch and I don’t know too well Tensorflow. I am currently re-writting a code in Tensorflow to Pytorch and they use tf.nn.dilation2d. I didn’t find any function close to the tensorflow’s one. Can anyone help me ?</p>
<p>Thanks !</p>

ptrblck: <p>I don’t think a dilation is implemented yet.<br>
You could <code>unfold</code> the image tensor and apply the max operation on each patch.<br>
Have a look at <a href="https://discuss.pytorch.org/t/efficiently-slicing-tensor-like-a-convolution/44840/2">this example</a> to see how to create the patches.</p>

cthnguyen: <p>Thank you ! I will look into it !</p>

yxchng: <p><a class="mention" href="/u/cthnguyen">@cthnguyen</a> have you figured out how to do this?</p>

AndrewYB: <p>So how to tackle the gradient function of the structural element in the dialtion in the back propagation ?</p>

AndrewYB: <p>So how to tackle the gradient function of the structural element in the dialtion in the back propagation ?</p>

ptrblck: <p>If you are using the <code>unfold</code> approach with an input tensor, which requires gradients, Autograd should be able to create the backward function automatically as long as you don’t detach the tensor from the computation graph.</p>

AndrewYB: <p>thanks!But I am confused why the dilation operation in tensorflow explicitly defines a backpropagation, so if I customize the dilation in pytorch, don’t I?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6e5cfdc1bd760e2c9cc88595ae5463de04d97e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6e5cfdc1bd760e2c9cc88595ae5463de04d97e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6e5cfdc1bd760e2c9cc88595ae5463de04d97e1.png" alt="image" data-base62-sha1="nOrD3slKKlRnc8DQQ3uXOJu2WfD" width="690" height="236" data-dominant-color="2F2F2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1185×407 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure, how Tensorflow is implemented in the backend, but note that in PyTorch also backward functions are implemented for you.<br>
Autograd will call these functions, so that you don’t have to manually write them.</p>

AndrewYB: <p>oh,I will try,thank you very much!</p>
Using GPU or CPU kernel in C++ Extensions

HallerPatrick: <p>Hello,</p>
<p>I am currently working on porting parts of my functionality to C++.</p>
<p>In one of my functions I am iterating over a input tensor to produce two new output tensors.<br>
For this I am using <code>TensorIterator</code>.</p>
<p>For the iteration  I am using <code>cpu_kernel_multiple_outputs</code>, which works fine.<br>
Now I try to include the logic for running on GPU, therefore I am using <code>gpu_kernel_multiple_outputs</code>.</p>
<p>Here a little code snippet:</p>
<pre><code class="lang-cpp">// Check if running in CPU or GPU, based on input tensor
if (self.device(),type() == at:kCUDA) {
  cpu_kernel_multiple_outputs(
        iter, [=](int64_t a) -&gt; std::tuple&lt;int64_t, int64_t&gt; { ... }
  );
} else {
  gpu_kernel_multiple_outputs(
        iter, [=] GPU_LAMBDA (int64_t a) -&gt; std::tuple&lt;int64_t, int64_t&gt; { ... }
  );
}

</code></pre>
<p>To make this run I had to include: <code>#include &lt;ATen/native/cpu/Loops.h&gt;</code> for the cpu kernel<br>
and <code>#include &lt;ATen/native/cuda/Loops.h&gt;</code>for the gpu kernel.</p>
<p>First of all, is this the right approach or am I including internals I should not be including?</p>
<p>Second, the import of the GPU kernel does not work, due to the missing <code>&lt;thrust/tuple.h&gt;</code> header, which looks like a external library. Do I manually have to include it to build the extension?</p>
<p>To be sure, here are all includes I am using:</p>
<pre><code class="lang-cpp">#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/native/TensorIterator.h&gt;
#include &lt;ATen/native/cpu/Loops.h&gt;
// #include &lt;ATen/native/cuda/Loops.h&gt;

#include &lt;torch/extension.h&gt;
#include &lt;tuple&gt;
#include &lt;vector&gt;
</code></pre>
<p>I am not a C++ developer, so please excuse me.</p>
<p>With best regards,<br>
Patrick</p>
Boolean mask + ellipsis lead to incorrect indexing?

stevethesteve: <p>Indexing with boolean masks and ellipsis leads to inconsistent behavior.</p>
<p>MWE:</p>
<pre><code class="lang-auto">import torch
t = torch.arange(12).view(3,2,-1)
mask = torch.BoolTensor([[True, False], [False, True], [False, False]])
</code></pre>
<p>the expression</p>
<pre><code class="lang-auto">t[mask][...,0]
</code></pre>
<p>returns</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor([0, 6])
</code></pre>
<p>whereas</p>
<pre><code class="lang-auto">t[mask,..., 0]
</code></pre>
<p>returns</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor([0, 5])
</code></pre>
<p>With numpy arrays, both <code>[mask,..., 0]</code> and <code>[mask][..., 0]</code> return <code>&gt;&gt;&gt; array([0, 6])</code>.</p>

ptrblck: <p>This seems indeed unexpected. Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track and fix it, please?</p>

stevethesteve: <p>github issue submitted: <a href="https://github.com/pytorch/pytorch/issues/70347" class="inline-onebox" rel="noopener nofollow ugc">boolean mask + ellipsis lead to incorrect indexing · Issue #70347 · pytorch/pytorch · GitHub</a></p>
About torch.cuda.amp model resuming

Yangmin: <p>Hi,</p>
<p>I have some questions about using torch.cuda.amp() related resuming models.</p>
<ol>
<li>
<p>For example, after I trained a model without torch.cuda.amp(), then saved the model weights and optimizer using torch.save() function. If I resumed a model with torch.cuda.amp() function running, I think there is an error in training the model. Am I right?</p>
</li>
<li>
<p>If I want to train properly when resuming, should it be same circumstance on using torch.cuda.amp()??</p>
</li>
</ol>
<p>For example,  is below okay?</p>
<p>Train using amp → resume → Train using amp<br>
or<br>
Don’t train using amp → resume → Don’t train using amp</p>
<p>Thanks.</p>

ptrblck: <ol>
<li>
<p>No, there shouldn’t be an error and your use case would be comparable to using <code>torch.cuda.amp</code> for fine tuning a pre-trained model.</p>
</li>
<li>
<p>The model parameters are all stored on <code>float32</code> so you could continue the training either using amp or the standard <code>float32</code> precision.</p>
</li>
</ol>
Torch::from_blob with const pointer C++

Unlikus: <p>Under which circumstances is it save to use a const pointer for <code>torch::from_blob</code> (via const cast)?<br>
I do not intend to change the values through the tensor, but use slicing, indexing and broadcasting. So could pytorch change the underlying data in any way?<br>
Would it be safe if I clone the tensor immediately?</p>
Input type disagreed with the model type

FengMu1995: <p>I quantized both my model and input to “int8”, but an error generated,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04139689f44a5e045a05d534c3a9ed2bdd9f4c30.png" data-download-href="https://discuss.pytorch.org/uploads/default/04139689f44a5e045a05d534c3a9ed2bdd9f4c30" title="Screenshot from 2021-12-22 16-23-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04139689f44a5e045a05d534c3a9ed2bdd9f4c30_2_690x138.png" alt="Screenshot from 2021-12-22 16-23-05" data-base62-sha1="A3SnsbaJ0rCp0V4BUXhLFuW7g4" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04139689f44a5e045a05d534c3a9ed2bdd9f4c30_2_690x138.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04139689f44a5e045a05d534c3a9ed2bdd9f4c30_2_1035x207.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04139689f44a5e045a05d534c3a9ed2bdd9f4c30_2_1380x276.png 2x" data-dominant-color="2F3031"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-12-22 16-23-05</span><span class="informations">1631×328 31.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>why does the model require that input type is float</p>

Vasiliy_Kuznetsov: <p>could you provide more context, ideally a reproducible example?</p>

FengMu1995: <p>Hi， I’ve found the error and remove the “normalize”, it worked.</p>
What is an efficient implementation of a custom map-style dataset for a hdf5 file with irregular structure?

flaviomaia77: <p>I have a hdf5 file that contains picture of a certain number of people, from a certain number of source cameras, for many seconds. So it is like this: file[seconds][person][camera] But this is quite irregular, such that for a given second, there may be different number of persons, and for a given second and person there may be picture from different cameras. I want to create a map-style pytorch dataset, so I need to implement get_item(idx) that will return a unique second, person and camera for that idx.</p>
<p>My first idea is to iterate through the whole dataset and create dictionaries that can be accessed with idx, that is, second[idx] = this_second, person[idx] = this_person, camera[idx] = this_camera. So I can use all of that to get a unique data from the dataset with file[this_second][this_person][this_camera].</p>
<p>However this solution seems to complicated for me. I wonder if there is a better way to solve that, since this is probably a common problem.</p>
Clip discriminator output in a WGAN to a certain range

dgrat: <p>Hi there<br>
In order to learn about GANs I adapted code for a WGAN and played around a bit (from <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/tree/master/ML/Pytorch/GANs/3.%20WGAN" class="inline-onebox" rel="noopener nofollow ugc">Machine-Learning-Collection/ML/Pytorch/GANs/3. WGAN at master · aladdinpersson/Machine-Learning-Collection · GitHub</a>).</p>
<ol>
<li>
<p>I noticed that if I add a TanH function to the discriminator model. Training becomes unstable and generally much slower. Why is that?</p>
</li>
<li>
<p>Does someone has an idea how I can converge during training the output of the discriminator of a WGAN to match a certain label (e.g. -1 and +1 or 0 and 1) without adding a transfer function to the model itself?</p>
</li>
</ol>
<p>This is the current training loop:</p>
<pre><code class="lang-auto">        # Train Critic: max E[critic(real)] - E[critic(fake)]
        for _ in range(CRITIC_ITERATIONS):
            noise = torch.randn(cur_batch_size, RAND_DIM, 1, 1).to(device)
            fake = gen(noise)
            critic_real = critic(real).reshape(-1)
            critic_fake = critic(fake).reshape(-1)

            mean_real = torch.mean(critic_real)
            mean_fake = torch.mean(critic_fake)

            loss_critic = mean_fake - mean_real
            critic.zero_grad()
            loss_critic.backward(retain_graph=True)
            opt_crit.step()

            # clip critic weights between -0.01, 0.01
            for p in critic.parameters():
                p.data.clamp_(-0.01, 0.01)

        # Train Generator: max E[critic(gen_fake)] &lt;-&gt; min -E[critic(gen_fake)]
        loss_gen = -torch.mean(critic(fake).reshape(-1))
        gen.zero_grad()
        loss_gen.backward()
        opt_gen.step()
</code></pre>
RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [2] at entry 1

alpha2210: <p>I am getting an above error , I don’t have an idea why I am getting this error. <a class="mention" href="/u/ptrblck">@ptrblck</a> sampler Module code Snippet for specific part where I am getting an error is below:</p>
Using NN module List in C++ API

Aaditya_Chandrasekha: <p>Hi,</p>
<p>I am aware of this wonderful functionality in the Python API to declare a simple feedforward network in a loop using ModuleList. I am now trying to port my code into C++ and having trouble replicating the code there. I am not sure how the ModuleList works in C++. Any help would be great!</p>
<pre><code class="lang-auto">def __init__(self, numLayers, numNeuronsPerLyr):
     
        super().__init__();
        self.layers = nn.ModuleList();
        current_dim = 2;

        for lyr in range(numLayers): # define the layers in loop
            l = nn.Linear(current_dim, numNeuronsPerLyr);
            nn.init.xavier_normal_(l.weight);
            nn.init.zeros_(l.bias);
            self.layers.append(l);
            current_dim = numNeuronsPerLyr;
        self.layers.append(nn.Linear(current_dim, 3));
        
def forward(self, x):
        m = nn.LeakyReLU();
        ctr = 0;
        for layer in self.layers[:-1]: # forward prop
            x = m(self.bnLayer[ctr](layer(x))); # ReLU in all hidden layers
            ctr += 1;
        out = torch.softmax(self.layers[-1](x), dim = 1); # output layer softmax
        return out 
</code></pre>

glaringlee: <p><a class="mention" href="/u/aaditya_chandrasekha">@Aaditya_Chandrasekha</a><br>
We have a simple instruction in the comment here:<br>
‘<a href="https://github.com/pytorch/pytorch/blob/cd0724f9f1b57dae12be2c3fc6be1bd41210ee88/torch/csrc/api/include/torch/nn/modules/container/modulelist.h#L11" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cd0724f9f1b57dae12be2c3fc6be1bd41210ee88/torch/csrc/api/include/torch/nn/modules/container/modulelist.h#L11</a>’</p>
<p>We have tests here, it contains more examples.<br>
‘<a href="https://github.com/ShahriarSS/pytorch/blob/678873103191c329e2ca4a53db1d398599ad9443/test/cpp/api/modulelist.cpp" rel="nofollow noopener">https://github.com/ShahriarSS/pytorch/blob/678873103191c329e2ca4a53db1d398599ad9443/test/cpp/api/modulelist.cpp</a>’</p>

Willem: <p>I’m trying to use a modulelist to inside a module to create a multi-layer perceptron where the user can specify the number and size of hidden elements. Signature of constructor is:</p>
<p>MLPImpl(int64_t inputdim, int64_t outputdim, std::vector&lt;int64_t&gt; hidden)</p>
<p>MLPImpl has member variable</p>
<pre><code>torch::nn::ModuleList layers;
</code></pre>
<p>I call register_module(“layers”, layers)  in constructor, and add some linear layers to the “layers” member via push_Back. Looking at implementation, this should also register those layers, so I don’t separately register those.</p>
<p>But somehow, this won’t work when I use torch::save and torch::load to store the neural network. I.e. the linear submodules contained “layers” won’t load.</p>
<p>Am I missing something here?</p>

Willem: <p>So it works when I initialize the module with the right number of elements in “layers”, but that kind of defeats the point of saving…</p>
How are unfold and fold implemented?

liao123: <p>I want to refer to fold and unfold in torch and use numpy to implement unfold and fold operations.</p>
<p>When I followed the torch source code, I found: torch/nn/modules/fold.py===&gt;&gt;torch/nn/functional.py===&gt;&gt;torch/nn/_functions/thnn/fold.py<br>
ctx._backend.Im2Col_updateOutput(ctx._backend.library_state,<br>
input, output,<br>
kernel_size[0], kernel_size[1],<br>
dilation[0], dilation[1],<br>
padding[0], padding[1],<br>
stride[0], stride[1])</p>
<p>This seems to be the implemented by C,  and I can’t check it.</p>
<p>So I am curious about how to implement the unfold and fold?</p>
<p>When I read other people’s implementation, im2col was performed through multiple for loops sliding</p>
<p><strong>I want to speed up the  im2col by matrix multiplication</strong>, like AXB where X is the input and A and B is auxiliary matrixs. Hope you can give me some suggestions</p>

ptrblck: <p>You can find the CPU implementation <a href="https://github.com/pytorch/pytorch/blob/4b1d044498f49ad5f770354174070a4277279c2d/aten/src/ATen/native/im2col.h#L13-L51">here</a> and the CUDA implementation <a href="https://github.com/pytorch/pytorch/blob/46b3fc032aad4ff71209aea287e9258d5c1256e4/aten/src/ATen/native/cuda/im2col.cuh#L24-L67">here</a>.</p>
Torch.cuda.is_available() is false after CUDA 11.3.109 installed. Could anyone help me with this? Thanks!

tsuijenk: <p>Hello,</p>
<p>I believe that I have PyTorch, CUDA, cudatoolkit, nvcc and everything else installed and most up-to-date, but I am still getting torch.cuda.is_available() = False. I appreciate any advice! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:37:49) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1050
Nvidia driver version: 465.89
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.4
[pip3] pytorch-lightning==1.4.9
[pip3] torch==1.10.1
[pip3] torchinfo==1.5.4
[pip3] torchmetrics==0.6.2
[pip3] torchvision==0.11.2
[conda] blas                      1.0                         mkl
[conda] blas-devel                3.9.0              12_win64_mkl    conda-forge
[conda] cudatoolkit               11.3.1               h280eb24_9    conda-forge
[conda] libblas                   3.9.0              12_win64_mkl    conda-forge
[conda] libcblas                  3.9.0              12_win64_mkl    conda-forge
[conda] liblapack                 3.9.0              12_win64_mkl    conda-forge
[conda] liblapacke                3.9.0              12_win64_mkl    conda-forge
[conda] mkl                       2021.4.0           h0e2418a_729    conda-forge
[conda] mkl-devel                 2021.4.0           h57928b3_730    conda-forge
[conda] mkl-include               2021.4.0           h0e2418a_729    conda-forge
[conda] mkl-service               2.4.0            py37h2bbff1b_0
[conda] numpy                     1.21.4           py37h940b05c_0    conda-forge
[conda] pytorch                   1.10.1          py3.7_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-lightning         1.4.9              pyhd8ed1ab_0    conda-forge
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchinfo                 1.5.4              pyhd8ed1ab_0    conda-forge
[conda] torchmetrics              0.6.2              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.11.2               py37_cu113    pytorch

(env) PS C:\Users\tsuijenk&gt; nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:41:42_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0
</code></pre>
Batching a nested list of graphs in torch_geometric

tangolin: <p>I am currently training a model which is a mix of graph neural networks and LSTM. However that means for each of my training sample, I need to pass in a list of graphs. The current batch class in torch_geometric supports batching with <code>torch_geometric.data.Batch.from_data_list()</code> but this only allows one graph for each data point. How else can I go about batching the graphs when each data consist of multiple graphs?</p>

ejguan: <p>Do you want to post the question/request to torch_geometric repo? We are not familiar with that.</p>

tangolin: <p>ah I didn’t think of that, thanks for the suggestion!</p>
Binary classification shows 99% for totally different input

Sherzod_Bek: <p>Hi,<br>
I did binary classification(dog&amp;cat) output is between 0 and 1. It works perfectly with two classes.<br>
But when I test totally different image(human, nature, etc)  output is closer to 1(class1) or closer to 0(class2).</p>
<p>I think it should show around 0.5.   because image doesn’t look like any class.</p>
<p>could you explain reason? is there something should I know more about classification?</p>

KFrank: <p>Hi Sherzod!</p>
<aside class="quote no-group" data-username="Sherzod_Bek" data-post="1" data-topic="139964" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sherzod_bek/48/13945_2.png" class="avatar"> Sherzod_Bek:</div>
<blockquote>
<p>I think it should show around 0.5.   because image doesn’t look like any class.</p>
</blockquote>
</aside>
<p>Here’s my speculative way of looking at this:</p>
<p>You train your classifier to predict one of two classes.  When it gets it<br>
right, it gets rewarded for predicting that class with high certainty, so<br>
there is some bias towards making high-certainty predictions.  Because<br>
you <em>don’t</em> train with not-cat, not-dog images, there is no penalty for<br>
predicting a not-cat, not-dog image – say an image of a person with a<br>
vaguely dog-like face – with high certainty as a dog.</p>
<p>If your actual use case is that at inference time your model will be presented<br>
with cat, dog, and neither images and it’s important that you not predict<br>
the neither images as cat or dog, then you need train you model to get<br>
the neither images right.</p>
<p>So, in this case, you would want to train a three-class classifier (rather<br>
than a binary classifier), where your three classes would be “cat,” “dog,”<br>
and “everything else.”  And your training would include not only “cat”<br>
and “dog” samples, but “everything else” (human, nature, etc.) samples,<br>
as well.</p>
<p>If such a model trains well – and there’s no reason it shouldn’t – when<br>
presented with an “everything else” image, it should predict “everything<br>
else” with a probability close to one, and correspondingly, with probabilities<br>
for “cat” and “dog” close to zero.</p>
<p>Best.</p>
<p>K. Frank</p>

Sherzod_Bek: <p>Thanks. I will try to use one more class.</p>
If I save a model in PyTorch is there way to load it into Libtorch, and vice versa?

Alio: <p>I have a project that must use Libtorch to train a model for one portion.<br>
If I could use PyTorch for the other portion it would make things easier.</p>
<p>If I have a model saved in the C++ version of torch can I get the model loaded in Python’s version (Pytorch, of course) and can I do the opposite? Load a model from Pytorch into the C++ version Libtorch?</p>
<p>I know I should try, and find this out for myself. It’s taking me a while to get to know Libtorch, so maybe someone can get me a faster answer here. Maybe the models are usable one way but not the other. Maybe not at all. Maybe yes to all.</p>
Torchtext.vocab Vocab constructor error- no keyword specials

markmorr: <p>Hi,<br>
I am trying to go through the tutorial on langauge translation at <a href="https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with TorchText — PyTorch Tutorials 1.7.1 documentation</a><br>
Running the first cell in the tutorial produces an error at this line:<br>
return Vocab(counter, specials=[’&lt;unk&gt;’, ‘&lt;pad&gt;’, ‘&lt;bos&gt;’, ‘&lt;eos&gt;’])<br>
<strong>init</strong>() got an unexpected keyword argument ‘specials’</p>
<p>I have seen the answer here but according to the answer my version is high enough:<br>
torchtext: 1.11.0<br>
torch: 1.10.0<br>
python: 3.7</p>
<p>I also get another error later in the tutorial (this line PAD_IDX = en_vocab.stoi[’&lt;pad&gt;’] results in the error AttributeError: ‘Vocab’ object has no attribute ‘stoi’) so I think it may be a version issue but I can’t figure out what versions will work. Thank you for any help!</p>

ptrblck: <p>It seems the <code>specials</code> argument was removed in <a href="https://github.com/pytorch/text/pull/891">this PR</a>.<br>
The new usage seems to be shown in e.g. <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html">this tutorial</a>.</p>

markmorr: <p>Thank you! You are correct. Using the new tutorial now.</p>
Import SummaryWriter gives AttributeError: AttributeError: module 'setuptools._distutils' has no attribute 'version'

Sycor4x: <p>I’m using PyTorch on a MacBook with an Apple M1 processor. I am using miniforge to install ARM PyTorch and ARM Tensorflow.</p>
<p>As far as I can tell, everything works except for Tensorboard. For instance, if I refer to the tutorial at <a href="https://pytorch.org/docs/stable/tensorboard.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.tensorboard — PyTorch 1.10.1 documentation</a> I am stopped at the third line:</p>
<p><code>from torch.utils.tensorboard import SummaryWriter</code></p>
<p>raises an error:</p>
<p><code> AttributeError: module 'setuptools._distutils' has no attribute 'version'</code>.</p>
<p>These are the versions I’m using:</p>
<pre><code class="lang-auto">python                    3.9.9
tensorboard               2.6.0
pytorch                   1.10.0
tensorflow-deps           2.7.0
tensorflow-estimator      2.7.0
tensorflow-macos          2.7.0
tensorflow-metal          0.3.0
</code></pre>
<p>Is this a known issue? And is there a workaround?</p>

ptrblck: <p>Yes, this was a known issue which was already fixed in <a href="https://github.com/pytorch/pytorch/pull/69904">this PR</a>. You could thus either update PyTorch to the nightly release or downgrade <code>setuptools==59.5.0</code> as a workaround.</p>

Sycor4x: <p>Is there a way to update without building from source for ARM?</p>
<p>As an alternative, I just monkey-patched by replacing the old <code>torch/utils/tensorboard/__init__.py</code> file with the new one. It works fine.</p>
On SubsetRandomSampler(indices)

hitbuyi: <p>on torch.utils.data.sampler.SubsetRandomSampler(),  if I want to choose 5 indices out of 100 randomly,  is it right?<br>
torch.utils.data.sampler.SubsetRandomSampler(idx, 5)<br>
where  len(idx) = 100</p>
<p>in tutorial of Pytorch, I can’t find using it at this way, why?</p>

ptrblck: <p>No, your usage is wrong and won’t chose 5 out of 100 indices.<br>
As described in the <a href="https://pytorch.org/docs/stable/data.html?highlight=subsetrandomsampler#torch.utils.data.SubsetRandomSampler">docs</a>:</p>
<blockquote>
<p>Samples elements randomly from a given list of indices, without replacement.</p>
</blockquote>
<p>which means you are supposed to pass the indices the sampler should randomly sample from as the first argument.<br>
E.g. if your original index is defined as <code>idx = torch.arange(100)</code>, you could select the desired 5 indices and pass it to the sampler: <code>SubsetRandomSampler(idx[:5])</code>.<br>
If you want to randomly pick these 5 indices, you could use:</p>
<pre data-code-wrap="python"><code class="lang-python">idx_subset = idx[torch.randperm(len(idx))[:5]]
</code></pre>

hitbuyi: <p>so, the name of this function  is kinda ambiguous,  where is the meaning of ramdom of this function itself?<br>
and where is the meaning of “subset” this function itself?<br>
e.g.<br>
if  idx = [1,2,3,4,5]<br>
torch.utils.data.sampler.SubsetRandomSampler(idx, 100)<br>
what’s sequeece of after call this function?<br>
it looks like  idx =[4,5,3,1,2], or [3,2,1,5,4] ramdomly?</p>

ptrblck: <p>Your won’t work, since the second argument is expected to be a <code>generator</code> as given in the docs.<br>
If you remove it,</p>
<pre data-code-wrap="python"><code class="lang-python">for i in sampler:
    print(i)
</code></pre>
<p>will return a random sequence of the passed indices.</p>
<aside class="quote no-group" data-username="hitbuyi" data-post="3" data-topic="139991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ee59a6/48.png" class="avatar"> hitbuyi:</div>
<blockquote>
<p>where is the meaning of ramdom of this function</p>
</blockquote>
</aside>
<p>The passed indices will be randomly sampled.</p>
<aside class="quote no-group" data-username="hitbuyi" data-post="3" data-topic="139991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ee59a6/48.png" class="avatar"> hitbuyi:</div>
<blockquote>
<p>and where is the meaning of “subset” this function itself?</p>
</blockquote>
</aside>
<p>You are able to pass the indices of a subset to the sampler instead of a data source.</p>
Learning rate scheduling at each step

Torcione: <p>Hi everyone,<br>
I want to modify the value of my learning rate <strong>at each step</strong> instead of doing it at the end of each epoch.<br>
In particular at each step of my training I compute a generic measure <strong>x</strong> and I want to modify the learning rate for the next step as a function of <strong>x</strong>.<br>
Just to give a more explicit idea of the pipeline, given the constant <strong>C</strong> what I want to do is something like:</p>
<pre><code class="lang-auto">for epoch in range (0, n_epochs):
    for data, label in train_loader:
        #training
        #computation of x
        #modify of the learning rate as a function of x (for example lr = C*(1+x))

</code></pre>
<p>Till now I’ve only used <code>lr_schedule = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda = lr_rule) </code> to modify the learning rate value.<br>
Is there a more flexible method in pytorch to modify it at each step and with a generic function of a generic computed measure?</p>

ptrblck: <p>I’m not sure if there is a more flexible class for your use case, but I guess you might skip using the scheduler and update the learning rate manually as you wish via:</p>
<pre><code class="lang-python">for param_group in self.optimizer.param_groups:
    param_group['lr'] = lr
</code></pre>
Second initialisation of bias parameter has no effect on return value of forward on GPU, but does on CPU

JakobHavtorn: <p>In trying to implement data-dependent initialisation for weight-normalisation, I’ve run into a confusing behaviour with parameter initialization.</p>
<p>I’ve boiled it down to a minimal example. Let’s say that for a linear layer I do the following:</p>
<ol>
<li>Initialize the bias to 0.0.</li>
<li>Run the first forward pass.</li>
<li>Initialize the bias to 1.0.</li>
<li>Run the second forward pass.</li>
</ol>
<p>I would then expect to see the average value of the second forward pass be exactly 1 larger than the one of the first.</p>
<p>On CPU, this is indeed the case. On GPU however, I get the same output in both forward passes, regardless of the value of the bias. You can inspect the module to see that the bias parameter has indeed been updated and is indeed located on the GPU.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


B = 2
C = 2
I = 256
O = 16

x = torch.randn(B, C, I)

module = nn.Linear(I, O)

# On CPU, changing the bias affects the output.
# On GPU, changing the bias does not affect the output.
for device in [torch.device("cpu"), torch.device("cuda:0")]:

    print(device)

    x = x.to(device)
    module = module.to(device)

    # Set bias to 0.0
    nn.init.constant_(module.bias, 0.0)

    o = module(x)
    print(o.mean().item(), o.std().item())

    # Set bias to 1.0
    nn.init.constant_(module.bias, 1.0)

    o = module(x)
    print(o.mean().item(), o.std().item())
</code></pre>

ptrblck: <p>I can’t reproduce the issue and get the expected results:</p>
<pre><code class="lang-python">cpu
-0.06411010026931763 0.5944567918777466
0.9358899593353271 0.5944567918777466
cuda:0
-0.06415127217769623 0.5944625735282898
0.9358487129211426 0.5944625735282898
</code></pre>
How to train LSTM Auto Encoder with multiple independent timeseries?

fafaw10110: <p>Hi,<br>
I’d like to build lstm autoencoder for roads. I read this <a href="https://curiousily.com/posts/time-series-anomaly-detection-using-lstm-autoencoder-with-pytorch-in-python/" rel="noopener nofollow ugc">article</a>, and it is quite clear to build encoder-decoder for one road. However, my questions are (1) is it possible to pass all roads data to the encoder?. I don’t want to do one road at the time, but do it for each road for each batch while training the model. I am thinking  of reshaping data to (1) include road id, or (2) combine all the times regardless of road id, for the example below (feature_1, 01:00:00 → road 1 (0.23), road 2 (0,13))?.</p>
<p>I have data that looks like the following. For each road (id), we have data for each 2 minutes, for two features.</p>
<pre><code class="lang-auto">|road_id|timestamp|feature_1|feature_2
| --- | --- | --- | --- | --- |
|1|2020-12-20 01:00:00|0.23|0.1
|1|2020-12-20 01:02:00|0.3|0.12
|1|2020-12-20 01:04:00|0.3|0.12
|2|2020-12-20 01:00:00|0.13|0.2
|2|2020-12-20 01:02:00|0.2|0.4
|2|2020-12-20 01:04:00|0.3|0.13
</code></pre>
<p>This is the encoder for one road, based on the link shared earlier.</p>
<pre><code class="lang-auto">class LSTMEncoder(nn.Module):
    def __init__(self, input_size,hidden_size,num_layers):
        super(LSTMEncoder,self).__init__()
        self.n_features = n_features
        self.hidden_size = hidden_size
        self.n_layers = n_layers
        self.device = device

        self.encoder = nn.LSTM(
            self.n_features,
            self.hidden_size,
            batch_first=True,
            num_layers=self.n_layers,
            bias=True,
        )


    def forward(self, x):
        enc_hidden = self.init_hidden_state(x.shape[0])
        _, enc_hidden = self.encoder(x.float(), enc_hidden)
       return enc_hidden

</code></pre>
TBPTT through the product of two parameters

B_W: <p>Hello,<br>
This is my first question and I wanted to say thanks for all the good answers I have found here.</p>
<p>I am having trouble backpropagating  through an activation I am testing.</p>
<pre><code>    self.a = Parameter(torch.mul(torch.ones(1,N),.5))
    self.g = Parameter(torch.rand(1,N))
    self.a_list = []
    self.g_list = []
    
def forward(self, x, z):
    return (1-self.a)*x + self.a*self.g*torch.tanh(x+z)
</code></pre>
<p>I would like to learn parameters a and g however the product of self.a and self.g is an issue during backprop since self.a is changed in place before the new self.g is calculated.</p>
<p>Inputs x and z are outputs from separate linear layers.</p>
<p>I have attempted different cloning schemes for these parameters among other things.</p>
<p>As of now (different from the lines above),  I am treating them as states similar to LSTM’s “hidden” and passing it through the full network repeatedly until calulating loss on the last prediction.</p>
<p>Is there better way to do this? Also, aside from using make_dot() is there a tool to see how the graph deletes  during backprop. From what I have seen it looks like the list and deletions form from left to right and this is why I believe self.a is being changed first.</p>
<p>Thanks for your time!</p>

B_W: <p>Going through the autodiff tutorials and other online discussions I came up with a function which specifically handles the multiplication of the two learnable parameters and returns the partial derivative with respect to each.</p>
<p>class Pmul(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, a, g):<br>
ctx.save_for_backward(a,g)<br>
return torch.mul(a,g)</p>
<pre><code>@staticmethod
def backward(ctx, grad_output):
    a,g = ctx.saved_tensors
    da = grad_output*g 
    dg = grad_output*a
    return da, dg
</code></pre>
<p>With this I alias as pmul = Pmul.apply and then replace,</p>
<p>…+  self.a * self.g * torch.tanh(…)<br>
with<br>
…+ pmul(self.a,self.g) * torch.tanh(…)</p>
<p>I am testing it more but it seems to work.  Any feedback is much appreciated.</p>
<p>Thanks in advance!</p>
Does torch.utils.data.random_split split every class equally?

shashi: <p>So I have a directory with subdirectories, each subdirectory is a class.</p>
<pre><code class="lang-auto">+Directory
--+class1
--+class2 
... etc
</code></pre>
<p>If I load them using torchvision.datasets.ImageFolder, and then split into training and testing like,</p>
<pre><code class="lang-auto">train_size = int(0.8*len(dataset_total))
test_size = len(dataset_total) - train_size
train_dataset,test_dataset = torch.utils.data.random_split(dataset_total,[train_size,test_size])
</code></pre>
<p>and then make a train_loader and test_loader for the train_dataset and test_dataset.<br>
Could I expect an equal division of data from all classes? What is the best practice here to send data to network?</p>
<p>Thanks!</p>

serser: <p>It is less likely to evenly split the dataset into two. It simply creates a list of permuted indices to sample the dataset into subsets, which doesn’t care about the classes. Say if random falls back to sequential, other classes are doom lost.<br>
Check <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#Subset" rel="nofollow noopener">link</a> for original code.</p>

Vikram_M: <p>So how to split it into equal classes using torch.utils.data.random_split</p>

Thytu: <p>Personally I ended up using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" rel="noopener nofollow ugc">train_test_split</a> from sklearn.</p>
Bug? torch.cdist gives similar results for different p values

izzatum: <p>Hi!</p>
<p>I discovered that <code>torch.cdist(x1, x2, p=2.0, compute_mode='use_mm_for_euclid_dist_if_necessary')</code> function keeps giving similar results even after I changed the <code>p</code> values. Does it means there is a bug in the code?</p>
<p>Thanks!</p>
<p>Here are the code snippets:</p>
<pre><code class="lang-auto">x = torch.linspace(0.0, 1.0, 2500).reshape(-1,1)
cdist = torch.cdist(x,x,p=2.0)
print(cdist)

tensor([[0.0000e+00, 4.0016e-04, 8.0032e-04,  ..., 9.9920e-01, 9.9960e-01,
         1.0000e+00],
        [4.0016e-04, 0.0000e+00, 4.0016e-04,  ..., 9.9880e-01, 9.9920e-01,
         9.9960e-01],
        [8.0032e-04, 4.0016e-04, 0.0000e+00,  ..., 9.9840e-01, 9.9880e-01,
         9.9920e-01],
        ...,
        [9.9920e-01, 9.9880e-01, 9.9840e-01,  ..., 0.0000e+00, 4.8828e-04,
         8.0972e-04],
        [9.9960e-01, 9.9920e-01, 9.9880e-01,  ..., 4.8828e-04, 0.0000e+00,
         4.2286e-04],
        [1.0000e+00, 9.9960e-01, 9.9920e-01,  ..., 8.0972e-04, 4.2286e-04,
         0.0000e+00]])

cdist = torch.cdist(x,x,p=1.0)
print(cdist)

tensor([[0.0000e+00, 4.0016e-04, 8.0032e-04,  ..., 9.9920e-01, 9.9960e-01,
         1.0000e+00],
        [4.0016e-04, 0.0000e+00, 4.0016e-04,  ..., 9.9880e-01, 9.9920e-01,
         9.9960e-01],
        [8.0032e-04, 4.0016e-04, 0.0000e+00,  ..., 9.9840e-01, 9.9880e-01,
         9.9920e-01],
        ...,
        [9.9920e-01, 9.9880e-01, 9.9840e-01,  ..., 0.0000e+00, 4.8828e-04,
         8.0972e-04],
        [9.9960e-01, 9.9920e-01, 9.9880e-01,  ..., 4.8828e-04, 0.0000e+00,
         4.2286e-04],
        [1.0000e+00, 9.9960e-01, 9.9920e-01,  ..., 8.0972e-04, 4.2286e-04,
         0.0000e+00]])

cdist = torch.cdist(x,x,p=float('inf'))
print(cdist)

tensor([[0.0000e+00, 4.0016e-04, 8.0032e-04,  ..., 9.9920e-01, 9.9960e-01,
         1.0000e+00],
        [4.0016e-04, 0.0000e+00, 4.0016e-04,  ..., 9.9880e-01, 9.9920e-01,
         9.9960e-01],
        [8.0032e-04, 4.0016e-04, 0.0000e+00,  ..., 9.9840e-01, 9.9880e-01,
         9.9920e-01],
        ...,
        [9.9920e-01, 9.9880e-01, 9.9840e-01,  ..., 0.0000e+00, 4.8828e-04,
         8.0972e-04],
        [9.9960e-01, 9.9920e-01, 9.9880e-01,  ..., 4.8828e-04, 0.0000e+00,
         4.2286e-04],
        [1.0000e+00, 9.9960e-01, 9.9920e-01,  ..., 8.0972e-04, 4.2286e-04,
         0.0000e+00]])
</code></pre>

tom: <p>For 1 element vectors, the norms for different p are all the absolute value, so you are seeing numerical differences of computing thesamw result.</p>
<p>Best regards</p>
<p>Thomas</p>

izzatum: <p>Hi <a class="mention" href="/u/tom">@tom</a>! Great thanks for your explanation! I was wondering why the results are different when I compute the same distance using <code>((x[:, None, :] - x[:, :, None])**2).sum(-1)</code>. Perhaps, there is something that I don’t really understand.</p>

tom: <p>I think you want a the None in the first and second position and a square toot at the end.</p>