akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How should the padding mask be expanded in tthe implementation of a transformer from scratch?

FeryET: <p>Hi.</p>
<p>I’m implementing a transformer from scratch and I was wondering how the padding mask should be generated. Should it make the generated attention vector in ScaledDotProduct Attention (the softmax of QK^T) filled with NaNs in only the columns or both the rows and columns? What is the correct approach here?</p>
Does Pytorch do garbage collection for modules whose parameters were registered and added to an optimizer but who are no longer referenced anywhere?

Sam_Lerman: <p>Are those parameters still tracked?</p>

ptrblck: <p>Unreferenced tensors should be freed by default. Are you seeing any issues with it? If so, did you make sure to delete all references?</p>

Sam_Lerman: <p>Not seeing any issues. However, it’s a whole Module (well, a sub-module, whose parameters are added to the super-module’s registered parameter’s and the optimizer’s param_group, but is then later deleted), not a tensor.</p>
torch.nn.Upsample layer slow in forwards pass

felixems: <p>I’m running a version of StarGAN where the ConvTranspose2d layers which up-sample the signal have been replaced by a resize convolution as described below. For some reason, the forwards pass through the Upsample layers are very slow in comparison to the rest of the network, which is surprising to me, considering that the process doesn’t depend on any matrices of parameters.</p>
<pre><code class="lang-auto">layers.append(nn.ConvTranspose2d(curr_dim, curr_dim//2, kernel_size=4, stride=2, padding=1, bias=False))
</code></pre>
<p>is replaced by</p>
<pre><code class="lang-auto">layers.append(nn.Upsample(scale_factor=2, mode='nearest'))
layers.append(nn.ReflectionPad2d(1))
layers.append(nn.Conv2d(curr_dim, curr_dim // 2, kernel_size=3, stride=1, padding=0, bias=False))
</code></pre>
<p>The first pass through the Upsample layer has an input size of (256, 256, 256) and takes roughly 9.5s.<br>
The second pass has an input size of (128, 512, 512) and takes roughly 17.5s.<br>
When the same up-sampling is performed by a script I wrote using NumPy, it takes 0.3s and 0.5s respectively.</p>
<p>A colleague suggested that this could be caused by the fact that a GPU is designed to process images with fewer channels than the signals processed in this network, but I find this unsatisfying as the rest of the layers in the network run in a fraction of a second. I’m wondering if anyone else has any thoughts as to the cause?</p>

tom: <p>Are you sure you’re measuring that right? I’m asking because most people don’t.<br>
For integral scaling, it can be faster (and with deterministic backward and all) to use array operations. You might split this into several lines, but to give you an idea, this would upscale a BCHW inp by 2x:</p>
<pre><code class="lang-python">out = inp[:,:, :,None, :, None].expand(-1, -1, -1, 2, -1, 2).reshape(inp.size(0), inp.size(1), 2 * inp.size(2), 2* inp.size(3))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

felixems: <p>I’m not totally sure that I am measuring the times correctly, but the reality was that the network was taking too long to run with the resize convolution, compared to the original network with the transpose convolution.</p>
<p>Nevertheless, you’re absolutely right, the array operation method is vastly more efficient. The time for a forwards pass through the network decreases from ~65s to ~40s by up-sampling your way.</p>

Jide_Olabisi: <p>True, upsampling in pytoch is very slow.</p>
How to get determistic behavior with different GPUs?

Kalle: <p>I have 3 different GPUs available and do not experience deterministic behavior. My GPUs are:</p>
<p>GeForce GTX 1080 Ti<br>
GeForce RTX 2080 Ti<br>
Tesla P100-PCIE-16GB</p>
<p>Im using PyTorch version 1.3.1 and currently using the following settings to try to achieve the same reproducible result:</p>
<pre><code>seed = 3
torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.cuda.manual_seed_all(seed)
random.seed(seed)
np.random.seed(seed)
</code></pre>
<p>I run resnet32 and running 3 different experiments for 2 epochs and predicting with the trained model. I then get the following results:</p>
<p>GeForce GTX 1080 Ti   : 0.33333<br>
GeForce RTX 2080 Ti   : 0.20000<br>
Tesla P100-PCIE-16GB : 0.33333</p>
<p>I dont really know why I achieve the same results on the GTX 1080 and Tesla P100 and not with the RTX 2080?</p>
<p>Is there something I am missing?</p>

ptrblck: <p>Determinism cannot be guaranteed between different hardware families, so you cannot expect to get bitwise accurate results between all three devices.<br>
You could update to the latest PyTorch release (<code>1.9.0</code>) and set all settings  as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> to get deterministic behavior between different runs on the same device.</p>
'str' object has no attribute 'to'

Hasan_Khan: <p>Error while loading waveglow model</p>
<pre><code class="lang-auto">waveglow_path = '/home/ec2-user/SageMaker/waveglow'
waveglow = torch.load(waveglow_path)
#waveglow.load_state_dict(torch.load(waveglow_path))
waveglow = waveglow_path.to('cuda')
waveglow.eval()
#waveglow.cpu().eval()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c721f1a51db869553f3c2b89de615ef2527d3a4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/c721f1a51db869553f3c2b89de615ef2527d3a4d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c721f1a51db869553f3c2b89de615ef2527d3a4d.png" alt="image" data-base62-sha1="spBJWXyRi3bmMWuLnQLFmtPvTR3" width="690" height="263" data-dominant-color="F7D6D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">715×273 9.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are trying to push the <code>waveglow_path</code> to the device, which is just the string defining the PATH to a directory, so replace it with <code>waveglow.to('cuda')</code>.</p>

Hasan_Khan: <p>oh yes. That was too silly of me. Well, after applying the changes I’m getting this error.</p>
<pre><code class="lang-auto">'dict' object has no attribute 'to'
</code></pre>

ptrblck: <p><code>torch.load(waveglow_path)</code> seems to load a <code>state_dict</code>, so you would need to create the model instance and load this <code>state_dict</code> afterwards.</p>
<pre><code class="lang-python">model = WaveGlow() # or whatever class name is used for the actual model
state_dict = torch.load(waveglow_path)
model.load_state_dict(state_dict)
</code></pre>
Newton-Method / Multidimensional autograd

JosAr: <p>Hi there,<br>
For my problem I try to minimize a functional (like a loss function) L: C → R over a Neural Network N which is acting on some data x in X.<br>
So i seek to minimize L(N(x)) for many x by adjusting the parameters of N.<br>
With all sorts of SGD / ADAM my results were not really good so I tried using <code>torch.optim.LBFGS</code> but this is not working as I want it to (max_eval is not always reached even for very very small tolerances and the result is far away from the solution), so i would like to implement the Newton-Method on my own.</p>
<p>To do this i would like to update the Networks weight on my own:</p>
<pre><code class="lang-auto">loss = LossFunctional(model, data)
loss.backward()
with torch.no_grad():
    for p in value_function.parameters():
        p -= torch.matmul(torch.linalg.inv(p.grad), loss)
</code></pre>
<p>But: this is not working if data is in batch-format because backward() is only allowed for scalar outputs. Therefore also inverting p.grad obviously does not work since its not a square matrix. So my question is:</p>
<p>How can I achieve to get a Jacobian for multiple batches?  Is this possible? Is there any other better implementation of a Newton-method solver then doing it by hand or using LBFGS?</p>
Torch Tensorboard graph not reflecting code

3nomis: <p>I built a video encoder made up of a simple combination of VGG + Linear layer</p>
<pre><code class="lang-auto">class Encoder(torch.nn.Module):
  def __init__(self, emb_dimension):
    super(Encoder, self).__init__()

    self.vgg16 = vgg16
    self.encoder = torch.nn.Sequential(
        torch.nn.Linear(4096, emb_dimension),
        # torch.nn.ReLU()
    )

  def forward(self, x):

    batch_size, time_steps, *dims = x.shape
    x = x.view(batch_size * time_steps, *dims)
    x = vgg16(x)
    x = x.view(batch_size, time_steps, -1)
    x = self.encoder(x)

    return x
</code></pre>
<p>I don’t get why the tensorBoard graph shows 20 tensors coming back from the sequential layer to VGG. I am afraid it might affect the gradient calculation.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb_2_517x373.png" alt="image" data-base62-sha1="w72c3XzLEnGri8VUfWd4mQoQpYv" width="517" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb_2_517x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb_2_775x559.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e11128cada9e9ec568d37a4ee55794e0ffd7b3eb_2_1034x746.png 2x" data-dominant-color="E8E0DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2056×1484 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Anyone has any idea?</p>
System reboot when training

kaka_zhao: <p>I am training a DDP model. However, the system always reboots.<br>
I use a machine with two RTX3090, 10900k CPU – Ubuntu18, CUDA11.0, Nvidia455.38, Pytorch1.7.1, and Python3.8.<br>
I also test my code on another machine with two 2080ti, it runs well.</p>

Yanli_Zhao: <p>it is hard to help without more details like code details and etc.</p>

pritamdamania87: <p><a class="mention" href="/u/kaka_zhao">@kaka_zhao</a> Regarding system reboots this could be a system issue rather than a PyTorch issue. Probably check system logs like <code>/var/log/messages</code> to see why the system rebooted.</p>

kaka_zhao: <p>I think this is a hardware issue. I use a 1200w PSU for two RTX3090.<br>
After I limited the GPU power from 350w to 250w by <code>nvidia-smi -i 0,1 -pl 250</code>, the problem has gone away!</p>

Alpha: <p>Limiting GPU power works for me.</p>
<p>A detailed instruction:</p>
<pre><code class="lang-auto"># enable persistence mode
sudo nvidia-smi -pm 1

# limite power from 350W to 250W
sudo nvidia-smi -i 0,1,...,3 -pl 250
</code></pre>
Does loss.backword() work if I filter the tensor manually?

EasonC13: <p>Hi, I’m working on an NER model with multi label on one word. So I choose <code>BCELoss</code> instead of <code>CrossEntropyLoss</code>.</p>
<p>While <code>CrossEntropyLoss</code> have <code>ignore_index</code> feature, so it will ignore the part where attention_mask is 0.</p>
<p>In BCELoss I only able to do it via manual filter unwanted part like the following code.</p>
<pre><code class="lang-auto">loss_fct = torch.nn.BCELoss()
# Only keep active parts of the loss
if attention_mask is not None:
	active_logits = logits.view(-1, self.num_labels)[attention_mask.view(-1)== 1]
	active_labels = labels.view(-1, self.num_labels)[attention_mask.view(-1)== 1]
	loss = loss_fct(active_logits, active_labels)
</code></pre>
<p>Now the model has train successfully. Loss goes down, evaluate goes well.</p>
<p>While I am curious to know that does I manually filter output with attention_mask affect the loss.backword()? And Why?</p>
<p>Thank you.</p>
<p>edit:</p>
<ol>
<li>the full code is at <a href="https://github.com/accordproject/labs-cicero-classify/blob/dev/Practice/PyTorch/BERT_practice/BERT_BASE_Custom_model.ipynb" rel="noopener nofollow ugc">here</a>
</li>
<li>I see there is a attribute call <code>grad_fn</code> and the loss output after <code>loss_fct</code>, it’s <code>loss.grad_fn</code> is an <code>index_select</code> object. So when I call the loss backword, it will only affect those which index been selected?</li>
</ol>

ptrblck: <p>Yes, indexing the model output and target should work and the gradient would be backpropagated to the selected values:</p>
<pre><code class="lang-python">output = torch.randn(10, 1, requires_grad=True)
target = torch.randint(0, 2, (10, 1)).float()
criterion = nn.BCEWithLogitsLoss()

batch_idx = torch.tensor([1, 3, 5, 7])
loss = criterion(output[batch_idx], target[batch_idx])
loss.backward()

print(output.grad)
&gt; tensor([[ 0.0000],
          [-0.1686],
          [ 0.0000],
          [ 0.1138],
          [ 0.0000],
          [ 0.2076],
          [ 0.0000],
          [-0.1147],
          [ 0.0000],
          [ 0.0000]])
</code></pre>

EasonC13: <p>Thanks for your reply! That is clear and easy to understand <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Pytorch.hub.load

Hasan_Khan: <p>I’m downloading my waveglow model using:</p>
<pre><code class="lang-auto">torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_waveglow')
</code></pre>
<p>but what if I want to directly download it locally on my pc or into the s3 bucket. What would be the procedure please help.</p>

ptrblck: <p>The code should be stored in the default cache directory, which can be changed via <code>torch.hub.set_dir</code>.<br>
Alternatively to changing the cache directory, you could also directly <code>git clone</code> the WaveGlow repository from <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/c481324031ecf0f70f8939516c02e16cac60446d/PyTorch/SpeechSynthesis/Tacotron2">GitHub</a>.</p>
Get RuntimeError in torch.onnx.export

rabbit_hog: <p>I tried exporting pretained MobileNetV3 and got RuntimeError:</p>
<p>RuntimeError: Exporting the operator hardsigmoid to ONNX opset version 9 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.</p>
<p>So how to export hardsigmoid to onnx? Thanks.</p>
Early Stopping in Deep metric learning

Bhavin_Jawade: <p>I am training a network using ArcFace loss. My eventual goal is to learn robust embeddings that can differentiate similar and dissimilar images (even for unseen classes).<br>
As this is a verification kind of task, but training ArcFace is a classification task, I am confused about how should I early stop on validation dataset because my validation dataset has unseen classes because validation tasks is of verification. Without early stopping ArcFace loss network overfits on training classification data and gives very poor performance on validation verification task.</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="Bhavin_Jawade" data-post="1" data-topic="124086">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bhavin_jawade/48/39312_2.png" class="avatar"> Bhavin_Jawade:</div>
<blockquote>
<p>Without early stopping ArcFace loss network overfits on training classification data and gives very poor performance on validation verification task.</p>
</blockquote>
</aside>
<p>This seems to give the answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Based on this statement it seems that you can (somehow) quantify that the model is overfitting and gives poor performance on the validation set. Assuming you can measure this overfitting as well as the performance on the training and validation set, you could use this metric also for early stopping.</p>

Bhavin_Jawade: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. You were right, instead of using a traditional sample wise train-test split, I reserved a few set of classes for validation from my training set, and for validation sampled pairs from this validation set and computed the pair-wise matching accuracy.</p>
Performing custom transformations on model output before computing loss

deepanshu: <p>I am working with an autoencoder model which takes an input image and reconstructs it for the output. I am using MSELoss here. I want to perform certain transformations on the model output (matrix multiplication with some other matrix, addition, then matrix multiplication again) before computing the loss on it. I tried implementing this but got the error:</p>
<p>one of the variables needed for gradient computation has been modified by an inplace operation.</p>
<p>Is there a workaround to perform these operations on the output?</p>

ptrblck: <p>Based on the error message it seems you are manipulating a tensor inplace.<br>
Could you check, if you are using inplace operations (e.g. via <code>a += b</code>) and remove them?</p>

deepanshu: <p>Thank you! I tried removing them and made the operations store their result in a different variable. Now, when I pass the new variable to the loss and backpropagate, there is no gradient being passed through the network. The code looks something like this:</p>
<pre><code class="lang-auto">        output = model(input)

        results = torch.zeros((batch_size, 16384))


        # Applying the custom transformation (each sample in the batch
        # undergoes the same operation but with different parameters)

        for i in range (0, batch_size):
            results[i] = measurement(output[i], i)


        results = Variable(results, requires_grad = True)

        loss = criterion(results, target)
        

        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>Is there a reason why the flow of gradients might be getting interrupted?</p>

ptrblck: <p>Rewrapping a tensor as seen here:</p>
<pre><code class="lang-python">results = Variable(results, requires_grad = True)
</code></pre>
<p>will detach it from the computation graph, so remove this line of code.<br>
Also, <code>Variable</code>s are deprecated since PyTorch <code>0.4.0</code> so you shouldn’t use them anymore in any case.</p>
ToTensor()(image) arguments

Hananel: <p>Hi:)</p>
<p>I am trying to iterate over pictures from folder, using PIL.Image to open and resize them, and then torchvision.transforms.ToTensor() to save them as tensors.</p>
<p>On the ToTensor() command  I get this error:<br>
TypeError: <strong>array</strong>() takes 1 positional argument but 2 were given</p>
<pre><code class="lang-python"># read from given dir, resize to imageSizeXimageSize
def readImages(dir, imageSize): 
    # process one given raw image
    def processImg(rawImg):
        image = Image.open(rawImg)
        image.thumbnail((imageSize, imageSize), Image.ANTIALIAS)
        image = image.resize((imageSize, imageSize), Image.ANTIALIAS)
        image = torchvision.transforms.ToTensor()(image)
        return image

    images = []
    for rawImg in (Path(dir).glob('*.png')):
        images.append(processImg(rawImg))
    return images
</code></pre>
<p>someone help?</p>

eqy: <p>This looks like an issue that has recently surfaced; you might want to see if downgrading to PIL 8.2 works.<br>
Issues:<br>
<a href="https://github.com/pytorch/vision/issues/4146" rel="noopener nofollow ugc">transform toTensor fails with PIL image · Issue #4146 · pytorch/vision (github.com)</a><br>
<a href="https://github.com/python-pillow/Pillow/issues/5571" rel="noopener nofollow ugc">Pillow 8.3 and NumPy · Issue #5571 · python-pillow/Pillow (github.com)</a></p>

Hananel: <p>It works thanks very much!</p>
Why the hidden state is not multiplied with output weights in rnn?

Madhu_Varun: <p>Hi,</p>
<p>I have a simple rnn code below.</p>
<p>rnn = nn.RNN(1, 1, 1, bias = False, batch_first = True)<br>
t = torch.ones(size = (1, 2, 1))<br>
output, hidden = rnn(t)</p>
<p>My understanding from the documentation is that the output from above is the hidden state.</p>
<p>I tried to manually calculate the output using the below</p>
<p>hidden_state1 = torch.tanh(t[0][0] * rnn.weight_ih_l0)</p>
<p>hidden_state2 = torch.tanh(t[0][1] * rnn.weight_ih_l0 + hidden_state1 * rnn.weight_hh_l0)</p>
<p>The result was correct. hidden_state1 and hidden_state2 match the output.</p>
<p>Shouldn’t the hidden_states get multiplied with output weights to get the output?</p>
<p>I checked for weights connecting from hidden state to output. But there are no weights at all.</p>
<p>Is there anything I’m missing? Please help me.</p>
Using a model to filter gradients from a classifier

Adrian_Orenstein: <h3>
<a name="system-explanation-1" class="anchor" href="#system-explanation-1"></a>System explanation</h3>
<p>What I’m trying to accomplish is a system where you have two models; a classifier, and what can call a “gradient_model”.</p>
<p>The goal of the classifier is learn features from a training dataset as normal.</p>
<p>The goal of the gradient_model is to learn to filter/aggregate/transform the gradients from the classifier so that the classifier performs well on the validation dataset; explicitly without having any feature information from the validation set.</p>
<h3>
<a name="colab-2" class="anchor" href="#colab-2"></a>Colab</h3>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/15AOjVItLCaQywp5XjSfu94ikY_PmKixJ?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210629-060140-RC00_382053243" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/15AOjVItLCaQywp5XjSfu94ikY_PmKixJ?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/15AOjVItLCaQywp5XjSfu94ikY_PmKixJ?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<h3>
<a name="algorithm-explanation-3" class="anchor" href="#algorithm-explanation-3"></a>Algorithm explanation</h3>
<ol>
<li>Get train_batch</li>
<li>Compute loss for classifier on train_batch</li>
<li>Compute gradients w.r.t classifer weights and the loss for every environment<br>
This should result in a tensor of size [num_envs, …]</li>
<li>Update the classifier’s weights by applying the gradient model<br>
new_weight = old_weight - gradient_model(gradients).squeeze()<br>
The gradient_model converts the [num_envs, …] tensor to [1, …] which we then squeeze.</li>
<li>Get val_batch</li>
<li>Compute val_loss for new_classifier on val_batch</li>
<li>Inorder to update the gradient_model, <em>i think</em> we need to:<br>
a) compute the gradient_of the of the new model and val_loss<br>
b) as the new model depends on the gradient_model, we can then compute the gradient of the gradient_model and a)</li>
<li>Update the gradient model</li>
</ol>
<h3>
<a name="the-problem-4" class="anchor" href="#the-problem-4"></a>The problem</h3>
<p>In order for me to train the gradient model, I think I need to train it using the gradient from the updated weights of the classifier.</p>
<p>Using torchviz, I can see that the gradient breaks between step 4 where we created new classifier weights and step 5 where we want to use the new classifier weights on a new forward pass.</p>
<p>If possible, I would like the gradient tape to continue onto a new forward pass of the model.</p>
Difference between deterministic and nondeterministic operation?

tuttelikz: <p>Hi,</p>
<p>I was reading docs from <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">here</a> and came to this:</p>
<blockquote>
<p>Deterministic operations are often slower than nondeterministic operations, so single-run performance may decrease for your model.</p>
</blockquote>
<p>My question is, what is a definition of deterministic operation and/or the difference with its counterpart?</p>

ptrblck: <p>Deterministic operations will return the same result for each execution, which might be helpful for e.g. debugging. Algorithmic non-determinism can be caused by round-off errors, since the associative law no longer holds as described by <a class="mention" href="/u/kfrank">@KFrank</a> in <a href="https://discuss.pytorch.org/t/discrepancy-in-matmul-while-batching/125023/2">this post</a> (with some additional links).<br>
While seeing these numerical errors due to the limited floating point precision is usually not concerning, your use case might depend on deterministic results, which can then be optionally enabled.</p>

tuttelikz: <p>More or less clear on the surface, enough for now, thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
Train.py: error: unrecognized arguments: False

Faysof: <p>Hello everyone, I’m new to pytorch.</p>
<p>I’m trying to run a code available on github.<br>
When I run the line that allows me to do the training as indicated on the code page by doing:</p>
<pre><code class="lang-auto">python train.py --config configs/pretrain.txt --distributed False
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">python train.py --config configs/pretrain.txt --distributed False
usage: train.py [-h] [--config CONFIG] [--rootdir ROOTDIR] [--expname EXPNAME]
                [--distributed] [--local_rank LOCAL_RANK] [-j N]
                [--train_dataset TRAIN_DATASET]
                [--dataset_weights DATASET_WEIGHTS [DATASET_WEIGHTS ...]]
                [--train_scenes TRAIN_SCENES [TRAIN_SCENES ...]]
                [--eval_dataset EVAL_DATASET]
                [--eval_scenes EVAL_SCENES [EVAL_SCENES ...]]
                [--testskip TESTSKIP] [--sample_mode SAMPLE_MODE]
                [--center_ratio CENTER_RATIO] [--N_rand N_RAND]
                [--chunk_size CHUNK_SIZE] [--coarse_feat_dim COARSE_FEAT_DIM]
                [--fine_feat_dim FINE_FEAT_DIM]
                [--num_source_views NUM_SOURCE_VIEWS]
                [--rectify_inplane_rotation] [--coarse_only]
                [--anti_alias_pooling ANTI_ALIAS_POOLING] [--no_reload]
                [--ckpt_path CKPT_PATH] [--no_load_opt] [--no_load_scheduler]
                [--n_iters N_ITERS] [--lrate_feature LRATE_FEATURE]
                [--lrate_mlp LRATE_MLP]
                [--lrate_decay_factor LRATE_DECAY_FACTOR]
                [--lrate_decay_steps LRATE_DECAY_STEPS]
                [--N_samples N_SAMPLES] [--N_importance N_IMPORTANCE]
                [--inv_uniform] [--det] [--white_bkgd]
                [--render_stride RENDER_STRIDE] [--i_print I_PRINT]
                [--i_img I_IMG] [--i_weights I_WEIGHTS] [--llffhold LLFFHOLD]

</code></pre>
<p>The code : <a href="https://github.com/googleinterns/IBRNet/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">IBRNet/train.py at master · googleinterns/IBRNet · GitHub</a></p>
<p>Any help ?</p>

ptrblck: <p>I guess the <code>--distributed</code> argument is used as a switch and doesn’t expect another <code>bool</code> argument.<br>
Try to use:</p>
<pre><code class="lang-python">python train.py --config configs/pretrain.txt --distributed
</code></pre>
BERT Pre-training

Agatha: <p>Hi,<br>
From BERT paper, they say</p>
<p><code>To speed up pretraing in our experiments, we pre-train the model with sequence length of 128 for 90% (900k) of the steps. Then, we train the rest 10% (100k) of the steps of sequence of 512 to learn the positional embeddings. </code></p>
<p>Does this mean,<br>
The model was first trained with a seq_length of 128, then the second train (of 512 seq_length) use the checkpoint of the first training?</p>
What’s the efficient way to convert Python objects of various types to Tensor objects?

Adherer: <p>Hi.<br>
I want to know is there any efficient way to convert Python objects of various types to Tensor objects,like convert_to_tensor in tensorflow, It accepts Tensor objects, numpy arrays, Python lists, and Python scalars.<br>
Although I know that I can determine the type of data and then call the corresponding API, but I think this is too bloated, and I don’t want to write a bunch of ‘if judgment statements’ to determine the type of data, so I would like to ask if there is a corresponding pytorch API or is there any efficient way</p>
<p>Appreciate for any replies.</p>

Adherer: <p>looking forward for your help!</p>

ptrblck: <p>The easiest way would be probably to use <code>torch.tensor</code>.<br>
Would that work or are you running into some issues?</p>

windson: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
HI, i have one question…<br>
so i have a list of list of list…sth like this</p>
<pre><code class="lang-auto">img1
   bbox
      cls1_bbox_list (100*5 ndarray)
      cls2_bbox_list (95*5 ndarray)
      cls3_bbox_list ([ ] )  ##empty
  masks
      cls1_masks_list  (5*256*256)
       cls2_mask_list ([ ])
img2
  ....
</code></pre>
<p>how I can quickly convert them into torch tensor and put into GPU?</p>
<p>I did</p>
<pre><code class="lang-auto"> map(torch.cuda.FloatTensor,  the_whole_listoflistoflist)
</code></pre>
<p>but it seems a bit slow<br>
any ideas?  Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I’m unfortunately not aware of a better approach, so let’s wait for others to chime in <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Is deep copy needed when doing load_state_dict?

Sam_Lerman: <p>I want to copy <code>Module1</code>'s params into <code>Module2</code>, but I want the 2 modules to be independent thereafter.</p>
<p>Is it enough to do <code>Module2.load_state_dict(Module1.state_dict())</code>?</p>
<p>Or do I have to do <code>Module2.load_state_dict(copy.deepcopy(Module1.state_dict()))</code>?</p>

pascal_notsawo: <p>According to the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module.load_state_dict" rel="noopener nofollow ugc">documentation</a>, <code>module.load_state_dict</code> copies parameters and buffers from state_dict into this module and its descendants.<br>
So you don’t need to copy again.</p>
Error when splitting a Node

bucky: <pre><code class="lang-auto">resnet-runtime: /home/bucky/Desktop/compiler/glow/include/qsnn/Support/Error.h:186: void glow::detail::CheckState&lt;true&gt;::ensureChecked() const: Assertion `checked_ &amp;&amp; "Unchecked Error or Expected"' failed.
Aborted (core dumped)

</code></pre>
<p>I am facing this error and after a little debugging I understood that below line of code is responsible for the error. But couldnt figure out how to get rid of this error.<br>
Also, I am only facing this error in debug mode, it runs fine in the release mode and produce correct results.</p>
<pre><code class="lang-auto">glow::splitNode(Node, SplitNodeByNumChunks({1}, {2}));
</code></pre>

jfix: <p><code>splitNode</code> returns <code>Expected&lt;std::vector&lt;Node *&gt;&gt; </code>. So you need something like:</p>
<pre><code class="lang-auto">std::vector&lt;Node *&gt; splitNodes;
ASSIGN_VALUE_OR_FATAL(splitNodes, glow::splitNode(Node, SplitNodeByNumChunks({1}, {2})))
</code></pre>
<p>Even if you don’t need <code>splitNodes</code>. Or else you can handle the error using your own custom code, check <code>Error.h</code> for <code>ASSIGN_VALUE_OR_FATAL</code> and other options.</p>
Pytorch AMP is too complicated and users need to change to much code

Zhaowei_Wang: <p>Nvidia Apex AMP only needs a few lines of code to do amp traning, while PyTorch amp need the training codes to be changed totally. Can we simplify the API?</p>

ptrblck: <p>While the deprecated <code>apex/amp</code> usage might work will fewer lines of code, it lacks e.g. flexibility and is not usable for a lot of use cases e.g. when models need to be recreated etc.</p>
<p>Suggestions to improve the API are always welcome, so let me know if you have some! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to embed Cartesian coordinates

lamppost: <p>Hi!</p>
<p>Im trying to use a transformer to predict a sequence of X,Y coordinates. My question is how I would go about embedding the coordinates to a D-dimensional space in order to input it in my Transformer model.</p>
<p>So far I’ve done something like the following:</p>
<p>First I scale the coordinates to numbers between 0-100.</p>
<pre><code class="lang-auto">input_dim = 100
embedding_dim = 256
embedding = nn.Embedding(input_dim, embedding_dim)

input_to_embed = torch.tensor([[20,30], [30, 20]])  ## [[X,Y],[X,Y]]  two positions of x,y coordinates
embed = embedding(input_to_embed)
print(embed.shape)
</code></pre>
<p>However, when the coordinates have decimals (like [20.3, 30.4]), I get an error because the values does not have scalar type Long.</p>
<p>I suspect that this is not the best way to solve the problem so I am happy to hear your thoughts and suggestions!</p>

alamin: <p>Hello, did you get any solution? I am facing similar situation.</p>
Is this a correct way to initialize a sparse model?

Igo312: <p>I’m tring to initialize a sparse model (in_channels are sparsed 50%) to train it from scratch, I don’t know it is a correct way as the code show below</p>
<pre><code class="lang-auto">for module_name, module, p_name, p, mask, pruned in cls.__sparse_parameters:
     kaiming_normal_(p, mask, nonlinearity='relu')
</code></pre>
<p><code>p</code> will be the weight tensor and mask is the binary-mask and I multiply it with weight then generating a sparse weight.And in the function <code>_calculate_fan_in_and_fan_out</code>, I just change the <code>num_input_fmaps </code></p>
<pre><code class="lang-auto">num_input_fmaps = tensor.size(1) -&gt; num_input_fmaps = mask[0,:,0,0].sum()
</code></pre>
<p>What confuse me is that by this way the recptive_field_size always equals <code>s</code> so the std will not grow when layer become deeper.<br>
I’m not familer with weight inilization so really hope someone can help me out!</p>
Layer.train() vs layer.eval() vs weight.requires_grad vs weight.grad = 0

sigma_x: <p>I’m still confused about these four settings. As far as I understand, they have the following functionality:</p>
<ol>
<li>
<p><code>layer.train()</code> vs <code>layer.eval()</code> make a difference only for batch normalization layers, as they determine whether means, variances will be computed and their number tracked or not. Only layers can be set to these modes, not the weights. These settings don’t affect the gradient computation and weight update in the model.</p>
</li>
<li>
<p><code>weight.requires_grad</code> determines whether gradients for this tensor’s weights will be computed or not, but it doesn’t say anythin about actually upgrading the weights in this tensor. This applies only to the weights, not layers. Weights can be accessed either through <code>layer.weight, layer.bias</code> or looping through <code>model.named_parameters()</code> generator. At the same time, <code>running_mean, running_var, num_batches_tracked</code> in BatchNorm layers are not <code>named_parameters</code>, they are <code>named_buffers</code>, but for some reason can also have <code>requires.grad</code> set to <code>True</code> and <code>grad</code> values, but I’m not sure whether the gradients are in fact computed or updated somehow.</p>
</li>
<li>
<p>In case I don’t want to upgrade weights in a particular layer, I loop through <code>model.named_parameters()</code>, to set the named parameter’s <code>weight.grad=0</code>, these obviously don’t include BatchNorm layers mentioned above. In case ‘weight.requires_grad=True’, gradients for these weights are still computed in this case.</p>
</li>
</ol>
<p>Is this all correct?</p>
Pytorch can not move tensor to cuda

pravin382: <p>System information:<br>
PyTorch version: 1.5.1<br>
CUDA used to build PyTorch: 9.2</p>
<p>OS: Ubuntu 18.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: version 3.10.2<br>
Libc version: glibc-2.27</p>
<p>Python version: 3.8.10 (default, Jun  4 2021, 15:09:15)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.4.0-77-generic-x86_64-with-glibc2.17<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce RTX 3070 Laptop GPU<br>
Nvidia driver version: 460.80<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.2<br>
[pip3] pytorch3d==0.4.0<br>
[pip3] torch==1.5.1<br>
[pip3] torchvision==0.6.0a0+35d732a<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               9.2                           0<br>
[conda] mkl                       2021.2.0           h06a4308_296<br>
[conda] mkl-service               2.3.0            py38h27cfd23_1<br>
[conda] mkl_fft                   1.3.0            py38h42c9631_2<br>
[conda] mkl_random                1.2.1            py38ha9443f7_2<br>
[conda] numpy                     1.20.2           py38h2d18471_0<br>
[conda] numpy-base                1.20.2           py38hfae3a4d_0<br>
[conda] pytorch                   1.5.1           py3.8_cuda9.2.148_cudnn7.6.3_0    pytorch<br>
[conda] pytorch3d                 0.4.0           py38_cu92_pyt151    pytorch3d<br>
[conda] torchvision               0.6.1                 py38_cu92    pytorch</p>
<p>When I run the following code, it simply gets stuck in the fourth statement.</p>
<pre><code class="lang-auto">import torch
device = torch.device("cpu" if False else "cuda")
a = torch.randn(10,10)
a = a.to(device)
</code></pre>
<p>The same code works properly if I use pytorch 1.7.1, cuda 11.0.</p>

ptrblck: <p>Since you are using an Ampere GPU (3070), you would need to use CUDA&gt;=11.0, so the old PyTorch <code>1.5.1</code> release with CUDA9.2 won’t work.<br>
Update to the latest release with CUDA11.1 and it should work.</p>
Setting 'requires_grad = True' increase the GPU allocated memory at each iteration

saeed_karimi: <p>I illustrate my issue with the following simple example:</p>
<p>import torch<br>
import sys</p>
<p>x = torch.randn(3,4)<br>
x=x.cuda()<br>
x.requires_grad = True<br>
q=x+1<br>
<span class="hashtag">#q</span>.requires_grad=True<br>
z=q*q<br>
z.sum().backward()<br>
print(‘before del x’,torch.cuda.memory_allocated(device=0)<br>
del x<br>
print(‘after del x’,torch.cuda.memory_allocated(device=0))</p>
<p>by running the above code ‘del x’ does not delete the tensor x and allocated memory does not change. However if i change ‘x.requires_grad= False’ and ‘q.requires_grad = True’, I can run the above code and ‘del x’ removes the tensor x.<br>
The question is how can I delete tensor x by setting ‘x.requires_grad = True’ ?</p>
Override datasets.ImageFolder for training with two kinds of ground truth labels

James_Lee: <p>When I tried to override <code> datasets.ImageFolder</code>  for training with two kinds of ground truth labels, I met such errors.</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “train.py”, line 205, in <br>
main()<br>
File “train.py”, line 105, in main<br>
train(net, optimizer)<br>
File “train.py”, line 112, in train<br>
for i, data in enumerate(train_loader):<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 363, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 989, in _next_data<br>
return self._process_data(data)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1014, in _process_data<br>
data.reraise()<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/_utils.py”, line 395, in reraise<br>
raise self.exc_type(msg)<br>
TypeError: Caught TypeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py”, line 185, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/public/test/Mymodel/datasets.py”, line 69, in <strong>getitem</strong><br>
img1, img2, target1, target2 = self.joint_transform(img1, img2, target1, target2)<br>
TypeError: <strong>call</strong>() takes 4 positional arguments but 5 were given</p>
</blockquote>
<p>Does <code>datasets.ImageFolder</code> support such occasion? How could I fix it? Many thanks!</p>
Change the particular layer name of in Pytorch

Nikhil_Chowdary_Gutl: <p>Hi,<br>
A portion of the model looks like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c" title="Screenshot (979)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c_2_690x387.png" alt="Screenshot (979)" data-base62-sha1="quJF5AnRDJfwQDOsxLOSFoO8bOk" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9b1fc1ceee96c1d0d430fe718d3d37efd72707c.png 2x" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (979)</span><span class="informations">1366×768 83.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can access my one of the relu function as “print(model.bottleneck.layer1[0].relu)”</p>
<p>This print out as “ReLU(inplace=True)”.</p>
<p>Now I want to replace the variable name relu to relu1. So that I finally need to access as model.bottleneck.layer1[0].relu1</p>
<p>Can some one plz help in this</p>

Nikhil_Chowdary_Gutl: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Can you please check this out?</p>
Building wheels for C++/Cuda extensions

Seledorn: <p>Hi,</p>
<p>I have c++/cuda extension, and to simplify deployment I would like to build wheels for it.</p>
<p>However, there seems to be a combinatorial explosion in the number of wheels necessary to build.</p>
<p>In particular, if I’m not mistaken, I need to provide one wheel for every combination of (supported)</p>
<ul>
<li>minor python versions</li>
<li>minor torch versions</li>
<li>cuda toolkit versions</li>
<li>operating systems</li>
</ul>
<p>This seems a bit much. Is there a better way of dealing with this? Note that the c++ extension part is essentially just starting the cuda kernels, and is certainly not performance critical.</p>
<p>In theory I would assume that providing a single compiled ptx file should be enough &amp; then torch could contain the minimal code neccessary for dispatching kernels with provided dims etc. But I don’t think any such functionality exists. How do people normally deal with this?</p>

ptrblck: <p>I don’t think that providing the <code>ptx</code> would be enough, as the user would still need to compile the code and thus would need a local CUDA toolkit installation, which doesn’t seem to be a huge benefit compared to building your extension.<br>
The advantage of building wheels would be to avoid the necessity of a local CUDA toolkit and I think you are right that the support matrix is not small in case you want to support different architectures etc.</p>

Seledorn: <p>Thanks for the answer!</p>
<p>Hmm, cuModuleLoad seemingly accepts PTX and is part of the driver api.  As long as torch would provide the cuModuleLoad + functionality for launching the kernels I think a single ptx would be all that is needed, independent of user configuration. I might well be missing something though.</p>
<p>Thanks though, my main question was if there was something already existing that provides functionality to simplify this process. Good to know that I’m not missing something</p>
Not be able to run training process with convert_sync_batchnorm

two_Two: <p>Hi I have a problem with using  <code>convert_sync_batchnorm</code> ,When I was trying to use DDP everything works fine ,but when I turn on the sync_bn mode ,the training process start and get stuck right away…</p>
<p>Here’s some info</p>
<pre><code class="lang-auto">pytorch  version 1.8.0
 
#How I run the script : 
python -m torch.distributed.launch  \
--nproc_per_node 4  \ 
--master_addr $master_addr  \ 
--master_port $port  train.py \ 
--train.py
--batch 256  --weights yolov5s.pt --device 0,1,2,3  \
--sync_bn

#How I init :
   #note : opt.local_rank is -1 here 
    if opt.local_rank != -1:
        assert torch.cuda.device_count() &gt; opt.local_rank
        torch.cuda.set_device(opt.local_rank)
        device = torch.device('cuda', opt.local_rank)
        dist.init_process_group(backend='nccl', init_method='env://')
</code></pre>
<p>I would like to know why when I trun on the sync batchnormalization and the training process stop at the beginning of training… Thanks</p>

Yanli_Zhao: <p>did not see anything wrong with your init_process_group, also do not think sync batch norm could cause initialization failure in common cases.</p>
<p>do you want to share your reproducible code snippet?</p>

two_Two: <p>Sorry for late reply , I actually raised a new issue here <a href="https://github.com/ultralytics/yolov5/issues/3754" rel="noopener nofollow ugc">yolov5 </a> .</p>
What information does nn.functional.interpolate use?

mohit117: <p>I have a tensor <code>img</code> in PyTorch of size <code>bx2xhxw</code> and want to upsample it using <code>torch.nn.functional.interpolate</code>. But while interpolation I do not wish channel 1 to use information from channel 2. To do this should I do,</p>
<pre><code class="lang-auto">img2 = torch.rand(b,2,2*h,2*w) # create a random torch tensor.
img2[:,0,:,:] = nn.functional.interpolate(img[:,0,:,:], [2*h,2*w], mode='bilinear', align_corners=True)
img2[:,1,:,:] = nn.functional.interpolate(img[:,1,:,:], [2*h,2*w], mode='bilinear', align_corners=True)
img=img2
</code></pre>
<p>or simply using</p>
<pre><code class="lang-auto">img = nn.functional.interpolate(img, [2*h,2*w], mode='bilinear', align_corners=True)
</code></pre>
<p>will solve my purpose?</p>

eqy: <p>I think if you pass a 4-D tensor it will be interpolated spatially by default (so channel and batch dimensions are not affected):</p>
<pre><code class="lang-auto">import torch
x = torch.randn(16, 16, 32, 32)
y = torch.nn.functional.interpolate(x, size=(64, 64))
y2 = torch.empty(16, 16, 64, 64)
for n in range(16):
    for c in range(16):
        inp = x[n,c,:,:].reshape(1, 1, 32, 32)
        y2[n,c,:,:] = torch.nn.functional.interpolate(inp, size=(64, 64))
print(torch.all(y == y2))
</code></pre>
<pre><code class="lang-auto">tensor(True)
</code></pre>
<pre><code class="lang-auto"></code></pre>