akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Holding certain weights fixed during training

omer_as: <p>Hello,<br>
I have a certain tensor calculated during the forward pass and I want some of its values to contribute nothing to the training of the model - neither in the forward nor in the backward pass.<br>
If I understand correctly, the way to do it is by adding these lines to <code>__forward__()</code>:</p>
<pre><code class="lang-auto">t = input_a @ mat_b # Create tensor
with torch.no_grad():
    t[i,j] = 0 # eliminate passing any information during FP
mask = torch.ones_like(t)
mask[i,j] = 0
t.register_hook(lambda grad: grad.mul_(mask)) # Set gradient to 0 so it doesn't affect BP
</code></pre>
<p>Am I correct? Is there a better way to do so?</p>
<p>Thanks!</p>
Changing network architecture after first round of training

Omroth: <p>Hi all,</p>
<p>I’m looking to train my model to do one thing (maybe that’s officially called “initialising” the network or something), then train it to do something else once it has learned some features.  I’d like to add some more nodes at each layer at this point.</p>
<p>What’s the recommended way of doing this in pytorch?</p>
<p>Thanks,<br>
Ian</p>
Get grads of parameters w.r.t a loss term in pytorch

vmercier: <p>I my Pytorch training i use a composite loss function defined as : <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/598e77bbce8e588c1531dd4de5936fdaff67e4cd.png" alt="CodeCogsEqn (9)" data-base62-sha1="cMfJi9GVf6INzWuECdhLB1NFctL" width="171" height="18">.<br>
In order to update the weights alpha and beta, i need to compute three values : <img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e3bc573c3b6386c5b357e1a91152db21d417eb0.png" alt="CodeCogsEqn (11)" data-base62-sha1="mzNuZ1n9Z6IZjlr03zVv0dgcLqE" width="150" height="21"><br>
which are the the means of the gradients of the loss terms w.r.t to all the weights in the network.</p>
<p>Is there an efficient way to write it in pytorch ?</p>
<p>My training code look like :</p>
<pre><code>for epoc in range(1, nb_epochs+1):
  #init
  optimizer_fo.zero_grad()
  #get the current loss
  loss_total = mynet_fo.loss(tensor_xy_dirichlet,g_boundaries_d,tensor_xy_inside,tensor_f_inter,tensor_xy_neuman,g_boundaries_n)
  #compute gradients
  loss_total.backward(retain_graph=True)
  #optimize
  optimizer_fo.step()
</code></pre>
<p>Where my .loss() function directly return the sum of the terms. I’ve thinking of make a second forward pass and call backward on each Loss term independently but it would be very expensive.</p>
Set values of 1D tensor using list of start/end indices

Yujian_Liu: <p>Given a 1D tensor x, and a 2D tensor (N * 2) of start and end indices, is there a way to set the values in the start and end range to a specific value (say 0)?<br>
I’m looking for a vectorized way to do the following:</p>
<blockquote>
<p>&gt;&gt;&gt; x = torch.randint(10, (8,))<br>
&gt;&gt;&gt; x<br>
tensor([1, 6, 1, 9, 3, 2, 6, 2])<br>
&gt;&gt;&gt; indices = torch.tensor([[1,2], [4,6]])<br>
&gt;&gt;&gt; indices<br>
tensor([[1, 2],<br>
[4, 6]])<br>
&gt;&gt;&gt; desired_output = torch.tensor([1, 0, 1, 9, 0, 0, 6, 2])<br>
&gt;&gt;&gt; desired_output<br>
tensor([1, 0, 1, 9, 0, 0, 6, 2])</p>
</blockquote>
<p>The range between start and end is not guaranteed to have the same length (as 1 and 2 here). Any help would be appreciated!</p>

ptrblck: <p>You could create a mask using <code>cumsum</code> and set the values to zero as seen here:</p>
<pre><code class="lang-python">x = torch.tensor([1, 6, 1, 9, 3, 2, 6, 2])

indices = torch.tensor([[1,2], [4,6]])
val = torch.tensor([[1, -1], [1, -1]])
mask = torch.zeros_like(x).scatter_(
    0, indices.view(-1), val.view(-1)).cumsum(0).bool()

x[mask] = 0
print(x)
&gt; tensor([1, 0, 1, 9, 0, 0, 6, 2])
</code></pre>
<p>Depending on the needed flexibility of the input tensors you could simplify the <code>val</code> creation etc.</p>

Yujian_Liu: <p>Thanks, this is brilliant!</p>

Yujian_Liu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry I have a follow up question: the solution seems to assume that the start and end ranges won’t overlap with each other. Is there a way to modify it to work for overlapping cases? For example, when <code>indices = torch.tensor([[1,3], [2,3]])</code> or <code>indices = torch.tensor([[1,3], [1,2]])</code>.</p>
Simplest beginner glossary editor

julkhami: <p>I’m new to machine learning and I’d like to try to make a simple algorithm which can format text better (the entries in a glossary).</p>
<p>I guess I have a few fundamental questions.</p>
<ol start="0">
<li>
<p>Is it likely I will need to find a good dataset to train the algorithm to format the text in the way I want it to? Is it possible for me to generate that data myself? How much would I need? Or is it possible for the algorithm to cluster like with like based on an unsupervised learning algorithm?</p>
</li>
<li>
<p>Is there a single most standard neural network to use or how do you decide which machine learning architecture to use, given the application?</p>
</li>
</ol>
<p>Thanks very much.</p>
How to use Torch index select for this use case?

soulslicer: <p>I have an index tensor of size [N], that contains indices from 0 to 200</p>
<p>I have a tensor of size [200, N, 3]</p>
<p>I want to somehow do <code>torch.index_select(tensor, 0, index)</code> and get a tensor of size [1, N, 3]. Basically, it needs to select the index in a batch friendly manner etc. How do i do this?</p>

ptrblck: <p>I don’t quite understand the desired output shape given the input tensor and index.<br>
If <code>tensor</code> is defined as:</p>
<pre><code class="lang-python">B, N, C = 5, 4, 3
x = torch.arange(B*N*C).view(B, N, C)
print(x)
</code></pre>
<p>what would be the desired output?<br>
If you want to create an output of <code>[1, N, 3]</code> you would index <code>x</code> in <code>dim0</code> with a single value.</p>
Problem about fine-tune semantic segmentation with Semantic Segmentation on MIT ADE20K dataset in PyTorch

111575: <p>I want to fine-tune semantic segmentation with datasets contain 2100 images that are collected in Habitat-sim(<a href="https://github.com/facebookresearch/habitat-sim" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/habitat-sim: A flexible, high-performance 3D simulator for Embodied AI research.</a>).</p>
<p>I use the model HRnetv2 in Semantic Segmentation on MIT ADE20K dataset in PyTorch, and it pre-trained 30 epoch on ADE20K.</p>
<p>Semantic Segmentation on MIT ADE20K dataset in PyTorch separates the model into two parts, encoder and decoder.</p>
<p>Because I want to do transfer learning, I only load the weight of the encoder.<br>
MIT ADE20K owns 150 categories.  My dataset owns 101, so I change the num_class to 101.</p>
<p>After training, I do an evaluation on another dataset.<br>
I got disastrous results.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee509493419a809506dd5de679687a9c03a13518.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ee509493419a809506dd5de679687a9c03a13518" title="apartment_01_step93"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee509493419a809506dd5de679687a9c03a13518_2_690x230.jpeg" alt="apartment_01_step93" data-base62-sha1="y0ehoH1oEI8ClCRW8GIp3Z1SMdO" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee509493419a809506dd5de679687a9c03a13518_2_690x230.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee509493419a809506dd5de679687a9c03a13518_2_1035x345.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee509493419a809506dd5de679687a9c03a13518_2_1380x460.jpeg 2x" data-dominant-color="819799"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">apartment_01_step93</span><span class="informations">1536×512 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
From left to right is scene, ground truth, and predict.<br>
The shape of the masks is approximately correct, but the color is all wrong.<br>
Does anybody have an idea of how this problem arises and how it can be fixed? Thank you!</p>
Issue passing x through stacked gru

cwoolfo1: <p>I have defined the hidden layers of my sequential model using the following code:</p>
<blockquote>
<p>self.hiddenlayers = nn.GRU(input_size, hidden_size, num_layers=num_layers, batch_first=True)</p>
</blockquote>
<p>When I try to pass input through that code as such:</p>
<blockquote>
<p>x = F.relu(self.hiddenlayers(x))</p>
</blockquote>
<p>I get the following error:</p>
<blockquote>
<p>TypeError: relu(): argument ‘input’ (position 1) must be Tensor, not tuple</p>
</blockquote>
<p>The problem seems to me that when I pass x through the hidden layers it returns a tuple instead of a tensor that I can pass to the relu function. How do I convert the self.hiddenlayers(x) to a tensor that I can pass to relu()?.</p>
<p>I am trying to create a stacked gru where each layer has a activation function of ReLU. Is this approach a correct method of accomplishing that?</p>

ptrblck: <p><code>nn.GRU</code> returns two tensors as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru#torch.nn.GRU">docs</a>. You could unwrap the outputs and pass the desired one to <code>F.relu</code>:</p>
<pre><code class="lang-python">output, hidden = self.hiddenlayers(x)
x = F.relu(output)
</code></pre>
How the calculate the BiGAN discriminator loss after the training

hajar: <p>Hi,<br>
After the training, I want to feed the generated images into the discriminator. This way seems correct? Please find in those pictures the discriminator architecture and the function to calculate the discriminator loss. What do you think?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00b0fc7a17c1ad9c3d90f56f8cee601462c11199.png" alt="Capture d’écran 2021-10-04 à 21.49.31" data-base62-sha1="67bRe3QaB2J7fmYYG3Q0FLcHDb" width="584" height="350"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/2971eff226f527e5885ff9ab949d2f4a4552fc8f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2971eff226f527e5885ff9ab949d2f4a4552fc8f" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2971eff226f527e5885ff9ab949d2f4a4552fc8f_2_690x387.png" alt="1" data-base62-sha1="5UDGwecPIZDNaLgHZJbmvgTm7ZJ" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2971eff226f527e5885ff9ab949d2f4a4552fc8f_2_690x387.png, https://discuss.pytorch.org/uploads/default/original/3X/2/9/2971eff226f527e5885ff9ab949d2f4a4552fc8f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/2971eff226f527e5885ff9ab949d2f4a4552fc8f.png 2x" data-dominant-color="EFF0EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">695×390 52.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I want to do that in order to keep the fake images that seems really like the real ones.<br>
Thank you,</p>

ptrblck: <p>Based on your code snippet I guess you are using TensorFlow?<br>
If so, I would recommend to post the question in their discussion board as you would find the experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
Best way to transfer multiple tensors at once?

Cow_woC: <p>I want to transfer multiple unrelated tensors at once from the GPU to the CPU. What’s the best way to accomplish this from a performance and readability perspective? I tried something like this:</p>
<pre><code class="lang-auto">batch_transfer = torch.cat((first.unsqueeze(0), second.unsqueeze(0), third.unsqueeze(0))).cpu()
first = batch_transfer[0]
second = batch_transfer[1]
third = batch_transfer[2]
</code></pre>
<p>This works but it’s problematic from a readability perspective and I’m also forced to group tensors batch on their shape. If I want to transfer tensors with different shapes, I need to transfer them separately. Is there a better way to do this?</p>
Interrupted by signal 11: SIGSEGV

Sam_Lerman: <p>I’m running a relatively big model on my M1 MacBook and I’m getting this error after a varying number of iterations:</p>
<p><code>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</code></p>
<p>I would share code, but it’s all a bit too complicated and entangled to give a clear, minimal example. I think it should be fine though…</p>

ptrblck: <p>You could use <code>gdb</code> via:</p>
<pre><code class="lang-auto">gdb --args python script.py args
...
run
...
bt
</code></pre>
<p>to isolate the issue further. A minimal code snippet would be indeed helpful, but I also understand that it can be a huge amount of work to create one.</p>

Sam_Lerman: <p>I believe I narrowed down the problem to a code snippet that replaces one <code>Sequential</code> with another <code>Sequential</code>. I have a list of sequentials called <code>sequentials</code> that I initialized in a <code>Module</code>. Periodically, I do <code>sequentials[i] = initialize_new_sequential()</code>. It seems memory isn’t being freed up properly.</p>

ptrblck: <p>What does the backtrace show? Could you post the output here, please?</p>

Sam_Lerman: <p>Sorry, I managed to get rid of the error, but now I get memory errors in the middle of training:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 250.00 MiB (GPU 0; 10.92 GiB total capacity; 9.19 GiB already allocated; 45.31 MiB free; 10.10 GiB reserved in total by PyTorch)</p>
<p>Memory is constant throughout training, so this shouldn’t happen. However, I do create new sequentials in the middle of training and replace old ones, as in <code>list_of_sequentials[index] = new_sequential</code></p>
<p>No matter how much I reduce memory requirements, I get this memory leak</p>

Sam_Lerman: <p>The “list” is a <code>ModuleList</code>, so perhaps the sequentials that are being replaced are registered and some information about them continues to persist?</p>

ptrblck: <p>I don’t know as I’ve never tried this approach. Could you post a minimal, executable code snippet to reproduce the memory leak?</p>
Why einsum not equal as matmul?

Xin_Wan: <p>Hi, I try to use <code>torch.einsum</code> to replace <code>torch.matmul</code> in my code for more intuitive, but the results between two op is not exactly equal.</p>
<pre><code class="lang-auto">import torch

x = torch.randn(3, 10)
y = torch.einsum('nc,mc-&gt;nm', x, x) - torch.matmul(x, x.T)
print(y)
-----------
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 0.0000e+00, -1.9073e-06, -1.1921e-07],
        [ 0.0000e+00, -1.1921e-07,  4.7684e-07]])
</code></pre>
<p>Can someone explain to me why the result is unexpected?</p>

JuanFMontesinos: <p>I’d say that, given the fact einsum and matmul operates differently, the error is small enough to be considered as the same.</p><aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:200/150;"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Z3_Deutsches_Museum.JPG/200px-Z3_Deutsches_Museum.JPG" class="thumbnail" width="200" height="150"></div>

<h3><a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers" target="_blank" rel="noopener nofollow ugc">Floating-point arithmetic | IEEE 754: floating point in modern computers</a></h3>

<p>The IEEE standardized the computer representation for binary floating-point numbers in IEEE 754 (a.k.a. IEC 60559) in 1985. This first standard is followed by almost all modern machines. It was revised in 2008. IBM mainframes support IBM's own hexadecimal floating point format and IEEE 754-2008 decimal floating point in addition to the IEEE 754 binary format. The Cray T90 series had an IEEE version, but the SV1 still uses Cray floating-point format.[citation needed]
 The standard provides for man...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Speech Recognition from ASR model

Hengineer327: <p>Hello! Can I have some help making a real-time speech recognizer from a pre-trained ASR model? If you need the model, please tell me. I have searched on google, but I couldn’t find anything. Thanks!</p>

JuanFMontesinos: <p><a href="https://speechbrain.github.io/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://speechbrain.github.io/</a></p>

Hengineer327: <p>Thank you! Have you done this before? If so, do you know of any good tutorials?</p>

JuanFMontesinos: <p>Sorry I’m afraid I don’t have time at all to look for that.<br>
I’m pretty sure if you look for tutorials of interspeech of icaspp you will find something useful.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/espnet/interspeech2019-tutorial">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/espnet/interspeech2019-tutorial" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/77f882b9c0005247df0b3d1abd42d3d1b1b3b483946690317197202df1ac1f35/espnet/interspeech2019-tutorial" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/espnet/interspeech2019-tutorial" target="_blank" rel="noopener nofollow ugc">GitHub - espnet/interspeech2019-tutorial: INTERSPEECH 2019 Tutorial Materials</a></h3>

  <p>INTERSPEECH 2019 Tutorial Materials. Contribute to espnet/interspeech2019-tutorial development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Detect overlapping rows in N x 4 tensors when N is large

martinr: <p>Hi!<br>
I have 2 large tensors, about [100K x 4] each and I need to find overlapping “rows”. The <strong>limiting factors</strong> are that rows are <strong>not in the same location</strong>, e.g., [1000, 45, 23, 0] might be 100th element in one and 125th element in another tensor AND tensors are <strong>not of equal length</strong>, e.g. one might be [101000 x 4] and other [89500 x 4], actual lengths vary from tensor to tensor. An example where there is one duplicate row <code>[5,6,7,8]</code>:</p>
<pre><code class="lang-auto">a = torch.tensor([[1,2,3,4],
[5,6,7,8],
[9,10,11,12]])

b = torch.tensor([[11,22,33,44],
[55,65,57,58],
[95,10,11,124],
[5,6,7,8]])
</code></pre>
<p>I have a simple loop-based solution to get the indices of matching rows:</p>
<pre><code class="lang-auto">matching = []
for coord in a:
    result = (b == coord).all(dim=1).nonzero()
    if result.shape[0] &gt; 0:
        matching.append(result.item())
print("Matching indices in B:", matching)
</code></pre>
<p>This works well but is <strong>painfully slow</strong> for 100k x 100k comparisons. I also have one constraint that might be helpful - it is guaranteed that there are no duplicate rows in <em>a</em> or <em>b</em>, i.e., each row is unique within a same tensor. Any ideas on <strong>how to speed this thing up</strong>?</p>
<p>Background for understanding my use case: I have somewhere between 0 to 1000 matching rows in tensors (I am working with 3D data and merging point clouds from different sensors so the rows of [x,y,z,time] are unique integer coordinates + integer timestep) and I need to locate rows that are located on the same coordinates at the same time before merging and inputting them into my network.</p>
Nan gradients with Torch.angle()

MinkyuChoi: <p>Seeing the <code>torch.angle()</code> description (<a href="https://pytorch.org/docs/stable/generated/torch.angle.html" class="inline-onebox" rel="noopener nofollow ugc">torch.angle — PyTorch 1.9.1 documentation</a>), it says that the behavior of <code>torch.angle()</code> has been changed since 1.8.0. Following is the note from the link.</p>
<h1>
<a name="note-starting-in-pytorch-18-angle-returns-pi-for-negative-real-numbers-zero-for-non-negative-real-numbers-and-propagates-nans-previously-the-function-would-return-zero-for-all-real-numbers-and-not-propagate-floating-point-nans-1" class="anchor" href="#note-starting-in-pytorch-18-angle-returns-pi-for-negative-real-numbers-zero-for-non-negative-real-numbers-and-propagates-nans-previously-the-function-would-return-zero-for-all-real-numbers-and-not-propagate-floating-point-nans-1"></a>====== Note =======<br>
Starting in PyTorch 1.8, angle returns pi for negative real numbers, zero for non-negative real numbers, and propagates NaNs. Previously the function would return zero for all real numbers and not propagate floating-point NaNs.</h1>
<p>Here, I don’t understand what it means by ‘propagates NaNs’. I understood it as its gradient will be NaN when the input to <code>Torch.angle()</code> is real value. However, when I tested, it seems its gradient is not Nan.</p>
<pre><code class="lang-auto">    x1=torch.tensor([[0, 4, -4, 1+4j]], dtype=torch.cfloat, requires_grad=True)
    out = x1.angle()
    print(out)
    out.mean().backward()
    print(x1.grad)
</code></pre>
<p>The corresponding outputs from print are</p>
<pre><code class="lang-auto">    tensor([[0.0000, 0.0000, 3.1416, 1.3258]], grad_fn=&lt;AngleBackward&gt;)
    tensor([[ 0.0000+0.0000j,  0.0000+0.0625j, -0.0000-0.0625j, -0.0588+0.0147j]])
</code></pre>
<p>If my understanding to the note is correct, the gradient from <code>angle()</code> when its input is real value should be Nan, but it is not.</p>
<p>Could anyone help me understand when <code>torch.angle()</code> returns Nan as its gradient?<br>
It looks like I am facing Nan issue because of angle() function, because when I remove <code>angle()</code>, I don’t see Nan anymore. I am trying to look into this function to see when its gradient becomes Nan.</p>
<p>(Code is tested in pytorch 1.9.0)</p>

ptrblck: <p>“Propagates NaNs” means that this module will output NaN values if the input was already containing NaNs (so it propagates/forwards them). This should apply for the forward and backward methods. Based on the description I would guess that NaN inputs were mapped to zeros in previous versions.</p>
Should I create optimizer after sending the model to GPU?

abhidipbhattacharyya: <p>Does it matter if I create optimizer before or after uploading the model in GPU. For example</p>
<pre><code class="lang-auto">model = myModel()
optimizer = torch.optim.Adamax(model.parameters())
model = model.to(device)
</code></pre>
<p>vs.</p>
<pre><code class="lang-auto">model = myModel()
model = model.to(device)
optimizer = torch.optim.Adamax(model.parameters())
</code></pre>
<p>Thanks,<br>
Abhidip</p>

ptrblck: <p>It shouldn’t matter, as the optimizer should hold the references to the parameter (even after moving them). However, the “safer” approach would be to move the model to the device first and create the optimizer afterwards.</p>

abhidipbhattacharyya: <p>Thank you. The reason I asked this question is because I am trying to train a model which is very demanding in terms of GPU space. Now if I create the optimizer after moving the model to GPU then it is taking more space in GPU and sometime my GPU is overflowing. But by creating the optimizer before moving model to GPU I am able to save some space in GPU.</p>

ptrblck: <p>Could you post an executable code snippet showing this behavior, please?</p>
Accessing batchnorm mean from nn.Sequential container

avalon1511: <p>I have a generator defined as:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)
</code></pre>
<p>I want to obtain the mean and standard dev values for each batchnorm layer during training - how can I do this?</p>
SGD: specifying lr in SGD constructor vs. per-param -- why different behavior?

Bart: <p>Hi,</p>
<p>The following two pieces of code produce different training results and I don’t understand why. My understanding is that named_parameters() returns all model params (trainable or not). What is the difference between specifying the learning rate on each of them vs. in the SGD constructor?</p>
<p>Code <span class="hashtag">#1:</span></p>
<pre><code class="lang-auto">def get_optimizer(self):
        lr = opt.lr 
        params = []
        for key, value in dict(self.named_parameters()).items():
            if value.requires_grad:
                if 'bias' in key:
                    params += [{'params': [value], 'lr': lr * 1, 'weight_decay': 0}]
                else:
                    params += [{'params': [value], 'lr': lr, 'weight_decay': 0}]
        self.optimizer = t.optim.SGD(params, momentum=0.9)
        return self.optimizer
</code></pre>
<p>Code <span class="hashtag">#2:</span></p>
<pre><code class="lang-auto">def get_optimizer(self):
        self.optimizer = t.optim.SGD(self.parameters(), lr = opt.lr, momentum=0.9)
        return self.optimizer
</code></pre>
<p>parameters() and named_parameters() have the same items.</p>
<p>Thank you,</p>
<p>Bart</p>

ptrblck: <p>Both approaches work identical as seen here:</p>
<pre><code class="lang-python">def get_optimizerA(model):
    lr = 1.
    params = []
    for key, value in model.named_parameters():
        if value.requires_grad:
            params += [{'params': [value], 'lr': lr, 'weight_decay': 0}]
    optimizer = torch.optim.SGD(params, momentum=0.9)
    return optimizer

def get_optimizerB(model):
    lr = 1.
    optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)
    return optimizer

# setup
modelA = models.resnet18().eval()
modelB = models.resnet18().eval()
modelB.load_state_dict(modelA.state_dict())
x = torch.randn(2, 3, 224, 224)

optimizerA = get_optimizerA(modelA)
optimizerB = get_optimizerB(modelB)

outA = modelA(x)
outB = modelB(x)

lossA = outA.mean()
lossB = outB.mean()
print((lossA - lossB).abs())

lossA.backward()
lossB.backward()

optimizerA.step()
optimizerB.step()

for (nameA, paramA), (nameB, paramB) in zip(modelA.named_parameters(), modelB.named_parameters()):
    if not nameA == nameB:
        print("Error in iterating param dicts")
        break
    print('{}: abs.max error {}'.format(nameA, (paramA - paramB).abs().max()))
</code></pre>
<p>Could you post an executable code snippet, which would reproduce the issue you are seeing?</p>

Bart: <p>Thanks – you are correct, they are identical. My apologies. I’m not sure how I got it into that state but I can no longer reproduce the issue.</p>
Partial dependence plots

Shar: <p>Hi,<br>
Is there a library for calculating partial dependence plots (PDP) for a trained model on pytorch?<br>
I found pdpbox in GitHub but it works with scikit-learn algorithms only.</p>
<p>Alternatively, can I somehow convert my model in pytorch to a scikit-learn model in order to use the developed tools for them?</p>
<p>Thanks</p>
Saving parameters of a Bayesian model (posterior) and loading them as the Prior

samin_hamidi: <p>Is there a way to save the parameters of a Bayesian neural network (each neuron has two parameters, mean and variance) as we save the parameters of  a normal neural network by state_dict() ? is there any easier way than overwriting existing classes and modules of Pytorch?</p>

googlebot: <p>No, pytorch is basically a framework for array computations, not a “probabilistic programming language” by itself. “distributions” module has a limited scope and won’t help you with priors, there is pyro project, more suitable for Bayesian models, though I don’t like its slowness.</p>
Diffusion Models Beat GANs - implementation

Bagg: <p>Less experienced one here. I would like to implement something along the line of the paper: <a href="https://arxiv.org/abs/2105.05233" class="inline-onebox" rel="noopener nofollow ugc">[2105.05233] Diffusion Models Beat GANs on Image Synthesis</a>. Apply the same concepts but in a simpler implementation.<br>
The code from OpenAI is quite complex and computationally demanding.<br>
How would you suggest proceeding to tackle this project?</p>
Mixed input (image data and 1x3 array) and image output deep learning

roky: <p>I have an input image and input RGB color with an image output. I want to isolate only a region of interest from the image and train the model to fit with the ROI of the output image. How to implement this model?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0e6712a467da4fc870eeedadda891cd380e1c80.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0e6712a467da4fc870eeedadda891cd380e1c80" title="CV Problem"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6712a467da4fc870eeedadda891cd380e1c80_2_690x347.png" alt="CV Problem" data-base62-sha1="mXo78p7qPT3EQeSnQheYEj09V16" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6712a467da4fc870eeedadda891cd380e1c80_2_690x347.png, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0e6712a467da4fc870eeedadda891cd380e1c80.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0e6712a467da4fc870eeedadda891cd380e1c80.png 2x" data-dominant-color="E3D6D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CV Problem</span><span class="informations">886×446 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
RuntimeError: one of the variables needed for gradient (torch==1.3.0) computation has been modified by an inplace operation:

A_Shamsadini: <p>Hi, I used CrossEntropyLoss() for segmentation but I have this error. Please help me<br>
"“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 512, 1, 1]], which is output 0 of ReluBackward1, is at version 6; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</p>
<p>I can’t loss.backward(),</p>
<p>j=0<br>
criterion1 = nn.CrossEntropyLoss()<br>
best_loss = 1000000000<br>
for epoch in range(2):</p>
<p>model.train()</p>
<p>for i, dat in enumerate(train_generator):</p>
<pre><code>  j+=1
  features, masks,labels = dat

  features = features.to(device).float()

  masks = masks.to(device)

  

  labels = labels.to(device)

  optimizer.zero_grad()  

  ### FORWARD AND BACK PROP

  features = features.permute(0, 3, 1,2)

  #print(features.shape)

  logits_labels,logits_masks = model((features))

  

  #masks=masks[...,0].squeeze()

  masks = masks.squeeze_()

  #Loss

  loss1 = criterion1(logits_masks,masks)

  loss2 = criterion1(logits_labels,labels)

  loss=loss1+loss2

  loss.backward()

  running_loss += loss.item()

    ### UPDATE MODEL PARAMETERS

  optimizer.step()</code></pre>

AlphaBetaGamma96: <p>Within your forward pass do you have <code>torch.nn.ReLU(inplace=True)</code>? Set it to <code>torch.nn.ReLU(inplace=False)</code></p>

A_Shamsadini: <p>Thank you so much. It solved, but Why should (inplace = True) be false?</p>

A_Shamsadini: <p>Thank you so much. It solved, but Why should (inplace = True) be false?</p>

AlphaBetaGamma96: <p>Because you’re trying to compute a gradient. You can’t compute gradients on Tensor which have been modified via an in-place operation, so make sure it’s set to False.</p>
<p>There’s a nice dicussion about it on the forums here: <a href="https://discuss.pytorch.org/t/what-is-in-place-operation/16244/15" class="inline-onebox">What is `in-place operation`? - #15 by Alex_Fann</a></p>

A_Shamsadini: <p>Does it have an effect on loss? I mean, isn’t loss too much?</p>

AlphaBetaGamma96: <p>On the output of the loss, No. On the calculation of the gradient of the loss with respect the parameters, yes.</p>

A_Shamsadini: <p>Thank you so mutch. <img src="https://discuss.pytorch.org/images/emoji/apple/rose.png?v=10" title=":rose:" class="emoji" alt=":rose:"></p>
Not possible to torch.jit.trace a faster r cnn with the cuda device

Nathb26: <p>Hi I tried to run this code on google collab and got an error. I don’t understand why this error occurs while I just dowloaded a faster r cnn model and called the torh.jit.trace function.</p>
<pre><code class="lang-auto">import torch
import torchvision

device = "cuda" if torch.cuda.is_available() else "cpu"
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.to(device)
model.eval()

input_t = torch.rand(1,3,224,224).to(device)
module = torch.jit.trace(model,input_t)

module.save("module.pth")
</code></pre>
<p>I got this error :</p>
<pre><code class="lang-auto">  for i in range(dim)
/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/anchor_utils.py:123: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  torch.tensor(image_size[1] // g[1], dtype=torch.int64, device=device)] for g in grid_sizes]

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

&lt;ipython-input-3-cfa0d532d4aa&gt; in &lt;module&gt;()
      8 
      9 input_t = torch.rand(1,3,224,224).to(device)
---&gt; 10 module = torch.jit.trace(model,input_t)
     11 
     12 module.save("module.pth")

9 frames

/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/rpn.py in _get_top_n_idx(self, objectness, num_anchors_per_level)
    223                 pre_nms_top_n = min(self.pre_nms_top_n(), num_anchors)
    224             _, top_n_idx = ob.topk(pre_nms_top_n, dim=1)
--&gt; 225             r.append(top_n_idx + offset)
    226             offset += num_anchors
    227         return torch.cat(r, dim=1)

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
</code></pre>
<p>Please does someone have a solution ?</p>

Nathb26: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> sorry to mention but I still didn’t find any solution do you have an idea ?</p>

ptrblck: <p>I’m not sure what exactly is causing the error (I can reproduce it), so would recommend to create an issue in the <code>torchvision</code> repository. As a workaround you could <code>torch.jit.script</code> the model, which should work.</p>
Change specified indices of torch tensors

negreanu1: <p>Hi, I come with a reproducible example of how to change some values of a tensor to 0 in some previously defined indices. Imagine that I have a tensor <code>a</code> of dimension <code>(n_sensors,batch_size,dim1,dim2)</code>. For each batch I want to set to 0 the 5 highest values of <code>dim2</code> for each sensor and each value of <code>dim1</code>. That is, I want to do this:</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
import torch 

a = torch.rand((5,4,10,15))
top_att = torch.sum(a.mean(0),axis=1)
top_att = top_att.argsort(axis = 1)[:, -5:] 
top_att = top_att.sort()[0]
print(top_att.shape)

plt.figure()
plt.imshow(a.mean(0)[0])
plt.colorbar(fraction=0.046, pad=0.04)
plt.clim(0,1)
plt.show()

for sensor in range(a.shape[0]):
    for time_step in range(a.shape[2]):
        for batch in range(a.shape[1]):
            indexes = top_att[batch]
            a[sensor,batch,time_step,indexes] = 0
            

plt.figure()
plt.imshow(a.mean(0)[0])
plt.colorbar(fraction=0.046, pad=0.04)
plt.clim(0,1)
plt.show()

</code></pre>
<p>This way it works fine for me, but I want to look for a more optimal way, without using loops, how can I do it?</p>
Can I create multiple DataLoaders on one dataset?

legendu: <p>If I do so, will iterating of 2 DataLoaders (backed by the same dataset) intervent with each other?</p>

ptrblck: <p>Yes, you can create multiple <code>DataLoader</code>s and could use them. I’m not sure what the concern is, but in case you are using multiple workers in these <code>DataLoader</code>s, note that each loader will create the workers and they will create batches in the background once you start to iterate them, which might or might not be desirable.</p>

legendu: <p>Let’s say that I have 2 <code>DataLoaders</code>: <code>dl1</code> and <code>dl2</code> backed by the same dataset. While I’m iterating through <code>dl</code> (hasn’t finished yet), I then iterate through <code>dl2</code> completely. Will they still both behave correctly?</p>

legendu: <p>It is essentially a question on how a DataLoader is implemented. Does a DataLoader mutate its underlying dataset? If a DataLoader does not mutate its underlying dataset in anyway but just create randomly shuffle indexes to access the dataset, then multiple DataLoaders backed by the same dataset won’t intervent with each other no matter they are used at the same time or not.</p>

ptrblck: <p>The <code>DataLoader</code> itself will not mutate the <code>Dataset</code>, as it’s calling into the <code>Dataset</code> to get the data, create batches, shuffle etc.<br>
However, the <code>Dataset.__getitem__</code> could mutate the data in case you are manipulating it inplace (this is usually not wanted and caused errors in the past).<br>
There is also a difference between the behavior of a single worker (main thread) or multiple workers, as the latter will create copies in each worker. So even if you are manipulating the data in the <code>__getitem__</code> method inplace, these manipulations won’t be stored in the original <code>Dataset</code>.</p>
<p>TL;DR: check the <code>Dataset.__getitem__</code> and make sure the data is not manipulated inplace.</p>

legendu: <p>That’s a very good point. In my use case, I don’t leverage multi-threading or multi-processing, so even if <code>Dataset.__getitem__</code> mutate the data, e.g., creating cache, it won’t affect iterating of 2 DataLoaders on top it (which might iterate in turn).</p>

ptrblck: <p>It’s the other way around. Since you are <em>not</em> using multiple processes, the <code>Dataset</code> will not be copied and manipulations in the <code>Dataset</code> will be visible in all <code>DataLoader</code>s, so be careful about it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
Easy way to get Kineto to use the new profiler?

alannnna: <p>Hello, I’m trying out the PyTorch Profiler as per <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/" rel="noopener nofollow ugc">this blog post</a>, but I get this error:</p>
<pre><code class="lang-auto">AssertionError: Requested Kineto profiling but Kineto is not available,
                  make sure PyTorch is built with USE_KINETO=1
</code></pre>
<p>I dug into the code to see if I could profile without Kineto but it seems the profiler requires it. Is there an easy way to get Pytorch with USE_KINETO set, or does this require building from source?</p>

crcrpar: <p>I also hope there is a way to do that other than rebuild.</p>
<p>Though I don’t think that possible reading <a href="https://github.com/pytorch/pytorch/blob/728d18f97695def22f237ce343a3a9679aca49b0/torch/csrc/autograd/init.cpp#L180-L186" class="inline-onebox" rel="noopener nofollow ugc">pytorch/init.cpp at 728d18f97695def22f237ce343a3a9679aca49b0 · pytorch/pytorch · GitHub</a>.</p>

Elena_Neroslavskaya: <p><a class="mention" href="/u/alannnna">@alannnna</a>  what pytorch version and cuda toolkit have you installed?<br>
Profiler is available starting from pytorch 1.8.1 release</p>

shi.feng: <p>Hi,<br>
I also encountered the same issue, I installed from the source of version 1.9.0a0 on my MacBook Air M1.<br>
Thanks</p>

alannnna: <p><a class="mention" href="/u/elena_neroslavskaya">@Elena_Neroslavskaya</a> Pytorch 1.8.1, no CUDA toolkit as I was just trying it out on Mac.</p>

alannnna: <p>Did installing 1.9.0a0 from source fix it for you, or are you saying you had the same problem when you installed that version? Thanks.</p>

Kiiiwii: <p>Hey, I also have the same problem (torch 1.8.1 on windows). Does anyone have a solution to this?</p>

yangyong: <p>I also encounter the same problem on torch1.8.1 with cuda on windows, is any solution?</p>

Patchie: <p>Got the same problem here. 1.8.1.</p>
<p>Installed using command: pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>

emezh: <p>I had the same issue. As a workaround I’ve switched to the legacy API<br>
<code>torch.autograd.profiler.profile()</code><br>
and then it worked.</p>

joesph_nguyen: <p>Same issue. Using Windows, pytorch 1.8.1, torchvision 0.9.1, torch-tb-profiler 0.2.1, cuda 11.4</p>
RuntimeError: input must have 3 dimensions, got 2. Help with GRU

cwoolfo1: <p>class CustomDataset(Dataset):<br>
def <strong>init</strong>(self, X_data, Y_data):<br>
super(CustomDataset, self).<strong>init</strong>()<br>
self.X_data = X_data<br>
self.Y_data = Y_data<br>
# Initializes the data and preprocessing.<br>
def <strong>getitem</strong>(self, index):<br>
return self.X_data[index,:,:], self.Y_data[index,:,:]<br>
# Returns data (input and output) in batches.<br>
def <strong>len</strong> (self):<br>
return len(self.X_data[:,1,1])<br>
# Returns the size of the input data.</p>
<p>def train_loop(dataloader, model, loss_fn, optimizer, lossvalue, recall, accuracy, f1score):<br>
<span class="hashtag">#size</span> = len(dataloader.dataset)<br>
size = dataloader.<strong>len</strong>()<br>
for batch, (X, y) in enumerate(dataloader):<br>
# Compute prediction and loss<br>
pred = model(X)<br>
loss = loss_fn(pred, y)<br>
# Backpropagation<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
if batch % 100 == 0:<br>
loss, current = loss.item(), batch * len(X)<br>
print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")<br>
lossvalue.append(loss)<br>
recall = recall.append(torchmetrics.functional.precision_recall(pred, y))<br>
accuracy = accuracy.append(torchmetrics.functional.accuracy(pred, y))<br>
f1score = f1score.append(torchmetrics.functional.f1(pred,y))</p>
<p>I am running a stacked gru. I have written the program to print the tensorshape before initializing the train dataload and got 3 dim (batch,seq,feature). I can’t seem to find what is wrong with my code? Help would be much appreciated</p>
Multiple gradient calculation using the same output

ckanbak: <p>Hello,<br>
In my project, I need to compute the gradient wrt the input for each output, basically by doing something like this:</p>
<pre><code class="lang-auto">for index in selected_outputs:
    output[:,index].backward(retrain_variables=True)
    # the code continues from here
</code></pre>
<p>The batch size at the input is only one here. I observed that each <code>backward()</code> takes about the same time in each iteration and causing the loop to take a lot of time. Is there a better way to do this?</p>

albanD: <p>That depends on what you do with the gradients after.<br>
If you need a linear combination of them, you can apply it before and then backward once.<br>
If you need something more complex, I’m afraid you have to call multiple backwards <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

ckanbak: <p>Thanks for your reply. It’s more complex than simply a linear combination, so I guess I have to wait for the backward to be done at each iteration <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:">.</p>

ckanbak: <p>Can I ask one more question?<br>
If I understand correctly from <a href="http://pytorch.org/docs/notes/autograd.html" rel="nofollow noopener">here</a>, the backward creates a directed graph that will do the computations and give the output and I think that even though retain_variables are set True, the graph is not retained. Now the <a href="http://pytorch.org/docs/autograd.html" rel="nofollow noopener">docs page of autograd</a> show that there will a <code>retain_graph</code> option in the new version of backward(I think). Do you know if that will help? I know I still have to call multiple backwards, but will that make consecutive backwards quicker?</p>

albanD: <p>The graph is actually created while forwarding, what the backward is doing is just traversing it.<br>
The difference between <code>retain_variables</code> (or <code>retain_graph</code> which is its new name) being True and False is that in the first case, the graph will be left unchanged during backward, in the second case, the graph will be destroyed as you go through it (to reduce memory usage) and thus you won’t be able to reuse it.</p>

ckanbak: <p>Ok, I see. Thanks for your reply.</p>

neouyghur: <p><a class="mention" href="/u/ckanbak">@ckanbak</a> did you find any solution to speed it up? I am also facing the same problem.</p>
Specifying CUDA Location when Building from Source

agu: <p>I am trying to build the <code>pytorch/master</code> branch from source on an AWS instance. The default CUDA <code>nvcc</code> version is 10.0, found in <code>/usr/local/cuda/bin/nvcc</code>, and is incompatible with PyTorch version 1.9+ (since version 10.2 or above is required).</p>
<p>On the instance, I also have newer versions of CUDA (e.g. in <code>/usr/local/cuda-11.0/</code>). I cannot figure out how to reconfigure the PyTorch build to use that version instead of the default. Any help?</p>

ptrblck: <p>Try to use <code>CUDA_HOME=/usr/local/cuda-11.0/ python setup.py install</code> .</p>
Automatic Type Casting

ziqipang: <p>Hi, I recently find that <code>torch.Tensor</code> will automatically cast the input <code>numpy.ndarray</code> to the float type. And I am wondering if there are anyway to bypass this.</p>
<p>For example:</p>
<pre><code class="lang-auto">import numpy as np, torch

a = np.arange(5).astype(np.int32)
b = torch.Tensor(a)
</code></pre>
<p>Then the tensor <code>b</code> here will automatically get the type <code>float</code>. I am aware that using <code>b.int()</code> will cast <code>b</code> into int, but I am wondering if there are any ways to avoid this by some mechanisms in <code>torch.Tensor</code>.</p>
<p>Thank you!</p>

ziqipang: <p>Never mind, I just found that <code>torch.tensor</code> can solve this.</p>