akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loading training data to a dataset in Mask RCNN

M_S: <p>I want to train my own <code>torchvision.models.detection.</code> <code>maskrcnn_resnet50_fpn</code> model with my own data. How should I organize my data and load it to a Dataset \ DataLoader?<br>
The example in <a href="https://pytorch.org/docs/stable/torchvision/models.html#mask-r-cnn" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#mask-r-cnn</a> is a toy example with random data.</p>
<p>Thanks</p>
Loss decrease different between the pytorch and keras

K_Li: <p>hi,all:<br>
I meet a problem when I rewrite the network form keras<br>
in keras, i use<br>
optimizer = keras.optimizers.Adam(lr = 1e-4)<br>
model.compile(optimizer=optimizer, loss=‘binary_crossentropy’, metrics=[‘accuracy’])<br>
in keras,and use<br>
optimizer = optim.Adam(model.parameters(),lr=0.0001)<br>
in pytorch<br>
but keras’s loss can keep decrease util 0.04,but in pytroch ,it will stuck at 0.2<br>
I use same data in both and BCELoss in pytorch , how can I fix that problem?<br>
thanks</p>

ptrblck: <p>Often the discrepancy is due to different parameter initializations.<br>
Could you make sure to use the same <code>torch.nn.init</code> methods for your layers as in your Keras model?</p>

K_Li: <p>I use he_normal to initialize the layers in keras,so what should I use in pytorch？</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_" rel="nofollow noopener"><code>torch.nn.init.kaiming_normal</code></a> corresponds to Keras’ <code>he_normal</code>.</p>
requires_grad=False in nn.Embedding

vainaijr: <p>why does this give an error?</p>
<p>A:</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.embedding_to_learn = nn.Embedding(10, 5, requires_grad=True)
    self.embedding_to_not_learn = nn.Embedding(10, 5, requires_grad=False)
</code></pre>
<p>If I want to not learn some embeddings, then I have to do,</p>
<p>B:</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.embedding_to_learn = nn.Embedding(10, 5)
    self.embedding_to_not_learn = nn.Embedding(10, 5)

model = Net()
model.embedding_to_not_learn.requires_grad_(requires_grad=False)
</code></pre>
<p>what is wrong with A?</p>

tom: <p>The requires_grad keyword argument only works for the tensor factory functions.<br>
Initially, <code>requires_grad_</code> only worked on Tensors / Parameters, too, but now <code>module.requires_grad_(rq)</code> is a convenience shorthand for <code>for p in module.parameters(): p.requires_grad_(rq)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
Is it possible for tensor to support different element sizes? Also, is it possible that the convolution layer can support inputs of different sizes in one batch?

Andybert: <p>Is it possible for tensor to support different element sizes?  For example:<br>
tensor A contains two elements, B and C. The dimensions of B and C are 10x20, 15x25, respectively.</p>
<p>Also, is it possible that the convolution layer can support inputs of different sizes in one batch? For example:<br>
There are two tensors, A and B, in one batch. The dimensions of A and B are 10x20, 15x25, respectively.</p>

ptrblck: <p>The <a href="https://github.com/pytorch/pytorch/issues/22169" rel="nofollow noopener">NestedTensors</a> proposal seems to discuss your use cases. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Andybert: <p>Thank you for telling me this!</p>
Where is the source code for mathematical operations?

C-Weed: <p>Specifically I want the source code for torch.mean(). I can’t seem to find it anywhere in the torch folder. Where is it?</p>

tom: <p>Implementations are in the aten folder, here <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/ReduceOps.cpp" rel="nofollow noopener">ReduceOps.cpp</a>. The <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">selective excursion</a> blog post attempts to show you how things are tied together.<br>
Note that reductions use a mechanism called TensorIterators that is one of the more elaborate things in PyTorch, so their actual implementation is spread across several files and is a bit less straightforward to understand than one might expect.</p>
<p>Best regards</p>
<p>Thomas</p>
Adding layers to Inception v3 Transfer Learning

Amrit_Das: <p>I am training a flower classifier and in order to increase the performance, I want to add 3 layers to the feature extracted by the pre-trained Inception v3 model and calculate the confusion matrix.</p>
<p>The three-layers are -</p>
<ol>
<li>An average pooling layer followed by</li>
<li>2 Dense Layers (512)</li>
</ol>
<p>I would really appreciate if you can help me with it.</p>
<pre><code class="lang-auto">model_ft = models.inception_v3(pretrained=True)
model_ft.aux_logits=False

# I want to add the layers here

num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 102)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=10, gamma=0.1)

</code></pre>

ptrblck: <p>Would you like to add these additional layers right before the pooling layer (<a href="https://github.com/pytorch/vision/blob/95131de394543a7c34bd51932bdfce21dae516c1/torchvision/models/inception.py#L172" rel="nofollow noopener">line of code</a>)?<br>
If so, I think the best way would be to create your own custom model and derive from <code>Inception3</code> as the base class.<br>
This would make manipulating the <code>forward</code> method quite easy.</p>
<p>Let me know, if you get stuck somewhere.</p>

Amrit_Das: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yes I would like to add these line before that the problem is I have never done these kind of modifications earlier and hence I am completely stuck. Thank you for your help. Could you please help me out on how to do it.</p>
Is the graph rebuilt every iteration?

BlueBlazin: <p>Is the computation graph created every time you evaluate your model on an input?</p>

ptrblck: <p>Yes, the computation graph will be created while running the specified operations (usually in your model’s <code>forward</code>, but it can be created by any arbitrary operation with a tensor which requires gradients).</p>
<p>If you don’t specify <code>retain_graph=True</code>, the computation graph will be cleared after the <code>backward()</code> call.</p>
Could not use all of the GPU power

MrRobot: <h1><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h1>
<p>As could be seen from the following snapshot, there are two things</p>
<ul>
<li>When I use less than half of the GPU memory (2392 vs. 5904). The Volatile GPU-Util is almost 100%.</li>
<li>The maximum batch size I could have is 128. If I made it larger (like 256). My code could not run because of following error</li>
</ul>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 5.77 GiB total capacity; 4.11 GiB already allocated; 18.81 MiB free; 395.30 MiB cached)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c30ae71e0f769757035042db90b1802d4f4ad4f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/c30ae71e0f769757035042db90b1802d4f4ad4f9" title="Screenshot%20from%202019-11-09%2019-45-16"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c30ae71e0f769757035042db90b1802d4f4ad4f9.png" alt="Screenshot%20from%202019-11-09%2019-45-16" data-base62-sha1="rPqsSRdhMsydEK7FUbPH15l8UwF" width="690" height="389" data-dominant-color="3C1A31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-11-09%2019-45-16</span><span class="informations">733×414 52.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So my question is</p>
<ul>
<li>Why it is 100% when I just use half of the GPU memory.</li>
<li>How to make most of my GPU so that I could faster training and (perhaps) better training results.</li>
</ul>
<p>Thank you in advance!</p>

ptrblck: <ol>
<li>
<p>The utilization gives the percentage of executing kernels during a time frame. It neither gives you the percentage of used memory nor the percentage of busy multiprocessors.</p>
</li>
<li>
<p>You could try to increase the batch size with by a smaller amount and try to max out the memory.<br>
However, not that too large batch sizes might yield worse results for your model, so you might want to experiment with the batch size a bit.</p>
</li>
</ol>
Target Transforms for keypoints in Image

umarkhalid: <p>Hi,<br>
If my training image is (3,128,96), but my target is just a coordinate (x,y), can I apply transform=transforms.Compose ([ transforms.ColorJitter(), transforms.ToTensor()] on both the label(target) and training image? I can see it gives me some sort of error when I use ColorJitter. One of the errors is TypeError: brightness should be a single number or a list/tuple with lenght 2</p>

ptrblck: <p>This won’t work, since you are trying to apply image transformations on coordinates.<br>
Neither <code>ColorJitter</code> not <code>ToTensor</code> should be used on your target.</p>
<p>If you are using <code>RandomCrop</code>, you have to crop the target coordinates in the same way of course.<br>
The easiest way would probably be to use the fuctional API of <code>torchvision.transforms</code>.</p>
Strange CUDA error with pytorch 1.1.0 CUDA 9

alwynmathew: <p>Strange error while running on the server with pytorch 1.1.0, cudatoolkit 9.0.</p>
<pre><code class="lang-auto">File "main.py", line 61, in main
    model = Model(args)
  File "train.py", line 63, in __init__
    norm_layer=None).to(self.device)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 386, in to
    return self._apply(convert)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 199, in _apply
    param.data = fn(param.data)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 384, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA error: unknown error
</code></pre>
<p>I really like to know the source of this error? Any help is appreciated.</p>

ptrblck: <p>Which GPU are you using?<br>
Does your code work with the latest PyTorch release (<code>1.3.1</code>) or with CUDA10.1?</p>

alwynmathew: <p>It worked on PyTorch <code>1.1.0</code> with CUDA 10 and PyTorch <code>1.0.0</code> with CUDA 9 but not on PyTorch <code>1.1.0</code> with CUDA 9. I wonder what exactly is the issue?</p>
<p>GPU: GTX 1080Ti</p>

ptrblck: <p>I’m not sure, what the error might be pointing to.<br>
Do you necessarily need to use PyTorch <code>1.1.0</code> or could you upgrade?<br>
The error might have been fixed in the latest release, so that debugging it might not be worth the effort.</p>

alwynmathew: <p>Our server still runs Nvidia Driver Version 384.130, thus I can max go up to CUDA 9. I downgraded Pytorch 1.1.0 to Pytorch 1.0 with CUDA 9. Thank you.</p>
<p>On a different note, What is the safest way to update Nvidia driver on a Ubuntu 16.04 server with around 20 active users at any instance without disturbing any running processes? Because whenever I have try to update Nvidia driver on my Ubuntu 16.04 workstation, something or the other breaks.</p>
Runtime Error: Expected scalar Long, but got Float for argument #2 'mat2'

George_Wu: <p>I am currently trying to do binary classification, so I’ve set up a simple one input to 2 output neural network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1,2)
    def forward(self, x):
        x = self.fc1(x)
        return x
</code></pre>
<p>Using cross entropy loss, I attempt to classify inputs to a 0 or 1 labelling of data.<br>
However, when attempting to train this model, there seems to be an issue when calculating a prediction:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
    for x_batch, y_batch in train_loader:
        x_batch = x_batch.to('cpu')
        print(x_batch.type())
        y_batch = y_batch.to('cpu')
        print(y_batch.type())

        net.train()

        yhat = net(x_batch.long()).unsqueeze(dim=0)

        loss = loss_fn(yhat, y_batch)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
</code></pre>
<p>x_batch, and y_batch are both long, but when calculating x = self.fc1(x) in forward, this error comes up:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "learned_bloom_filter.py", line 65, in &lt;module&gt;
    yhat = net(x_batch).unsqueeze(dim=0)
  File "/home/george/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "learned_bloom_filter.py", line 16, in forward
    x = self.fc1(x)
  File "/home/george/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/george/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/george/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2' in call to _th_mm
</code></pre>
<p>I’ve searched around and found that I have to turn the weight of a Linear to scalar, but I’m not sure if this is the correct solution, nor do I know how to do it. Can someone advise?</p>

ptrblck: <p><code>nn.Linear</code> uses <code>float32</code> parameters by default, so your input tensor should also have the same data type.<br>
Cast your <code>x_batch</code> to <code>float()</code> and run the code again.</p>
Which one to use? Dataparallel vs DistributedDataParallel

surya00060: <p>I have a machine with 32v CPUs and 4 GPUs. I need to train an imagenet model from scratch. So which of these Dataparallel or DistributedDataParallel would be faster to have high utilization of 4 GPUs?<br>
Thanks in advance for helping</p>

tom: <p>DistributedDataParallel is more well-liked even on a single machine when there is benefit in also parallelizing the CPU overhead of the model.</p>
<p>Best regards</p>
<p>Thomas</p>
Reshape 4-dimensional tensor

SU801T: <p>Hi All,</p>
<p>I have a 4 dimensional numpy input array that was created from extracting features using ResNet50.</p>
<p>With a batch size of 5, I have created tensors of:</p>
<pre><code class="lang-auto">torch.Size([5, 2048, 1, 1])
</code></pre>
<p>I’m not really sure what to do if I want to feed this into a simple autoencoder. Would anyone be able to clarify how I should reshape this tensor to feed into an nn.linear layer?</p>
<p>Cheers,</p>

KFrank: <p>Hi Taran!</p>
<aside class="quote no-group" data-username="SU801T" data-post="1" data-topic="60420" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>With a batch size of 5, I have created tensors of:</p>
<pre><code class="lang-auto">torch.Size([5, 2048, 1, 1])
</code></pre>
<p>I’m not really sure what to do if I want to feed this into a simple autoencoder. Would anyone be able to clarify how I should reshape this tensor to feed into an nn.linear layer?</p>
</blockquote>
</aside>
<p>I assume that your problem is the last two length-1 dimensions<br>
of your tensor.  If so, you can use <a href="https://pytorch.org/docs/stable/torch.html#torch.squeeze" rel="noopener nofollow ugc">torch.squeeze()</a> to get rid of<br>
them.</p>
<p>You could then feed your (now <code>torch.Size([5, 2048])</code>)<br>
tensor to, for example, a <code>nn.Linear (2048, 256)</code> layer.</p>
<p>(Note, if you ever have a batch size of 1, this will break by also<br>
killing off the batch-size dimension.  In such a case, you would<br>
have to use <code>torch.squeeze()</code>'s <code>dim</code> argument to specify the<br>
length-1 dimensions your need to get rid of.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

SU801T: <p>Hi K. Frank,</p>
<p>Thanks for the reply.</p>
<p>Great, I had a look into torch.squeeze and created an autoencoder with those inputs. It works fine so thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Taran</p>
Padding 0 in specific index

huangbiubiu: <p><a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad" rel="nofollow noopener"><code>torch.nn.functional.pad</code></a> could only pad number at the edge of tensors. How can I <em>insert</em> numbers, e.g. zeros in specific index?</p>
<p>For example, after pad 0 in index <code>[1 2 3]</code>, the tensor <code>[1000 1001 1002 1003]</code> comes <code>[1000 0 0 0 1001 1002 1003]</code>.</p>
<p>Now I implement it by</p>
<ol>
<li>create a all zero tensor</li>
<li>assign the value by index</li>
</ol>
<pre><code class="lang-auto"># assume x is a tensor with shape NCHW
data = torch.zeros(x.size()[0], full_channel_num, x.size()[2], x.size()[3], device=x.device)
data[:, idx, :, :] = x
</code></pre>
<p>However, I don’t know if it’s recommended to create a new tensor in <code>forward</code> method. What’s the best practice for this operation?</p>
Change the shape of tensor

zahra: <p>Hi,</p>
<p>I want to change the tensor shape from:</p>
<pre><code class="lang-auto">[[[[ 1.,  2.,  3.,  4.],
   [ 5., 6., 7.,  8.]],

  [[ 9.,  10.,  11.,  12.],
   [ 13.,  14.,  15.,  16.]]]]
</code></pre>
<p>to</p>
<pre><code class="lang-auto">[[[[ 1, 2, 3, 4, 9, 10, 11, 12],
   [ 5, 6, 7, 8, 13, 14, 15, 16]]]]
</code></pre>
<p>Thanks</p>

zahra: <p>I found this solution:</p>
<pre><code class="lang-auto">temp1 = test[0,0]
for i in range(1, test.shape[1]):
    temp1 = torch.cat([temp1, test[0, i]], dim=1)
</code></pre>
<p>Is there any faster solution than that?</p>

MalumaDev: <p>Hi,<br>
you can try to use <a href="https://pytorch.org/docs/stable/torch.html?highlight=reshape#torch.reshape" rel="nofollow noopener">torch.reshape</a>:</p>
<pre><code class="lang-auto">x = torch.tensor([[[[ 1.,  2.,  3.,  4.],
   [ 5., 6., 7.,  8.]],
  [[ 9.,  10.,  11.,  12.],
   [ 13.,  14.,  15.,  16.]]]])
x = x.reshape(1,1,2,8)
</code></pre>

zahra: <aside class="quote no-group" data-username="MalumaDev" data-post="3" data-topic="60443">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malumadev/48/17463_2.png" class="avatar"> MalumaDev:</div>
<blockquote>
<p>x = x.reshape(1,1,2,8)</p>
</blockquote>
</aside>
<p>Thanks for your reply.<br>
Your code generates following tensor that is not what I want:</p>
<pre><code class="lang-auto">tensor([[[[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.],
          [ 9., 10., 11., 12., 13., 14., 15., 16.]]]])
</code></pre>

MalumaDev: <p>Sorry,<br>
I had read output too fast, try this:</p>
<pre><code class="lang-auto">x.reshape(4,4).index_select(0,torch.tensor([0,2,1,3])).reshape(1,1,2,8)
</code></pre>

zahra: <p>Thanks for your reply.<br>
The shape of x is different. I think I can not determine the numbers of elements manually because it changes every time.<br>
Is there any solution?</p>

vainaijr: <p>will this work</p>
<pre><code class="lang-auto">a, b = y.split(split_size=1, dim=-3)
torch.cat((a, b), dim=-1)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">torch.cat(y.split(split_size=1, dim=-3), dim=-1)
</code></pre>
Out of memory error with pytorch installation

Nguyen_Minh_Chau: <p>Hi,</p>
<p>I been trying to install pytorch from anaconda and keep getting out of memory issue. My first try is</p>
<p>conda install pytorch torchvision -c pytorch</p>
<p>this feedback out of memory</p>
<p>After research, many sites suggested to include a no cache command, so I try the command to</p>
<p>conda install pytorch torchvision -c pytorch --no-cache-dir</p>
<p>and<br>
–no-cache-dir conda install pytorch torchvision -c pytorch</p>
<p>However, the system reports that --no-cache-dir as unrecognized arguments.</p>
<p>My python is 3.6.5. tk version; 8.6.6</p>

rasbt: <aside class="quote no-group" data-username="Nguyen_Minh_Chau" data-post="1" data-topic="18734">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nguyen_minh_chau/48/4927_2.png" class="avatar"> Nguyen_Minh_Chau:</div>
<blockquote>
<p>–no-cache-dir</p>
</blockquote>
</aside>
<p>I think this is a <code>pip</code>, not a <code>conda</code> command line argument. In this case, try to install it via</p>
<pre><code>pip3 install torch torchvision --no-cache-dir
</code></pre>
<p>then</p>

CSRabbit: <p>This work for me. Thanks:+1:</p>

Akshit_jain: <p>Hi! are you working on windows?<br>
I’m trying this on Ubuntu 18.04 but it’s giving me memoryerror</p>

khaerulumam42: <p>This is also work for me, thanks</p>

xavier: <p>same for me, it always show an error</p>

ptrblck: <p>Could you post the error message here please?</p>

xavier: <p>i have reboot the computer and i works now (with --no-cache-dir).</p>
Tf py_function equivalent in pytorch

ishanic: <p>Tensorflow’s py_function wraps a python function into a TensorFlow op and allows autograd through the graph. Is there an equivalent function in Pytorch - where a numpy tensor is passed into an arbitrary function and wrapped as a Pytorch op.</p>
<p>Thanks.</p>

rustagiadi95: <p>Hello There,</p>
<p>There is an equivalent of this functionality in PyTorch.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a><br>
This article shows how to define a custom function with autograd tracking. You have to define the forward and backward pass of the function in a custom class inheriting torch.autograd.Function and type in ‘your_custom_function.apply’ before its usage to use it as a regular function.</p>
<p>Hope this helps,<br>
All the best.</p>

ishanic: <p>Yes, that works. These scripts are also relevant: <a href="https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html</a></p>
<p>Thanks!</p>
GRU included CNN for text Generator

Pisit_Bee: <p>Hi everybody,<br>
I try to integrate CNN to GRU. My model gets the image through CNN. The features from CNN will pass to the GRU frame by frame. The structure is shown in the picture.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/6635cea4e5b969bf255a08833b0443bf41591cc7.png" data-download-href="https://discuss.pytorch.org/uploads/default/6635cea4e5b969bf255a08833b0443bf41591cc7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/6635cea4e5b969bf255a08833b0443bf41591cc7.png" alt="image" data-base62-sha1="eAbYY8ShxtWFVgkoTTiAKvtN5jN" width="690" height="327" data-dominant-color="C5E1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">881×418 17.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is my example code that implemented follows the above structure.<br>
encoder :</p>
<pre><code class="lang-auto">### input Image size [batch,seq,colorch,hight,weight]
### expext output 
class CNNencoder(nn.Module):
    def __init__(self, input_size, hidden_size,batch_size =5):
        super(CNNencoder, self).__init__()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.hidden_size = hidden_size
        self.modelVGG = models.vgg11(pretrained = False)
        self.modelVGG = self.modelVGG.to(self.device)

        self.adaptor = nn.Linear(8192, self.hidden_size)
        self.adaptor = self.adaptor.to(self.device)
        
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True, bidirectional=False)  ## (inputSize, hidden_size)
        self.gru = self.gru.to(self.device)
        self.batch_size = batch_size

    def forward(self, input, hidden):
        seqs = input.size()[1]
        for indexseq in range(0, seqs):
            inputImageBatch = input[:, indexseq,:,:,:].view(-1,3,128,128)
            features = self.modelVGG.features(inputImageBatch)
            
            flat_features = features.view(features.size(0), 1,-1)  # flatten
            if indexseq == 0:
                output = flat_features
            else:
                output = torch.cat((output, flat_features), dim=1)

        # output = flat_features    ## expected   [batch,seq,features]
        outputAdaptor = self.adaptor(output)
        outputGru, hidden = self.gru(outputAdaptor, hidden)
        return outputGru, hidden

    def initHidden(self):
        return torch.zeros(1, self.batch_size, self.hidden_size, device=self.device)
</code></pre>
<p>If I calculate loss and gradient from loss.backword(). I would like to khow, How CNN parameter will get gradient through time?<br>
Could anyone suggest me to implement follow above structure?</p>
How do i unfreeze the last layer

Jordan_Howell: <p>Hello,<br>
I brought in a resnet model and set the requires_grad to false.  However I changed the last layer and want the requires grad to true.  How do I do that?  I tried this:</p>
<blockquote>
<p>model = models.resnet50(pretrained = True)<br>
for param in model.parameters():<br>
param.requires_grad = False<br>
model.Linear = nn.Sequential(nn.Linear(in_features = 2048, out_features = 1000),<br>
nn.ReLU(),<br>
nn.Dropout(0.4),<br>
nn.Linear(1000, 2),<br>
nn.LogSoftmax(dim=1))<br>
model.Linear.parameters(requires_grad = True)</p>
</blockquote>
<p>That brought this error:</p>
<blockquote>
<p>TypeError: parameters() got an unexpected keyword argument ‘requires_grad’</p>
</blockquote>

albanD: <p>You need to do the same thing you do to set it to false:</p>
<pre><code class="lang-auto">for param in model.Linear.parameters():
    param.requires_grad = True
</code></pre>
Torchvision.__version__

prabu-github: <p>I updated PyTorch as recommended to get version 1.3.1. After the update/uninstall+install, I tried to verify the torch and torchvision version. While torch.<strong>version</strong> prints out 1.3.1 as expected, for torchvision.<strong>version</strong> I get an AttributeError. Is the attempt to find the version of torchvision correct? Can some one help me figure out this error? Also dir(torchvision) does not show <strong>version</strong>.</p>
<p>EDIT: I am on a Windows10 machine without GPU.</p>

prabu-github: <p>Just to add to my question I am on a Windows 10 machine and I do NOT have a GPU.</p>

spanev: <p>Hi <a class="mention" href="/u/prabu-github">@prabu-github</a>,</p>
<p>This has problably something to do with the fact that you do not have a GPU, can you please run:</p>
<pre><code class="lang-auto">from torchvision import _C
if hasattr(_C, 'CUDA_VERSION'):
    cuda = _C.CUDA_VERSION
</code></pre>
<p>this is the generated content of <code>version.py</code>, the code is <a href="https://github.com/pytorch/vision/blob/efb0b265ce62bb9205ba93701628c4b57012001c/setup.py#L53" rel="nofollow noopener">here</a>.</p>

spanev: <p>Also, please try to uninstall all the torch/torchvision packages before reinstalling, by running</p>
<pre><code class="lang-auto">pip uninstall torch torchvision
</code></pre>
<p>until you get the “Cannot uninstall” error.</p>

prabu-github: <p>Thanks <a class="mention" href="/u/spanev">@spanev</a>. As you pointed out, that was the issue.</p>
<p>To summarize my use case based on <a class="mention" href="/u/spanev">@spanev</a> suggestion:<br>
(1) I uninstalled (using conda) pytorch+torchvision.<br>
(2) I realized I did have a MX150 on my new laptop - so installed the GPU version of PyTorch.<br>
(3) Now torchvision.<strong>version</strong> prints out the version as expected.</p>
Torch cpu as a dependency of package

jdb100: <p>Hi, I have a trained model and created a Django app to host this model. Everything works great in development but now as I am trying to package the Django app for production I have the problem that setuptools can’t seem to install pytorch correctly. In my setup.py I have install_requires containing <code>"torch==1.2.0+cpu", "torchvision==0.4.0+cpu</code> and the dependency_links containing <code>https://download.pytorch.org/whl/torch_stable.html</code>.</p>
<p>This seems to be finding torch on pip instead and deciding it can’t install.</p>
<p>I have also tried with install_requires containing</p>
<pre><code class="lang-auto">"torch==1.2.0",
"torchvision=0.4.0",
</code></pre>
<p>and dependency_links containing</p>
<pre><code class="lang-auto">https://download.pytorch.org/whl/cpu/torch_stable.html
</code></pre>
<p>Neither of these seem to work and I can’t find anything online about this issue. Has anyone done this before and how did you do it?</p>

mlazowik: <p>I’ve had some success using URL dependency syntax from <a href="https://www.python.org/dev/peps/pep-0508/" rel="nofollow noopener">PEP 508</a>. I had to commit to a specific python version and architecture, though. It’s fine in my case, I know the environment where the package will be used. I’ve looked for a way to leave out the environment spec, but haven’t found anything.</p>
<p>Example entry in <code>install_requires</code>: <code>"torch @ https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-linux_x86_64.whl"</code></p>
<p>The links can be found here: <a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a>.</p>
Creating a confusion matrix between Baseline Model and Modified Model

Amrit_Das: <p>While working on a problem statement, I came across the following, I would really appreciate if someone could explain to me what it is -</p>
<p>Train  at least two  models</p>
<ol>
<li>To establish a Baseline (using a pre-trained model)</li>
<li>Your Modified algorithm (using the same pre-trained model)</li>
</ol>
<p>Could someone explain to me how to create the two model and calculate a confusion matrix between them. Assuming I am working on Transfer Learning Tutorial as per the following link.</p>
<p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>Any help will be appreciated.</p>
Train dynamic models in parallel

GIOVANNI_COLUCCI: <p>Hi everyone,<br>
I created a small dynamic neural network based on multiple feedforward NN (based on the input data it constricts the computational graph), the input of this network is composed of two dict (one composed of tensors and the other used to choose the network path in the output calculation); the size of each single network is very small and varies from 1 to 9 elements (each element is a feedforward NN).<br>
With a dataset of about 150 elements it takes about 0.3 seconds for a single epoch.<br>
For each network except the result and use it with the ‘real’ value by calculating the ‘difference’ with MSELoss, I sum up all the losses and calculate the gradient with respect to the weights of the feedforward NN (shared weight trick); finally I update the weights with an optimizer.<br>
Since the individual models can be called separately (they only share the feed forward weights NN) is there a way to reduce the calculation time by running the models in parallel?<br>
I tried to use the GPU but the computation time is greater than that of the CPU (why is the model very small?)</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Net, self).__init__()                    # Inherited from the parent class nn.Module
        self.fc1 = nn.Linear(input_size, hidden_size,bias=False)  # 1st Full-Connected Layer:  (input data) -&gt;  (hidden node)
        nn.init.kaiming_normal_(self.fc1.weight)
        self.fc2 = nn.Linear(hidden_size, output_size,bias=False) # 2nd Full-Connected Layer:  (hidden node) -&gt;  (output class)        
        nn.init.kaiming_normal_(self.fc2.weight)
        self.LRelu = nn.LeakyReLU(0.25)
        self.fnn = nn.Sequential(
            self.fc1,
            self.LRelu,
            self.fc2
        ) 
    
    def forward(self,g,gTensor):
        f={}
        root=0
        for key, value in g.items():
            lenghtV=len(value)
            if (lenghtV==0):
                f[key]=self.fnn(gTensor[key])
            else:
                i=1
                z=gTensor[key].clone()
                for n in value:
                    z[i]=f[n]
                    i=i+1
                f[key]=self.fnn(z)
        root=f[key]
        del f
        return root


def train(net,opt,trainGraphSet,testGraphSet,num_epochs,save,deviceName):
        startTime=time.time()
        criterion = nn.MSELoss()
        optimizer = opt
        # # Training loop
        RMSE=[]
        start = time.time()
        for epoch in range(num_epochs): 
            losses=0
            for g,gT,y in trainGraphSet:
                res=net(g,gT)
                loss = criterion(res,y) 
                losses+=loss
            optimizer.zero_grad()
            losses+=net.fc1.weight.norm()+net.fc2.weight.norm()
            losses.backward() 
            optimizer.step()
</code></pre>
<p>I thank in advance who can help me</p>
How should I keep gradients for reassiged variable?

Goldname: <p>I am trying to implement a version of iterative FGSM. I want to iteratively backpropagate and then reassign a variable like so:</p>
<pre><code class="lang-auto">def fgsm_attack(image, epsilon, data_grad):
    # Collect the element-wise sign of the data gradient
    sign_data_grad = data_grad.sign()
    # Create the perturbed image by adjusting each pixel of the input image
    perturbed_image = image + epsilon*sign_data_grad
    # Adding clipping to maintain [0,1] range
    perturbed_image = torch.clamp(perturbed_image, 0, 1)
    # Return the perturbed image
    return perturbed_image
</code></pre>
<p>I iteratively call this function, but the problem is that “perturbed_image” loses its gradients from “image.” But, I cannot set perturbed_image.requires_grad = True because it is not a leaf variable. What’s the correct solution to this?</p>

albanD: <p>So you want to change the value inside image to the perturbed value? Or you want to keep both?</p>
<p>Do you actually want to have the same <code>.grad</code> field as image? Or just make the perturbed image a leaf so that you can get gradients for it?</p>
An error in create Dataset

az009a: <p>I get the data from torchvision.datasets.MNIST.<br>
Then, I try to save the path and labels in a txt named mnist_test.txt,but here is an error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1cb69c843f24ba6e7bfe866504b22aec6807a47.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1cb69c843f24ba6e7bfe866504b22aec6807a47" title="que1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1cb69c843f24ba6e7bfe866504b22aec6807a47.png" alt="que1" data-base62-sha1="yv0SDeHBZoubRm8HsPX9d4QqYDR" width="543" height="500" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">que1</span><span class="informations">958×881 32.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i try to change the type of img,but it doesn’t work.<br>
Thanks for your help !</p>

az009a: <p>i found delete the sentence  “if arr.dtype == bool:<br>
warn(‘%s is a boolean image: setting True to 1 and False to 0’ % fname)”   in _io.py.<br>
it works!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/3536dbd5a0eeef63678fc3f6f3a500f739c23b3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3536dbd5a0eeef63678fc3f6f3a500f739c23b3d" title="qu2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/3536dbd5a0eeef63678fc3f6f3a500f739c23b3d.png" alt="qu2" data-base62-sha1="7AKQ5NlGE5XdfOPSVowSzC0wr7v" width="690" height="308" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">qu2</span><span class="informations">732×327 29.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Unable to import 'weak_module'

Soniya: <p>Hi Everyone,<br>
I’m unable to import ‘weak_module’ from ‘torch._jit_internal’. It says 'ImportError: cannot import name ‘weak_module’. I’m using Pytorch 1.3.0a0+24ae9b5 version. Please help me.</p>

albanD: <p>This is some internal module, it might have been removed/renamed. Why do you want to use it?</p>

Soniya: <p>I’m trying to run EfficientNet which has used the module in the following manner.<br>
‘from torch._jit_internal import weak_module, weak_script_method’</p>

driazati: <aside class="quote no-group" data-username="Soniya" data-post="3" data-topic="60289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/5fc32e/48.png" class="avatar"> Soniya:</div>
<blockquote>
<p>EfficientNet</p>
</blockquote>
</aside>
<p>Which EfficientNet implementation are you using? The decorators its pulling from <code>_jit_internal</code> were deleted in v1.2. Their functionality is now automatic, so they can just be deleted and everything should work the same.</p>

Soniya: <p>I’m using the implementation which is given here <a href="https://github.com/katsura-jp/efficientnet-pytorch" rel="nofollow noopener">https://github.com/katsura-jp/efficientnet-pytorch</a></p>

driazati: <p>It should work if you apply this patch</p>
<pre><code class="lang-diff">diff --git a/model/swish.py b/model/swish.py
index 66adfa5..3f68678 100644
--- a/model/swish.py
+++ b/model/swish.py
@@ -1,10 +1,8 @@
 import torch
 import torch.nn as nn
 from torch.nn.parameter import Parameter
-from torch._jit_internal import weak_module, weak_script_method
 
 
-@weak_module
 class Swish(nn.Module):
     def __init__(self, train_beta=False):
         super(Swish, self).__init__()
@@ -13,7 +11,6 @@ class Swish(nn.Module):
         else:
             self.weight = 1.0
 
-    @weak_script_method
     def forward(self, input):
         return input * torch.sigmoid(self.weight * input)
</code></pre>

Soniya: <p>Where to apply this patch? I’m unable to find the corresponding file from where i have used it as ‘from torch._jit_internal’.</p>

albanD: <p>Hi,</p>
<p>He refers to <a href="https://github.com/katsura-jp/efficientnet-pytorch/blob/master/model/swish.py" rel="nofollow noopener">this file</a> in the repo you linked above.</p>
About the value of CTX

Yongjie_Shi: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/78b920064e05e85b1a0eda184fa6137198dfa97e.png" data-download-href="https://discuss.pytorch.org/uploads/default/78b920064e05e85b1a0eda184fa6137198dfa97e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78b920064e05e85b1a0eda184fa6137198dfa97e_2_690x445.png" alt="image" data-base62-sha1="hdXVmfxNVcKeoHJ0DXf0cbwz6bQ" width="690" height="445" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78b920064e05e85b1a0eda184fa6137198dfa97e_2_690x445.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78b920064e05e85b1a0eda184fa6137198dfa97e_2_1035x667.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78b920064e05e85b1a0eda184fa6137198dfa97e_2_1380x890.png 2x" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1488×960 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t understand what is the meaning of the CTX, while I use it. I didn’t find the default value of CTX.<br>
When I use the grid_sample function, I do not give CTX a value.<br>
So what does the CTX mean?</p>

albanD: <p>Hi,</p>
<p><code>ctx</code> can be seen as the context in which this <code>Function</code> is running. You will get an empty one during the forward that only contains helper functions. The same <code>ctx</code> will be passed to the backward function so you can use it to store stuff for the backward.<br>
It is similar to the <code>self</code> argument when working with python classes.</p>

ecdrid: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="16821">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>ctx</code> can be seen as the context in which this <code>Function</code> is running.</p>
</blockquote>
</aside>
<p>Can an example be given as to using <code>ctx</code> in SDE let’s say?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>Could you specify what you mean by SDE please?</p>

ecdrid: <p>Sorry for a late reply,<br>
I mean where can i use this “ctx” behaviour in Software Development let’s say?<br>
Thanks.</p>

albanD: <p>You can see in my first comment how you should use it.</p>
How do I successfully unpack values from batches?

Jordan_Howell: <p>Hello:</p>
<p>I am running images through an encoder in batches of 10 in order to get vectors back. The data loader unpacks the ID, image and label (even though I’m not using the label).  After running the image through the encoder, I append the vector to a list and the ID to another list.  I would like to append each image and ID to the list by themselves but the code is appending lists of 10 ids and 10 vectors.  How can i append the vectors and IDs separately and not batches of 10?</p>
<p>Below is my code:</p>
<blockquote>
<p>epochs = 1<br>
max_train_batch = 6361<br>
max_test_batch = 1559</p>
<p>policy_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
test_policy_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
train_output = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
test_output = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for i in range(epochs):<br>
model.eval()<br>
for b, (image, label, policy) in enumerate(train_loader):<br>
policy_list.append(policy.data.cpu().numpy())<br>
image = Variable(image).cuda()</p>
<pre><code>    with torch.no_grad():
        output = model(image)
        train_output.append(output.data.cpu().numpy())

    if b == max_train_batch:
        break
        
    b += 1

    print(f'train_set, epoch: {i}, batch: {b}')
    
    #run test set thru
for b, (image, label, policy) in enumerate(test_loader):

    test_policy_list.append(policy.data.cpu().numpy())
    image = Variable(image).cuda()

    with torch.no_grad():
        output = model(image)
        test_output.append(output.data.cpu().numpy())

        if b == max_test_batch:
            break

        b += 1

        print(f'test_set, epoch: {i}, batch: {b}')
</code></pre>
</blockquote>

tom: <p><a href="https://pytorch.org/docs/master/torch.html#torch.split" rel="nofollow noopener"><code>torch.split</code></a> will get you a list of tensors.</p>
<p>Best regards</p>
<p>Thomas</p>

Jordan_Howell: <p>Thanks Thomas.  I tried the following and got an error.</p>
<blockquote>
<p>for b, (image, label, policy) in enumerate(train_loader):<br>
policy_list.append(policy.split.data.cpu().numpy())<br>
image = Variable(image).cuda()</p>
<pre><code>    with torch.no_grad():
        output = model(image)
        train_output.append(output.split.data.cpu().numpy())
</code></pre>
</blockquote>
<p>The policy (ID) and image are tensors so should I split them, save those to a variable then append them to my list?</p>

Jordan_Howell: <p>After reading the docs, it seems this method results in a tensor.  I want the ID and/or outcome of the model to be a float value that is appended to my list, so I can later turn those into a dataframe to be joined to a tabular data set.</p>
Load weights into a nn.Embedding with different size

Audrey: <p>I have two <em>nn.Embeddings</em>, A and B, with different embeddings num.  B contains A. For an example:<br>
A:</p>
<pre><code class="lang-auto">index    weight
1        [0.1, 0.2]
2        [0.4, 0.6]
</code></pre>
<p>B:</p>
<pre><code class="lang-auto">index    weight
1        zeros
2        zeros
3        zeros
</code></pre>
<p>Now I want to copy the weights in A to B. After copy B should be like:<br>
B:</p>
<pre><code class="lang-auto">index    weight
1        [0.1, 0.2]
2        [0.4, 0.6]
3        zeros
</code></pre>
<p>Any workable method? I only know a function <strong>from_pretrained</strong> can achieve weight loading, but it requires the same structure.</p>
<p>Or… anyway to expand A?  I don’t know if nn.Embedding is static.</p>

vainaijr: <p>do you mean something like this,</p>
<pre><code class="lang-auto">a = nn.Embedding(3, 2, padding_idx=2)
list(a.parameters())
</code></pre>
<pre><code class="lang-auto">[Parameter containing:
 tensor([[ 0.7825, -1.5956],
         [-1.9341,  0.7428],
         [ 0.0000,  0.0000]], requires_grad=True)]
</code></pre>
Iterating over tensor in C++

cll: <p>I’m trying to iterate over a tensor element-wize in C++, which is proving difficult.</p>
<p>I’ve been able to get the number of dimensions of a tensor (torch::Tensor tensor) using tensor.dim(), and I’m able to get the size of each dimension using torch::size(tensor, dim), however I can’t figure out to iterate over a tensor. for (int i = 0; i &lt; sizeof(tensor) / sizeof(tensor[0]); i++) only iterates over each element of the first row.</p>
<p>Is there a way to iterate element-wize over a nth dimensional torch::Tensor?</p>

Matej_Kompanek: <p>You could try using tensor.sizes() or tensor.numel()</p>
<pre><code class="lang-auto">torch::Tensor t = torch::randn({ 2, 3, 4 });

float* ptr = (float*)t.data_ptr();
	
// iterate by each dimension
for (int z = 0; z &lt; t.sizes()[0]; ++z)
{
	for (int y = 0; y &lt; t.sizes()[1]; ++y)
	{
		for (int x = 0; x &lt; t.sizes()[2]; ++x)
		{
			printf("Element at [%d %d %d]: %f\n", x, y, z, *ptr++);
		}
	}
}

ptr = (float*)t.data_ptr();

// iterate through all elements
for (int i = 0; i &lt; t.numel(); ++i)
{
	printf("%dth Element: %f\n", i, *ptr++);
}
</code></pre>

tom: <p>Note that this <em>only</em> works for contiguous tensors, so you want to either TORCH_CHECK <code>is_contiguous</code> or call <code>auto tc = t.contiguous()</code>, but in the latter case you cannot reliably modify <code>t</code> anymore.<br>
As a convenient (but not terribly efficient for CPU because you likely won’t be using SIMD with it) method to index an arbitrarily strided tensor, you can use the array-like interface of <code>t.accessor&lt;...&gt;()</code>, but that requires you to know the number of dimensions.<br>
As an alternative, you can do the index calcs yourself using stride and storage_offset.</p>
<p>Best regards</p>
<p>Thomas</p>