Efficient way to accumulate item into 1D tensor and not in a list

jpainam: <p>Hi!<br>
Most of the time; i do</p>
<pre><code class="lang-python">my_list = []  # will create the list on the CPU
for x in some_tensor:  # elements of some_tensor are on GPU
     v =  process(x) # Return a tensor(val, device='cuda:0')
     my_list.append(v)

# convert my_list back into a tensor ---&gt; and move the list back to the GPU
my_list = torch.from_numpy(my_list)
my_list = my_list.to(device)
</code></pre>
<p>Is it possible to create an empty tensor <code>my_list = torch.cuda.FloatTensor(...)</code> and directly use it as an accumulator? If yes, how to initialize the size of the <code>list</code>  when i don’t know the size.<br>
Using a list and then converting the list back to a tensor and moving the tensor back on the GPU  is overloading. (I guess). I assume creating a numpy list in the <code>forward</code> function happens on the CPU.<br>
How can I accumulate tensor and do the work while on the GPU?<br>
Thank you.</p>

JuanFMontesinos: <p>I’m sorry but I don’t understand your problem.</p>
<p>You iterate over a tensor? for? In fact when you do that torch enables iteration over the batch size but it’s not usually useful… Numpy does not enable iterators over np.arrays</p>
<p>Anyway you cannot create tensors in the forward pass without breaking the backpropagation.</p>

jpainam: <p>Thank you.</p>
<p>I was trying to generalize my question. I’m not creating a new tensor in the forward and i don’t intend to iterate through my batch.<br>
My problem is, how do I accumulate tensors in a new tensor. I don’t want to use <code>mlist = []</code> and <code>.append</code> which I assume run on the CPU. That basically means moving my data from the GPU to the CPU, accumulate and moving them back. Is there a way to directly accumulate tensors in another tensor.</p>
<p>This usually happens in a loop.</p>

JuanFMontesinos: <p>There is no I guess. If you think about it, a list is agnostic to backpropagation. It does not appear in the graph cut. You could, for example, accumulate tensors of the same dimensions on a new dimension using torch.concatenate, however this would create a joint in the graph cut at the time of backpropagating that may be undesirable for you .<br>
I cannot say 100% it’s not possible but according to pytorch working, which follows the flow of data to backpropagate, it’d always create this joint unless developers created a “gpu list” which I unknown at the time of replying.</p>
<p>Anyway I find this topic very interesting. Based on the code i’ve seen everone uses python lists.</p>

jpainam: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="4" data-topic="31928">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Based on the code i’ve seen everone uses python lists</p>
</blockquote>
</aside>
<p>Thanks, this is exactly what i’m worry about. Is  there an alternative to python lists when we are dealing with tensor running on GPU?<br>
An example</p>
<pre><code class="lang-auto">    def forward(self, xx, neg):
        # xx shape [View, B, C, H, W]
        # neg shape [B, C, H, W]

        xx = xx.transpose(0, 1)
        combined_views = []
        for v in xx:
            v = self.base(v)
            v = self.avgpool(v)
            v = v.view(v.size(0), -1)
            combined_views.append(v)
        p_view = combined_views[0]
        for i in range(1, len(combined_views)):
            p_view = torch.max(p_view , combined_views[i])
</code></pre>
<p>where is this <code>combined_views</code> located?</p>

JuanFMontesinos: <p>I cannot confirm the issue of moving back to cpu and to gpu.</p>
<p>However since you are using cuda tensors they are allocated in the gpu.<br>
import torch</p>
<pre><code class="lang-auto">a=torch.rand(100).cuda()

l=[a,a,a]

l[0].device
Out[4]: device(type='cuda', index=0)
</code></pre>
<p>However I unknow if, as you said, to create this list it is moving back to cpu and then to gpu</p>

albanD: <p>Hi,</p>
<p>If your original list contains <code>Tensor</code>s as your first post indicate, you can just call <a href="https://pytorch.org/docs/stable/torch.html#torch.cat"><code>torch.cat()</code></a> or <a href="https://pytorch.org/docs/stable/torch.html#torch.stack"><code>torch.stack()</code></a> on it to concatenate the tensors into a new one. All elements of the list should be on the same device and the concatenated version will be on the same device as the Tensors in the list.</p>

drevicko: <aside class="quote no-group" data-username="jpainam" data-post="5" data-topic="31928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpainam/48/5870_2.png" class="avatar"> jpainam:</div>
<blockquote>
<p>where is this <code>combined_views</code> located?</p>
</blockquote>
</aside>
<p>Unless I’m mistaken, it’s located on CPU, but it’s a list of references to objects whose data is on GPU. All the data calculations in your code are done on GPU.</p>