How to create dataloader with variable length 3D arrays without padding, then nn.AdaptiveAvgPool3d?

username: Johnny_Tam
cooked: <p>I have a <code>embed_arr_list</code> which is a list of 3d arrays with variable 3rd dimension shape, like this:</p>
<pre><code class="lang-auto">for arr in embed_arr_list:
    print(arr.shape)

(50, 128, 331)
(50, 128, 331)
(50, 128, 331)
(50, 128, 331)
(50, 128, 201)
(50, 128, 201)
(50, 128, 532)
(50, 128, 532)
(50, 128, 532)
(50, 128, 532)

len(embed_arr_list)
10
</code></pre>
<p>I also have a <code>target_list</code>, containing the targets (floats).</p>
<pre><code class="lang-auto">len(target_list)
10
</code></pre>
<p>Then, I want to build a dataloader</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import TensorDataset, DataLoader

tensor_x = torch.Tensor(embed_arr_list) # transform to torch tensor
tensor_y = torch.Tensor(target_list)

my_dataset = TensorDataset(tensor_x,tensor_y) # create your datset
my_dataloader = DataLoader(my_dataset) # create your dataloader
</code></pre>
<p>However, I got the following error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-84-f293256d52d8&gt; in &lt;module&gt;
      3 from torch.utils.data import TensorDataset, DataLoader
      4 
----&gt; 5 tensor_x = torch.Tensor(embed_arr_list) # transform to torch tensor
      6 tensor_y = torch.Tensor(target_list)
      7 

ValueError: expected sequence of length 331 at dim 3 (got 201)
</code></pre>
<p>May I know how to feed the variable length 3d arrays to the dataloader <em>without using padding</em>?</p>
<p>Because I want to use <code>nn.AdaptiveAvgPool3d</code>, which I understand should adapt to variable input dimensions.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I assume you either want to return a single sample from the <code>__getitem__</code> method of your <code>Dataset</code> and apply the pooling layer afterwards, apply the pooling inside the <code>__getitem__</code>, or return a <code>list</code> with samples of different shapes from <code>__getitem__</code> and apply the pooling on each sample inside the training loop.<br>
In all cases you should write a custom <code>Dataset</code> and implement your logic there as <code>TensorDataset</code> expects tensors as its input, which will not work since you won’t be able to stack/concatenate the samples to a single tensor. (Nested tensors could work, but I think this util. is still work in progress.)</p>

username: Johnny_Tam
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ! If I do the second option (i.e. return a <code>list</code> with samples of different shapes from <code>__getitem__</code> and apply the pooling on each sample inside the training loop.), does it mean each training cycle will use one sample only? I think it will make the training quite slow (?).</p>

username: ptrblck
cooked: <p>Yes, that’s correct if the pooling layer is inside the model. However, I thought you would like to apply the pooling layer on the samples first before passing them to the model:</p>
<pre><code class="lang-python">for data_list in loader:
    x = []
    for d in data_list:
        x.append(pool(d))
    x = torch.stack(x)
    out = model(x)
</code></pre>