Is Numpy array a DataSet?

Intel_Novel: <p>Can someone explain how this construct works w/o problems when the <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">PyTorch documentation</a> shows we need TensorDataset to feed a DataLoader</p>
<pre><code class="lang-auto">x1 = np.array([1,2,3])
d1 = DataLoader( x1, batch_size=3)
</code></pre>

LeviViana: <p><code>x1 = np.array([1,2,3])</code> isn’t a <code>Dataset</code> as properly defined by PyTorch. Actually, <code>Dataset</code> is just a <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#Dataset" rel="nofollow noopener">very simple abstract class (pure Python)</a>.</p>
<p>Indeed, the snippet below works as expected, i.e., it will sample correctly:</p>
<pre><code class="lang-python">import torch
import numpy as np
x = np.arange(6)
d = DataLoader(x, batch_size=2)
for e in d:print(e)
</code></pre>
<p>It works mainly because the methods <code>__len__</code> and <code>__getitem__</code> are well defined for numpy arrays. The <code>__add__</code> method is implemented too, but it doesn’t perform concatenation. Maybe the snippet below will make it clear:</p>
<pre><code class="lang-python">import torch
from torch.utils.data import Dataset, DataLoader

class NpDataset(Dataset):
  def __init__(self, array):
    self.array = array
  def __len__(self): return len(self.array)
  def __getitem__(self, i): return self.array[i]

x = np.arange(3)
y = np.arange(3, 6)

data_x = NpDataset(x)
data_y = NpDataset(y)

for e in DataLoader(x + y, batch_size=1):print(e) # prints 3, 5, 7

for e in DataLoader(data_x + data_y, batch_size=1):print(e) # prints 0, 1, 2, 3, 4, 5

</code></pre>

Intel_Novel: <p>It’s nice when you can share these goodies with your friends. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">n1 = np.array([1,2,3])
d1 = DataLoader( n1, batch_size=3)
t1 = torch.Tensor(n1) 
d2 = DataLoader( t1, batch_size=3)
print(d1)
print(d1.dataset)
print(type(d1.dataset))
print(d2)
print(d2.dataset)
print(type(d2.dataset))
</code></pre>
<hr>
<pre><code class="lang-auto">&lt;torch.utils.data.dataloader.DataLoader object at 0x0000024C52514588&gt;
[1 2 3]
&lt;class 'numpy.ndarray'&gt;
&lt;torch.utils.data.dataloader.DataLoader object at 0x0000024C52515A20&gt;
tensor([1., 2., 3.])
&lt;class 'torch.Tensor'&gt;
</code></pre>
<p>As you can see, in the first case we feed with the numpy the dataset inside dataloader will be numpy, and in the second case will be torch.Tensor.</p>
<p>Looks like no internal conversion as I can see and I would say it is better to feed with Tensors because this way we  can relay on cuda stuff.<br>
Any comments?</p>
<hr>
<h3>Update</h3>
<p>Further check shows that the feed element need to be a sequence. Any kind of sequence where insde are numeric values. There is no container type checking. It can work for single int  <code>0</code>, or <code>list [0,1]</code>, …</p>
<pre><code class="lang-auto">n1 = 0#[0,1] #np.array([])#[1,2,3,...])
d1 = DataLoader( n1, batch_size=3)
t1 = torch.Tensor(n1) 
d2 = DataLoader( t1, batch_size=3)
print(d1)
print(d1.dataset)
print(type(d1.dataset))
print(d2)
print(d2.dataset)
print(type(d2.dataset))
</code></pre>

LeviViana: <aside class="quote no-group" data-username="Intel_Novel" data-post="3" data-topic="47376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/intel_novel/48/9466_2.png" class="avatar"> Intel_Novel:</div>
<blockquote>
<p>Looks like no internal conversion as I can see and I would say it is better to feed with Tensors because this way we can relay on cuda stuff.<br>
Any comments?</p>
</blockquote>
</aside>
<p>No conversion is made. You are right, I’d recommend you to use Tensors instead of Numpy arrays. I encourage you to spend some time checking out the implementation of Dataset and <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader" rel="noopener nofollow ugc">Dataloaders</a>, it’s pure Python. You will see eventually that no conversion is made.</p>
<aside class="quote no-group" data-username="Intel_Novel" data-post="3" data-topic="47376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/intel_novel/48/9466_2.png" class="avatar"> Intel_Novel:</div>
<blockquote>
<p>Further check shows that the feed element need to be a sequence. Any kind of sequence where insde are numeric values.</p>
</blockquote>
</aside>
<p>As I said, the object must have at least the methods <strong>len</strong> and <strong>getitem</strong> defined, otherwise it won’t work. For instance, the method <strong>getitem</strong> in what allows you to use the brackets operator:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; [1, 2, 3].__getitem__(1)
2
&gt;&gt;&gt; [1, 2, 3][1]
2
&gt;&gt;&gt; 
</code></pre>

Intel_Novel: <p>Yes, I checked the example you sent. Great one.</p>
<p>It may be even smart to implement <code>__add__</code> in your original example, because this way you can alter <code>ConcatDataset</code> type when you print the type.</p>
<p><code>d1= DataLoader(x + y, batch_size=1)</code><br>
<code>print(d1.dataset)</code><br>
<code>print(type(d1.dataset))</code> <span class="hashtag">#ConcatDataset</span> type</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import Dataset, DataLoader
import torch.utils.data as data_utils

class NpDataset(Dataset):
  def __init__(self, array):
    self.array = array
  def __len__(self): return len(self.array)
  def __add__(self, ds): 
        return data_utils.TensorDataset(  torch.from_numpy(self.array + ds))
        
        
        
  def __getitem__(self, i): return self.array[i]

x = np.arange(3)
print(x)
y = np.arange(3, 6)
print(y)

data_x = NpDataset(x)
data_y = NpDataset(y)

d1 =DataLoader(x + y, batch_size=1)
    
print("...")
d2 = DataLoader(data_x + data_y, batch_size=1)

print(d1.dataset)    
print(d2.dataset)
</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">[0 1 2]
[3 4 5]
...
[3 5 7]
&lt;torch.utils.data.dataset.TensorDataset object at 0x000001ABF4C8F208&gt;
</code></pre>
<p>I am not sure what is the best practice.<br>
Should we have our <code>DataLoader</code>. <code>dataset</code> as <code>Tensor</code> or <code>np array</code>, or <code>TensorDataset</code>, or ContactDataset.</p>
<p>I assume <code>ContactDataset</code> is very good as I can do <code>d2.dataset.datasets</code>, and list all the <code>`NpDataset`s</code> inside.</p>
<p>But why this may be useful?</p>