What is the recommended format to save data in pytorch?

username: Brando_Miranda
cooked: <p>This is mainly out of curiosity (since one can always use the transform <code>ToTensor()</code> in the dataloader).</p>
<p>But when storing data (e.g. pre-processed data, synthetic data etc.), is there any advantage to saving the data already as a <code>torch.tensor </code> rather than as an image, PIL image thing, numpy array, etc?</p>
<p>fully functionable example e.g.</p>
<pre><code class="lang-auto">#%%

import torch

from pathlib import Path

path = Path('~/data/tmp/').expanduser()

tensor_a = torch.rand(2,3)
tensor_b = torch.rand(1,3)

db = {'a': tensor_a, 'b': tensor_b}

torch.save(db, path/'torch_db')
loaded = torch.load(path/'torch_db')
print( loaded['a'] == tensor_a )
print( loaded['b'] == tensor_b )
</code></pre>
<hr>
<p>Q:</p>
<ul>
<li>if it’s already in tensor format, does that cause issues with <code>ToTensor()</code> Op?</li>
</ul>
<hr>
<p>Note: that we probably can’t preload things to speed things in gpu since the dataloader loading of data is subtle due to cuda multithtreading subtlities.</p>
<hr>
<p>related: <a href="https://discuss.pytorch.org/t/save-a-tensor-to-file/37136" class="inline-onebox">Save a tensor to file</a></p>

username: Brando_Miranda
cooked: <p>Seems that the <code>ToTensor</code> op gives problems. Most likely the best way to save data is in numpy format so that transforms do not complain.</p>
<p>Error:</p>
<pre><code class="lang-auto">TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>sample code that gave error:</p>
<pre><code class="lang-auto">
# saving torch tensors

import torch
import torch.nn as nn
import torchvision

from pathlib import Path
from collections import OrderedDict

path = Path('~/data/tmp/').expanduser()
path.mkdir(parents=True, exist_ok=True)

tensor_a = torch.rand(2,3)
tensor_b = torch.rand(1,3)

db = {'a': tensor_a, 'b': tensor_b}

torch.save(db, path/'torch_db')
loaded = torch.load(path/'torch_db')
print( loaded['a'] == tensor_a )
print( loaded['b'] == tensor_b )

# testing if ToTensor() screws things up
lb, ub = -1, 1
N, Din, Dout = 3, 1, 1
x = torch.distributions.Uniform(low=lb, high=ub).sample((N, Din))
print(x)

f = nn.Sequential(OrderedDict([
    ('f1', nn.Linear(Din,Dout)),
    ('out', nn.SELU())
]))
y = f(x)

transform = torchvision.transforms.transforms.ToTensor()
y_proc = transform(y)
print(y_proc)
</code></pre>
<p>reference for saving data in numpy:</p>
<pre><code class="lang-auto">
# saving data in numpy

import numpy as np
import pickle
from pathlib import Path

path = Path('~/data/tmp/').expanduser()
path.mkdir(parents=True, exist_ok=True)

lb,ub = -1,1
num_samples = 5
x = np.random.uniform(low=lb,high=ub,size=(1,num_samples))
y = x**2 + x + 2

# using save (to npy), savez (to npz)
np.save(path/'x', x)
np.save(path/'y', y)
np.savez(path/'db', x=x, y=y)
with open(path/'db.pkl', 'wb') as db_file:
    pickle.dump(obj={'x':x, 'y':y}, file=db_file)

## using loading npy, npz files
x_loaded = np.load(path/'x.npy')
y_load = np.load(path/'y.npy')
db = np.load(path/'db.npz')
with open(path/'db.pkl', 'rb') as db_file:
    db_pkl = pickle.load(db_file)

print(x is x_loaded)
print(x == x_loaded)
print(x == db['x'])
print(x == db_pkl['x'])
print('done')
</code></pre>
<p>reference: <a href="https://stackoverflow.com/a/62883249/1601580" rel="nofollow noopener">https://stackoverflow.com/a/62883249/1601580</a></p>