Data processing as a batch way

taiky: <p>Hi everyone,</p>
<p>Here is my problem:</p>
<p>suppose I have a 1G .csv file, then I will process it which will make it expand to 30G. It’s unacceptable to load the whole file into memory then process it, so I’m considering to use Dataset &amp; Dataloader to do that.</p>
<p>Anyone can tell me how to do that in detail?</p>
<p>Thanks!</p>

ptrblck: <p>You could try to use <code>pandas</code> to read the csv file in chunks.<br>
In your <code>Dataset</code> read the chunks in the <code>__getitem__</code> method with <code>pd.read_csv(..., skiprows=index*chunksize, chunksize=chunksize)</code>.<br>
Note that you have to take care of the <code>__len__</code> of the dataset, since the index should now be in <code>[0, nb_samples/chunksize]</code>.</p>

taiky: <p>Thanks!</p>
<p>I still have some confusions when I followed this method.</p>
<ol>
<li>
<p>What does batch mean<br>
Suppose the csv data has (360000 - 200) lines, so I set<br>
chunk_size = 3600<br>
batch_size = 10<br>
then I found the len(loader) is 10 (360000 = 3600 x 10 x 10) which indicate the batch is for chunks, but not for the lines in one chunk. Every batch it grabs 10 chunks, right?</p>
</li>
<li>
<p>In which place I should process data<br>
The data processing procedure can be inserted into two places, the first place is in " <strong>getitem</strong> (self, index)", the other place is to handle the data in enumerate(loader), which place is best?</p>
</li>
<li>
<p>RuntimeError: inconsistent tensor sizes at THTensorMath.c<br>
I think the data is just a batch of chunks(3600 x 10) in every enumerate(loader), maybe it’s ok to send 3600 x 10 lines to Net(lines), but before this step, it will meet a RuntimeError in enumerate(loader) place. The reason is the total lines is actually less than 360000, so there must be one loader which contains 9 batches of 3600 lines plus 1 batch which is less than 3600 lines(3600 - 200). I don’t how to fix this problem, should I expand every loader into lines?</p>
</li>
</ol>
<p>The pseudo code is like this:</p>
<p>class myDataset(Dataset):<br>
def <strong>init</strong> (self, path):<br>
# init<br>
def <strong>getitem</strong> (self, index):<br>
data = read_chunck(dp)<br>
label = read_chunck(lp)<br>
return data, label</p>
<pre><code>def __len__ (self):
    return math.ceil(filesize / chunksize)
</code></pre>
<p>…<br>
.<br>
.<br>
data = myDataset()<br>
loader = torch.utils.data.DataLoader(data, batch_size, shuffle=True)</p>
<p>for epoch in range(nepochs):<br>
for i, (samples, labels) in enumerate(loader):    # RUNTIME ERROR<br>
samples = Variable(samples)<br>
labels = Variable(labels)<br>
…<br>
outputs = net(samples.float())<br>
…</p>
<p>#!!!<br>
return torch.cat(inputs, dim)<br>
RuntimeError: inconsistent tensor sizes at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/TH/generic/THTensorMath.c:2864</p>

ptrblck: <ol>
<li>
<p>You are right. Every batch will grab 10 chunks of size 3600.</p>
</li>
<li>
<p>I would suggest to preprocess your data in the <code>__getitem__</code> method, since you will most likely wrap your <code>Dataset</code> into a <code>DataLoader</code>, which can load the batches using multi-processing.<br>
Using this, your <code>DataLoader</code> can grab some batches in the background, while your training loop is still busy.</p>
</li>
<li>
<p>I have created a small example snippet to play around:</p>
</li>
</ol>
<pre><code class="lang-auto"># Create dummy csv data
nb_samples = 110
a = np.arange(nb_samples)
df = pd.DataFrame(a, columns=['data'])
df.to_csv('data.csv', index=False)


# Create Dataset
class CSVDataset(Dataset):
    def __init__(self, path, chunksize, nb_samples):
        self.path = path
        self.chunksize = chunksize
        self.len = nb_samples / self.chunksize

    def __getitem__(self, index):
        x = next(
            pd.read_csv(
                self.path,
                skiprows=index * self.chunksize + 1,  #+1, since we skip the header
                chunksize=self.chunksize,
                names=['data']))
        x = torch.from_numpy(x.data.values)
        return x

    def __len__(self):
        return self.len


dataset = CSVDataset('data.csv', chunksize=10, nb_samples=nb_samples)
loader = DataLoader(dataset, batch_size=10, num_workers=1, shuffle=False)

for batch_idx, data in enumerate(loader):
    print('batch: {}\tdata: {}'.format(batch_idx, data))
</code></pre>
<p>The last batch will only have <code>10</code> samples.<br>
Is this code of any help?<br>
I think your data should be divisible without a remainder by <code>chunksize</code>, because otherwise the last batch will have different sizes. At least I cannot come up with a fast solution at the moment. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

rasbt: <p>So each batch will have 100 examples if I see that correctly? In general, I would maybe recommend to convert the CSV file to HDF5, so you could have true shuffling.</p>
<p>Once converted, in <code>__getitem__</code>, you could then use something like</p>
<pre><code>with h5py.File('your_dataset.h5', 'r') as h5f:
    features = h5f['features'][index])
    target = h5f['targets'][index])
return features, target
</code></pre>
<p>Could be a bit slower though since you’d require more <code>__get_item__</code> calls  (100 instead of 10) to get e.g. a batch size of 100</p>

taiky: <p>That’s exactly what I mean! It’s very helpful, thank you!</p>
<p>Right now I realize that the 3rd one is the key problem I met.</p>
<p>The real dataset usually has irregular line numbers such as 361234 which cannot be divisible without a remainder by chunk size. So the chunk size matters rather than batch size, and the lazy way to handle this runtime error is to make <span class="hashtag">#lines</span> be divisible by chunk size, right?</p>

ptrblck: <p>You are right, the lazy way it to drop the remainder of your data, if that’s possible. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
However, have a look at <a class="mention" href="/u/rasbt">@rasbt</a> 's suggestion as well, which seems to have some advantages (true shuffling etc.).</p>

taiky: <p>Yes, thanks for your mention, I also have problems with data persistence.</p>
<ol>
<li>
<p>Split data into train_set and test_set<br>
Previously I loaded all data into memory and random index it to generate train_set and test_set, but right now I don’t know how to do this better. The only way I know is to split .csv file in advance then process them by chunking, is there any better way?</p>
</li>
<li>
<p>Store the processed data<br>
The data processing procedure will expand data X20-30 times so it’s better to dump them. Is there any recommended way/format to do this? I guess I can do this in “”“<strong>getitem</strong>”"" function.</p>
</li>
</ol>

rasbt: <p>I would recommend looking into HDF5. The handling is similar to numpy arrays (with the indexing), but the dataset is not loaded into memory until you access it. I just wrote a quick example for converting a CSV into HDF5 using Iris for illustration purposes; here, imagine the iris dataset is a super large dataset that doesn’t fit into memory</p>
<pre><code class="lang-python">import pandas as pd
import numpy as np
import h5py

# suppose this is a large CSV that does not 
# fit into memory:
csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

# Get number of lines in the CSV file if it's on your hard drive:
#num_lines = subprocess.check_output(['wc', '-l', in_csv])
#num_lines = int(nlines.split()[0]) 
num_lines = 150
num_features = 4

class_dict = {'Iris-setosa': 0,
              'Iris-versicolor': 1,
              'Iris-virginica': 2}

# use 10,000 or 100,000 or so for large files
chunksize = 10

# this is your HDF5 database:
with h5py.File('iris.h5', 'w') as h5f:
    
    # use num_features-1 if the csv file has a column header
    dset1 = h5f.create_dataset('features',
                               shape=(num_lines, num_features),
                               compression=None,
                               dtype='float32')
    dset2 = h5f.create_dataset('labels',
                               shape=(num_lines,),
                               compression=None,
                               dtype='int32')

    # change range argument from 0 -&gt; 1 if your csv file contains a column header
    for i in range(0, num_lines, chunksize):  

        df = pd.read_csv(csv_path,  
                header=None,  # no header, define column header manually later
                nrows=chunksize, # number of rows to read at each iteration
                skiprows=i)   # skip rows that were already read
        
        df[4] = df[4].map(class_dict)

        features = df.values[:, :4]
        labels = df.values[:, -1]
        
        # use i-1 and i-1+10 if csv file has a column header
        dset1[i:i+10, :] = features
        dset2[i:i+10] = labels[0]
</code></pre>
<p>Once you have converted the dataset, you can check and work with it as follows:</p>
<pre><code class="lang-python">with h5py.File('iris.h5', 'r') as h5f:
    print(h5f['features'].shape)
    print(h5f['labels'].shape)
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">(150, 4)
(150,)
</code></pre>
<pre><code class="lang-python">with h5py.File('iris.h5', 'r') as h5f:
    print('First feature entry', h5f['features'][0])
    print('First label entry', h5f['labels'][0])
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">First feature entry [ 5.0999999   3.5         1.39999998  0.2       ]
First label entry 0
</code></pre>
<p>Afer that, you can use the HDF5 database to set up a DataLoader via PyTorch</p>

rasbt: <p>I just put up a full example here (<a href="https://github.com/rasbt/deep-learning-book/blob/master/code/model_zoo/pytorch_ipynb/custom-data-loader-csv.ipynb" rel="nofollow noopener">https://github.com/rasbt/deep-learning-book/blob/master/code/model_zoo/pytorch_ipynb/custom-data-loader-csv.ipynb</a>) that might be handy as a template for your scenario</p>

Egor_Kraev: <p>If you have put your data into an hdf5 dataset, with 0 being the batch dimension, you could use the below code to sample batches batches directly from disk, while splitting into test and training.</p>
<pre><code class="lang-auto">train_loader, valid_loader = train_valid_loaders(DatasetFromHDF5(data_path,'data'),
                                                 valid_fraction=0.1,
                                                 batch_size=BATCH_SIZE,
                                                 pin_memory=use_gpu)


import torch
import h5py
import math, np
from torch.utils.data.dataset import Dataset
from torch.utils.data.sampler import SubsetRandomSampler

class DatasetFromHDF5(Dataset):
    def __init__(self, filename, dataset):
        h5f = h5py.File(filename, 'r')
        self.data = h5f[dataset]

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self, item):
        return self.data[item].astype(float)

def train_valid_loaders(dataset, valid_fraction =0.1, **kwargs):
    num_train = len(dataset)
    indices = list(range(num_train))
    split = int(math.floor(valid_fraction* num_train))

    if not('shuffle' in kwargs and not kwargs['shuffle']):
            #np.random.seed(random_seed)
            np.random.shuffle(indices)
    if 'num_workers' not in kwargs:
        kwargs['num_workers'] = 1

    train_idx, valid_idx = indices[split:], indices[:split]
    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(dataset,
                                               sampler=train_sampler,
                                               **kwargs)
    valid_loader = torch.utils.data.DataLoader(dataset,
                                               sampler=valid_sampler,
                                               **kwargs)
    return train_loader, valid_loader
</code></pre>
<p>The source code lives <a href="https://github.com/ZmeiGorynych/basic_pytorch/blob/master/data_utils/data_sources.py" rel="nofollow noopener">here</a></p>

taiky: <p>I’m following your code to convert csv to hdf5. There is an error when I set num_workers greater than 2:<br>
KeyError: ‘Unable to open object (bad object header version number)’\n’”</p>
<p>Is it a safer way by using hdf5 to set the num_workers=1 in dataloader?</p>

taiky: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Is it a bottleneck to process data in <strong>getitem</strong> ?</p>
<p>I mean everytime when Dataset fetches data through <strong>getitem</strong>  method, it should wait for the data processing. This is done by CPU, right? Although DataLoader will load the batches using multi-processing(by CPU?), it’s still slower than batch-load simple data and batch-process this data.cuda(by GPU), right?</p>
<p>I found it will slow down when I use process(data) in <strong>getitem</strong>, but I haven’t compared with data.cuda since my GPU environment is on the remote server and difficult for debug.</p>

ptrblck: <p>It shouldn’t be a bottleneck and I would even recommend using it!<br>
The <code>__getitem__</code> method is usually called by the CPU and I’m not sure if it’s possible to call it from the GPU or just to transfer the calculations to it.<br>
Using <code>num_workers &gt; 1</code> will use multi-processing on the CPU.<br>
AFAIK even <code>num_workers=1</code> creates another process, while <code>num_workers=0</code> processes the data in the main process.</p>
<p>It’s not slower necessarily, since the data can be loaded and processed by your CPUs while the GPU is busy performing the forward and backward call. After the GPU finishes it can just get the new batch, which is probably already waiting for it. You should also use <code>pin_memory=True</code> in your <code>DataLoader</code> to speed up the data transfer between CPU and GPU.</p>
<p>GPU operations are called in an asynchronously way, so that they are not blocking your CPU.</p>

Egor_Kraev: <p><code>num_workers=1</code> spawns a separate process to load the data (I double-checked in <code>top</code> when running <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>
<p>If you try to use more than one worker, my guess is they will get in each other’s way when trying to read/write from the file, I think that option is more for loading data from sources where it’s easy to have multiple simultaneous handles to it.</p>
<p>In any case I think disk i/o is likely to be the bottleneck in your case, so I woudn’t expect increasing the number of workers to increase performance much.</p>

Egor_Kraev: <aside class="quote no-group" data-username="taiky" data-post="13" data-topic="14154">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taiky/48/9446_2.png" class="avatar"> taiky:</div>
<blockquote>
<p>I found it will slow down when I use process(data) in getitem, but I haven’t compared with data.cuda since my GPU environment is on the remote server and difficult for debug.</p>
</blockquote>
</aside>
<p>By the way, Pycharm’s support for remote development is awesome. I’ve configured my pytorch project to have a <a href="https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html" rel="noopener nofollow ugc">remote interpreter</a> on an AWS box, and together with configuring it as a <a href="https://www.jetbrains.com/help/pycharm/creating-a-remote-server-configuration.html" rel="noopener nofollow ugc">remote server</a> I just write code on my laptop in PyCharm, press ‘debug’, and it auto-uploads it to AWS, and runs it there while debugging on my local laptop! So I can set breakpoints, inspect variables, etc, in my local IDE against the process running on AWS - really nice, I can’t recommed it enough.</p>

taiky: <p>yes, it’s really a good way to do remote work.<br>
But, duo authentication with a remote task queue now … <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=5" title=":sob:" class="emoji" alt=":sob:"></p>

taiky: <p>Finally, I finished the training procedure successfully!</p>
<p>I learn the chunk part from <a class="mention" href="/u/ptrblck">@ptrblck</a>, hdf5 part from <a class="mention" href="/u/rasbt">@rasbt</a>, and the sampler part from <a class="mention" href="/u/egor_kraev">@Egor_Kraev</a>, thanks for all of you guys!</p>
<p>There are two problems I have to mention, in case of any other people might cost time on them.</p>
<ol>
<li>KeyError of unable to open hdf5 file</li>
</ol>
<blockquote>
<p>[… in h5py.h5o.open\nKeyError: ‘Unable to open object (bad object header version number)’\n’]</p>
</blockquote>
<p>It’s easy to get this error when the num_workers is greater than 2, but even num_workers=1 can get this error except the frequency is much lower. It might be an h5py or hdf5 problem, rerun the code can bypass it.</p>
<ol start="2">
<li>RuntimeError: Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes’ failed</li>
</ol>
<blockquote>
<p>[return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)<br>
RuntimeError: Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes’ failed.  at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THNN/generic/ClassNLLCriterion.c:87]</p>
</blockquote>
<p>If you run on gpu, it might be a “triggle device side … error”.</p>
<blockquote>
<p>/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THCUNN/ClassNLLCriterion.cu:57: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCStorage.c line=32 error=59 : device-side assert triggered</p>
</blockquote>
<p>This is a very weird error since I’m definitely sure my target is in [0, n_classes-1], actually just 0 and 1, and also, this is not an always error. Sometimes everything is fine but sometimes it will show up, just like error 1. The only place I suspect is that the target is int32 in hdf5 on the contrary int64 in LongTensor. So I print them out and was shocked by the contents. Some batches can print out 0 and 1 but the error batch print out 0.0…0e0 and 1.12…6e9 something like that, before I cast them. I’m stuck on this error for a long time until I remove all num_workers setting. Besides, I found the smaller the batch size, the lower frequency this error show up.</p>
<p>I think the hdf5/h5py might have some multi-processing reading bugs which is far beyond my ability. So I have to sacrifice the training time to get the output model.<img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=12" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:" loading="lazy" width="20" height="20"></p>

rasbt: <p>Hm, are you trying to open the HDF5 file multiple times via <code>h5py.File</code> maybe?</p>

taiky: <p>I try 2 ways but both of them have those errors sometimes.</p>
<p>First way:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
   ...
   with h5py.File(h5_path, 'r') as h5f:
         dataset = HDF5Dataset(h5f, transform)
         train_loader, valid_loader = loaders(dataset, valid_fraction=0.1, **kwargs)
         model = train_model(net, train_loader, valid_loader, criterion, optimizer, nepochs)
   ...
</code></pre>
<p>Second way</p>
<pre><code class="lang-auto">class HDF5Dataset(Dataset):
    def __init__(self, file, transform=None):
        self.h5f = h5py.File(file, 'r')
...

if __name__ == '__main__':
    dataset = HDF5Dataset(file, transform)
    train_loader, valid_loader = loaders(dataset, valid_fraction=0.1, **kwargs)
    model = train_model(net, train_loader, valid_loader, criterion, optimizer, nepochs)

    dataset.h5f.close()
</code></pre>