How to use dataset larger than memory?

username: Hubert
cooked: <p>I have a dataset consisting of 1 large file which is larger than memory consisting of 150 millions records in csv format.</p>
<p>Should i split this info smaller files and treat each file length as the batch size ?</p>
<p>All the examples I’ve seen in tutorials refer to images. ie 1 file per test example or if using a csv load the entire file into memory first.</p>
<p>The examples for custom dataset classes I’ve seen are as below. <strong>len</strong> returns the entire file length and <strong>getitem</strong> returns an individual record.</p>
<p>I’d have thought files larger than memory would be a common issue in this time of big data ?</p>
<pre><code>def __init__(self, csv_path):

def __len__(self):
    return self.data_len
    
def __getitem__(self, index):
    return self.input[index], self.labels[index]
</code></pre>
<p>Any advice would be greatly appreciated.</p>
<p>thanks</p>

username: ptrblck
cooked: <p>You could use <code>pd.read_csv</code> with the <code>chunksize</code> argument, so that you will only read smaller chunks of your data. Have a look at <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#iterating-through-files-chunk-by-chunk" rel="nofollow noopener">this documentation</a> for an example.</p>

username: DoubtWang
cooked: <p>Can <a href="https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader" rel="nofollow noopener">torch.utils.data.DataLoader</a>  be used to read batch size of data to memory ?</p>
<p>Dataloader. Combines a dataset and a sampler, and provides single- or multi-process iterators over the dataset.</p>

username: ptrblck
cooked: <p>If your <code>Dataset</code> loads the data lazily in <code>__getitem__</code> each worker in your <code>DataLoader</code> will load one complete batch into memory.</p>

username: DoubtWang
cooked: <p>I don’t understand how Dataset loads the data lazily in <code>__getitem__</code>.<br>
It means that the <code>batch size</code> is 1 ?<br>
Thanks  in advance for your explanation.</p>

username: ptrblck
cooked: <p>If your <code>Dataset</code>'s <code>__getitem__</code> looks like this:</p>
<pre><code class="lang-python">def __getitem__(self, index):
      x = Image.open(self.paths[index])  # load lazily
      return x
</code></pre>
<p>then each call to <code>__getitem__</code> will load a single sample using the <code>index</code>.<br>
The <code>Dataloader</code> takes care of loading enough samples to create a whole batch specified by <code>batch_size</code>.</p>

username: DoubtWang
cooked: <p>Thanks for your detailed explanation.<br>
I know what your mean.</p>

username: Hubert
cooked: <p>thanks. I’ve made some progress but my setup doesn’t look to be iterating properly at the moment. the code doesn’t hit my print lines in the enumerate stage.</p>
<p>here’s my code:</p>
<pre><code class="lang-auto">import torch
from torch.utils import data
import pandas as pd

class MyDataset(data.Dataset):
    def __init__(self, csv_path, chunkSize):
        self.chunksize = chunkSize
        self.reader = pd.read_csv(csv_path, sep=',', chunksize=self.chunksize, header=None, iterator=True)
    
    def __len__(self):
        return self.chunksize

    def __getitem__(self, index):
        data = self.reader.get_chunk(self.chunksize)
        tensorData = torch.as_tensor(data.values, dtype=torch.float32)
        inputs = tensorData[:, :-1]
        labels = tensorData[:, 99]
        return inputs, labels 

def main():
    batch_size = 100

    kwargs = {}
    custom_data_from_csv = MyDataset('data/mydata.txt', batch_size)
    train_loader = data.DataLoader(dataset=custom_data_from_csv, batch_size=batch_size, shuffle=True, **kwargs)
            
    for inputData, target in enumerate(train_loader):
        print(inputData)
        print(target)

if __name__ == '__main__':
    main()
</code></pre>
<p>also i wasn’t sure if <strong>len</strong> was meant to be the size of each chunk or the total number of chunks in the entire file.</p>

username: thiagocrepaldi
cooked: <p>The new ChunkDataset API might help you!</p>
<p>The way it works is through a hierarchical sampling, meaning it splits the dataset in chunks (set of examples) which are shuffled. Each chunk has its samples shuffled too (second layer of shuffling). C++ or Python Dataloader will retrieve batches from an internal buffer that holds just a few chunks, not the whole corpus.</p>
<p>In order to use it, all you need to do is to implement your own C++ ChunkDataReader, which parses a single chunk of data. This new chunk datareader is then passed to ChunkDataset, which will handle all the shuffling and chunk by chunk loading for you.</p>
<p>Look at an test example at: DataLoaderTest::ChunkDataSetGetBatch</p>
<p>Currently there is only C++ support, but python bindings are on the way (<a href="https://github.com/pytorch/pytorch/pull/21232" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/21232</a>) and any feedback is welcome</p>

username: Junhao_Wen
cooked: <p>Hi,</p>
<p>Thanks for your explanation, but I am still a little confused. I think if we load the image lazily as you mentioned by using <strong>index</strong>, even the whole dataset requires much more memory than the machine has (let’t say that we have 20000+ 3D images). Pytorch will only take, in each iteration, batch_size * one image’s memory * number of batches &lt;= total memory?</p>
<p>I also encountered this situation, where I have more than 10000 MRI images to train. Here is my <strong><strong>getitem</strong></strong> for my <strong>Datatset</strong> customized class, but my job terminated with any error.</p>
<blockquote>
<p>class MRIDataset(Dataset):</p>
</blockquote>
<pre><code>def __init__(self, caps_directory, tsv_file, transformations=None):
    """
    Args:
        caps_directory (string): Directory of all the images.
        tsv_file (string): File name of the train/test split file.
        transformations (callable, optional): Optional transformations to be applied on a sample.

    """
    self.caps_directory = caps_directory
    self.transformations = transformations

    # Check the format of the tsv file here
    self.df = pd.read_csv(tsv_file, sep='\t')
    if ('age' not in list(self.df.columns.values)) or ('session_id' not in list(self.df.columns.values)) or \
       ('participant_id' not in list(self.df.columns.values)):
        raise Exception("the data file is not in the correct format."
                        "Columns should include ['participant_id', 'session_id', 'diagnosis']")
    self.participant_list = list(self.df['participant_id'])
    self.session_list = list(self.df['session_id'])
    self.age_list = list(self.df['age'])

def __len__(self):
    return len(self.participant_list)

def __getitem__(self, idx):
    img_name = self.participant_list[idx]
    sess_name = self.session_list[idx]
    age = self.age_list[idx]

    image_path = os.path.join(self.caps_directory, 'subjects', img_name, sess_name, 't1', 'dl',
                              img_name + '_' + sess_name + '_skull-removing_reg-linear.nii.gz')

    pt_path = os.path.join(self.caps_directory, 'subjects', img_name, sess_name, 't1', 'dl',
                              img_name + '_' + sess_name + '_skull-removing_reg-linear.pt')

    ## to accelarate the CPU to load image, we convert the nifti image to .pt format
    if not os.path.exists(pt_path):
        save_as_pt(image_path)

    image = torch.load(pt_path)

    # check if the data has NAN value
    if torch.isnan(image).any() == True:
        image[torch.isnan(image)] = 0

    if self.transformations:
        image = self.transformations(image)

    sample = {'image_id': img_name + '_' + sess_name, 'image': image, 'age': age}

    return sample
</code></pre>
<p>I do not know if this is caused by the memory limitation or not.</p>
<p>Thanks in advance</p>
<p>Hao</p>

username: duskybomb
cooked: <p>Hi could you find any solution to your problem? I am stuck with a similar problem where I have numpy array of size greater than my memory. If I split the data per instance/item then the process of accessing the disk for every item makes the dataloader very slow.</p>

username: Aiman_Mutasem-bellh
cooked: <p>Hello, sir about your method is it work with NMT dataset?</p>
<p>I have a NMT dataset in size of 199 MB for Training and 22.3 MB for dev. set. , batch size is 256, and the max-length of each sentence is 50 words. The data is loaded to GPU RAM without any problems when I start training I got  <code>Out of memory error</code>.</p>
<pre><code class="lang-auto">SRC = Field(tokenize= normalizeString, init_token='&lt;sos&gt;', eos_token='&lt;eos&gt;', fix_length = 50, batch_first=True)
TRG = Field(tokenize= normalizeString, init_token='&lt;sos&gt;', eos_token='&lt;eos&gt;', fix_length = 50, batch_first=True) 

train_data, valid_data = TabularDataset.splits(path='./data/',train='SCUT_train.csv',
    validation=''SCUT_.csv' , format='csv',
    fields=[('src', SRC), ('trg', TRG)], skip_header=True)

SRC.build_vocab(train_data, min_freq = 2) 
TRG.build_vocab(train_data, min_freq = 2)

BATCH_SIZE = 128

train_iterator, vali_iterator = BucketIterator.splits((train_data, valid_data), sort_key=lambda x: len(x.src),
     batch_size = BATCH_SIZE, device = device)
</code></pre>
<p>My dataset is small (212.3 MB) and I can’t use the whole training set, what if I use 1GB data or more?</p>

username: Afshin_Samani
cooked: <p>Hi,</p>
<p>following the discussion on the memory management, my original dataset can very well fit into the memory but before I’d able to provide my samples for training I have to do some pre-processing including up-sampling. if I up-sample all the entities in my dataset, that cannot be contained in the memory any longer. so using the custom dataset and dataloder, I was hoping that this would be done within a batch only and avoid surpassing the memory limit.</p>
<p>I have a code like this:</p>
<blockquote>
<p>Blockquote<br>
class NickiDataset(Dataset):</p>
</blockquote>
<pre><code>def __init__(self, X, y, transform=None, target_transform=None):

    self.target = y

    self.Xdata = X

    self.transform = transform

    self.target_transform = target_transform

def __len__(self):

    return len(self.target)

def __getitem__(self, idx):

    data_i = self.Xdata[idx,:,:]

    target_i = self.target[idx,:]

    if self.transform:

        data_i = self.transform(data_i)

    if self.target_transform:

        target_i = self.target_transform(target_i)

    return data_i, target_i
</code></pre>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>class TrnsDt(object):</p>
<pre><code>"""Convert ndarrays in sample to Tensors."""

def __call__(self, sample):

    

    global count

    if count==0:

      print(sample.shape)

    rsmpl=samplerate.resample(sample, 160 , 'sinc_best')

    

    pt_sample = torch.from_numpy(np.transpose(rsmpl))

    if count==0:

      print(pt_sample.shape)

    

    #to_net=pt_sample.tolist()

    #prc_Wv, mdl_wv

    input_values = prc_Wv(pt_sample, return_tensors="pt", padding="longest", sampling_rate= 16e3).input_values

    out=mdl_wv(torch.squeeze(input_values))

    out_proc = out[0].clone().detach()

    if count==0:

      print(out_proc.shape)

  

    return out_proc
</code></pre>
<p>class TrnsDtTrgt(object):</p>
<pre><code>"""Convert ndarrays in sample to Tensors."""

def __call__(self, sample):

    

    pt_sample =  torch.from_numpy(sample)

          

    return pt_sample
</code></pre>
<p>train_ds = NickiDataset(train_dt_in, train_trgt,<br>
transform=TrnsDt(), target_transform= TrnsDtTrgt())</p>
<p>batch_size=16</p>
<p>train_dataloader = DataLoader(dataset= train_ds, batch_size=batch_size, shuffle=True)</p>
<p>num_epoch=1</p>
<p>total_train_sample=len(train_ds)</p>
<p>n_iterations= np.ceil(total_train_sample/batch_size)</p>
<p>global count</p>
<p>count=0</p>
<p>for epoch in range(num_epoch):</p>
<pre><code>for inx, (dt_in, trgt) in enumerate(train_dataloader):

               

    if count==0:

        print("input tensor shape: ", dt_in.shape)

               

    if (inx+1) % 5 == 0:

        print(f'epoch {epoch}/{num_epoch}, step {inx+1}/{n_iterations}, input {dt_in.shape}')
</code></pre>
<p>Any suggestion why this still makes a trouble in memory management?</p>
<p>Best,</p>

username: ptrblck
cooked: <p>I assume “trouble in memory management” means you are running out of host RAM?<br>
If so, make sure that loading the entire (small) dataset fits into the RAM as well as a batch using the larger images (intermediates could also be stored). In case you are using multiple workers, note that each worker would copy the <code>Dataset</code> and thus also the preloaded small dataset.<br>
If you are using the CPU for the actual training then note that also the model training (parameters, forward activations, gradients etc.) also need to be stored in the RAM.</p>

username: Xuhui_Zhou
cooked: <p>Hi ptrblck,</p>
<p>What if my raw data (60000 x 4000 x 10) is in a single file (.pt)? 60000 is the number of training samples and 4000x10 is the input dimension. I did the lazy loading in this way but the ram cannot handle even if my batch_size is very small. Can you help me with this?</p>
<pre><code class="lang-auto">import os
import torch
from torch.utils.data import Dataset

class MyDataset(Dataset):  # create dataset class

    def __init__(self, feature_dir, label_dir):
        self.feature_dir = feature_dir
        self.label_dir = label_dir

        self.labels = torch.load(label_dir)

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        X_data = torch.load(self.feature_dir).float()[index]
        y_label = self.labels.float()[index]

        return X_data, y_label
</code></pre>

username: ptrblck
cooked: <p>In your code snippet you are loading the entire tensor in the <code>__init__</code> method which explains the high memory usage. You could try to store the tensor as a numpy array and use <code>mmap</code> to load chunks of the data.</p>

username: goodboy-hub
cooked: <p>I am confused that loading the dataset in <em>getitem</em> weather would load the whole dataset in memory?</p>

username: Xuhui_Zhou
cooked: <p>Hi ptrblck,</p>
<p>Thanks for your reply! I’ll try this method.</p>

username: aisven
cooked: <p>Hi,</p>
<p>I find this thread very interesting.</p>
<ol>
<li>
<p>I am too searching for a way to lazily load data in chunks or batches from 1 large CSV file (the file is too large to fit into memory of the particular device).</p>
</li>
<li>
<p>Moreover, thereby I am also searching for a way to somehow randomly split this data into X_train, X_valid, X_test, y_train, y_valid, y_test for training, validation, testing, respectively.</p>
</li>
</ol>
<p>2.1 However, the requirement of having validation data is optional, I would already be happy to have a split into just training and test data.</p>
<p>The problem is a classification with 1 target column and c target classes.</p>
<p>Vague ideas are currently to somehow use <code>Dataset</code> and <code>DataLoader</code>. Perhaps somehow using pre-computed indices, or splitting chunks or batches on the fly, or whatever solution might be available.</p>
<p>During my little journey I came by <code>utils.data.random_split</code> and <code>Subset</code> and <code>ChunkDataset</code>. However, so far, I could not figure out how to fulfill the above stated requirements in a pure PyTorch way.</p>
<p>As of today, maybe still in the works, is there a straight-forward way to do the above outlined things (1. with 2. and optionally with 2.1) purely with PyTorch, i.e. without integrating pandas or some self-written C++ code or something like that?</p>
<p>Any further ideas and of course also a comprehensive answer or a hint towards the most modern solution would be greatly appreciated.</p>
<p>Thanks</p>

username: harpone
cooked: <p>You could try to save the csv data as a numpy memmap. Works quite well for language models. Another great alternative is to save it on disk as e.g. chunked zarr array (or use tensorstore). Should be really fast to read with multiple processes since in practice it’s just a bunch of different files.</p>