Load entire dataset on GPU

harshildarji: <p>I am working on a project which involves implementing RNNs from scratch. As we know, training huge RNNs is a time-consuming task, but I am also facing a problem with <strong>optimizing dataset</strong> import.</p>
<p>For my project work, I have customized <strong><code>torch.utils.data.Dataset</code></strong> as:</p>
<pre><code class="lang-python">class MakeDataset(Dataset):
    def __init__(self, data):
        self.strings = list(data['string'])
        self.valid = list(data['valid'])
        self.len = len(self.valid)
        self.valid_list = [0, 1]

    def __getitem__(self, index):
        return self.strings[index], self.valid[index]

    def __len__(self):
        return self.len
</code></pre>
<p>Following is how I generate <strong>dataloaders</strong> for use in training:</p>
<pre><code class="lang-python">train = MakeDataset(train_data)
test = MakeDataset(test_data)
train_loader = DataLoader(dataset=train, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test, batch_size=batch_size, shuffle=True)
</code></pre>
<p>As of now, these methods are working perfectly without any errors.</p>
<p>The problem arises when trying to train my model on <strong><code>Cuda</code></strong>. My <strong>GPU</strong> utilization is around <strong>15%</strong> while the <strong>CPU</strong> is at <strong>maximum</strong>. I believe this affecting the speed of my training.</p>
<p>I read various answers on the forum about <strong>loading the dataset on the GPU</strong>, but none of which are working for me. It would be a great help if someone could point out a better way to do this.</p>

ptrblck: <aside class="quote no-group" data-username="harshildarji" data-post="1" data-topic="79165">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harshildarji/48/17915_2.png" class="avatar"> harshildarji:</div>
<blockquote>
<p>but none of which are working for me. It would be a great help if someone could point out a better way to do this.</p>
</blockquote>
</aside>
<p>Which approaches have you tried and what is not working for you?<br>
Also, how large is your dataset and how much memory is currently used during the training?</p>

harshildarji: <p>I tried the following approaches:</p>
<ol>
<li>using <code>pin_memory=True</code>
</li>
<li>loading entire dataset in <code>__init__</code> of <code>Dataset</code> as:</li>
</ol>
<pre><code class="lang-python">class MakeDataset(Dataset):
    def __init__(self, key):
        data = pd.read_csv('../input/reberseq/train_data.csv') if key == 'train' else pd.read_csv('../input/reberseq/test_data.csv')
        
        self.strings = list(data['string'])
        self.valid = list(data['valid'])
        
        self.strings, self.valid = self.make_variables(self.strings, self.valid)
        
        self.len = len(self.valid)
        self.valid_list = [0, 1]
        
    def str2ascii(self, string):
        ascii_arr = [ord(s) for s in string]
        return ascii_arr, len(ascii_arr)

    def pad_seq(self, vect_seqs, seq_lens, valid):
        seq_tensor = torch.zeros((len(vect_seqs), seq_lens.max())).long()

        for index, (seq, seq_len) in enumerate(zip(vect_seqs, seq_lens)):
            seq_tensor[index, :seq_len] = torch.LongTensor(seq)

        return seq_tensor, valid

    def make_variables(self, strings, valid):
        seqs_and_lens = [self.str2ascii(string) for string in strings]
        vect_seqs = [s[0] for s in seqs_and_lens]
        seq_lens = torch.LongTensor([s[1] for s in seqs_and_lens])
        valid = torch.LongTensor(valid)
        return self.pad_seq(vect_seqs, seq_lens, valid)

    def __getitem__(self, index):
        return Variable(self.strings[index].cuda()), Variable(self.valid[index].cuda())

    def __len__(self):
        return self.len
</code></pre>
<p>Even after trying all this, My GPU utilization is still around 40% while CPU is always 100%.</p>
<p>My dataset set consists of 25000 <a href="https://cnl.salk.edu/~schraudo/teach/NNcourse/reber.html" rel="nofollow noopener">reber sequence</a>. In size, the entire csv is no more than 700 kB, but still during training, 3 GB of CPU and 1.5 GB of GPU is occupied.</p>

ptrblck: <ol>
<li>
<p><code>pin_memory</code> will used page-locked memory to speed up the transfer between the device and host. You could also use <code>to(non_blocking=True)</code> to transfer the data asynchronously.</p>
</li>
<li>
<p>Your <code>MakeDataset</code> transfers each sample separately in <code>__getitem__</code>, which might be slower than transferring the complete batch to the device in the training loop. If you have enough memory on the device, you should push the complete data in <code>__init__</code> to the device.</p>
</li>
</ol>