How train_test_split and dataloader work together

username: John_Deatherage
cooked: <p>I have a Dataset Class (DS) with 21006 rows by 75 feature columns and 1 output column. my Dataset class splits the data into X &amp; y. (and converts X &amp; y into torch tensors…)<br>
(DS.X is 21006 x 75)  (DS.y is 21006 x 1)…  I’ve verified this with print (len(DS.X), len(DS.y)) &amp; print(DS.X.shape, DS.y.shape)</p>
<p>len of DS.X &amp; DS.y  21006 21006<br>
shape of … torch.Size([21006, 75]) torch.Size([21006])</p>
<p>I want to pass it through the function ( X_train, X_test, y_train, y_test = train_test_split(DS.X, DS.y, test_size=.25, shuffle=True))</p>
<p>After the train_test_split, I get:<br>
len of X_train &amp; y_train  15754 15754<br>
shape of … torch.Size([15754, 75]) torch.Size([15754]) (note: 75% of 21006 is 15754)</p>
<p>Now I want to pass X_train &amp; y_train into a dataloader but how???  the dataloader seems to accept a dataset, not a tuple. " dl = DataLoader(ds, batch_size=2, shuffle=True)"</p>
<p>I think I want dl = DataLoader(ds=(X_train,  y_train), batch_size=2, shuffle = True) …</p>
<pre><code>    training_DS = (X_train, y_train)
    dataloader = DataLoader(ds=training_DS, batch_size=2, shuffle = True) 	


    for epoch in range(epochs): #outer loop
	    for step, (X_train, y_train) in enumerate(dataloader): #inner loop
            print (epoch, step, len(X_train), y_train[0]) 
</code></pre>
<p>I get this error message: for step, (X_train, y_train) in enumerate(dataloader):<br>
ValueError: not enough values to unpack (expected 2, got 1)</p>
<p>What am I missing / not understanding?</p>
<p>THANK YOU</p>
<p><a href="mailto:john.david.deatherage@gmail.com">john.david.deatherage@gmail.com</a>  417 527 4042</p>

username: AbdulsalamBande
cooked: <p>The issue here is that the DataLoader expects a Dataset object as its input, not a tuple. You can create a custom Dataset class that takes X_train and y_train as inputs and implements the necessary methods required by DataLoader. Check this site on creating <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">Custom Dataloaders</a>. Or something like below</p>
<pre><code class="lang-auto">class CustomTensorDataset(Dataset):
    def __init__(self, X, y):
        self.X = X
        self.y = y

    def __getitem__(self, index):
        return (self.X[index], self.y[index])

    def __len__(self):
        return len(self.X)
</code></pre>

username: John_Deatherage
cooked: <p>Thank you for taking the time to help me with this… Here is my custom data class:</p>
<p>class Dow30_Dataset(Dataset): # inherits the Dataset class</p>
<p>def <strong>init</strong>(self): # download, read into pandas dataframe<br>
df_Xy = pd.read_csv(r’C:\Users\user\OneDrive\Desktop\NN_1OUT\DataSet_DOW30.csv’)</p>
<h1>
<a name="define-the-x-features-1" class="anchor" href="#define-the-x-features-1"></a>define the X features</h1>
<p>self.X = (df_Xy[[‘RSI_Sc’, ‘RSIsc-1’, ‘RSIsc-2’, ‘RSIsc-3’, ‘RSIsc-4’,<br>
‘RSIsc-5’, ‘RSIsc-6’, ‘RSIsc-7’, ‘RSIsc-8’, ‘RSIsc-9’, ‘ROC_Sc’,<br>
‘ROCsc-1’, ‘ROCsc-2’, ‘ROCsc-3’, ‘ROCsc-4’, ‘ROCsc-5’, ‘ROCsc-6’,<br>
‘ROCsc-7’, ‘ROCsc-8’, ‘ROCsc-9’, ‘STOCH_Sc’, ‘STOCHsc-1’, ‘STOCHsc-2’,<br>
‘STOCHsc-3’, ‘STOCHsc-4’, ‘STOCHsc-5’, ‘STOCHsc-6’, ‘STOCHsc-7’,<br>
‘STOCHsc-8’, ‘STOCHsc-9’, ‘CCI_Sc’, ‘CCIsc-1’, ‘CCIsc-2’, ‘CCIsc-3’,<br>
‘CCIsc-4’, ‘CCIsc-5’, ‘CCIsc-6’, ‘CCIsc-7’, ‘CCIsc-8’, ‘CCIsc-9’,<br>
‘Price_Sc-0’, ‘Price_Sc-1’, ‘Price_Sc-2’, ‘Price_Sc-3’, ‘Price_Sc-4’,<br>
‘Price_Sc-5’, ‘Price_Sc-6’, ‘Price_Sc-7’, ‘Price_Sc-8’, ‘Price_Sc-9’,<br>
‘Volume_Sc-0’, ‘Volume_Sc-1’, ‘Volume_Sc-2’, ‘Volume_Sc-3’,<br>
‘Volume_Sc-4’, ‘Volume_Sc-5’, ‘Volume_Sc-6’, ‘Volume_Sc-7’,<br>
‘Volume_Sc-8’, ‘Volume_Sc-9’, ‘VIX_cls_0’, ‘VIX_cls_1’, ‘VIX_cls_2’,<br>
‘VIX_cls_3’, ‘VIX_cls_4’, ‘VIX_cls_5’, ‘VIX_cls_6’, ‘VIX_cls_7’,<br>
‘VIX_cls_8’, ‘VIX_cls_9’, ‘3MoSc’, ‘1YrSc’, ‘5YrSc’, ‘10YrSc’, ‘30YrSc’]]) # upto but excludes last column</p>
<p>self.X = torch.from_numpy(self.X.to_numpy()).float()</p>
<h1>
<a name="define-the-y-output-class-columns-2" class="anchor" href="#define-the-y-output-class-columns-2"></a>define the Y output class column(s)</h1>
<p>self.y = (df_Xy[‘Output_NB’]) <span class="hashtag">#includes</span> only the last column<br>
self.y = torch.from_numpy(self.y.to_numpy()).float()</p>
<h1>
<a name="determine-the-number-of-rows-in-the-dataframe-df_xy-3" class="anchor" href="#determine-the-number-of-rows-in-the-dataframe-df_xy-3"></a>determine the number of rows in the dataframe df_Xy</h1>
<p>self.n_samples = df_Xy.shape[0]</p>
<h1>
<a name="support-indexing-such-that-dsi-can-be-used-to-get-the-i-th-sample-4" class="anchor" href="#support-indexing-such-that-dsi-can-be-used-to-get-the-i-th-sample-4"></a>support indexing such that DS[i] can be used to get the i-th sample</h1>
<p>def <strong>getitem</strong> (self, index):<br>
return (self.X[index], self.y[index]) # returns a tuple</p>
<p>def <strong>len</strong>(self):<br>
return self.n_samples</p>
<p>you are suggesting I need a new method in the custom data class? Can you explain in pseudo code what you mean?</p>
<p>Thank you!</p>