High-level question: how did you implement dataset splitting and standardisation of training data?

username: fkempf92
cooked: <p>Hi Everyone! I have a general question and was hoping for your advice. This is my first question in this forum – I have searched the forum for a while but could not quite find what I am looking for. So if this has been asked before, I am sorry!</p>
<p>I have a dataset, let’s say with 5000 observations (rows) and 10 variables (columns). I want to randomly split my data into training, validation and testing. Moreover, I want my training dataset to be standardised so each column has zero mean and std of 1. Most importantly, there should be no information leakage from the test dataset, so that the mean and std must be calculated from the training set only!</p>
<p>My question is: on a high level, how would you implement this?</p>
<p>I have tried different things, but am not quite happy yet.</p>
<p>First implementation: I split my data into training-validation-testing in my <em>Dataset Class</em> and return standardised datasets - training, validation and testing. Here, by standardised I mean that I standardised all three datasets using the training mean and training std. I subsequently use <strong>one</strong> Dataloader which returns batches from the training-validation-test datasets into my training loop. The batches are not further standardised.</p>
<p>Second implementation: I again have a Dataset Class, but this time it simply returns x and y. In a next step I use <strong>SubsetRandomSampler</strong> to have three different DataLoaders – one for each split. However, I am unsure about how I would now standardise my dataset in this scenario, as to me it seems that I could only standardise my individual batches in the training loop rather than the entire training-validation-test dataset as the DataLoader returns batches. Or would this be more appropriate?</p>
<p>Is there a more elegant way of doing this? I have very little experience with the batch normalisation tool that exists within Pytorch. Can I standardise my data ‘inside the network’ so I do not actually need to return standardised data in the first place?</p>
<p>This may be more of a general ML question – I hope this makes sense!</p>

username: ptrblck
cooked: <p>I would recommend to split the initial dataset either manually or with e.g. <code>sklearn.model_selection.train_test_split</code>.<br>
Once you have the splits, you could create 3 separate <code>Dataset</code> objects, each with an own transformation (where the normalization stats were calculated from the training set) as well as three separate <code>DataLoaders</code>.<br>
This would create a clean splitting in my opinion, which would be easily readable and might thus avoid potential data leaking.</p>
<aside class="quote no-group" data-username="fkempf92" data-post="1" data-topic="89857">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/d6d6ee/48.png" class="avatar"> fkempf92:</div>
<blockquote>
<p>I have very little experience with the batch normalisation tool that exists within Pytorch. Can I standardise my data ‘inside the network’ so I do not actually need to return standardised data in the first place?</p>
</blockquote>
</aside>
<p>BatchNorm layers normalize the activations inside the model, but usually you would still apply a normalization on the model inputs regardless, if batchnorm is used or not.</p>

username: fkempf92
cooked: <p>Hi ptrblck! thank you for your quick response – I greatly appreciate it. This is great advice as I haven’t thought about having three separate Dataset objects yet, but I think this could add to the readability of my code.</p>
<p>Have a great day!</p>

username: CJROCKBALL
cooked: <p>Hi ptrblck,</p>
<p>I tried building on your suggestions;</p>
<ul>
<li>Split data</li>
<li>Make <code>Dataset</code> for each data set, do transformation (I used sklearn’s <code>StandardScaler</code>)</li>
<li>Make separate <code>DataLoaders</code></li>
</ul>
<p>and came up with this</p>
<pre><code class="lang-auto">    class InsData(Dataset):
        def __init__(self, X, y, X_transform=None):
            self.X = X
            self.y = y
            self.X_transform = X_transform
            
        def __len__(self):
            return len(self.y)
        
        def __getitem__(self, idx):

            if self.X_transform is not None:
                # Because it's only reading one line data will be T automatically
                X_data = self.X.iloc[idx,:].to_frame().T
                # Flatten the [1,C] to [C]
                X_data = self.X_transform.transform(X_data).reshape(10)
                X_data = torch.FloatTensor(X_data)
                y_data = torch.LongTensor(self.y.iloc[idx,:].values)
                # y_data will be [Rx1] no [R] flatten in training
                return X_data, y_data
            
            X_data = torch.FloatTensor(self.X.iloc[idx,:].values)
            y_data = torch.LongTensor(self.y.iloc[idx,:].values)
            
            return X_data, y_data
</code></pre>
<pre><code class="lang-auto">
    scaler = StandardScaler()
    scaler.fit(Xtrain)

    trainset = InsData(Xtrain, ytrain, scaler)
    testset = InsData(Xtest, ytest, scaler)
</code></pre>
<p>This works but using the transformation slows down the training a lot. I think it’s because <code>Dataset</code> does the transformation on each row individually. Am I structuring this wrongly or does this structure not make sense for tabular data?</p>
<p>I would also like to add feature engineering to this dataset. As I see it I have these options</p>
<ul>
<li>Do data transformation outside the <code>Dataset</code></li>
<li>Do data transformation on the whole of X inside the <code>Dataset</code> possibly in the <strong>init</strong> section</li>
</ul>
<p>Would you have any suggestions?</p>

username: TheEliasBe
cooked: <p>I recommend a tool such as <a href="https://pypi.org/project/line-profiler/" rel="noopener nofollow ugc">line profiler</a> to debug any performance bottlenecks.</p>
<p>Since <code>__getitem__</code> does not need <code>X_transform</code> as a parameter, I would suggest to move the transformation to the <code>__init__</code> function, so it is only executed once.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="CJROCKBALL" data-post="4" data-topic="89857">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e274bd/48.png" class="avatar"> CJROCKBALL:</div>
<blockquote>
<p>I would also like to add feature engineering to this dataset. As I see it I have these options</p>
<ul>
<li>Do data transformation outside the <code>Dataset</code></li>
<li>Do data transformation on the whole of X inside the <code>Dataset</code> possibly in the <strong>init</strong> section</li>
</ul>
<p>Would you have any suggestions?</p>
</blockquote>
</aside>
<p>It depends on the applied transformation.<br>
I.e. you would need to check if any randomness is needed or if you would “leak” information into different splits if the transformation is applied onto the entire dataset before the splitting.</p>
<p>E.g. the <code>StandardScaler</code> should use the stats of the training dataset only and should then be applied onto the validation and testing dataset. Applying it on all samples, would be considered a data leak.<br>
Any random transformation should be applied on the fly in the <code>__getitem__</code> since you would otherwise just transform all samples once (and keep them static) for the entire training run.</p>

username: wzsmith
cooked: <p>Hi ptrblck,</p>
<p>Thank you for your in-depth responses on the forums! I’m currently working with sequential image classifier using a CNN-Bi-LSTM architecture. Fixed-length sequences (length ~5) of images are fed into my network, which outputs a single label for classification.</p>
<p>My current approach to splitting the data has been creating a <code>Dataset</code> that reads in the image paths and class IDs, and processes the index from <code>__getitem__()</code> to generate sequences in a sliding window fashion. This works perfectly fine, but is it more standard to have this sequence partitioning logic implemented in a <code>Sampler</code> (similar to your reply <a href="https://discuss.pytorch.org/t/how-upload-sequence-of-image-on-video-classification/24865/9">here</a>)?</p>
<p>I’m currently using <code>random_split</code> for my train, validation, and test splits, and have since realized there is data leakage due to the sliding window approach. How do you recommend fixing this while keeping the model robust (without breaking the sequence partitioning logic)? If I don’t allocate my testing data in a large chunk (i.e. random sampling), I’ll lose a good amount of potential sequences.</p>
<p>Here is my <code>Dataset</code> class below:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, img_dir, annotations_file, seq_length, transform=None, target_transform=None):
        self.img_dir = img_dir
        self.img_labels = pd.read_csv(annotations_file, header=None, names=['image', 'class'])
        self.seq_length = seq_length
        self.transform = transform
        self.target_transform = target_transform
        self.class_groups = self.img_labels.groupby('class')

    def __len__(self):
        return sum(len(group) - self.seq_length + 1 for _, group in self.class_groups)

    def __getitem__(self, idx):
        # Find class index and index within the class
        for class_label, group in self.class_groups:
            group_size = len(group) - self.seq_length + 1
            if idx &lt; group_size:
                group_idx = idx
                break
            else:
                idx -= group_size
        
        # Read sequence images
        img_paths = group.iloc[group_idx : group_idx + self.seq_length, 0].tolist()
        images = []
        for img_path in img_paths:
            image = read_image(os.path.join(self.img_dir, img_path)).float()
            if self.transform:
                image = self.transform(image)
            images.append(image)

        images = torch.stack(images)    
        label = torch.tensor(class_label)

        if self.target_transform:
            label = self.target_transform(label)

        return images, label
</code></pre>
<p>Also on another note, will the LSTM learn effectively if I am training on sequences from different classes in my batches?</p>
<p>Really appreciate your help!</p>

username: ptrblck
cooked: <p>Sequential datasets are often split in the temporal dimension, e.g. per-day or multiple of days to make sure data won’t be leaked if sequences are used.<br>
Would this work in your use case? E.g. could you split the dataset into temporal sequences and use the first one for training, the next for validation, and the last temp. sequence for testing?<br>
If so, you could create the corresponding indices (with a sliding window offset at the end) and pass these to <code>Subset</code>s to create the data splits.</p>