5 minutes per epoch on tabular data?

username: Wesley_Neill
cooked: <p>Hello all,</p>
<p>I am training an auto encoder on tabular data that has 272 features that are mostly sparse one-hot encodings. I am experiencing incredibly slow training. In fact it appears to possibly be training faster on CPU than on GPU. <strong>The time to train one epoch with a batch size of 5 is 5 minutes.</strong></p>
<p>I’m hoping someone can provide some insight into why this is happening. Thank you in advance.</p>
<p>My set-up is a very basic fully connected network:</p>
<pre><code class="lang-auto">class FCNetwork(nn.Module):
    """Fully Connected Network Class"""
    def __init__(self, n_input, layers, n_output, act=('relu', nn.ReLU())):
        """
        :param n_input: Integer. Size of input vector.
        :param layers: Tuple. containing the desired hidden layer architecture.
        :param n_output: Size of output vector.
        :param act: Tuple ('name', act_func). The first element should be a string
            describing the activation function. The second element is the activation
            function itself. Default is ``('ReLU', nn.ReLU())``.
        """
        super().__init__()
        self.input = nn.Linear(n_input, layers[0])
        self.hidden = self.init_hidden(layers, activation=act)
        self.output = nn.Linear(layers[-1], n_output)
        
    def init_hidden(self, layers, activation, dropout=0.0):

        n_layers = len(layers)
        modules = OrderedDict()
        a_name = activation[0]
            
        modules[f'{a_name}_in'] = activation[1]
        
        for i in range(n_layers - 1):
            modules[f'fc{i}'] = nn.Linear(layers[i], layers[i + 1])
            modules[f'{a_name}{i}'] = activation[1]
            modules[f'drop{i}'] = nn.Dropout(p=dropout)
            
        modules[f'{a_name}_out'] = activation[1]
        
        return nn.Sequential(modules)
            
    def forward(self, x):
        x = x.float()
        x = self.input(x)
        x = self.hidden(x)
        return self.output(x)
    
</code></pre>
<p>And this is my training method:</p>
<pre><code class="lang-auto">def train_encoder(model, trainload, epochs, criterion=nn.MSELoss(), optimizer=optim.Adam, lr=1e-4, testload=None):
    """
    Train auto-encoder reconstruction for given number of epochs.

    :param trainload: a DataLoader object containing training variables and targets used for training.
    :param testload: Optional. a DataLoader containing the validation set. If included, both training and
        validation loss will be tracked and can be plotted using model.plot_loss().
    :param epochs: Number of times the network will view the entire data set
    :param optimizer: Learning method. Default optim.Adam
    :param lr: Learning Rate. Default 0.003
    :param criterion: Loss function
    :return:
    """
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(f'Training model on {device}')
    model = model.to(device)
    opt = optimizer(model.parameters(), lr)

    train_loss = []
    valid_loss = []

    for e in tqdm(range(epochs)):
        running_tl = 0
        running_vl = 0
        for x in trainload:
            x = x.to(device).float()
            opt.zero_grad()
            loss = criterion(model(x), x)
            loss.backward()
            opt.step()
            running_tl += loss.item()

        if testload is not None:
            model.eval()
            with torch.no_grad():
                for x in testload:
                    x = x.to(device).float()
                    loss = criterion(model(x), x)
                    running_vl += loss.item()
                valid_loss.append(running_vl / len(testload))
            model.train()
            
        train_loss.append(running_tl / len(trainload))
    
    return train_loss, valid_loss
</code></pre>

username: Henry_Chibueze
cooked: <p>Number of features = 272<br>
And it’s one hot encoded at that<br>
And u have data of of Length = 5 / batch<br>
Assuming this tabular data is of 5 columns,<br>
The u are having a data of (5, 5, 272) per batch.<br>
This is really really large.</p>
<p>U should try to use a different vectorizing method other than the one hot encoding method simply because the number of feature is way way much and can lead to higher computational time.</p>

username: Wesley_Neill
cooked: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="2" data-topic="103127">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>U should try to use a different vectorizing method other than the one hot encoding method simply because the number of feature is way way much and can lead to higher computational time.</p>
</blockquote>
</aside>
<p>Hey, thanks for the response. The 272 columns are AFTER one-hot encoding. Do you still think that is too large?</p>

username: Henry_Chibueze
cooked: <p>Sorry I don’t really get this<br>
Can u write the shape of the data out for me?</p>
<p>Eg:<br>
(N, C, F) sth like this.</p>

username: Wesley_Neill
cooked: <p>Okay, input looks like this:</p>
<p><code>(batch_size, n_columns) = (5, 272)</code></p>
<p>Since it’s an auto-encoder output is the same shape as the input.</p>

username: Henry_Chibueze
cooked: <p>Ok I see.<br>
Still, I don’t really recommend one-hot vectorization for data this long except there’s really no other way.</p>
<p>What exactly is the context of ur data?</p>

username: Wesley_Neill
cooked: <p>The context of my data is that I have 3 columns worth of continuous data, plus one column that is categorical (with 269 unique categories). These categories are nominal and this is why I chose one-hot.</p>
<p>I am only aware of one-hot and label encoding. I know label-encoding is not good for this use-case. Is there a 3rd option?</p>

username: Henry_Chibueze
cooked: <p>So the the one column that is categorical is ur target data?</p>
<p>I want to confirm b4 I go further to explain something.</p>

username: Wesley_Neill
cooked: <p>Sure, no problem.</p>
<p>It’s an auto-encoder, so the target is the same the input. The network reduces the input to a smaller-dimensional feature vector (embedding) and then tries to re-construct it.</p>
<p>For example, if input is 272 element vector, the network reduces this vector down to something that is say a 100 element vector. It then tries to reconstruct the original vector based on this reduced version.</p>
<p>so <code>x</code> in and <code>x</code> out.</p>

username: Henry_Chibueze
cooked: <p>Ok ok I understand now<br>
So basically there’s an embedder that takes in the input, resizes it into smaller vector, then this smaller vector is passed to another layer (a dense layer) that then tries to predict the actual vector that was initially inputed.</p>
<p>Hmmmm🤔<br>
One hot encoding might actually be the way forward…but then 272 length one hot vector  might be too sparse</p>
<p>Well try encoding ur data with number labels still so that the shape can go from (5, 272) to (5, 1) then the input will be 1 and the output will be a probability score vector of length = 272 so that the index with the highest probability will be the number of that category.</p>
<p>Eg:<br>
Output: [0.8, 0.1, 0.1] will belong to category 0 coz the highest value is at the 0th index.<br>
Sth like that.</p>
<p>How about that?<img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=9" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"><br>
What do u think?</p>

username: Wesley_Neill
cooked: <p>Well, there’s only one way to find out. I will just try it!</p>
<p>Thanks for the time and your ideas!</p>

username: Henry_Chibueze
cooked: <p>Cool<br>
I would like to get ur feedback after🙃</p>