Embedding Layer and loss

Henrik_Mad: <p>So I am training an LSTM with pytorch which gets a bunch of sequential data (made with a sliding window).</p>
<p>Since those sequential data are categorical data, I want to encode them somehow. I started out with one hot encoding and now I want to do it with embedding.</p>
<p>This is my Model:<br>
class LSTM(nn.Module):<br>
def <strong>init</strong>(self, num_features_df, hidden_size, num_class, num_layers, value_embedding):<br>
super(LSTM, self).<strong>init</strong>()</p>
<pre><code>    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.embedding = nn.Embedding(num_features_df, value_embedding)
    self.lstm = nn.LSTM(value_embedding + 1, hidden_size, num_layers, batch_first=True) 
    self.output_layer = nn.Linear(hidden_size, 1)

def forward(self, X, timeArray):
    batch_size, seq_len = X.size()
    embeddings = self.embedding(X)
    X_combined = torch.cat((embeddings, timeArray.unsqueeze(-1)), dim=-1)
    hidden_states = torch.zeros(self.num_layers, X.size(0), self.hidden_size)#.to(device)
    cell_states = torch.zeros(self.num_layers, X.size(0), self.hidden_size)#.to(device)
    out, _ = self.lstm(X_combined, (hidden_states, cell_states))
    # Take the output from the last layer
    out = self.output_layer(out[:, -1, :])
    return out
</code></pre>
<p>When I change the embedding dimension to lets say 20, then my loss explodes. When I have it at 5, then my loss doesnt decrease below (18) even for only one single sample!</p>
<p>So this is my data before embedding</p>
<p>tensor([[12, 13, 14, 15, 11],<br>
[ 1,  2,  3,  3,  4],<br>
[11,  9, 11, 12, 13],<br>
[ 6,  7,  8,  9,  3],<br>
[ 9,  9, 10, 11,  9],<br>
[11, 12, 13, 14, 15],<br>
[ 0,  1,  2,  3,  3],<br>
[ 9, 10, 11,  9, 11],<br>
[ 0,  0,  0,  1,  2],<br>
[ 2,  3,  3,  4,  5],<br>
[ 9,  9,  9, 10, 11],<br>
[ 3,  4,  5,  6,  7],<br>
[ 8,  9,  3,  9,  9],<br>
[ 5,  6,  7,  8,  9],<br>
[ 4,  5,  6,  7,  8],<br>
[ 7,  8,  9,  3,  9],<br>
[ 0,  0,  0,  0,  1],<br>
[ 9,  9,  9,  9,  9],<br>
[ 3,  3,  4,  5,  6],<br>
[10, 11,  9, 11, 12],<br>
[ 0,  0,  1,  2,  3],<br>
[ 0,  0,  0,  0,  0],<br>
[ 9,  3,  9,  9,  9],<br>
[ 9, 11, 12, 13, 14],<br>
[ 3,  9,  9,  9,  9],<br>
[ 9,  9,  9,  9, 10]])</p>
<p>(Sequence length = 5 and batch size = something like 20)</p>
<p>And this after the embedding (5 dimensions)</p>
<p>tensor([[[ 7.9585e-01, -8.6597e-01,  6.6819e-01, -9.6632e-01, -8.9836e-01],<br>
[ 2.9030e-01, -1.4120e+00,  1.9102e+00, -1.0922e+00,  1.1509e+00],<br>
[-6.6070e-01,  1.4359e+00,  7.1594e-01,  1.0106e+00, -9.4437e-01],<br>
[ 1.8024e-01,  4.5509e-01, -2.2949e-01, -9.4121e-01, -1.3281e+00],<br>
[ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00]],</p>
<pre><code>    [[-1.4323e+00,  1.9156e+00,  8.8717e-01, -3.0498e+00,  1.9944e+00],
     [-2.4164e+00,  2.9312e+00,  1.0857e+00, -3.4247e+00,  6.4080e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [ 7.4022e-01, -1.7842e+00, -5.0039e-01,  1.2958e+00,  2.2418e+00]],

    [[ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00],
     [ 7.9585e-01, -8.6597e-01,  6.6819e-01, -9.6632e-01, -8.9836e-01],
     [ 2.9030e-01, -1.4120e+00,  1.9102e+00, -1.0922e+00,  1.1509e+00]],

    [[ 5.8842e-01, -1.5198e-01,  9.7243e-01, -9.3604e-01, -2.3747e-01],
     [-6.1877e-01, -8.4997e-01, -2.1185e+00, -3.7339e-01,  7.0051e-01],
     [-1.5926e+00,  9.9540e-01, -3.5235e-01,  1.5699e+00, -9.5930e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00]],

    [[-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [ 3.7007e-01, -2.0332e+00, -5.3068e-01,  1.0414e+00, -2.1472e-01],
     [ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01]],

    [[ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00],
     [ 7.9585e-01, -8.6597e-01,  6.6819e-01, -9.6632e-01, -8.9836e-01],
     [ 2.9030e-01, -1.4120e+00,  1.9102e+00, -1.0922e+00,  1.1509e+00],
     [-6.6070e-01,  1.4359e+00,  7.1594e-01,  1.0106e+00, -9.4437e-01],
     [ 1.8024e-01,  4.5509e-01, -2.2949e-01, -9.4121e-01, -1.3281e+00]],

    [[ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [-1.4323e+00,  1.9156e+00,  8.8717e-01, -3.0498e+00,  1.9944e+00],
     [-2.4164e+00,  2.9312e+00,  1.0857e+00, -3.4247e+00,  6.4080e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00]],

    [[-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [ 3.7007e-01, -2.0332e+00, -5.3068e-01,  1.0414e+00, -2.1472e-01],
     [ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00]],

    [[ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [-1.4323e+00,  1.9156e+00,  8.8717e-01, -3.0498e+00,  1.9944e+00],
     [-2.4164e+00,  2.9312e+00,  1.0857e+00, -3.4247e+00,  6.4080e+00]],

    [[-2.4164e+00,  2.9312e+00,  1.0857e+00, -3.4247e+00,  6.4080e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [ 7.4022e-01, -1.7842e+00, -5.0039e-01,  1.2958e+00,  2.2418e+00],
     [ 1.5402e-01,  6.3792e-01,  7.9866e-01, -6.7601e-03,  8.6886e-01]],

    [[-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [ 3.7007e-01, -2.0332e+00, -5.3068e-01,  1.0414e+00, -2.1472e-01],
     [ 4.7741e-02,  1.1636e+00, -1.3477e+00,  9.7652e-01,  1.1309e+00]],

    [[-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [ 7.4022e-01, -1.7842e+00, -5.0039e-01,  1.2958e+00,  2.2418e+00],
     [ 1.5402e-01,  6.3792e-01,  7.9866e-01, -6.7601e-03,  8.6886e-01],
     [ 5.8842e-01, -1.5198e-01,  9.7243e-01, -9.3604e-01, -2.3747e-01],
     [-6.1877e-01, -8.4997e-01, -2.1185e+00, -3.7339e-01,  7.0051e-01]],

    [[-1.5926e+00,  9.9540e-01, -3.5235e-01,  1.5699e+00, -9.5930e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01]],

    [[ 1.5402e-01,  6.3792e-01,  7.9866e-01, -6.7601e-03,  8.6886e-01],
     [ 5.8842e-01, -1.5198e-01,  9.7243e-01, -9.3604e-01, -2.3747e-01],
     [-6.1877e-01, -8.4997e-01, -2.1185e+00, -3.7339e-01,  7.0051e-01],
     [-1.5926e+00,  9.9540e-01, -3.5235e-01,  1.5699e+00, -9.5930e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01]],

    [[ 7.4022e-01, -1.7842e+00, -5.0039e-01,  1.2958e+00,  2.2418e+00],
     [ 1.5402e-01,  6.3792e-01,  7.9866e-01, -6.7601e-03,  8.6886e-01],
     [ 5.8842e-01, -1.5198e-01,  9.7243e-01, -9.3604e-01, -2.3747e-01],
     [-6.1877e-01, -8.4997e-01, -2.1185e+00, -3.7339e-01,  7.0051e-01],
     [-1.5926e+00,  9.9540e-01, -3.5235e-01,  1.5699e+00, -9.5930e-01]],

    [[-6.1877e-01, -8.4997e-01, -2.1185e+00, -3.7339e-01,  7.0051e-01],
     [-1.5926e+00,  9.9540e-01, -3.5235e-01,  1.5699e+00, -9.5930e-01],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01],
     [-1.0042e+00,  9.7925e-01,  3.9989e-01, -1.2930e+00,  2.3922e+00],
     [-3.1364e-01,  5.1960e-02,  8.2539e-01, -1.0360e+00,  3.4423e-01]],

    [[ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [ 5.5398e+00, -1.1879e+01, -3.9847e+00,  7.2435e-01, -1.4676e+01],
     [-1.4323e+00,  1.9156e+00,  8.8717e-01, -3.0498e+00,  1.9944e+00]],
</code></pre>
<p>Any idea of why this happens?</p>