PyTorch: How to get data from an LSTM

Vlad_Dumitru: <p>I am new to PyTorch and I am trying to build a reinforcement learning system that uses OpenAI for trying to predict whether or not a stock should be bought or not and at what time.</p>
<pre><code>class NeuronalNetwork(nn.Module):
    def __init__(self, stock_env: StockEnv):
        super(NeuronalNetwork, self).__init__()
        self.stock_env = stock_env
        input_size = len(self.stock_env.normalized_dataframe.columns)
        self.hidden_size = 128
        self.num_layers = 4
        self.kernel = 2
        output_size = self.stock_env.action_space.n
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=self.hidden_size, num_layers=self.num_layers, batch_first=True)
        self.output_layer = nn.Linear(self.hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=output_size)
        self.tanh = nn.Tanh()

    def forward(self, x, hidden=None):
        # N x T x D
        # N - the number of windows sizes
        # T - the window size
        # D - the number of indicators and OHLCV in total

        if len(x.shape) &gt; 2:
            batch_size = x.shape[0]
        else:
            batch_size = 1

        if hidden is None:
            hidden = (
                torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device),
                torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device),
            )
        D = len(self.stock_env.normalized_dataframe.columns)
        T = self.stock_env.window_size
        N = batch_size
        x = x.view(N, T, D).type(torch.FloatTensor).to(device)

        out, (ht, ct) = self.lstm(x, hidden)
        out = self.tanh(out)
        out = self.output_layer(out)
        return out
</code></pre>
<p>My x from <code>forward</code> is representing my data under the form <code>[Number_of_batches x Window_size x Features]</code></p>
<p>For the moment my <code>out</code> will be the shape <code>Number_of_batches x Window_size x Action</code> but what I want to make my model learn is to predict the best action <strong>ONLY</strong> for the 250th element. So does anyone know what can I do in order to obtain an <code>out</code> with a shape of (batch_size x action) where the action is going to be the last element from the column window_sie?</p>
<p>Example:</p>
<pre><code>out.shape =&gt; (batch_size, windows_size, features)

FOR **b** all batch_size:
   batch = [] 
   FOR _ all actions
      batch.append(out[b][250])
   new_out.append(batch)
</code></pre>
<p>And on the end, I will have an <code>out</code> that is going to be batch_size x action where the action is only going to be the action of the 250th element from the window_size.</p>
<p>I’m not sure if makes sense for you what is my question, but it doesn’t just let me know and I will try to explain it differently.</p>

THEKROLL: <p>It is important to consider the bigger picture. Since there is very little information on LSTM and RL, I took the time to <a href="https://medium.thekroll.ltd/practical-application-of-reinforcement-learning-with-lstm-and-pytorch-a-simple-solution-for-a-1578d56a5a49" rel="noopener nofollow ugc">document</a> my experiences. Feel free to go through the related <a href="https://github.com/svenkroll/simple_RL-LSTM" rel="noopener nofollow ugc">code of the GitHub project</a> as well; you can adapt it directly for your purposes and feed the model with stock data.</p>