How much deep a Neural Network Required for 12 inputs of ranging from -5000 to 5000 in a3c Reinforcement Learning

username: granth_jain
cooked: <p>I am trying to use A3C with LSTM for an environment where states has 12 inputs ranging from -5000 to 5000. I am using an LSTM layer of size 12 and then 2 fully connected hidden layers of size 256, then 1 fc for 3 action dim and 1 fc for 1 value function. The reward is in range (-1,1).</p>
<p>However during initial training I am unable to get good results.</p>
<p>My question is- Is this Neural Network good enough for this kind of environment? Or this bad performance initially is due to lstm?</p>
<p>Below is the code for Actor Critic</p>
<pre><code class="lang-auto">class ActorCritic(torch.nn.Module):

    def __init__(self, params):
        super(ActorCritic, self).__init__()

        self.state_dim = params.state_dim
        self.action_space = params.action_dim
        self.hidden_size = params.hidden_size
        state_dim = params.state_dim
        self.lstm = nn.LSTMCell(state_dim, state_dim)
        self.lstm.bias_ih.data.fill_(0)
        self.lstm.bias_hh.data.fill_(0)
        lst = [state_dim]
        for i in range(params.layers):
            lst.append(params.hidden_size)
        
        self.hidden = nn.ModuleList()
        for k in range(len(lst)-1):
            self.hidden.append(nn.Linear(lst[k], lst[k+1]))
        for layer in self.hidden:
            layer.apply(init_weights)

        self.critic_linear = nn.Linear(params.hidden_size, 1)
        self.critic_linear.apply(init_weights)
        self.actor_linear = nn.Linear(params.hidden_size, self.action_space)
        self.actor_linear.apply(init_weights)
        self.train()

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        inputs = inputs.reshape(1,-1)
        hx, cx = self.lstm(inputs, (hx, cx))
        x = hx
        for layer in self.hidden:
            x = torch.tanh(layer(x))
        return self.critic_linear(x), self.actor_linear(x), (hx, cx)

class Params():
    def __init__(self):
        self.lr = 0.0001
        self.gamma = 0.99
        self.tau = 1.
        self.num_processes = os.cpu_count()
        self.state_dim = 12
        self.action_dim = 3
        self.hidden_size = 256
        self.layers = 2
        self.lstm_layers = 1
        self.lstm_size = self.state_dim
        self.num_steps = 20
</code></pre>

username: Henry_Chibueze
cooked: <p>The LSTM network is a type of Recurrent Neural network used for detecting and recognizing sequencial patterns for some given time steps or time series data.</p>
<p>So except the rewards of u RL environment are gotten in a sequencial manner of exploration / exploitation then I suggest u use a different architecture</p>
<p>Anyways even without all these details I’ll still suggest u change ur architecture just to check if it’s really from the network or from sth else u did or failed to do.</p>
<p>Just saying <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=9" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"><img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=9" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"></p>

username: granth_jain
cooked: <p>Hi,<br>
Thanks for the reply.</p>
<p>Actually I want my model to remember some information about the past and therefore I am using LSTM.<br>
However I am not sure if this precision of states can be handled by the neural network.</p>

username: Henry_Chibueze
cooked: <p>Yes it can be handled by a Neural network<br>
I don’t really see anything other practical way of doing this without a neural network except for some strange reasons, u have access to petabytes of ram and a processor that can process petabytes of data in seconds then the standard Q-learning algorithm will suffice, but u and I know that’s not really possible tho lol😅</p>
<p>So yah</p>
<p>Then again can u give me brief run down of this environment u r using?</p>

username: Henry_Chibueze
cooked: <p>Also when u said:</p>
<aside class="quote no-group" data-username="granth_jain" data-post="3" data-topic="102688">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/granth_jain/48/27523_2.png" class="avatar"> granth_jain:</div>
<blockquote>
<p>I want my model to remember some information about the past and therefore I am using LSTM.</p>
</blockquote>
</aside>
<p>What exactly is it u want the model to remember?<br>
Is it the previous states and actions it took when it was there?</p>

username: granth_jain
cooked: <p>yes, I want it to remember the state information from the past.</p>

username: Henry_Chibueze
cooked: <p>Well u can still use an LSTM with some dense layers as u initially did<br>
There’s really no rule of thumb here or sth like that.</p>
<p>Hmmmm🤔</p>
<p>I’m kinda curious tho. How would it affect the decision it takes next?<br>
Coz from my knowledge given a state the action selected is the one with the maximum approximate value of Q outputted by the network</p>
<p>Or is there sth else u wish to do with this?</p>