## 3 Research Design and Methodology
### Asynchronous Advantage Actor-Critic (A3C) Model
#### 3.2.2 Actor-Critic Model
The primary goal of the actor is to maximize the expected returns, which are represented by the policy loss functions:  
\[L_{\text{actor}}=-\log\pi(a|s;\theta)\cdot A(s,a)+\beta H(\pi) \tag{14}\]  
In this equation:  
* \(-\log\pi(a|s;\theta)\) represents the probability of choosing action \(a\) given state \(s\), as determined by the actor network with parameters \(\theta\). * \((A(s,a))\) denotes the advantage function
* \((H(\pi))\) is an entropy term that encourages exploration by ensuring the policy does not become overly deterministic, with \((\beta)\) being a hyperparameter that influences the agent into prioritizing learning policies that consistently favor a specific action over another with greater probability. **Critic Network:** The critic network evaluates the value of the current state, providing a baseline to assist the actor in determining the quality of actions. It takes the state (s) as input and outputs a scalar value V(s;w), where (w) represents the critic network's parameters. The critic's objective is to minimize the mean squared error between the estimated value function (V(s;w)) and the actual discounted return 'R' observed during training. The formula for the critic loss function L_critic is:  
\[L_{\text{critic}}=(R-V(s;w))^{2} \tag{15}\]  
where R is the cumulative discounted return, and V(s,w) is the estimated value function for state (s) with parameters (w). **Neural Layer Structure of Actor and Critic Model** This section presents the neural network architectures for the actor and critic models utilized by our A3C model. Each neural network is implemented using an input layer, multiple hidden layers, and output layers. In this section, we define the action network (actor) and the critic network as neural networks (NNs), integral components of our reinforcement learning model. Each network is designed with specific layers and sizes tailored to their respective roles within the model. **Actor Network:** The neural network layer structure of the Action Network model:  
The actor network's architecture, as shown in Algorithm 1, consists of one input, two hidden, and two output layers. The input layer has twelve neurons and accepts the state representation of the environment. The hidden layers consist of two dense layers, each with 32 neurons and ReLU activation functions. This layer is responsible for the model's ability to learn the input state's complexities. The two output layers then compute two outputs--mean output (mu) and standard deviation (std), respectively. ```
state_input=Input((self.state_dim,)) dense_1=Dense(32,activation='relu')(state_input) dense_2=Dense(32,activation='relu')(dense_1) out_mu=Dense(self.action_dim,activation='tanh')(dense_2) mu_output=Lambda(lambda x: x * self.action_bound)(out_mu) std_output=Dense(self.action_dim,activation='softplus')(dense_2) returntf.keras.models.Model(state_input, [mu_output, std_output])
```  
**Algorithm 1** Actor Neural Network Architecture  
**Critic Network:**  
The neural network layer structure of the critic model is as follows:  
The critic network's architecture, as shown in Algorithm 2, consists of one input layer, two hidden layers, and one output layer. Like the actor network, the input layer has 12 neurons that accept the state representation. The next three layers are the hidden layers. The first two are dense layers, each consisting of 32 neurons and a Relu activation function, followed by a third layer with 16 neurons and a Relu activation function. The final output layer is a dense layer with a single neuron and a linear activation function, which outputs the given state's estimated value, which is crucial in calculating the advantage estimates to train our actor network.