## 3 Integration of the PPO Algorithm for Robot Navigation
### PPO Workflow  
The PPO update script is structured as in Figure 3. At each time step, the actor network within the PPO agent exhibits some action \(a_{t}\) and receives a tuple of the current state \(s_{t}\), the action taken \(a_{t}\), the reward received \(R_{t+1}\), the logarithmic probability of the action taken \(log_{prob}(a_{t})\), and the binary episode termination flag \(Flag\) (the flag takes a value of either 1 or 0). The tuple is then stored into a trajectory, which will be used to update the PPO agent every other 500 time steps, where each time step is at the millisecond scale. During each update, the actor network outputs the policy of the current state \(\pi_{\theta}\), which will be used to calculate the probability ratio \(r_{t}(\theta)\). The state value returned by the critic network is then used to estimate the advantage function \(\hat{A}_{t}\). Then both \(\hat{A}_{t}\) and \(r_{t}(\theta)\) are used to calculate the actor network objective function \(L^{CLIP}(\theta)\), which is used to update the actor neural network weights. The aforementioned calculations are boxed in blue to show that they are specific to the updates of the actor network. At the end, the current policy \(\pi_{\theta}\) is saved as the previous policy \(\pi_{\theta_{old}}\), and the current policy \(\pi_{\theta}\) is updated using stochastic gradient descent (SGD). The current value function \(V_{\mu}(s_{t})\) is updated using SGD, based on the error between predicted and actual values. The aforementioned calculations are boxed in red to show that they are specific to the updates of the critic network. Note that the solid arrow lines represent direct maneuver of the variables, whereas the dotted arrow lines represent the updates of the networks.  
Figure 1: (a) Agent learning its surroundings using LiDAR data while navigating to the goal. The agent has four discrete actions available in this environment, where Action 1 is not moving, Action 2 is turning clockwise, Action 3 is turning counter-clockwise, and Action 4 is moving forward. (b) Coordinate frame of the agent, where 0 rad is the forward direction of the agent.  
Figure 3: PPO agent update workflow. The PPO agent is updated every other 500 time steps. Each of the trajectory memory tuple contains the current state \(s_{t}\), the action taken \(a_{t}\), the reward received \(R_{t+1}\), the logarithmic probability of the action taken \(log_{prob}(a_{t})\), and the binary episode termination flag \(Flag\). The goal of the actor network here is to maximize the value function output by the critic network. The operations within the blue box are specific to the actor network updates, whereas the operations within the red box are specific to the critic network updates. Note that the solid arrow lines represent direct maneuver of the variables, whereas the dotted arrow lines represent the updates of the networks.  
Figure 2: (**a**) Structure of the actor network. The network takes 6 inputs. The LiDAR data input is composed of 360 infrared readings of distances between the agent and the obstacles around it. The past action input is composed of left and right wheel speeds from the previous time step. The other 4 inputs are of single values. The inputs are sent through a ReLU activation function to a hidden layer of 256 nodes. Afterwards, they are sent through another ReLU activation function to another hidden layer of 256 nodes. The output is then sent through first a ReLU and then a SoftMax activation function, which would be either of the 4 action values. (**b**) Structure of the critic network. The network takes the same inputs as the actor network. The output is the value of the agent’s current state.