## 6 Actor-Critic Methods
### Proximal Policy Optimization (PPO)
#### 6.3.2 PPO-Based Path Planning for Cleaning Robot  
[PERSON] et al. [42] propose a PPO method integrated with transfer learning (TL), detection of nearest uncleaned tiles (DNUT), reward shaping (RS), and an elite set (ES) to improve cleaning performance by robots. These additions address challenges such as reward sparsity, inefficient exploration, and data inefficiency in reinforcement learning tasks for cleaning robots. Transfer learning allows the agent to leverage knowledge from previously trained environments to initialize training in new ones. This ensures improved convergence and robustness. The transferred policy \(\pi_{\text{TL}}\) initializes the primary policy \(\pi_{\theta}\):  
\[\pi_{\theta}\leftarrow\pi_{\text{TL}}. \tag{176}\]  
The DNUT technique ensures that the agent focuses on uncleaned tiles. The agent detects the nearest uncleaned tile and adjusts the reward \(r_{t}\) accordingly:  
\[r_{t}= \begin{cases}+1,&\text{if moving to an uncleaned tile,}\\ -1,&\text{if moving to a cleaned tile,}\\ -2,&\text{if moving to an obstacle.}\end{cases}  \tag{177}\]  
The reward shaping introduces a stacked value mechanism to encourage consistent positive behavior. The stacked value \(R_{\text{stacked}}\) is computed as follows:  
\[R_{\text{stacked}}=\sum_{i=1}^{k}1.5^{i}, \tag{178}\]  
where \(k\) is the number of consecutive positive rewards. The total reward becomes the following:  
\[R_{t}=r_{t}+R_{\text{stacked}}. \tag{179}\]  
The elite set (ES) ensures the agent avoids learning from poor-performance episodes. If the agent takes more than 500 steps in a single episode, the episode is terminated, and the data are excluded from training. Let \(E_{\text{elite}}\) denote the set of high-performance episodes:  
\[E_{\text{elite}}=\{e\in E\mid\text{performance}(e)>\text{threshold}\}. \tag{180}\]  
The PPO loss function is modified to incorporate these components. The clipped objective is given by the following:  
\[L_{\text{CLIP}}(\theta)=\mathbb{E}_{t}[\min(r_{t}(\theta)A_{t},\text{clip}(r_ {t}(\theta),1-\epsilon,1+\epsilon)A_{t})], \tag{181}\]  
where \(r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{ t}|s_{t})}\) is the probability ratio and \(A_{t}\) is the advantage function:  
\[A_{t}=R_{t}+\gamma V_{\theta}(s_{t+1})-V_{\theta}(s_{t}). \tag{182}\]  
The value network minimizes the temporal difference loss:  
\[L_{V}=\frac{1}{2}\big{(}V_{\theta}(s_{t})-(R_{t}+\gamma V_{\theta}(s_{t+1})) \big{)}^{2}. \tag{183}\]  
As presented in Algorithm 10, the agent learns the optimal policy by combining TL for initialization, DNUT for focus, RS for behavior shaping, and ES for data curation. These components significantly enhance the efficiency and robustness of PPO in cleaning environments. ```
Input: Height-map images, environment observations, transferred policy \(\pi_{\text{TL}}\) Output: Optimized navigation policy \(\pi_{\theta}(a|s)\) Initialize: Policy \(\pi_{\theta}\) with \(\pi_{\text{TL}}\), value network \(V_{\theta}\), elite set \(E_{\text{elite}}\); whileEpisode not terminateddo // Step 1: Feature Extraction Process height-map images and observations through convolutional and dense layers to extract features; // Step 2: Nearest Uncleaned Tile Detection (DNUT) Detect the nearest uncleaned tile and adjust reward: \(r_{t}= \begin{cases}+1,&\text{if moving to an uncleaned tile,}\\ -1,&\text{if moving to a cleaned tile,}\\ -2,&\text{if moving to an obstacle.}\end{cases} \) // Step 3: Reward Shaping (RS) Compute stacked rewards: \(R_{\text{stacked}}=\sum_{i=1}^{k}1.5^{i}\) Compute total reward: \(R_{t}=r_{t}+R_{\text{stacked}}\) // Step 4: Elite Set (ES) Filtering If the episode exceeds 500 steps, terminate and exclude the episode from \(E_{\text{elite}}\).