## 3 Analysis of Literature Resources
### AI Techniques to Support APS Systems
#### 3.2.2 Reinforcement Learning and Deep Learning Techniques  
Reinforcement Learning (RL) is a subfield of ML that has gained attention for its ability to solve complex decision-making problems through trial-and-error interactions with dynamic environments [3; 47; 36]. Unlike supervised learning, which relies on labeled data but enables an agent to learn optimal actions by maximizing cumulative rewards based on feedback from past experiences--similar to how humans or animals learn [1; 32]--in RL, an agent observes a state (St), takes an action (At), receives a reward (Rt+1), and transitions to a new state (St + 1) [32; 38]. The goal is to learn an optimal policy \(\pi\): S \(\rightarrow\) A that maps states to actions to maximize long-term returns, often guided by value functions like V\(\pi\)(st) and Q\(\pi\)(st, at) [3; 32; 38; 80]. RL excels in its ability to handle environments with uncertainty, making it particularly suitable for real-time control tasks in manufacturing, production scheduling, and maintenance planning [28; 32]. Effective RL implementation requires the careful design of state representation, the action space, and the reward function [32]. A typical RL setup includes an agent, a process representing the real system (e.g., a factory), and an optimizer that mediates decision-making [24]. Multi-Agent RL (MARL) has proven effective in decentralized settings, where agents manage local tasks like order dispatching or machine scheduling [1]. Despite its strengths, RL faces challenges such as reward design, convergence, and computational demands [32]. It performs best when environmental patterns are at least partially deterministic, improving learning efficiency [28; 51]. Modern DRL integrates RL with Deep Neural Networks (DNNs), allowing agents to learn from high-dimensional, unstructured data [47; 22; 41]. Algorithms such as Q-learning, Monte Carlo methods, and SARSA are widely used to approximate optimal policies without full environment models [52]. While powerful, RL is best viewed as complementary to traditional optimization methods [51]. Hybrid approaches that combine RL with classical stochastic optimization or heuristic methods have been shown to improve performance in complex industrial scenarios [28; 32]. Overall, RL offers a flexible and adaptive framework for dynamic industrial environments, particularly when integrated with Deep Learning to enhance state representation and generalization [18; 22; 32; 53]. Deep Learning (DL) is a branch of ML that uses ANNs to map complex, high-dimensional input data to meaningful outputs, making it particularly well-suited for decision-making in unstructured industrial environments [19]. While DL was previously limited by insufficient data and computational power, recent advancements in big data (e.g., IoT sensors) and high-performance computing have unlocked its full potential, especially in tackling NP-hard problems common in manufacturing [54]. ML and D techniques are increasingly being applied in smart manufacturing to extract actionable insights, optimize processes, and improve productivity. A significant advancement in this area is DRL, which combines DL's capacity for handling unstructured data with RL's adaptive decision-making [19; 47]. DRL is particularly effective in dynamic and uncertain environments, such as supply chain and production systems. One promising application is in Zero Defect Management, where DRL dynamically adjusts MPS to maximize service levels while minimizing inefficiencies [13]. By leveraging DNNs, DRL eliminates the need for manual feature engineering or predefined state spaces [18; 47; 56; 57]. This makes it suitable for autonomous planning and scheduling in complex industrial settings. However, DRL is still challenged by high data and computational requirements, limited real-world deployment, and difficulties in generalizing across different environments [28; 51].