# Advancing Additive Manufacturing Through Machine Learning Techniques: A State-of-the-Art Review
## 2 Machine Learning
### Reinforcement Learning  
Unlike supervised learning, RL, another subset of ML, does not rely on pre-existing datasets [39]. Instead, an RL agent (typically a computer program) learns from its experiences by interacting with its surroundings or environment, as depicted in Figure 3. A fundamental assumption in RL is that the environment is fully observable, meaning the agent can determine its environment's configuration or state through observations. Consequently, the agent can choose an action based on the current state. Upon executing the selected action, the environment transitions to another state, and the agent receives a reward as feedback. This learning process is iterative. A mathematical framework known as the Markov decision process is commonly used to describe the interaction between the agent and its environment during learning [96; 97]. The key components of this framework include a state space, an action space, a transition function, and a reward function. The transition function denotes the probability of the agent moving from the current state to the next state after taking an action, while the reward function determines the feedback the agent receives during this transition. The objective of RL for an agent is to accumulate rewards, often referred to as the expected return or utility, to the maximum extent possible. The output of RL corresponds to the agent's optimal behavior function, or the optimal policy, which maps a state to the action the agent should take. There are two primary classes of methods for solving RL problems: model-based and model-free, as compared in Table 4. Model-based RL methods, such as policy iteration and value interaction, aim to determine optimal value functions using dynamic programming techniques [98]. These value functions quantify the expected return an agent can achieve from specific states or state-action pairs. The state value function represents the total reward an agent can attain starting from a particular state. Conversely, the state-action value function, also known as the action value function, indicates the expected return an agent can obtain by starting from a state and taking a specific action. Once the optimal value functions are derived, determining the optimal policy becomes feasible, as the agent tends to select actions associated with the highest values. Another type of RL method is model-free, which assumes that agents have no knowledge of transition probabilities and reward functions, reflecting most practical applications. Early model-free RL methods include the Monte Carlo and temporal difference methods, which are value-based RL methods. Q-learning is another widely used value-based RL method [99]. During the learning process, the agent selects the best action according to the current knowledge of the action value function at each step, receives a reward as feedback, and updates the action values through the Bellman equation [39]. It should be noted that the epsilon-greedy action selection technique is usually adopted to balance exploration and exploitation. Over many episodes, the action values converge to their optimal values, from which the optimal policy can be derived. Conventional Q-learning is a tabular method where a table stores action values with respect to the finite state and action spaces. If the state and action spaces are extremely large or continuous, an ANN can be employed to approximate action values. This method is called deep Q-Network (DQN) and is one of the methods in deep reinforcement learning (DRL) [100]. In addition to value-based RL methods, another type of model-free RL method is policy-based, as explained in Table 5. These methods directly update and converge to the optimal policy rather than the optimal value function. Policy gradient methods are a typical family of policy-based RL methods, with proximal policy optimization (PPO) being a notable example [101; 102].