## 2 Related Work
### RL  
RL is a machine learning method that involves rewarding a behavior in a specific environment to determine its effectiveness and subsequently training to maximize the reward through repetition. Figure 2 shows that reinforcement learning consists of two components: the environment and the agent. RL is defined by the Markov decision process (MDP), which is the assumption that the state at any point in time is only affected by the state immediately prior to that point in time [15]. The Markov assumption [16] is expressed as \(P(s_{t}|s_{1}),s_{2},\ldots,s_{t-1})=P(s_{t}|s_{t-1})\). An MDP is a tuple of an action \(A,\ \text{a reward}\ R,\ \text{and a depreciation rate}\ \gamma\) based on a Markov process (MP). An MDP can be summarized as follows [16]:  
1. \(\text{MDP}=(S,A,P,R,\gamma)\);
2. The state set \(S\ =\ \left\{s_{1},s_{2},\ldots,s_{||s||}\right\}\), representing the set of all possible states in the MDP;
3. The action set \(A=\left\{a_{1},a_{2},\ldots,a_{||a||}\right\}\), representing the set of all actions that an agent can take;
4. The state transition probability \(P_{ss^{\prime}}^{a}=P(s^{\prime}\ |\ s,a)=P[s_{t+1}=s^{\prime}\ |\ s_{t}=s,a_{t}=a]\), representing the probability of changing from any state \(s\) to state \(s^{\prime}\ \text{when taking action}\ a\);
5. The reward function is a function that rewards an agent for an action taken in any state where \(R_{t+1}=R(s_{t},a_{t})\). The environment rewards the agent for taking the action \(a_{t}\) in a certain state \(s_{t}\). Unlike MP, MDP adds a decision-making process, and thus the agent should determine its behavior for each state: 1. The agent must decide what action \(a\) to take in any state \(s_{t}\), which is called a policy. A policy is defined as follows [16]: \[\pi(a\ |\ s)=P[a_{t}=a\ |\ s_{t}=s]\]
2. RL is a learning policy that involves trial and error to maximize the reward. The goodness or badness of an action is determined by the sum of the rewards, which is defined as the return value \(G_{I}\). Figure 2: Reinforcement learning (RL) diagram. The depreciation rate \(\gamma\) and the return determine the weight of the reward, with the weighted sum of the rewards expressed as follows:  
\[G_{t}=R_{t+1}+\gamma R_{t+2}+\ldots+\gamma^{T-t}R_{T+1}\]  
The depreciation rate is typically expressed as \(\gamma\). The value can be set to a value between 0 and 1, with a value closer to 1 revealing more weight is placed on future rewards. In RL, maximizing the reward denotes maximizing the return \(G_{t}\) and not just one reward [16]. In RL, an agent receives a reward for performing a specific action in a particular state and learns to optimize this reward. Through trial and error, the agent selects actions that maximize its reward, and this iterative process optimizes its reward. Unlike supervised learning, RL does not require any pre-prepared data and can be trained with small amounts of data. To address challenges, such as accurately detecting the positions and orientations of objects which robots encounter during pick-and-place tasks, deep learning techniques are used to predict their location and orientation. To accomplish this task, the robot uses its camera to identify objects, and a deep learning model is used to anticipate their positions and alignments. To address the problem of a robot inaccurately picking objects during a pick-and-place task, we used RL to train the robot to accurately grasp objects. By using an RL algorithm, the robot receives rewards for accurately picking up objects and is penalized for any incorrect pickups. Studies have revealed that this method exhibits high accuracy and stability when robots perform pick-and-place tasks [17]. Deep Q-learning (DQL) enables a robot to collect goods in a warehouse and automatically fulfill orders. The robot receives a list of orders, collects the necessary goods, and processes them using DQL algorithms to determine the most efficient path. The optimal behavior is selected by approximating the Q-function using a deep learning model.