# A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems
## 4 Action Space  
The following chapter details popular actions made available to the agent in different literature approaches. Table 4 groups all the cited papers in relevant categories. Two common strategies compatible with multiple action spaces are \(\epsilon\)-greedy exploration and action masking.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Group** & **Features** & **References** \\ \hline Q-value estimation & & [20,23,30,45,73,78,79,83,84,91,107,110,138,149,150,153,156,160,164,165,178,200,206–208,211,263,280,290,294–298] \\ \hline \multirow{7}{*}{List selection} & Resources & [20,38,49,64,74,81,86,91,105,111,112,121,122,124,126,131,149,150,153,157,160,164,168,173,204,230,237,241,242,288,296] \\ \cline{2-3}  & Entity & [31,32,47,49,50,52,54,57,59,64,68,73,79,80,83,84,87,94,96,108,109,115,130–132,134,135,139,143,148,154,163,172,176,177, 179,198,199,210,213,215,220,232,271,273,274,277,278,280, 286,287,291] \\ \cline{2-3}  & Locations, positions and nodes & [70,71,75,76,81,97,102,128,136,137,142,152,158,166,175,186–188,190,191,195,219,221,222,229,239,242,251,268,269,276, 281,282,284,285,299–301] \\ \cline{2-3}  & Network-specific & [20,23,24,38,63,82,91,103,104,121–123,127,147,149,153,173,194,201,211,241,243,245,250] \\ \hline \multirow{7}{*}{Sequential decisions} & Resource reallocation & [17,40,46,49,55,69,80,89,113,114,125,129,146,164,167,170,246, 249,266,302] \\ \cline{2-3}  & Accept or reject entities & [49,55,80,89,161,217,226,267,303,304] \\ \cline{2-3}  & State change & [17,39,40,43,49,55,59,61,69,80,89,90,113,114,164,167,168,223, 246,249,266,303,304] \\ \cline{2-3}  & Cardinal and ordinal directions & [118,156,206,240,305] \\ \cline{2-3}  & Maintenance actions & [45,48,60,151,231,238] \\ \hline \hline \end{tabular}
\end{table}
Table 4: Popular agent action categories.  
The classic approach is to have the RL agent estimate the **Q-value** of a certain state or state-action pair. However, it has become more popular to have the agent directly select a candidate option from a list, such as the next entity to consider or the next resource to allocate.  
Alternatively, the list of actions can be a smaller, repeatable set of **decisions**, for example, accepting or rejecting a certain entity into a resource. Some approaches conduct hyperheuristic selection, having the agent choose the most appropriate heuristic based on the current state. The agent can also make multiple selections at once, combining any examples from this section.  
Depending on the architecture, the agent might output a full **solution**. This is typically from approaches that work with variable-sized outputs. Finally, the output can produce one or multiple number estimates. The use cases include simulation or metaheuristic parameters.  
In \(\epsilon\)-greedy approaches [45; 175; 225; 250; 310], \(\epsilon\) is a small number. With probability 1-\(\epsilon\), the action that will lead to the highest Q-value is selected. Otherwise, with \(\epsilon\), a random action will be selected. This strategy is often used to balance the exploration of new solutions versus exploiting currently promising solutions. This is a common alternative to always selecting the highest Q-value [267; 280; 289]. The \(\epsilon\)-greedy threshold can also decrease over the training [95; 260].  
Action **masking** refers to only allowing the agent to select feasible actions [139; 204; 233; 271], such as unavailable resources [84; 121], entities [57; 80; 96; 132; 179] or locations [39]. In [24], all options are available, but if the candidate option returns an unfeasible solution, the agent does nothing instead. The authors of [170] include the feasible actions in the state.