# A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems
## 3 State Representation
### State Format  
The reviewed state representations always include some of the features previously described. The most common state format that gathers these features is a fixed list of variables. However, there are other noteworthy formats or considerations, as illustrated in  
**Entity lists**, for example, resources or jobs, often present one of two approaches. When the entity features are averaged over all resources [16; 18; 27; 34], it results in a compact, low-dimensional representation that is easy to handle but might overlook some relevant information. Alternatively, state features can each directly relate to individual entities [20; 35; 247], resulting in a more granular representation. However, this option requires a longer state that must also be fixed-sized, hurting generality. **Spatial-based** representations allow for the data position within the structure to also play a role. States represented as matrices [71; 221; 232; 233] leverage structured environments, such as grids and images, where the spatial proximity and order are relevant. An example of a specific matrix representation is a heightmap [195; 221; 233; 232], often used to encode three-dimensional space into a two-dimensional representation. It is also important to highlight applications using convolutional neural networks (CNNs) [61; 205; 232; 266] that, while more cumbersome to train, can automatically extract spatial features. **Graph-based** representations are useful for relational and entity-based problems. Directed and undirected graphs [219; 246; 270; 142] capture relational dependencies, respectively, \begin{table}
\begin{tabular}{p{56.9 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Format** & **Content** & **References** & **Insights** \\ \hline \multirow{3}{*}{Entity lists} & Averages & [16; 18; 27; 34; 35; 37; 42; 78; 203; 264] & Compact and simple, loses individual details \\ \cline{2-4}  & Per resource & [20; 35; 37; 74; 95; 101; 108; 122; 127; 169; 205; 226; 247; 265] & Granular details but less scalable \\ \hline \multirow{4}{*}{Spatial representations} & Matrices & [22; 70; 71; 76; 112; 120; 152; 166; 167; 172; 193; 195; 208; 210; 212; 221; 228; 232 – 234; 266; 267] & For structured environments and spatial reasoning \\ \cline{2-4}  & Heightmaps & [195; 221; 222; 232 – 234] & Capture 3D variations in a 2D representation \\ \cline{2-4}  & Convolutional approaches & [22; 61; 173; 181; 198; 205; 232; 234; 266] & Automatic feature extraction \\ \hline \multirow{4}{*}{Graph solutions} & Undirected and directed & [19; 21; 142; 219; 220; 228; 246; 254; 268 – 270] & Symmetric and asymmetric relational dependencies \\ \cline{2-4}  & Disjunctive graphs & [49; 78; 80; 117; 130; 271; 272; 273; 274; 275; 276; 277; 278] & Complex relationships and multi-entity interactions \\ \cline{2-4}  & Graph node features & [81; 219; 229; 246; 261; 268; 278 – 283] & Per-entity attributes \\ \cline{2-4}  & Graph edge features & [19; 21; 113; 136; 150; 220; 229; 239; 273, 284] & Captures relationships and dependencies between entities \\ \cline{2-4}  & Graph NN & [21; 70; 78; 80; 130; 142; 188; 219; 246, 254; 261; 268 – 271; 274; 275; 277 – 279, 283; 285 – 288] & Process graph states, improving generalization at computational cost \\ \hline \multirow{3}{*}{Variable-sized} & & [141; 179; 181; 220] & Flexible, adapts to dynamic state spaces \\ \cline{2-4}  & Recurrent neural networks & [20; 47; 65; 104; 115; 136; 156; 178; 184; 198; 238; 243; 271; 288 – 293] & Capture temporal dependencies in sequential decisions \\ \hline \multirow{2}{*}{Fuzzy} & & [83; 246; 258] & Model uncertainty, useful for imprecise states \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Categorization of state representation formats with representative examples. asymmetric and symmetric interactions. An example of a directed arc would be a precedence constraint between operations, while an undirected arc could connect all job nodes compatible with the same resource.