# A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems
## 3 State Representation  
The state is the representation of the environment available to the agent. This section groups and details different state representations found in this review. Table 2 categorizes the most common state features in the reviewed papers, which are then further detailed in the following sections.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Group** & **Features** & **References** \\ \hline \multirow{4}{*}{Resource features} & Utilization and efficiency & [16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42] \\ \cline{2-3}  & Queues & [43; 40; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68] \\ \cline{2-3}  & Load & [17; 25; 63; 69; 100; 101] \\ \cline{2-3}  & State and status & [17; 48; 60; 68; 69; 80; 93; 96; 100; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136] \\ \hline \multirow{4}{*}{Entity features} & Demand & [18; 32; 58; 79; 81; 94; 137; 138; 139; 142] \\ \cline{2-3}  & State and status & [26; 27; 59; 73; 86; 93; 96; 110; 117; 120, 124, 132; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 169; 161; 164; 166; 168; 169; 169; 170; 171] \\ \hline \multirow{4}{*}{Time-related} & Lateness and due dates & [18; 27; 52; 66; 75; 134; 168; 170; 171] \\ \cline{2-3}  & Earliness and slack time & [18; 29; 33; 34; 41; 57; 59; 134; 171; 172] \\ \cline{2-3}  & Idle and waiting time & [16; 27; 39; 40; 60; 78; 93; 123; 132; 173; 174] \\ \hline \multirow{4}{*}{Solution} & Full & [79; 81; 153; 175; 176; 177; 178; 179; 184; 179; 185; 170; 172; 186; 173; 174; 175; 176; 177; 178; 179; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 199; 190; 199; 191; 192; 193; 194; 19 The most common approach is to use **environment variables** as different state features. These variables, representing different model states, change over time as the environment reacts to the actions given. By tracking key variables and related features, many approaches manage to create a short yet representative state.  
Some of the most popular state features are **resource-related**. This includes machines, workers, vehicles, channels, etc. Other popular features relate to system entities such as jobs, operations, tasks, orders, products, etc. Job and order are used interchangeably to refer to the highest-level entity. A job can be the manufacturing of an object. Operation, sometimes used interchangeably with task, is a specific step in a job's production process. Each step requires different resources. However, tasks can also be used as smaller units of work within an operation [29].  
When the **current environment** should be represented, time-related metrics or solution representations can be part of the state. Time-related metrics can be computed for resources or entities and track lateness, wait times, etc. When representing the solution or its encoding, the objective function values or certain infeasibilities can be informative to the agent.  
Static problem parameters provide **extra context** to the agent. These are frequent when the same agent must solve different problem instances, as it needs to be aware of changes in available resources, number and types of jobs, processing times, etc.  
Some state representations are very **problem-specific**, only making sense for certain topics. This is the case for networks and energy problems. When the agent has a hybrid role, the state representation is also problem-dependent.  
Lastly, it is also common for approaches to **normalize** state features [41; 57; 90; 132]. In [132], the state is a list of seven features, such as percentage of finished jobs or idle time since last operation. Each feature has a unique scaling factor to keep all state values between zero and one. In [57], min-max scaling is applied to each job feature of the state.