# A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems
## 5 Reward Design
### Mirror Objective Function  
One of the simplest and yet most common approaches is to reward based on the current objective function value [49; 79; 81; 175]. The current value, the difference between iterations or some linear transformation of either case can also be used. The values measured can be summations, averages and standard deviations. Makespan, or completion time, is very common in scheduling problems [113; 279; 286; 307].  
Non-linear transformations to the objective function are also used. Using the _max_ operator [110; 153; 184] breaks linearity but forces measurements to only take meaningful  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Group** & **Features** & **References** \\ \hline \multirow{6}{*}{Conditional rewards} & & [24; 36; 40; 42; 58; 76; 90; 106; 120; 156; 161; 165; 169; 185; 193; 217, 219; 249; 250; 284; 308; 309] \\ \cline{2-3}  & \multirow{2}{*}{Events and states} & [26; 40; 57; 68; 69; 71; 86; 88; 97; 116; 118; 124; 129; 145; 166; 168; 206, 219; 231, 243; 249; 250, 279; 281, 282; 298; 306; 315] \\ \cline{2-3}  & \multirow{2}{*}{Action selection or infeasibility} & [17; 24; 53; 74; 76; 108; 123; 125; 132; 133; 136; 146; 156; 163; 165; 168, 185; 189, 200; 206; 224; 241, 242; 262; 263; 294; 299; 303; 305] \\ \hline \multirow{6}{*}{Conditional rewards} & \multirow{2}{*}{Metric threshold} & [18; 48; 60; 70; 71; 111; 116; 132; 133; 159; 167; 168; 183; 207, 218, 238; 279; 291, 311] \\ \cline{2-3}  & \multirow{2}{*}{Solution evolution} & [92; 96; 160; 192; 196; 197; 203; 257; 270; 311; 313] \\ \cline{2-3}  & \multirow{2}{*}{Terminal} & [24; 34; 35; 69; 76; 92; 136; 143; 172; 176; 177; 182; 234; 282; 287; 288; 302] \\ \hline \multirow{6}{*}{Problem-specific} & \multirow{2}{*}{Continuous production systems} & [49; 59; 80; 100; 173] \\ \cline{2-3}  & \multirow{2}{*}{Due dates} & [50; 65; 87, 102; 119; 170; 295; 307; 308] \\ \cline{2-3}  & \multirow{2}{*}{Profit and costs} & [46; 79; 89; 102; 105; 134; 135; 137; 162; 208; 227; 304] \\ \cline{2-3}  & \multirow{2}{*}{Energy} & [72; 88, 100; 128; 194; 225; 289; 312] \\ \cline{2-3}  & \multirow{2}{*}{Routing} & [81; 175; 206; 249] \\ \cline{2-3}  & \multirow{2}{*}{Non-linear objective function} & [18; 27; 67; 85; 110; 153; 184; 210; 215; 290] \\ \cline{2-3}  & \multirow{2}{*}{Multiple and many objective} & [18; 23; 32; 63; 87; 104; 132; 193; 196; 226; 257; 280] \\ \hline Multi-agent & & [20; 78; 164; 172; 206; 287] \\ \hline \multirow{6}{*}{Alternative goals} & \multirow{2}{*}{Unavailable objective function} & [22; 31; 38; 39; 42; 54; 55; 61; 64; 66, 91; 99; 115; 161; 174, 186; 202, 221; 247; 304; 307; 308] \\ \cline{1-1} \cline{2-3}  & \multirow{2}{*}{Ratios} & [38; 39; 55; 71; 126; 158; 171; 185; 221; 256; 304; 308] \\ \cline{1-1} \cline{2-3}  & \multirow{2}{*}{Estimation} & [22; 38; 66; 91] \\ \cline{1-1} \cline{2-3}  & \multirow{2}{*}{Hybrid strategies} & [235; 252; 253; 263; 300] \\ \hline \hline \end{tabular}
\end{table}
Table 5: _Cont._numbers. For example, negative lateness is not desired [18; 27]. Ref. [290] punishes delay increments more severely when the total delay is small than when it is large.  
This approach is very popular due to its simplicity and ease of implementation, since it directly optimizes the objective function. However, it can also be too sensitive to problem scale and magnitude differences in returned reward values.