# Smart Grid for Industry Using Multi-Agent Reinforcement Learning
## 4 Proposed MARL Approach
### Reward Calculation and Assignment  
The reward function and assignment are crucial for the learning success in RL. Since the given environment consists of various elements and agents, there are several aspects to consider. In order to avoid the credit assignment problem, the reward function is partly divided into global and local rewards (see Figure 7). Whereas local rewards are only assigned to single agents, all agents are receiving global rewards. In addition, some parts of the reward are assigned immediately, and others are assigned at the end of a billing interval or training episode, which means the end of a shift for the case in question. \begin{table}
\begin{tabular}{c|c|l|l|c|c|c} \hline \hline \multicolumn{2}{c|}{} & \multicolumn{2}{c|}{**Observation Feature**} & \multicolumn{2}{c}{**Agent**} \\ \hline \multicolumn{2}{c|}{_Category_} & \multicolumn{2}{c|}{**Name**} & \multicolumn{2}{c|}{**Explanation**} & **Resource** & **Battery** & **Market** \\ \hline \multirow{8}{*}{**Agents**} & \multirow{4}{*}{**Specific**} & Setup state & Current setup state of the resource & x & & \\  & & Job deadlines & Three next deadlines of every job type & x & & \\  & & Number of jobs & Number of jobs in queue for every job type & x & & \\ \cline{2-6}  & \multirow{4}{*}{**Resource**} & Action & Currently selected action & x & x & x \\  & & Remaining steps & Remaining time steps for currently selected & x & x & x \\  & & & next deadlines & The three next upcoming job deadlines & x & x & x \\  & & Number of jobs & Total number of jobs in queue & x & x & x \\  & & Energy demand & Total energy demand of all jobs in queue & x & x & x \\ \cline{2-6}  & \multirow{2}{*}{**Battery**} & SCC & Current Battery State of Charge & x & x & x \\ \cline{2-6}  & & Purchased electricity & Purchased electricity in the current billing & x & x & x \\ \hline \multirow{8}{*}{**Environment**} & \multirow{4}{*}{**Cooperative**} & Billing interval & Count of current billing interval & x & x & x \\  & & Partial interval & Count of current partial interval & x & x & x \\  & & VPS power & VPS power generation forecast for the next 2 h & x & x & x \\  & & Prior purchased power for the next 2 h & x & x & x \\  & & electricity & Prior purchased power for the next 2 h & x & x & x \\  & & Electricity price & Electricity price & Electricity price & Electricity price & x & x \\  & & Known power & Power demand of all resource agents, & & & \\  & & demand & who have not finished the current action yet & & & \\  & & Number of agents with & Number of all resource agents, who have not & & & \\  & & known actions & finished the current action yet & & & \\  & & & Total power demand in the current partial & & & \\  & & Power demand & Interval (only available, when all resource & & & x \\ \cline{2-6}  & & & agents have already chosen their actions) & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Agent-specific observation space. As displayed in Figure 6, there are two components of the reward function: Energy costs and production costs. As it is the overall system objective to minimize total costs, the assigned rewards consist of the negative costs that the agents aim to maximize. Beginning with the resource agents, the time-dependent production costs of a job can only be influenced by the resource to which the specific job is assigned. As a result, these production rewards \(R_{t,a}^{p}\) are granted as an individual and local reward to the resource agents \(p\) immediately after an action \(a\) is finished and consist based on [45] of time-dependent delay and storage costs \(TC_{n}(t)\), setup costs \(SC_{n}\), and labor costs \(LC_{n}\). In case an action \(a\) lies outside the currently valid action set \(\left\{A_{RA,t}\right\}\), a penalty reward \(r_{BA}\) is assigned:  
\[R_{t,a}^{p}=\left\{ \begin{array}{c}-\left(\sum TC_{n}(t)+\sum SC_{n}+LC_{n} \right),\;a\in\left\{A_{RA,t}\right\}\\ -r_{BA}\;,\;a\
otin\left\{A_{RA,t}\right\}\end{array} \right.