## 4 Scheduling Optimization Framework
### Learning Algorithm  
At \(\epsilon_{p}\), the agents devise a sequence of actions, that is, a policy \(p\), for the manufacturing system. This policy is associated with a relative reward. The system attains this reward in the subsequent \(e_{p}+1\). The following expression estimates the expected reward under \(p\):  
\[\begin{split}\overline{A_{p}}\leftarrow\lim_{\rho\to\infty} \frac{1}{\rho}\sum_{\sigma=1}^{p}E(A_{exp})\leftarrow\lim_{\rho\to\infty} \frac{1}{\rho}\sum_{\sigma=1}^{p}E(K_{\alpha}+K_{\beta}+K_{\gamma}-X_{\alpha} \\ -X_{\beta}-X_{\gamma}-X_{\delta_{p}}-X_{\epsilon}-X_{\zeta}-X_{ \eta}-F_{\alpha})\end{split} \tag{5}\]  
In (5), \(X_{\alpha}\) and \(X_{\beta}\) denote the costs associated with producing and storing products. \(X_{\gamma}\) refers to the cost of procuring raw materials. \(X_{\delta_{p}}\) is the cost of maintenance occurred at \(\epsilon_{p}\), and \(X_{\epsilon}\) represents the cost of repair activities. Note that \(X_{\delta_{0}}<X_{\delta_{1}}<X_{\delta_{2}}<\ldots<X_{\delta_{p}}\). \(X_{\zeta}\), \(X_{\eta}\) and \(F_{\alpha}\) are the cost of remanufacturing, refurbishing, and the fee of returned goods, respectively. In regard to system profits, \(K_{\alpha}\) refers to the standard product revenue, and \(K_{\beta}\) denotes the remanufactured product revenue. Lastly, \(K_{\gamma}\) is the revenue associated with sold refurbished products.  
At every step, the mechanism attempts to increase \(\overline{A_{exp}}\) using (5). Clearly, this expression is the objective function of the agents. In other words, the proposed mechanism seeks to formulate a policy \(p\) that maximizes the defined goal. In this regard, the agents integrate a Markov Decision Process. During this process, the next environment state and the expected reward can be estimated on the basis of the present state and opted action using (6). According to [PERSON] and [PERSON] [PERSON] and [PERSON] (2015), both expressions represent the dynamics of the interacting environment. Therefore, the dynamics of the studied manufacturing environment are estimated as follows (for the sake of brevity, the time variables are omitted):  
\[p(\theta^{\prime}|\theta,\psi)\gets P(\theta^{t}=\theta^{\prime}|\theta^{ t-1}=\theta,\psi^{t-1}=\psi) \tag{6}\]  
where \(\psi^{t}\) denotes the action at \(\epsilon_{p}\).  
For deriving a policy, the agents use a model-free average reward RL algorithm, called R-Learning. This algorithm strives to attain the optimum reward at every step [PERSON] and [PERSON] (2015); [PERSON] and [PERSON] (2015). To achieve that, it considers two values: a state-action value and a reward. The following expressions estimate these values:  
\[q(\theta,\psi)=q(\theta,\psi)+\beta[A-\overline{A}+max_{\psi^{\prime}}q(\theta ^{\prime},\psi^{\prime})-q(\theta,\psi)] \tag{7}\]  
where \(\theta\), \(\psi\) are the current state and the action at \(\epsilon_{p}\), \(\theta^{\prime}\) and \(\psi^{\prime}\) denote the subsequent state and action in the next epoch \(\epsilon_{p}+1\), \(A\) and \(\overline{A}\) correspond to the reward and average reward obtained by the agents, and \(\beta\) is a hyper-parameter that assumes real values. The state-action value corresponds to the action taken in a specific system state. Every value is stored in a table, called \"q-table\", which is updated at every \(\epsilon_{p}\) with (7).  
The actions are selected by exploring/exploiting the state-action space. To this end, the e-greedy strategy is used. This strategy can opt for a greedy action (i.e., the action with the optimum value in the q-table) with probability \(1-e\), or a random one with probability \(e\). When the greedy action is opted, the corresponding reward is calculated as follows:  
\[\overline{A}=\overline{A}+\gamma[A-\overline{A}+max_{\psi^{\prime}}q(\theta^{ \prime},\psi^{\prime})-max_{\psi}q(\theta,\psi)] \tag{8}\]  
where \(\gamma\) is a real-valued hyper-parameter.