## 2 Materials and Methods
### Proposed Solution: Deep Reinforcement Learning
#### 2.2.2 Action-Reward Schema  
When designing the action-reward schema, we take extreme care in giving the right entity to the penalty of resource over-utilization, as it seriously affects QoS. We also include a cost-related penalty to our reward function to jointly optimize QoS and Operational Costs. Recall that the actions taken by our agent are the single-VNF assignation decisions for each VNF request of a SFC. At each iteration \(\tau\), our agent observes the state information \(s_{\tau}\), takes an action \(a_{\tau}\), and receives a correspondent reward \(r(s_{\tau},a_{\tau})\) computed as:  
\[r(s_{\tau},a_{\tau})=w_{Q}\cdot r_{QoS}(s_{\tau},a_{\tau})-\
u\cdot(w_{D} \cdot\lambda_{D}^{t}+w_{H}\cdot\lambda_{H}^{t}) \tag{18}\]  
where:  
* The parameters \(w_{Q}\), \(w_{D}\), and \(w_{H}\) are weights for the reward contributions relating to the QoS, the DT costs, and the Hosting Costs, respectively,
* \(r_{QoS}(s_{\tau},a_{\tau})\) is the reward contribution that is related to the QoS optimization objective,
* \(\
u\) is a binary variable that equals 1 if action \(a\) corresponds to the last assignation step of the last session request arrived in the current simulation time-step \(t\), and 0 otherwise. Recall that \(\lambda_{D}^{t}\) and \(\lambda_{H}^{t}\) are the total DT costs and hosting costs of our vCDN at the end of the simulation time-step \(t\). Using (18), we subtract a penalty proportional to the current whole hosting and DT costs in the vCDN only at the last transition of each simulator time-step, i.e., when we assign the last VNF of the last SFC in \(R_{t}\). Such a sparse cost penalty was also proposed in [14]. When modeling the QoS-related contribution of the reward instead, we propose the usage an _inner delay-penalty function_, denoted as \(d(t)\). In practice, \(d(t)\) will be continuous and non-increasing. We design \(d(t)\) in such a way that \(d(t)<0,\forall t>T\). Recall that \(T\) is a fixed parameter indicating the maximum RTT threshold value for the incoming Live-Streaming requests. We specify the inner delay-penalty function used in our simulations in Appendix A.2. Whenever our agent performs an assignation action \(a\), for a VNF request \(\hat{f}_{r}^{k}\) in \(r\), we compute the generated contribution to the RTT of \(r\). In particular, we compute the processing time of \(r\) in the assigned VNF, eventual instantiation times, and the transmission delay to the chosen node. We sum such RTT contribution at each assignation step to form the current partial RTT, which we denote as \(t_{r}^{a}\). The QoS-related part of the reward assigned to \(a\) is then computed as:  
\[r_{QoS}(a)= \begin{cases}d(t_{r}^{a})\cdot 2^{-\hat{f}_{r}^{-}},&\text{if }\hat{f}_{r}^{-}>0 \text{ and }d(t_{r}^{a})>0\\ d(t_{r}^{a})\cdot 2^{\hat{f}_{r}^{-}},&\text{if }\hat{f}_{r}^{-}>0\text{ and }d(t_{r}^{a})<0\\ 1,&\text{if }\hat{f}_{r}^{-}=0\text{ and }d(t_{r}^{a})>0\\ 0,&\text{if }\hat{f}_{r}^{-}=0\text{ and }d(t_{r}^{a})<0\end{cases}  \tag{19}\]  
If we look at the first line of (19), we realize that a positive reward is given for every assignment that results in a non-prohibitive partial RTT. Moreover, such a positive reward is inversely proportional to \(\hat{f}_{r}^{-}\) (the number of pending assignations for the complete deployment of the SFC of \(r\)). Notice that, since \(t_{r}^{a}\) is cumulative, we give larger rewards to the latter assignation actions of an SFC, as it is more difficult to avoid surpassing the RTT limit at the end of the SFC deployment with respect to the beginning. The second line in (19) shows instead that a negative reward is given to the agent whenever \(t_{r}^{a}\) exceeds \(T\). Further, such a negative reward worsens proportionally to the prematureness of the assignation action that caused \(t_{r}^{a}\) to surpass \(T\). Such a worsening makes sense because bad assignation actions are easier to occur at the end of the SFC assignation process with respect to the beginning.