## 6 Connection-Aware Scheduling Heuristics
### Connection-Aware Reinforcement Learning Agent
#### 6.4.3 Environment Definition: States, Actions, and Rewards  
The first step in applying RL in dew computing is to define the environment in which the agent will interact adequately, that is, to define the states \(S\) of the environment, the actions \(A\) that the agent can perform, and the reward signal \(r(s,a,s^{\prime})\) that the agent will optimize for. We also have to define the transition probabilities \(p(s^{\prime}|s,a)\) if the agent does not interact with a real system (or with a predefined simulator). Whether the agent succeeds or fails at distributing jobs in a dew environment partially depends on how we define those four elements: \(S\), \(A\), \(r\), and \(p\). To define a dew environment in terms of \(S\), \(A\), \(r\), and \(p\), our starting point was EdgeDewSim Extended Simulator. As presented in Section 4, EdgeDewSim Extended Simulator inherits the simulation features of DewSim and EdgeDewSim, including the energy consumption of each device, battery-dependent and non-battery-dependent devices as contributors of an SCE, the job executions, and user behaviors, and, in this work, it incorporates features to represents network status changes due to nodes' mobility. Thus, the transition probabilities \(p\) in the environment are modeled by EdgeDewSim Extended Simulator. To define the state space \(S\), we considered two critical features of RL. First, most RL agents do not have memory. They learn a policy \(\pi(a_{t}|s_{t})\) that makes decisions purely based on the information available in the current state \(s_{t}\in S\). Thus, \(s_{t}\) must include all the relevant information the agent needs to assign a job to the correct device. Second, we note that any information identical for all the states \(s\in S\) is useless for the agent. The reason is that the agent has to discriminate whether action \(a\) is a good action in state \(s\). If a subset of information \(x\subset s\) is identical for all states, then \(x\) provides no discriminatory information to the agent. We then define the state space as follows: \(S=D\times J\times C\), where \((d,j,c)\in S\). Specifically, \(d\in D\) contains information on the current state of each device. This information includes, for each device, its CPU usage percentage, its remaining battery, the network strength, the connection status, and its current job queue. Then, \(j\in J\) provides information about the job the scheduler must assign next. This information includes the job's ops, input size, and output size. Finally, \(c\in C\) contains general statistics about the previously completed jobs. These statistics include the number of jobs that have arrived, the number of jobs that have been completed, the sum of the jobs' ops that have been completed, and the elapsed time since the beginning of the simulation (i.e., \(t-t_{0}\)). We note that \(s\in S\) comprises all the necessary information to correctly assign job \(j\) to a device in a fixed dew environment. If the environment is not fixed, we might want to add some additional features, such as the number of instructions per second a device can execute (or its battery capacity). We discuss this further in Section 8. Our definition of the action space \(A\) is as expected. We define one possible action per device in the dew environment. Then, whenever the agent executes action \(a_{i}\in A\) given the current state \((d,j,c)\in S\), the job \(j\) is assigned to the device associated with action \(a_{i}\). Finally, the reward function is equal to the number of successful jobs executed in the dew environment, as defined in Equation (6). Formally,  
\[r(s,a,s^{\prime})= \begin{cases}\frac{|I|}{|I|}&\text{if $s^{\prime}$ is terminal}\\ 0&\text{otherwise}\end{cases}  \tag{7}\]  
Note that this reward function only rewards the agent in terminal states. After the agent finishes distributing the whole sequence of jobs, the agent receives a reward equivalent to the successfully executed jobs in the dew environment.