## 2 System Model
### Proactive-Energy Saving Optimisation Based CO\({}_{2}\) Reduction
We define user \(u\) association criterion to the SCs as:  
\[\mathbb{U}_{i}(i\subseteq C)=\left[u\in\mathbb{U}\ |i=\underset{i\in C}{ \operatorname{argmax}}(P_{r,u}+P_{CIO})\right], \tag{11}\]  
where \(P_{r,u}\) (dBm) is user reference signal power from a cell, i.e., one MC or sum of nine SCs, and \(P_{CIO}\) (dB) is Cell Individual Offset (CIO), a biased parameter depends on load of the cell that has a main function to offset lower transmit power of all the SCs in the HetNet in order to transfer load when lightly used or idle. When under utilised, SCs are turned off, and the load would be transferred to MC provided there is sufficient capacity. The downside of this activity is that users would no longer be associated with the strongest SC and backhaul overhead would increase when loading and off-loading occurs. This would make the SINR lower with higher CIO values. However, CIO measures loads to be balanced, which would eventually drop the capacity due to SINR drop and affects the QoS. Therefore, to partially offset the SINR of the serving cell, available PRBs would be allocated to users \(u\) which are more abundant in comparison to the available PRBs in the previous serving SC. As a trade-off knob to control the load balancing, Energy Consumption and CO\({}_{2}\) Emissions of the cells and overall HetNet architecture, the CIO parameter is highly important. Now, we formulate the equation of general energy consumption for each of the time steps \(k+\bar{k}\) for all cells \(c\) within HetNet architecture as [3,12,17]:  
\[min(\rho,P_{CIO})\ =\ \sum\limits_{i\in C}\left[ECR\right]_{k+\bar{k}}\ \, \tag{12}\]The main objective is to optimise the HetNet energy consumption based on cell switching and traffic offloading so that CO2 emissions are reduced and optimal policies are enforced to automate the network. To do this, load and CIO parameters (\(\rho,P_{CIO}\)) are required to be optimised for all the SCs such that overall energy consumption ratio in all SCs within HetNet is minimised, consequently reducing the CO2 emissions. The first two limitations define the CIO limits and On-Off state array, respectively, and determine the solution search space size, whereas the third limitation is to ensure the minimum amount of coverage to all the users through means of HetNet collective contribution. The minimum received power that a user would require is \(P_{th}\), and \(\varpi\) is the area of coverage probability that the users would be within it that would maintain QoS requirements with the indicator function. The minimum required bit rate is the fourth constraint that depends upon the QoS requirements. To maintain the ECR minimum objective, switching off lightly loaded SCs would impact received \(P_{r,u}\) (dBm) minimum user reference signal power and make it worse, leading to worse SINR and throughputs. Therefore, the fourth constraint guarantees the minimum SINR to be maintained for all users in all cases. This would require cell load \(\rho\) to be less than total load threshold \(\rho_{T}\) (0, 1). Cell switching On-Off, CIOs and cell load index is a non-convex optimisation problem on a large scale [12, 18]. The complexity of user association in Equation (11) per SC is expected to grow exponentially when dealing with multiple constraints concurrently; therefore, we analyse some Reinforcement Learning (RL) techniques to compare and obtain optimal results. Our modelled scenario has CIO as an optimisation variable with ten possible values available at each SC and 1024 possible iterations. The Energy-Aware Framework devises optimal On-Off state array and all SCs CIO values proactively aiming to minimise the energy consumption ratio of the whole network that would further extrapolate the CO2 emissions. The Energy-Aware Framework has a direct impact on CO2 emissions and is directly proportional. The total integral sum of all the \(ECR\) ratio values in a HetNet architecture would be calculated with the help of CO2 conversion factors in [1, 3].