# Measurement of Regional Electric Vehicle Adoption Using Multiagent Deep Reinforcement Learning
## 3 Materials and Methods
### Environment in Reinforcement Learning  
The reinforcement learning environment necessitates defining its environment and components, including (1) the agent, (2) the state, (3) the action, and (4) the reward. Our study utilizes a multi-agent DQN. The two agents in our model represent either high or low-income communities, which were sampled based on the results of a hot spot analysis using median household income. Our sampling approach utilizes GIS to effectively represent spatial features [57; 58]. The state considers the areal interpolated PHEV and BEV counts, the number of vehicles, and the estimated total GHG emissions. The latter was calculated based on the state average in TX from the Alternative Fuel Data Center, U.S. Department of Energy [59]. It is estimated that 7635 and 9574 pounds of CO\({}_{2}\) are reduced annually when using PHEVs and BEVs, respectively, as opposed to ICEVs. ICEVs emit 12,594 pounds of CO\({}_{2}\) per vehicle annually [59]. The actions in our model are defined in three stages. The first is promoting PHEVs, the second is promoting BEVs, and the final is maintaining the current mode choice without any promotional activity. Based on the mean growth rate, promoting PHEVs will increase its state by 3%. Similarly, promoting BEVs results in a 5% increase in its state. The reward is based on CO\({}_{2}\) reduction estimates obtained by multiplying the updated constant by CO\({}_{2}\) reduction estimates. The rationale for designing the agents to seek the highest reward involving CO\({}_{2}\) reduction is derived from findings that environmental awareness is a significant factor impacting the purchase of EVs [6; 7; 8; 26]. Discretization was implemented to save on the cost of training. For instance, a reward is divided by 1000. One adjustment we make is that we assume the low-income agent is influenced by their neighboring high-income peers. EV adoption is impacted by peer influence [23; 25], and it is more widely adopted by White and wealthier individuals [10; 11; 12]. Thus, it is reasonable to hypothesize that low-income communities, who possess fewer EVs, could be influenced by their wealthier counterparts. Utilizing the epsilon decay function, we assumed that the low-income agent has a 10% probability of mirroring the action of the high-income agent. We hypothesize that the neighborhood peer effect in EV adoption exists similar to the diffusion of solar photovoltaics [60; 61]. Tiered subsidy scenarios were tested four times. The default scenario is to maintain the current uniform subsidy policy. The others assume the impact of a subsidy intended to boost EV sales among low-income agents by 10% incrementally. The total reward per episode was recorded and compared between agents. The agents' mode choice in the most optimal episode was also calculated. The agents are identifying the optimal policy to achieve maximum CO\({}_{2}\) reduction in this environment. The research question we pose is as follows: To what extent should we distribute tiered incentives to low-income communities to mitigate the existing disparities in EV adoption? The results would help us measure the magnitude of allocating tiered incentives. Other hyperparameters include a batch size of 32, a discount factor set at 0.98, and a learning rate of 0.01. Our multi-agent DQN has an input layer of 4, two hidden layers of 32, and an output layer of 3, which returns a set of three actions. Figure 4 illustrates the structure of the neural network utilized. The activation functions in the hidden layers follow a linear activation, while the output layers employ SoftMax activation. This hyperparameter setting is based on optimization to prevent catastrophic forgetting. We trained the model with over 100 episodes, with each episode consisting of 100 steps. Epsilon was designed to decrease linearly by 0.01 as it approached 70 steps. The experience replay buffer records 30 episodes. After that point, the agent would either use its memory or explore its action. Keras was used during the modeling process.