## 3 Problem Formulation
### Optimal Problem Formulation  
As depicted in Figure 2, the goal is to maximize the energy efficiency \(\eta_{EE}(t)\) by jointly optimizing the transmit power \(\mathbf{P}=[p_{1}(t),p_{2}(t),\ldots,p_{K}(t)]\) from the BS, RIS selection and the phase shift matrix \(\mathbf{\Phi}=[\mathbf{\phi_{1}}(t),\mathbf{\phi_{2}}(t),\ldots,\mathbf{\phi_ {M}}(t)]\) from the RIS.  
Markov Decision Process (MDP) formulation encompasses essential components including the state, action, transition probability function, reward, and environment. These components are elucidated as follows:  
* State space: Consider \(S\) as the state space, which comprises the following constituents: (i) the channel gains for communication links: \(h_{k}^{t}\) and \(g_{k}^{t}\), (ii) the velocity and position of intelligent vehicle agents \(v_{k},p_{k}\), (iii) actions involving the configuration of phase shifting for the RIS components and power distribution of \(VU_{k}\) implemented at time \(t-1\), and (iv) the energy efficiency at time \(t-1\). Hence, \(S\) encompasses: \[s^{(t)}=\{\{h_{k}^{t},g_{k}^{t}\}_{k\in K},v_{k},p_{k},a^{(t-1)},\{\eta_{EE,k}^{t }\}_{k\in K}\}\] (9)
* Action space: Symbolized as \(A\), the action space encompasses the array of actions available to the agent. It includes the manipulation of phase shifting for individual RIS components and the adjustment of transmission power at the base station. The action \(a^{(t)}\) is expressed as: \[a^{(t)}=\{\mathbf{\Theta},\{\mathbf{W}_{k}\}_{k\in K}\}\] (10)
* Transition Probability Function (\(P\)): This characterizes the likelihood of transitioning between states and a particular action. Formally, it is represented as \(P(s^{\prime}|s,a)\), where \(s^{\prime}\) denotes the subsequent state, \(s\) denotes the current state, and \(a\) signifies the action.
* Reward function: The agent is provided with an immediate reward \(r_{i}^{t}\), representing the energy efficiency as defined in Equation (8). \[r_{k}^{t}=\eta_{EE,k}^{t}\] (11)
* Value Function: This indicates the anticipated cumulative reward, originating from a specific state according to a predetermined policy \(\pi\). Employing \(V_{t}^{\pi}\) to symbolize the value function at time step \(t\) under policy \(\pi\), it signifies the anticipated total of rewards beginning from \(s_{t}=s\) and extending to the conclusion of the episode: \[V_{t}^{\pi}(\mathbf{P},\mathbf{\Phi},t)=E\Bigg{[}\sum_{\tau=t}^{T_{F}}R(s_{\tau}, \pi(s_{\tau},\tau)|s_{t}=s)\Bigg{]}\] (12)
* The \(Q\)_-value function_ denotes the anticipated return, commencing from a designated state \(s_{t}=s\), executing a particular action \(a_{t}=a\), and subsequently adhering to policy \(\pi\). It is articulated as follows: \[Q_{t}^{\pi}(s,a)=R(s,a)+E\Bigg{[}\sum_{\tau=t+1}^{T_{F}}R(s_{\tau},\pi(s_{\tau},\tau))|s_{t}=s,a_{t}=a\Bigg{]}\] (13)  
In line with the foundational principles of optimal control theory [34], the optimal value function, along with the optimal policies for optimal resource allocation modulation, can be derived as follows:  
\[V^{*}(\mathbf{\Phi},\mathbf{P},t)=\max_{\mathbf{u}_{\mathbf{\phi}},\mathbf{u}_{P}}V(\mathbf{\Phi}, \mathbf{P},t) \tag{14}\]  
Here, \(\mathbf{u}_{P}\in\mathbb{C}^{N\times N}\) and \(\mathbf{u}_{\mathbf{\phi}}\in\mathbb{C}^{M\times M}\) represent the resource allocation control policies. Specifically, \(\mathbf{u}_{P}\) pertains to the transmit power control policy, while \(\mathbf{u}_{\mathbf{\phi}}\) corresponds to the RIS phase shift control policy.  
Additionally, adhering to [PERSON]'s principle of optimality [35], the dynamic representation of the finite horizon optimal cost function unfolds as follows:  
\[V^{*}(\mathbf{\Phi},\mathbf{P},t)=\max_{\mathbf{u}_{\mathbf{\phi}},\mathbf{u}_{P}}\{r(\mathbf{ \Phi},\mathbf{P},t)\}+V^{*}(\mathbf{\Phi},\mathbf{P},t+1) \tag{15}\]