## 5 Algorithm Design
### Multi-Agent Collaborative Decision-Making Optimization  
The major advantage of DRL is that, in the process of continuous interaction between agents and wireless environments, the mobility of wireless networks, such as satellite mobility and link status, can be resolved, especially among multi-agent systems that can collaborate to solve problems through shared environments [35]. ### Multi-Agent Collaborative Decision-Making Optimization  
The next section revolves around constructing Markov decision processes (MDPs), the core reason being its ability to formalize sequential decision-making problems in dynamic environments, providing structured modeling for an AI to optimize long-term cumulative gains through a mathematical framework of states, actions, rewards, and state transfer probabilities. At the same time, the Markovian nature of the MDP (where the current state is only dependent on the previous state) drastically simplifies the policy learning process in complex environments, enabling reinforcement learning methods to efficiently solve optimal decision-making policies, as follows  
**State space:** At time slot \(t\), we define the state of the \(k\)-th agent as  
\[S_{k}(t)=(x_{meta}(t),x_{image}(t),x_{cache_{start}}(t),x_{cache_{success}}^{ pre}(t),x_{cache_{success}}^{cur}(t)), \tag{21}\]  
where \(x_{meta}(t)\in\mathbb{C}^{K\times 9}\) represents the attribute characteristics of the contents, \(x_{meta,j}(t)=id,shoottime_{i}\), \(latstart_{i}\), \(latend_{i}\), \(lonstart_{i}\), \(lonend_{i}\), \(payload_{i}\), \(resolution_{i}\), \(product_{i}\), \(i\leq K\), \(x_{meta}(t)\in\mathbb{C}^{p\times L_{safe}}\) represents the cache status of remote sensing satellites with cache capabilities, for remote sensing satellite \(i\), its cache status is denotes as \(x_{cache_{start},i}(t)=x_{cache_{start},i}^{1}(t),...,x_{cache_{start},i}^{ Late}(t);x_{cache_{user}}^{pre}(t)\in\mathbb{C}^{\widehat{M}\times L_{user}}\), \(x_{cache_{user}}^{cur}(t)\in\mathbb{C}^{\widehat{M}\times L_{user}}\) represents the cache status of users in the previous time slot and current time slot, respectively, \(\widehat{M}\) represents the number of users involved in a single decision, for user \(i\), its cache status in the previous time slot is denotes as \(x_{cache_{user},i}^{pre}(t)=x_{cache_{user},i}^{pre,1}(t),...,x_{cache_{ user},i}^{pre,L_{user}}(t)\). **Action space:** We define the multi-dimensional discrete action space \(a_{t}\in\mathbb{C}^{K\times\widehat{M}\times(T+1)}\) to represent the push decision actions, and the action vector can be represented as follows  
\[a(t)=\left( \begin{array}{c}a_{1,1}(t),...,a_{\widehat{1,\widehat{M}}}(t)\\ a_{K,1}(t),...,a_{K,\widehat{M}}(t)\\ \end{array} \right)_{K\times\widehat{M}}, \tag{22}\]where \(a_{i,j}(t)=a_{i,j,1}(t),...,a_{i,j,l}(t),...,a_{i,j,T+1}(t),a_{i,j,l}(t)e\{0,1\}\), \(\sum_{l=1}^{T+1}a_{i,j,l}(t)=1,\ \forall i,j\), with \(a_{i,j,l}(t)=1\) indicating that the content \(f_{i}\) is pushed to user \(j\) at time slot \(l\), otherwise \(a_{i,j,l}(t)=0\). If \(l=T+1\), the content is not pushed to user \(j\). **Reward function:** To minimize the total content delivery delay, we define the following reward function. \[r(s(t),a(t))=\left\{ \begin{array}{c}\textit{real\_d}\ (t),\ \text{if step }\% \frac{M}{\hat{M}}=0\\ 0,\ \text{else}\end{array}