## 3 Materials and Methods
### Data Forwarding Method
#### 3.3.3 Learning to Forward  
If \(v_{i}\) transmits a packet to a relay node or a sink node, the state of \(v_{i}\) at time slot \((t+1)\) turns to 1, \(x_{t+1}=1\). Otherwise, \(x_{t+1}=0\). The action \(a_{t}\) in our scenario is \(a_{i}(t)=(p_{i,t},v_{j})\) which denotes the action of \(v_{i}\) forwarding packet \(p_{i,t}\) to \(v_{j}\). Then, the reward of of taking action \(a_{t}\) to next state \(x_{t+1}\) isdescribed as \(r(p_{i,t},v_{j})\). Lastly, the \(Q\)-value is updated to \(Q(p_{i,t},v_{j})\) which indicates the fitness of \(v_{i}\) forwarding packet \(p_{i,t}\) to \(v_{j}\) at time slot \(t\). In our data forwarding scenario, each sensor node is an independent learning agent and actions are options of a relay node or a sink node within its communication range. The following describes details of the model solution, including time, actions, transmission probabilities, rewards, and \(Q\)-values. Agents Agents are underwater sensor nodes. Time A \(v_{i}\) handling a packet \(p\) is associated with a time slot \(t\in\{0,1,2,3,...,T\}\) defined by the sequential number of time slots. Actions Actions refer to the joint selection of a packet in the node's cache and of a relay node in its neighborhood. The set \(A_{i}(t)\) of available actions is \(A=\{a_{i}(t)=(p_{i,t},v_{j})|p_{i,t}\in P_{i,t},v_{j}\in H(i,t)\}\), where \(a_{i}(t)=(p_{i,t},v_{j})\) is the action of forwarding packet \(p_{i,t}\) to relay node \(v_{j}\). Transmission Probabilities Denote the probability of transmission from \(v_{i}\) to \(v_{j}\) at time slot \(t\) as \(Pr_{i,j}(t)\). Meanwhile, the transmission probability from the current relay node \(v_{j}\) to the next potential relay node \(v_{k}\) is denoted by \(Pr_{j,k}(t+1)\). \(Pr_{i,j}(t)\) is computed by \(v_{i}\) while \(Pr_{j,k}(t+1)\) is computed by \(v_{j}\) and sent to \(v_{i}\) in the header of the broadcast packet in each round. The transmission probabilities can be calculated via the orientation metric by Equation (12), as follows. \[ \begin{cases}\Pr_{i,j}(t)=1-\frac{1}{\pi}\arccos\frac{\mathbf{e}_{ori}(i,t) \mathbf{v}_{i}(i,t)}{|\mathbf{e}_{ori}(i,t)||\mathbf{v}_{i}(i,t)|}\\ \Pr_{j,k}(t+1)=1-\frac{1}{\pi}\arccos\frac{\mathbf{e}_{ori}(i,t)\mathbf{v}_{n_ {j}}(i,t)}{|\mathbf{e}_{ori}(i,t)||\mathbf{v}_{n_{j}}(i,t)|}\end{cases}  \tag{12}\]  
Note that \(Pr_{j,k}(t+1)\) is the prediction from the current time slot \(t\) because the topology at time slot \((t+1)\) cannot be ascertained yet due to the node mobility. Rewards The rewards mainly consist of two aspects, energy consumption and VoI, as shown in Equation (13),  
\[r(p_{i,t},v_{j})=VoI(p_{i,t})\cdot E_{r}(i,t) \tag{13}\]  
where \(r(p_{i,t},v_{j})\) represents the reward of \(v_{i}\) transmitting packet \(p_{i,t}\) to \(v_{j}\), \(VoI(p_{i,t})\) denotes the VoI of packet \(p_{i,t}\), and \(E_{r}(i,t)\) represents the residual energy of \(v_{i}\) after transmission, at time slot \(t\). Q-values represent the goodness of actions and agents aim to learn the actual fitness of potential actions. We initialize the \(Q\)-values as shown in Equation (14),  
\[Q(p_{i,t},v_{j})=VoI(p_{i,t})\cdot E_{r}(i,t) \tag{14}\]  
where \(Q(p_{i,t},v_{j})\) refers to the \(Q\)-value of \(v_{i}\) in response to the action of choosing \(v_{j}\) as the relay node, \(VoI(p_{i,t})\) denotes the VoI of packet \(p_{i,t}\) to be transmitted, and \(E_{r}(i,t)\) represents the residual energy of \(v_{i}\), in the beginning. Algorithm 2 describes the learning process of \(v_{i}\in V\) in each time slot as well as the corresponding determination of the packet to forward and its relay node. If sink node \(s_{m}\in S\) is within the transmission range of \(v_{i}\), \(v_{i}\) transmits the packet with the largest VoI in its cache to \(s_{m}\) directly. Otherwise, to identify an optimal forwarding decision, \(v_{i}\) learns the value of function \(Q(p_{i,t},v_{j})\) and updates the \(Q\)-value. Based on this value \(v_{i}\) determines the optimal forwarding action \(a_{i}(t)=(p_{i,t},v_{j})\). Each node starts with no knowledge of its surrounding environment.