## 3 Method
### Action Evaluation Network  
To address the existing problems in SR and to use latent states explicitly, we propose to use an action evaluation network (AEN) to compute continuous contribution. In the mainstream reinforcement learning algorithms, the input of most models is the agent's observation of the environment. This observation is generally expressed in the form of images. We think that simple images cannot provide complete environmental information and there is a more important latent state that cannot be displayed with single-image. Therefore, we want to extract the latent state from multiple consecutive images and use the latent state to complete the environmental information. Moreover, the agent's action will not only change the state of the environment but also affect the latent state. The feedback result of the former is the environmental reward \(r^{c}\), and the feedback result of the latter can be regarded as intrinsic reward \(r^{c}\). In a partially observable environment, the agent can only observe its surrounding environment and obtain local observation \(o_{i}\). Meanwhile, the extraction of the latent state requires sufficient environmental information. Therefore, we first combine the joint observation \((o_{1},o_{2}\ldots o_{N})\) and input it into the convolutional neural network (CNN) to extract the complete environment information \(O\). Then according to the continuity of the environmental state in time, our network adds a recurrent neural network with memory capabilities. We input this series of states into the long short-term memory (LSTM) network. Perceived changes in the environment are stated through remembering and forgetting, the LSTM outputs the latent state \(L\) of the environment \(L_{t},h_{t}=f_{L}(h_{t-1},O_{t};\theta_{t})\). The latent state extracted from the observation sequence \((O_{t},O_{t+1},\ldots,O_{t+k})\) reflects the change of the environmental state from \(t\) to \(t+k\). The influence of the joint action at timestep \(t\) on the latent state at timestep \(t+k\) can be regarded as its continuous contribution to the future collective. Therefore, the timestep \(t\) of the green part in Figure 1 corresponds to timestep \(t+k\) of the red part. Finally, we propose a novel evaluation network that takes latent state and joint action as input and outputs the continuous contribution of action during this time \(C=f_{a}(O_{t+k},A_{t};\psi_{a})\). We expect to use the AEN to get the effect of a joint action \(A_{t}\) on the latent state \(L_{t}\). Then an intuitive evaluation method is to calculate the difference between the state value during this period. So we use \(TargetV_{t}=v(s_{t})-v(s_{t+k})\) as target to AEN. The loss function can be written as:  
\[\mathcal{L}_{\theta}=E[(TargetV_{t}-C_{t})^{2}] \tag{10}\]  
Figure 1: The architecture of action evaluation network. The red part is used to process the input images and extract latent state \(L\). The green part is responsible for evaluating the continuous contribution \(C\) of agent actions in the latent state. In multi-agent reinforcement learning, the state value can be represented by the collective reward  
\[v(s_{t})=\sum_{s^{\prime}\in S^{+}}p(s^{\prime}|s,\pi(s))(R_{t}+\gamma v(s^{ \prime})) \tag{11}\]  
The continuous contribution \(C_{t}\) represents the influence of the joint action on the latent state at time \(t\), and evaluates the change of state value from \(t\) to \(t+k\). However, this is the contribution of the joint action and cannot be precisely assigned to a specific agent. Therefore, we use counterfactual reasoning to replace the action of agent \(i\) in the joint action. The joint action after the replacement is re-input into the network for evaluation, and a new evaluation value is obtained \(C_{i,t}\). Similar to the method in the previous section, we make a difference between the two contribution values to obtain the continuous contribution of the agent's action on the environment \(r_{i,t}^{c}=C_{t}-C_{i,t}\). In this way, replace the actions of each agent and get their contribution.