# A2C: Attention-Augmented Contrastive Learning for State Representation Extraction
## 5 Results and Analysis  
### Sample Efficiency  
As shown in Figure 5, agents with different state representations were trained with 2.5 million frames respectively, as well as rewarded with different efficiencies. The random agent failed in completing the navigation task, which indicated that the evaluation environment was valid. In conclusion, A2C is competitive with other representation learning methods. Even though it lost to autoencoding at the beginning, A2C surpassed autoencoding at the end. We explain this phenomenon as follows: In autoencoding, the agent needs to find the critical information that can restore the raw image in the latent space. If we regard the raw and the restored image as a pair of similar pictures, features extracted from autoencoding should be more compact than A2C because it tries to generate the same image rather than a similar one. Thus, it is not surprising that autoencoding had similar efficiency to A2C at the beginning. After the agent acquires the effective policy, A2C can help agents explore richer perception modes instead of being single-minded like autoencoding with the help of attention mechanism. What is more, the learning efficiency of autoencoding is not as stable as A2C. Table 3 shows that autoencoding had a higher standard deviation than A2C. There were even three phenomena in the reward curve of autoencoding in our experiments: basically unchanged, sharply rising, and suddenly falling, while A2C was always rising. The instability of the state representation is not acceptable in reality because it would cause the unreliability of control in reality. This concern is confirmed by Table 4. Autoencoding caused the most collisions in the testing phase. Therefore, although autoencoding was able to achieve higher learning efficiency, it did not have much practical value. On the contrary, A2C takes into account stability while ensuring better learning efficiency, and it has the best collision performance in Table 4. Therefore, in terms of learning efficiency, A2C obtained the best performance.  
In addition, there was a gap between A2C and attention-removed A2C. Figure 6 displays two contrastive leaning loss curves of A2C and attention-removed A2C. Both of them basically converged to the same loss value, which shows that the encoders in both contrastive learning models were able to capture efficient similarity information. However, in Figure 5, A2C has higher learning efficiency than the attention-removed A2C when applied to RL. This contrast shows that the attention mechanism helps learn the essential similarity information, which should be attributed to its ability of calculating the correlation between two arbitrary pixels.  
\begin{table}
\begin{tabular}{c l l l l l} \hline \hline
**NO.** & **Representations** & **Number** & **Mean Value** & **Standard Deviation** & **Maximum** \\ \hline
1 & A2C & 100 & 0.71 & 1.46 & 7 \\
2 & Euclidean depth & 100 & 1.15 & 4.03 & 27 \\
3 & Normal & 100 & 2.72 & 6.90 & 45 \\
4 & Reshading & 100 & 2.74 & 6.53 & 42 \\
5 & Segmentation & 100 & 6.60 & 13.12 & 46 \\
6 & Denoise & 100 & 9.24 & 23.55 & 119 \\
7 & Attention-removed A2C & 100 & 9.90 & 28.22 & 241 \\
8 & Curvature & 100 & 16.06 & 43.08 & 214 \\
9 & Texture edge & 100 & 17.81 & 32.89 & 154 \\
10 & Autoencoding & 100 & 30.19 & 77.65 & 256 \\ \hline \hline \end{tabular}
\end{table}
Table 4: (Sorted by Mean value in ascending order) Number of collisions caused by agents with different state representations  
Figure 5: Reward curves. Rewards obtained by agents with different state representations in the policy learning phase.  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Autoencoding** & **Normal** & **Curvature** & **Denoise** & **Texture Edge** & **A2C** & **Attention-Removed A2C** \\ \hline
164.16 & 78.18 & 30.68 & 51.15 & 53.62 & 80.84 & 32.03 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Standard deviation of the reward distribution. A2C, attention-augmented contrastive learning method.