EquiformerV2: Improved Equivariant Transformer for Scaling to
  Higher-Degree Representations
Authors: name: Yi-Lun Liao; name: Brandon Wood; name: Abhishek Das; name: Tess Smidt
Published: 2023-06-21 07:01:38

Abstract: Equivariant Transformers such as Equiformer have demonstrated the efficacy of
applying Transformers to the domain of 3D atomistic systems. However, they are
still limited to small degrees of equivariant representations due to their
computational complexity. In this paper, we investigate whether these
architectures can scale well to higher degrees. Starting from Equiformer, we
first replace SO(3) convolutions with eSCN convolutions to efficiently
incorporate higher-degree tensors. Then, to better leverage the power of higher
degrees, we propose three architectural improvements -- attention
re-normalization, separable S^2 activation and separable layer normalization.
Putting this all together, we propose EquiformerV2, which outperforms previous
state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
forces, 4% on energies, offers better speed-accuracy trade-offs, and
2times reduction in DFT calculations needed for computing adsorption
energies.