Before revision:  In this work, we investigate the problems in the previous formulations and   propose a new positional encoding method for BERT called Transformer with Untied Positional Encoding (TUPE).
After revision:  In this work, we investigate the problems in the previous formulations and the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method for BERT called Transformer with Untied Positional Encoding (TUPE).