Before revision:  In this work, we investigate the problems in the previous formulations and   propose a new positional encoding method for BERT called Transformer with Untied Positional Encoding (TUPE).
After revision:  In this work, we investigate the problems in the previous formulations and propose a new positional encoding method for BERT called Transformer with Untied Positional EncodingE (TUPE).