Before revision:  In this paper, we found that removing or duplicating most layers in BERT would not change their outputs .
After revision:  In this paper, we found that surprisingly the output layer of BERT can reconstruct the input sentence by directly taking each layer of BERT as input, even though the output layer has never seen the input other than the final hidden layer .