Before revision:  The pre-training consists of the model learning to apply the given change (an edit sequence) to the code in a correct way , and therefore requires only the code change itself. To increase the quality of the obtained embeddings, we only consider the changed tokens in the edit sequence .
After revision:  The pre-training consists of the model learning to apply the given change (an edit sequence) to the code in a correct way . This task requires only code changes themselves, which makes it unsupervised .