Before revision:  In the contextualization step, we use LMs to generate many left and right contexts which collectively capture the meaning of the input sentence.
After revision:  In the contextualization step, we use LMs to generate ensembles of past and future contexts which collectively capture the meaning of the input sentence.