Before revision: We demonstrate how transformer-based models can be redesigned in order to capture inductive biases across tasks on different granularities and perform inference in a zero-shot manner. Specifically, we show how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision.
After revision: We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision.