Before revision:  Our experiments with such metrics show that probe's performance curves often fail to align with widely accepted rankings between language representations (with, e .g.,  non-contextual representations outperforming contextual ones). These results lead us to argue, second, that commonsimplistic probe taskssuch as POS labeling and dependency arc labeling, are inadequate to evaluate the properties encoded in contextual word representations.
After revision:  Our experiments with such metrics show that probe's performance curves often fail to align with widely accepted rankings between language representations (with, e .g., non-contextual fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech labeling and dependency arc labeling, are inadequate to evaluate the properties encoded in contextual word representations.