Before revision:  To study this, we introduce a  probing task with a diagnostic dataset, NumerSense, containing 3,145 masked-word-prediction probes . Surprisingly, our experiments and analysis reveal that: (1) BERT and its stronger variant RoBERTa perform poorly on our dataset prior to any fine-tuning;
After revision:  To study this, we introduce a probing task with a diagnostic dataset, NumerSense, containing 3,145 masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our analysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on our dataset prior to any fine-tuning;