Before revision:  To study this, we introduce a  probing task with a diagnostic dataset, NumerSense, containing 3,145 masked-word-prediction probes . Surprisingly, our experiments and analysis reveal that: (1) BERT and its stronger variant RoBERTa perform poorly on our dataset prior to any fine-tuning;
After revision:  To study this, we introduce a probing task with a diagnostic dataset, NumerSense, containing 13.6k masked-word-prediction probes . Surprisingly, our experiments and analysis reveal that: (1) BERT and its stronger variant RoBERTa perform poorly on our dataset prior to any fine-tuning;