Before revision: Using the attention map based probing framework from (Clark et al., 2019), we observe that  BERT's attention heads have modest but well above-chance ability to spot event arguments sans any training or domain finetuning, varying from a low of 17.77\% for Place to a high of 51.61\% for Artifact.
After revision: Using the attention map based probing framework from (Clark et al., 2019), we observe that , on the RAMS dataset (Ebner et al., 2020), BERT's attention heads have modest but well above-chance ability to spot event arguments sans any training or domain finetuning, varying from a low of 17.77\% for Place to a high of 51.61\% for Artifact.