Before revision:  We show that model performance and robustness can be greatly improved using the dynamic data collection paradigm. The model error rate decreased across rounds , from 72.1\% in the first round to 35.8\% in the last round, showing that models became increasingly harder to trick -- even though content become progressively more adversarial as annotators became more experienced. Hate speech detection is an important and subtle problem that is still very challenging for existing AI methods. We hope that the models , dataset and dynamic system that we present here will help improve current approaches, having a positive social impact.
After revision:  We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also perform better on HateCheck, a suite of functional tests for online hate detection. We provide the code , dataset and dynamic system that we present here will help improve current approaches, having a positive social impact.