Question: How much higher quality is the resulting annotated data?
Answer: improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3.5 F1 score, much larger than when a random set of expert annotations are added