Gold standards and reliability
Channel: Applied Language Technology
Date: 22/02/2021

In this video, I discuss the notion of a gold standard and its role in evaluating the performance of natural language processing models. Because a gold standard should be reliable, I show you how to measure agreement between annotators using Cohen's kappa, a measure of inter-annotator agreement implemented in the scikit-learn library.

✨ Check out the learning materials associated with this video: https://applied-language-technology.mooc.fi/html/notebooks/part_ii/05_evaluating_nlp.html#what-is-a-gold-standard

✨ Clone the repository with Jupyter Notebooks for interactive computing from GitHub: https://github.com/Applied-Language-Technology/notebooks

✨ Check out other videos in the Working with Text in Python playlist: https://youtube.com/playlist?list=PL6cQi6qFlek0rF-aCox3YHOWNdTZSFakA