Refactor the data structure of annotations