Refactor annotation code

This depends on #2945