Complete rewrite of the annotation layer

*Thorough description is coming up...*