Annotations filter by viewport

Implementation for #540 and related minor improvements.
This explains EVERYTHING! (No, really, I was wondering why we're doing extend on empty objects etc.)