Add streams for annotated annotations, use lists for storage

Also use a specialized implementation for unannotated annotations (the
majority of cases)