The goal wasn't performance, nothing but a pure reusability of this method in other three:
The #addToAnnotationLists can add an empty collection, but #getAnnotatedFields and #getAnnotatedMethods cannot modify the Map.
Maybe I should rename the param canFill to canModify or add an extra method (#addAnnotatedMembers) without this param.