Prefer straight forward memoization

Instead of going through `attr_reader`, just use `@annotation_provider`.