Perf. optimizations and cleanup for annotation scanning