Various optimizations around access and authorization

Add some caching, change some semantics to match uses, update older
code.