[api] add some caching for very hot routes

The caching of visibility of packages is tricky, I tried to be as careful
as feasible