Speed up build caches

@bfulton @adamjt @tlunter