Only recalculate rectangle bounds when angles change