Only push widening casts inward