Make infill_only_where_needed idempotent. #2198