Your on-time delivery rate metrics will be affected.
However, Amazon does not hold this against you in a decision to kick you out of the Seller Fulfilled Prime program.