This depends on what your contract with the caterer states.  The contract should say how much notice you need to give to cancel the contract without having to pay.  If, for example, the contract states that you need to give 30 days notice to cancel and you only gave three weeks, you will have to pay the caterer.