Yes. The person you appoint can decline to act as your Executor which is one reason why you need to make sure the person you appoint is willing to serve.