You have no obligation to answer the door if you don’t want to. It’s your private home, and unless you’re expecting someone, there’s no need to respond to an unexpected knock. It’s perfectly fine to ignore it until the person leaves.