So He tells them they can destroy the true temple, His own body, and it would be raised again in three days. Of course the good church folks misunderstood. They think He's still talking about a building. If the place starts getting to be sacred for its own sake, and not for God's sake, you might as well tear it down. It has become an idol.
Idols are sneaky things.
Even good things, or perhaps especially good things, become idols and distract us from God and our neighbor. It was not that the animal sellers and the money changers served no purpose - Their priorities got out of whack. They got so caught up in having the right money, and sacrificing the right animals, in doing the right worship stuff, they lost sight of why the temple existed in the first place. It's like my computer. It's a good tool for some things.