I think that's pretty foolish when dealing with sociopaths and fanatics. Some forceful response is definitely in order, but a measured and rational one, not a Jihad of our own. To go a little further, I'd say that justice, for all the difficulties involved in defining it, is one of the more finite abstract nouns. Just retribution for a crime can never be infinite, any more than a just reward could. It's actually a pretty difficult thing to even talk about.
Justice means fairness, and proportionality, and a hundred other things.
To some people, like the Hebrews, it means a life for a life, and an eye for an eye. Some people want to take a little bit more than just the eye, saying that it's only fair to add a little bit of punishment on top of the retribution. Even they, usually have a limit, though. Someone who killed another man would have to pay his relatives two sheep and a cow, for instance. Since the US is a descendant of all these cultures, there are echoes of most of their mechanisms of justice in our own.