To answer this question, we first look at different types of force. In any group of people, from 2 to 20 billion, there is no use of force until some one uses it first. Initial force is aggression or coercion. Defensive force is the use of force to defend your safety, rights, or property. You have the right to defend yourself, and the right to authorize others, such as those in government, to use defensive force in your behalf.
Defensive force is survival.
If someone assaults you, you have the right to authorize government to punish those responsible in your behalf. Retaliatory force is justice. Let me say that again, Libertarians are those who oppose the initiation of force. Some libertarians are pacifists. They decline the use of any force.