The common factor is opposition to the initiation of force. Freedom is the absence of the initiation of force. The robber and the bully have initiated force and the condition of freedom doesn't exist unless there is an absence of the initiation of force. You have the right to free speech, but not to compel others to provide your forum. You have the right to earn a living, but not to compel others to provide your living.
Libertarians apply the non-coercion principle to all human behavior.
It doesn't matter if the initiators of force are in or out of government. Government doesn't confer some mystical right on some to violate the rights of others. If it is wrong for a person to commit a rape as an individual, it must be equally wrong for a person to commit a rape as an agent of government. It's theft regardless of whether the loot is used to buy drugs or to feed the poor. It is theft regardless of whether there is 1 thief or 20 million thieves.