Freedom is sometimes defined as the absence of the initiation of force. The robber and the bully have initiated force and the condition of freedom doesn't exist unless there is an absence of the initiation of force. You have the right to free speech, but not to compel others to provide your forum. You have the right to earn a living, but not to compel others to provide your living. You have the right to believe in whatever religion you choose, but if your god requires the sacrifice of virgins, you'd better find a virgin willing to be sacrificed or else it's an initiation of force.
Libertarians apply the non-coercion principle to all human behavior.
It doesn't matter if the initiators of force are in or out of government. Government doesn't confer some mystical right on some to violate the rights of others. If it is wrong for a person to commit a rape as an individual, it must be equally wrong for a person to commit a rape as an agent of government. It's theft regardless of whether the loot is used to buy drugs or to feed the poor. It is theft regardless of whether there is 1 thief or 20 million thieves.