It requires a judgement about the facts. In judging the facts one further concept for political philosophy is useful. People tend to speak as though legal freedom was an all or nothing affair - either something is against the law or it is not. But this ignores the important fact that there are devices for social control other than outright permission or prohibition. One of these is licence.
Licensed behaviour is behaviour permitted under certain conditions.
Licences are required, for example, for driving motor vehicles or experimenting with animals. Their intention is to secure the freedom of the individual and at the same time prevent the excesses of absolute freedom. For a very long time licences have also been required for the manufacture of alcohol and tobacco. No doubt the origins of those lie in the desire of governments to raise revenue, but they have also proved to be effective ways in which quality of product and restriction of sale can be controlled. The time may have come to consider licences for other drugs also.