There's a lot to be said for using diplomacy first. On the other hand, don't lose sight of the fact that the list is your turf. If one of your house guests spits on the carpet in your living room, you're at liberty tell them to stop or to show them the door. A lot of people believe that they have a greater right to freedom of speech online than they have in the tangible world. By the same token, you have no right to blast away with any off-topic subject the pops into your head on a mailing list.
All list have some topic drift.
A few lists are meant to cover an extremely wide range of topics. It's up to the list admin to write the ground rules and enforce the decorum that is appropriate for their list. Try gentle persuasion first. Send those who misbehave a private note. If that doesn't work, use the software tools you have available to do whatever it takes to maintain an appropriate level of order.