Just delete the tag that gives them permission to see it from the document. Or remove the permission that allows them to see it from the tag.
Note, however, that they may have taken a copy of the document while they could see it. There is no way of preventing that (just like with paper documents) and there is no way of controlling their access to that copy.