I'd still avoid being alone with him when you do this.
This may not be the best advice because I've never really dealt with a situation like this (annoying backstabbing co-workers, sure, but not outright aggressive and hostile).