An excuse for taking off the mask is that the catcher feels they can see the ball better without the mask on. With that philosophy the catcher should never wear a mask. If you can catch balls from a pitcher 60 feet away and not have problems seeing the baseball, you should be able to see a throw from the outfield. It is also a safety precaution. If the ball is short and takes a bad hop, an injury to the face and head could occur.
Safety comes first.
Protect yourself from injury at all times in as many ways possible.