Try to better protect agains unbalanced Add/RemoveFileHandler calls,

my first naive implementation was ... naive.

Thanks to Dario for spotting this problem with stressProof.