Fix performance issues by moving some code into a PARANOID-mode only