Remove stats_fast* variants

Given the locking that I'll soon introduce, the
fast variants are kind of a stupid, error prone optimization.