Remove FactMutator and deal with BytesWritable directly

I think the added/removed lines says something about how much real value this was providing.