hopefully finished refactoring for reduced alphabet compression