revert to end tokens instead of pure distribution-based lengths to preserve ending characters