Refine rate estimate in trellis

Take into account cutoff parameter to switch between sets of contexts
for rate estimation in arithmetic coding version of trellis quantization