fix level_prefix too large in residual decoding

prevent from level_prefix data too big according to standard 9.2.2.1 (<=15 for Baseline, main profile).
review request is: https://www.rbcommons.com/s/OpenH264/r/80/
also contain some minus formatting changes.