Fix bug RC_QUALITY_MODE not works as expected

review at:
https://rbcommons.com/s/OpenH264/r/808/