fix bitrate setting and add bitrate validation

approved at: https://rbcommons.com/s/OpenH264/r/426/