[Encoder] Remove duplicate constant macro and modify the num_ref check accordingly

being reviewed at: https://rbcommons.com/s/OpenH264/r/816/

to avoid duplicate macro constant and fix inconsistent checking of max-ref