[Encoder] Remove duplicate constant macro and modify the num_ref check accordingly

being reviewed at: https://rbcommons.com/s/OpenH264/r/816/

remove duplicate constant macro and modify the num_ref check accordingly
to avoid duplicate macro constant and fix inconsistent checking of max-ref