If lowering the maximum number of ref frames, check iNumRefFrames as well

Make sure we don't set iMaxNumRefFrames lower than iNumRefFrames.

This fixes occasional test failures in
EncodeDecodeTestAPI.SetOptionEncParamExt.

Review at https://rbcommons.com/s/OpenH264/r/1004/.