If lowering the maximum number of ref frames, check iNumRefFrames as well

Make sure we don't set iMaxNumRefFrames lower than iNumRefFrames.

This fixes occasional test failures in
EncodeDecodeTestAPI.SetOptionEncParamExt.
Merge pull request #1643 from mstorsjo/check-num-refs

If lowering the maximum number of ref frames, check iNumRefFrames as well