lavu/opt: fix range check logic in set_format()

In particular, allow to reject undefined values. Previously the code
was only accepting values in the range -1 .. NB_FORMATS-1.
lavu/opt: factorize setting of format values from string

Simplify incoming patch.
lavu/opt: apply range checks also when setting format string value

Previously when setting a pixel/sample format as a string range checks
were not performed. This is consistent with the
av_opt_set_pixel/sample_fmt() interface.
lavu,lavfi,lavd: do not hardcode AV_PIX_FMT_NB value when setting pixel format max value

The constant may change in libavutil but the library may be compiled
against an older version, thus rejecting a value which is otherwise
supported by the new libavutil.

INT_MAX is used here to denote the max allowed value for a pixel format.

The opt-test code is changed to provide a valid reference example.
lavu,lavc,lswr: do not hardcode AV_SAMPLE_FMT_NB value when setting sample format max value

The constant may change in libavutil but the library may be compiled
against an older version, thus rejecting a value which is otherwise
supported by the new libavutil.

INT_MAX is used here to denote the max allowed value for a sample format.

The opt-test code is changed to provide a valid reference example.
lavfi/abuffersrc: use AV_OPT_TYPE_SAMPLE_FMT for sample_fmt option

Simplify/extend syntax.