fix cast signedness when comparing to AV_NOPTS_VALUE

(gcc's -Wsign-compare)