util/primconvert: Avoid point arithmetic; apply offset on all cases.

Matches what u_vbuf_get_minmax_index() does.

Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
(cherry picked from commit f9098f09720d9b751e8529b2648971ee7d576b37)