Fix vmin/vmax to follow the PSP's rules

Also the interpreter.  Fixes #6107.

I'm not really sure of a good way to make x86 work correctly...

-[Unknown]
Well, I guess branching is the only simple way.

-[Unknown]