Fix vmin/vmax to follow the PSP's rules

Also the interpreter.  Fixes #6107.

I'm not really sure of a good way to make x86 work correctly...

-[Unknown]
Oh wait, maybe this is wrong... hmm.  I could do another branch, ugh.  What do you think is the best way?

-[Unknown]