Fix incorrect clamping in SWRenderer.

A previous PR changed a whole lot of min/maxes to std::min/std::max
but made a mistake here and used a templated min which cast it's
arguments to unsigned instead of casting return value.

This resulted in glitchy artifacts in bright areas (See [issue 7439](https://code.google.com/p/dolphin-emu/issues/detail?id=7439))

I rewrote the code to use a proper clamping function so it's cleaner
to read.