Fix incorrect clamping in SWRenderer.

A previous PR changed a whole lot of min/maxes to std::min/std::max
but made a mistake here and used a templated min which cast it's
arguments to unsigned instead of casting return value.

This resulted in glitchy artifacts in bright areas (See issue 7439)

I rewrote the code to use a proper clamping function so it's cleaner
to read.
Merge pull request #623 from phire/sw-xfb-clamping

Fix incorrect clamping in SWRenderer.