Catch the click event on the color span only

Thanks to WhetDawg for the bug report
Minor refactoring