Revert 265f59986b58b1d36709d50996c7c66d4b6356d5

This brokes the dragging in another color picker on the same page, see https://core.trac.wordpress.org/ticket/30537