Revert #35

#35 brokes dragging in another color picker on the same page, see https://core.trac.wordpress.org/ticket/30537.

For now this is just an hotfix.