3D viewer: make user colormap button toggle colormap dialog

This commit changes the behavior of the 'User colormap' button in the
3D viewer so that it hides the dialog when it is open already. Before, a
second click after the dialog was opened would render the dialog
unusable, even if would then be closed correctly. This fixes issue #576.