color: Do not crash when toggling devices and reopening the panel

Resolves: https://bugzilla.gnome.org/show_bug.cgi?id=728584