fix issue #2253

This only needs to happen if `color` is `off`, and consequently `action` is `remove`. Right now, this code will run even if the color assignment is just to "another color".