Remove unused random colors