Olive oil can effectively remove hair dye from your skin because it lifts and breaks up the pigment particles. For the same reason, a hot olive oil treatment is often recommended for removing semi-permanent color from your hair itself. Olive oil is a gentle and non-abrasive option when it comes to getting rid of dye stains. For that reason, it's ideal for sensitive areas like your face and ears.