No, you should not cause any permanent damage using Mineral Spirits. Don't wait longer than 20 minutes to wash it off your vinyl wall, and it will be okay.