Try using acetone, which you should be able to find at your local hardware store. Just dab a little bit onto a cotton ball and wipe it off.
Also, be sure to wear rubber gloves when using spray paint. This will keep you from accidentally getting a bunch of the chemicals onto your skin.