Yes, you should always wash off the lube after use because it will emerge inside your fabrics. You may always feel lube residues on your sheets or clothes and this can annoy you. Also, letting lube stain for a long time can cause some skin irritations. You may feel itchiness or some allergies especially if you’re allergic to stained sheets or clothes.