In British English, we'd say "clear the dirt off it". Possibly Americans say "off of it", though I'm not sure about that.