Yes, you must wash celery stalks with tap water to remove debris. Apart from that, you can also soak celery stalks in vinegar to kill insects, dust, debris and bacteria.