Remove submodule directory