Remove the submodule directory