Clean up submodule directory fragments