Repository: microsoft/DeepSpeed

Issue:
[BUG] Local `tmp` directory created when running `ds_report`
**Describe the bug**
When running `ds_report`, a "temporary" local directory is created but never removed when execution finishes.

**To Reproduce**
Steps to reproduce the behavior:
1. Run `ds_report`
2. See a local directory created named `tmp` (on Linux) or `var` (on macOS)

This also seems to happen during installation, but that is much more annoying to run repeatedly.

**Expected behavior**
There should not be any temporary local directories remaining after execution finishes.

**ds_report output**
```text
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-devel package with yum
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/nas/gaia/lestat/users/ahedges/lestat-curator/pycurator/venv/lib/python3.7/site-packages/torch']
torch version .................... 1.8.0+cu111
torch cuda version ............... 11.1
nvcc version ..................... 11.1
deepspeed install path ........... ['/nas/gaia/lestat/users/ahedges/lestat-curator/pycurator/venv/lib/python3.7/site-packages/deepspeed']
deepspeed info ................... 0.5.5+0b77d1d, 0b77d1d, master
deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1
```

**System info:**
 - OS: CentOS 7; macOS 11
 - GPU count and types: one machine with x4 RTX A6000; one machine with out GPUs
 - Python version: 3.7.11; 3.7.10

**Launcher context**
`ds_report`, installation

**Additional context**
I did some research into this issue myself before reporting.

The bug is in [`OpBuilder.has_function()`](https://github.com/microsoft/DeepSpeed/blob/8abdaee2430497438729daeded370db8a0e03c9b/op_builder/builder.py#L171), which is only called by [`AsyncIOBuilder.is_compatible()`](https://github.com/microsoft/DeepSpeed/blob/8abdaee2430497438729daeded370db8a0e03c9b/op_builder/async_io.py#L87). `has_function` creates a temporary directory to create files in and deletes it at the end of the function, but on [line 215](https://github.com/microsoft/DeepSpeed/blob/8abdaee2430497438729daeded370db8a0e03c9b/op_builder/builder.py#L215), a call to `distutils` writes to a local directory instead of the temporary directory. Digging into `distutils`, the problem is that [`distutils.ccompiler.CCompiler.object_filenames()`](https://github.com/python/cpython/blob/3.7/Lib/distutils/ccompiler.py#L847) intentionally "chop[s] off leading /" when processing paths, so instead of creating files the temporary directory in `/tmp`, it puts them in a local directory named `tmp`.

I have an idea for a workaround to fix it, which I'll release shortly. However, it is an awkward workaround to the behavior in `distutils`. The module is deprecated as of Python 3.10 (see [PEP 632](https://www.python.org/dev/peps/pep-0632/)), so the logic will likely need to get re-written, anyway.


Discussion:

Resolving patch:
diff --git a/op_builder/builder.py b/op_builder/builder.py
--- a/op_builder/builder.py
+++ b/op_builder/builder.py
@@ -210,9 +210,16 @@ def has_function(self, funcname, libraries, verbose=False):
                 oldstderr = os.dup(sys.stderr.fileno())
                 os.dup2(filestderr.fileno(), sys.stderr.fileno())
 
+            # Workaround for behavior in distutils.ccompiler.CCompiler.object_filenames()
+            # Otherwise, a local directory will be used instead of tempdir
+            drive, driveless_filename = os.path.splitdrive(filename)
+            root_dir = driveless_filename[0] if os.path.isabs(driveless_filename) else ''
+            output_dir = os.path.join(drive, root_dir)
+
             # Attempt to compile the C program into an object file.
             cflags = shlex.split(os.environ.get('CFLAGS', ""))
             objs = compiler.compile([filename],
+                                    output_dir=output_dir,
                                     extra_preargs=self.strip_empty_entries(cflags))
 
             # Attempt to link the object file into an executable.


Test patch: