Repository: spack/spack

Issue:
Too many open files
I believe I've found a possible file handle leak in Spack. This manifests itself when installing very large environments.

### Steps to reproduce the issue

I'm trying to install the following environment:
```yaml
spack:
  specs:

  # Shells
  - bash

  # Spack dependencies
  - clingo
  - gnupg
  - graphviz
  - kcov

  # Newer versions of Linux tools
  - watch
  - wget
  - unrar

  # Software installation tools
  - automake
  - autoconf
  - m4
  - cmake
  - ninja
  - patchelf
  - scons

  # Version control systems
  - mercurial
  - subversion

  # Python libraries
  - python
  - py-azureml-sdk
  - py-black
  - py-cartopy
  - py-cmocean
  - py-codecov
  - py-dask
  - py-fiona
  - py-flake8
  - py-flake8-import-order
  - py-geocube
  - py-geopandas
  - py-geoplot
  - py-inference-schema+numpy
  - py-ipywidgets
  - py-joblib
  - py-jupyterlab
  - py-matplotlib
  - py-metpy
  - py-mypy
  - py-numpy
  - py-openpyxl
  - py-pandas
  - py-pycocotools
  - py-pygeos
  - py-pyinstrument
  - py-pytest
  - py-pytest-cov
  - py-pytest-mock
  - py-pyyaml
  - py-rarfile
  - py-rasterio
  - py-scikit-learn
  - py-scipy
  - py-seaborn
  - py-setuptools
  - py-shapely
  - py-sphinx
  - py-sphinx-rtd-theme
  - py-sphinxcontrib-programoutput
  - py-statsmodels
  - py-tables
  - py-torch
  - py-torchvision
  - py-twine
  - py-vermin
  - py-wheel
  - py-xarray
  - py-xgboost

  # Research libraries
  - gdal
  - opencv

  concretization: together
```
A few of these packages aren't yet in Spack, but you should be able to substitute them with other packages and still reproduce this.

### Error Message

After successfully installing several packages, I eventually hit:
```
==> Installing py-pytest-cov-2.8.1-jvm5vpnwx3vgzanxse2putpvi6ft54o5
==> No binary for py-pytest-cov-2.8.1-jvm5vpnwx3vgzanxse2putpvi6ft54o5 found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Error: Failed to install py-pytest-cov due to OSError: [Errno 24] Too many open files
==> Installing py-pytest-mock-1.11.1-f2yunbyvz5f7sm6p5fkunwpu3lu6zc2f
==> No binary for py-pytest-mock-1.11.1-f2yunbyvz5f7sm6p5fkunwpu3lu6zc2f found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Error: Failed to install py-pytest-mock due to ProcessError: /usr/bin/git: Too many open files
    Command: '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> Installing py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq
==> No binary for py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq found: installing from source
==> Warning: Missing a source id for apple-libuuid@1353.100.2
==> Warning: Skipping build of py-azureml-core-1.23.0-ufhtrskom5dhrbf2ohomke6d75uaw5m5 since py-requests-2.24.0-cbnrnlrvv32qea7ofousm62yr7g5o4eq failed
==> Warning: Skipping build of py-azureml-pipeline-core-1.23.0-6o5oa3nknsfca3a2yrplbj6l2rikwwyp since py-azureml-core-1.23.0-ufhtrskom5dhrbf2ohomke6d75uaw5m5 failed
==> Error: [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/lock'
```

### Information on your system

* **Spack:** 0.16.1-2888-a284edb852
* **Python:** 3.8.10
* **Platform:** darwin-catalina-ivybridge
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output


Discussion:
When I run `spack -ddd install`, I also see:
```
==> [2021-05-25-12:10:56.640281, 54459] CorruptDatabaseError: error parsing database:
    [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/index.json'
==> [2021-05-25-12:10:56.640384, 54459] Error: error parsing database:
[Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/index.json'
Traceback (most recent call last):
  File "/Users/Adam/spack/lib/spack/spack/directory_layout.py", line 118, in remove_install_directory
    shutil.rmtree(path)
  File "/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/python-3.8.10-cloblksy6vn52cmys3wm23fvsrww2xox/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/python-3.8.10-cloblksy6vn52cmys3wm23fvsrww2xox/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/python-3.8.10-cloblksy6vn52cmys3wm23fvsrww2xox/lib/python3.8/shutil.py", line 631, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/python-3.8.10-cloblksy6vn52cmys3wm23fvsrww2xox/lib/python3.8/shutil.py", line 627, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/py-ndg-httpsclient-0.5.1-5p7eunxeoixykmmdtfzsjcj3flabg2kk/.spack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Adam/spack/lib/spack/spack/environment.py", line 1586, in install_specs
    builder.install()
  File "/Users/Adam/spack/lib/spack/spack/installer.py", line 1640, in install
    pkg.remove_prefix()
  File "/Users/Adam/spack/lib/spack/spack/package.py", line 1296, in remove_prefix
    spack.store.layout.remove_install_directory(self.spec)
  File "/Users/Adam/spack/lib/spack/spack/directory_layout.py", line 120, in remove_install_directory
    raise RemoveFailedError(spec, path, e)
spack.directory_layout.RemoveFailedError: Could not remove prefix /Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/py-ndg-httpsclient-0.5.1-5p7eunxeoixykmmdtfzsjcj3flabg2kk for py-ndg-httpsclient@0.5.1%apple-clang@12.0.0 arch=darwin-catalina-x86_64/5p7eunx : [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/py-ndg-httpsclient-0.5.1-5p7eunxeoixykmmdtfzsjcj3flabg2kk/.spack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 711, in _read_from_file
    with open(filename, 'r') as f:
OSError: [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/index.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 774, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 498, in _invoke_command
    return_val = command(parser, args)
  File "/Users/Adam/spack/lib/spack/spack/cmd/install.py", line 351, in install
    env.install_all(args, **kwargs)
  File "/Users/Adam/spack/lib/spack/spack/environment.py", line 1543, in install_all
    self.install_specs(None, args=args, **install_args)
  File "/Users/Adam/spack/lib/spack/spack/environment.py", line 1598, in install_specs
    self.regenerate_views()
  File "/Users/Adam/spack/lib/spack/spack/environment.py", line 1339, in regenerate_views
    view.regenerate(specs, self.roots())
  File "/Users/Adam/spack/lib/spack/spack/environment.py", line 604, in regenerate
    with spack.store.db.read_transaction():
  File "/Users/Adam/spack/lib/spack/llnl/util/lock.py", line 567, in __enter__
    self._as = self._acquire_fn()
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 1013, in _read
    self._read_from_file(self._index_path)
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 714, in _read_from_file
    raise CorruptDatabaseError("error parsing database:", str(e))
spack.database.CorruptDatabaseError: error parsing database:
    [Errno 24] Too many open files: '/Users/Adam/spack/opt/spack/.spack-db/index.json'
```
Thanks to Activity Monitor, it looks like Spack has hundreds of open file handles for `/Users/Adam/spack/opt/spack/.spack-db/prefix_lock`.
It looks like these file handles get cleaned up once each root spec in the environment finishes installing, but for specs with large DAGs like `py-azureml-sdk` we accrue too many open file handles and the ulimit is exceeded.
Hit this again today.

Resolving patch:
diff --git a/lib/spack/llnl/util/lock.py b/lib/spack/llnl/util/lock.py
--- a/lib/spack/llnl/util/lock.py
+++ b/lib/spack/llnl/util/lock.py
@@ -9,6 +9,7 @@
 import socket
 import time
 from datetime import datetime
+from typing import Dict, Tuple  # novm
 
 import llnl.util.tty as tty
 
@@ -36,6 +37,126 @@
 true_fn = lambda: True
 
 
+class OpenFile(object):
+    """Record for keeping track of open lockfiles (with reference counting).
+
+    There's really only one ``OpenFile`` per inode, per process, but we record the
+    filehandle here as it's the thing we end up using in python code.  You can get
+    the file descriptor from the file handle if needed -- or we could make this track
+    file descriptors as well in the future.
+    """
+    def __init__(self, fh):
+        self.fh = fh
+        self.refs = 0
+
+
+class OpenFileTracker(object):
+    """Track open lockfiles, to minimize number of open file descriptors.
+
+    The ``fcntl`` locks that Spack uses are associated with an inode and a process.
+    This is convenient, because if a process exits, it releases its locks.
+    Unfortunately, this also means that if you close a file, *all* locks associated
+    with that file's inode are released, regardless of whether the process has any
+    other open file descriptors on it.
+
+    Because of this, we need to track open lock files so that we only close them when
+    a process no longer needs them.  We do this by tracking each lockfile by its
+    inode and process id.  This has several nice properties:
+
+    1. Tracking by pid ensures that, if we fork, we don't inadvertently track the parent
+       process's lockfiles. ``fcntl`` locks are not inherited across forks, so we'll
+       just track new lockfiles in the child.
+    2. Tracking by inode ensures that referencs are counted per inode, and that we don't
+       inadvertently close a file whose inode still has open locks.
+    3. Tracking by both pid and inode ensures that we only open lockfiles the minimum
+       number of times necessary for the locks we have.
+
+    Note: as mentioned elsewhere, these locks aren't thread safe -- they're designed to
+    work in Python and assume the GIL.
+    """
+
+    def __init__(self):
+        """Create a new ``OpenFileTracker``."""
+        self._descriptors = {}  # type: Dict[Tuple[int, int], OpenFile]
+
+    def get_fh(self, path):
+        """Get a filehandle for a lockfile.
+
+        This routine will open writable files for read/write even if you're asking
+        for a shared (read-only) lock. This is so that we can upgrade to an exclusive
+        (write) lock later if requested.
+
+        Arguments:
+          path (str): path to lock file we want a filehandle for
+        """
+        # Open writable files as 'r+' so we can upgrade to write later
+        os_mode, fh_mode = (os.O_RDWR | os.O_CREAT), 'r+'
+
+        pid = os.getpid()
+        open_file = None  # OpenFile object, if there is one
+        stat = None       # stat result for the lockfile, if it exists
+
+        try:
+            # see whether we've seen this inode/pid before
+            stat = os.stat(path)
+            key = (stat.st_ino, pid)
+            open_file = self._descriptors.get(key)
+
+        except OSError as e:
+            if e.errno != errno.ENOENT:  # only handle file not found
+                raise
+
+            # path does not exist -- fail if we won't be able to create it
+            parent = os.path.dirname(path) or '.'
+            if not os.access(parent, os.W_OK):
+                raise CantCreateLockError(path)
+
+        # if there was no already open file, we'll need to open one
+        if not open_file:
+            if stat and not os.access(path, os.W_OK):
+                # we know path exists but not if it's writable. If it's read-only,
+                # only open the file for reading (and fail if we're trying to get
+                # an exclusive (write) lock on it)
+                os_mode, fh_mode = os.O_RDONLY, 'r'
+
+            fd = os.open(path, os_mode)
+            fh = os.fdopen(fd, fh_mode)
+            open_file = OpenFile(fh)
+
+            # if we just created the file, we'll need to get its inode here
+            if not stat:
+                inode = os.fstat(fd).st_ino
+                key = (inode, pid)
+
+            self._descriptors[key] = open_file
+
+        open_file.refs += 1
+        return open_file.fh
+
+    def release_fh(self, path):
+        """Release a filehandle, only closing it if there are no more references."""
+        try:
+            inode = os.stat(path).st_ino
+        except OSError as e:
+            if e.errno != errno.ENOENT:  # only handle file not found
+                raise
+            inode = None  # this will not be in self._descriptors
+
+        key = (inode, os.getpid())
+        open_file = self._descriptors.get(key)
+        assert open_file, "Attempted to close non-existing lock path: %s" % path
+
+        open_file.refs -= 1
+        if not open_file.refs:
+            del self._descriptors[key]
+            open_file.fh.close()
+
+
+#: Open file descriptors for locks in this process. Used to prevent one process
+#: from opening the sam file many times for different byte range locks
+file_tracker = OpenFileTracker()
+
+
 def _attempts_str(wait_time, nattempts):
     # Don't print anything if we succeeded on the first try
     if nattempts <= 1:
@@ -56,7 +177,8 @@ class Lock(object):
     Note that this is for managing contention over resources *between*
     processes and not for managing contention between threads in a process: the
     functions of this object are not thread-safe. A process also must not
-    maintain multiple locks on the same file.
+    maintain multiple locks on the same file (or, more specifically, on
+    overlapping byte ranges in the same file).
     """
 
     def __init__(self, path, start=0, length=0, default_timeout=None,
@@ -161,25 +283,10 @@ def _lock(self, op, timeout=None):
 
         # Create file and parent directories if they don't exist.
         if self._file is None:
-            parent = self._ensure_parent_directory()
+            self._ensure_parent_directory()
+            self._file = file_tracker.get_fh(self.path)
 
-            # Open writable files as 'r+' so we can upgrade to write later
-            os_mode, fd_mode = (os.O_RDWR | os.O_CREAT), 'r+'
-            if os.path.exists(self.path):
-                if not os.access(self.path, os.W_OK):
-                    if op == fcntl.LOCK_SH:
-                        # can still lock read-only files if we open 'r'
-                        os_mode, fd_mode = os.O_RDONLY, 'r'
-                    else:
-                        raise LockROFileError(self.path)
-
-            elif not os.access(parent, os.W_OK):
-                raise CantCreateLockError(self.path)
-
-            fd = os.open(self.path, os_mode)
-            self._file = os.fdopen(fd, fd_mode)
-
-        elif op == fcntl.LOCK_EX and self._file.mode == 'r':
+        if op == fcntl.LOCK_EX and self._file.mode == 'r':
             # Attempt to upgrade to write lock w/a read-only file.
             # If the file were writable, we'd have opened it 'r+'
             raise LockROFileError(self.path)
@@ -292,7 +399,8 @@ def _unlock(self):
         """
         fcntl.lockf(self._file, fcntl.LOCK_UN,
                     self._length, self._start, os.SEEK_SET)
-        self._file.close()
+
+        file_tracker.release_fh(self.path)
         self._file = None
         self._reads = 0
         self._writes = 0


Test patch: