Fix: Auto-create TMPDIR directory when it doesn't exist (Issue #7877)
State: closed

# Fix: Auto-create TMPDIR directory when it doesn't exist

Fixes #7877

## Description

This PR fixes issue #7877 by implementing automatic creation of the `TMPDIR` directory when it is set but doesn't exist. Previously, `tempfile.mkdtemp()` would silently ignore `TMPDIR` and fall back to `/tmp` if the specified directory didn't exist, causing confusion for users experiencing "No space left on device" errors.

## Problem

When users set `TMPDIR` to a non-existent directory (e.g., `export TMPDIR=/some/big/storage`), Python's `tempfile` module silently ignores it and falls back to the default temporary directory (`/tmp`). This leads to:

- Users unable to use their specified temporary directory
- Silent failures that are difficult to debug
- Continued "No space left on device" errors even after setting `TMPDIR`

## Solution

The fix automatically creates the `TMPDIR` directory if it is set but doesn't exist, ensuring that:

1. Users' `TMPDIR` settings are respected
2. Clear logging is provided when the directory is created
3. Graceful fallback with warnings if directory creation fails
4. The fix works even if `tempfile.gettempdir()` was already called and cached

## Changes

### Implementation (`src/datasets/fingerprint.py`)

- Modified `_TempCacheDir.__init__()` to check if `TMPDIR` environment variable is set
- Added logic to auto-create the directory if it doesn't exist using `os.makedirs()`
- Added informative logging when directory is created
- Added warning logging when directory creation fails, with graceful fallback
- Added path normalisation to handle path resolution issues
- Explicitly pass `dir` parameter to `tempfile.mkdtemp()` to ensure TMPDIR is respected

### Tests (`tests/test_fingerprint.py`)

Added comprehensive test coverage:

- `test_temp_cache_dir_with_tmpdir_nonexistent`: Tests auto-creation of non-existent TMPDIR
- `test_temp_cache_dir_with_tmpdir_existing`: Tests behaviour when TMPDIR already exists
- `test_temp_cache_dir_without_tmpdir`: Tests default behaviour when TMPDIR is not set
- `test_temp_cache_dir_tmpdir_creation_failure`: Tests error handling when directory creation fails

Also fixed incomplete `test_fingerprint_in_multiprocessing` test that was missing implementation.

## Testing

- All existing tests pass
- New tests added for TMPDIR handling scenarios
- Code formatted with `make style`
- No linter errors
- Manual testing confirms the fix works as expected

## Example Usage

Before this fix:
```bash
$ export TMPDIR='/tmp/username'  # Directory doesn't exist
$ python -c "import tempfile; print(tempfile.gettempdir())"
/tmp  # Silently falls back, ignoring TMPDIR
```

After this fix:
```bash
$ export TMPDIR='/tmp/username'  # Directory doesn't exist
$ python -c "from datasets.fingerprint import get_temporary_cache_files_directory; print(get_temporary_cache_files_directory())"
# Directory is automatically created and used
# Log: "Created TMPDIR directory: /tmp/username"
```

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update

---

**Note**: This implementation follows the approach suggested in the issue, automatically creating the TMPDIR directory when it doesn't exist, which provides the best user experience whilst maintaining security (we only create directories explicitly specified by the user via environment variable).


Comments:
> Thank you for implementing this, @ada-ggf25
> 
> I see there are a few more uses of `tempfile` in `datasets` besides the one treated here - should they be treated in the same way as well?

I had a look through the other tempfile call sites:
- In `arrow_dataset.py` the code always pass an explicit `dir=cache_dir` when using `NamedTemporaryFile`, so TMPDIR is not involved there.
- In `search.py` the `NamedTemporaryFile` is only used to generate a random Elasticsearch index name, it doesn’t control where big cache files are written, so it doesn’t have the same “No space left on device” impact.

Given that, the fingerprint temp directory is the only place where respecting TMPDIR is part of the user‑visible API for disk usage. If you’d like, I’m happy to also thread TMPDIR through the `search.py` name generation, but I think this PR should keep scoped to the place that actually writes large temporary files.


Thank you for checking the other instances of tempfile use, @ada-ggf25