fix(fingerprint): treat TMPDIR as strict API and fail (Issue #7877)
State: open

Fixes #7877 and follows up on feedback from [`#7890`](https://github.com/huggingface/datasets/pull/7890).

## Context

In [`#7890`](https://github.com/huggingface/datasets/pull/7890) it was introduced an automatic creation of the `TMPDIR` directory in `_TempCacheDir` so that `datasets` does not silently fall back to `/tmp` when `TMPDIR` points to a non‑existent path. This addressed the original “`tempfile` silently ignores TMPDIR” issue in [`#7877`](https://github.com/huggingface/datasets/issues/7877).

During review, it was pointed out by @stas00 (see [review comment](https://github.com/huggingface/datasets/pull/7890#discussion_r...)) that if the code treats `TMPDIR` as part of the public API, then failures to use it should **fail loudly**, not just emit warnings, because warnings are easy to miss in complex multi‑GPU setups.

This PR implements that stricter behaviour.

## What this PR changes


### `_TempCacheDir` behaviour In `src/datasets/fingerprint.py`:

- Continue to:
  - Detect `TMPDIR` from the environment
  - Normalise the path
  - Auto‑create the directory when it does not exist
  - Pass the (validated) directory explicitly to `tempfile.mkdtemp(...)` so `TMPDIR` is honoured even if `tempfile.gettempdir()` was already cached

- **New behaviour** (in response to review on [`#7890`](https://github.com/huggingface/datasets/pull/7890)):
  - If `TMPDIR` is set, but the directory cannot be created, we now **re‑raise an `OSError`** with a clear, actionable message:
    - “TMPDIR is set to '…' but the directory does not exist and could not be created: … Please create it manually or unset TMPDIR to fall back to the default temporary directory.”
  - If `TMPDIR` is set but points to something that is **not a directory**, we also **raise `OSError`** with guidance:
    - “TMPDIR is set to '…' but it is not a directory. Please point TMPDIR to a writable directory or unset it to fall back to the default temporary directory.”
  - When `TMPDIR` is **not** set, behaviour is unchanged: we pass `dir=None` and let `tempfile` use the system default temp directory.

This aligns with the @stas00’s suggestion that TMPDIR should be treated as a strict API contract: if the user chooses a TMPDIR, we either use it or clearly fail, rather than silently falling back.

### Tests

In `tests/test_fingerprint.py`:

- Updated tests that previously expected warning‑and‑fallback to now expect **hard failures**:
  - `test_temp_cache_dir_tmpdir_creation_failure`
    - Uses `unittest.mock.patch` to force `os.makedirs` to raise `OSError("Permission denied")`
    - Asserts that constructing `_TempCacheDir()` raises `OSError` and that the message contains both “TMPDIR is set to” and “could not be created”
  - `test_temp_cache_dir_tmpdir_not_directory`
    - Points `TMPDIR` to a regular file and asserts that `_TempCacheDir()` raises `OSError` with a message mentioning “is not a directory”

- Left the positive‑path tests in place:
  - `test_temp_cache_dir_with_tmpdir_nonexistent` – verifies that a non‑existent `TMPDIR` is created and used
  - `test_temp_cache_dir_with_tmpdir_existing` – verifies that an existing `TMPDIR` directory is used as the base for the temp cache dir
  - `test_temp_cache_dir_without_tmpdir` – verifies behaviour when `TMPDIR` is not set (default temp directory)

- Kept the earlier fix to `test_fingerprint_in_multiprocessing`, which now uses `Pool.map` and asserts that fingerprints are stable across processes.

## Rationale

- Treating `TMPDIR` as part of the API for cache placement means:
  - Users can rely on it to move large temporary Arrow files away from small `/tmp` partitions.
  - Misconfigured TMPDIR should be **immediately visible** as a hard error, not as a warning lost among many logs.
- The stricter failure mode matches the concern on [`#7890`](https://github.com/huggingface/datasets/pull/7890) that “warnings are very easy to miss in complex applications where there are already dozens of warnings multiplied by multiple GPU processes”.

## Testing

- `pytest tests/test_fingerprint.py`
- `make style`
- No new linter issues introduced.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7891). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.