embed_table_storage crashes (SIGKILL) on sharded datasets with Sequence() nested types
State: open

## Summary

`embed_table_storage` crashes with SIGKILL (exit code 137) when processing sharded datasets containing `Sequence()` nested types like `Sequence(Nifti())`. Likely affects `Sequence(Image())` and `Sequence(Audio())` as well.

The crash occurs at the C++ level with no Python traceback.

### Related Issues

- #7852 - Problems with NifTI (closed, but related embedding issues)
- #6790 - PyArrow 'Memory mapping file failed' (potentially related)
- #7893 - OOM issue (separate bug, but discovered together)

### Context

Discovered while uploading the [Aphasia Recovery Cohort (ARC)](https://openneuro.org/datasets/ds004884) neuroimaging dataset to HuggingFace Hub. Even after fixing the OOM issue (#7893), this crash blocked uploads.

Working implementation with workaround: [arc-aphasia-bids](https://github.com/The-Obstacle-Is-The-Way/arc-aphasia-bids)

## Reproduction

```python
from datasets import Dataset, Features, Sequence, Value
from datasets.features import Nifti
from datasets.table import embed_table_storage

features = Features({
    "id": Value("string"),
    "images": Sequence(Nifti()),
})

ds = Dataset.from_dict({
    "id": ["a", "b"],
    "images": [["/path/to/file.nii.gz"], []],
}).cast(features)

# This works fine:
table = ds._data.table.combine_chunks()
embedded = embed_table_storage(table)  # OK

# This crashes with SIGKILL:
shard = ds.shard(num_shards=2, index=0)
shard_table = shard._data.table.combine_chunks()
embedded = embed_table_storage(shard_table)  # CRASH - no Python traceback
```

## Key Observations

| Scenario | Result |
|----------|--------|
| Single `Nifti()` column | Works |
| `Sequence(Nifti())` on full dataset | Works |
| `Sequence(Nifti())` after `ds.shard()` | **CRASHES** |
| `Sequence(Nifti())` after `ds.select([i])` | **CRASHES** |
| Crash with empty Sequence `[]` | **YES** - not file-size related |

## Workaround

Convert shard to pandas and recreate the Dataset to break internal Arrow references:

```python
shard = ds.shard(num_shards=num_shards, index=i, contiguous=True)

# CRITICAL: Pandas round-trip breaks problematic references
shard_df = shard.to_pandas()
fresh_shard = Dataset.from_pandas(shard_df, preserve_index=False)
fresh_shard = fresh_shard.cast(ds.features)

# Now embedding works
table = fresh_shard._data.table.combine_chunks()
embedded = embed_table_storage(table)  # OK!
```

## Disproven Hypotheses

| Hypothesis | Test | Result |
|------------|------|--------|
| PyArrow 2GB binary limit | Monkey-patched `Nifti.pa_type` to `pa.large_binary()` | Still crashed |
| Memory fragmentation | Called `table.combine_chunks()` | Still crashed |
| File size issue | Tested with tiny NIfTI files | Still crashed |

## Root Cause Hypothesis

When `ds.shard()` or `ds.select()` creates a subset, the resulting Arrow table retains internal references/views to the parent table. When `embed_table_storage` processes nested struct types like `Sequence(Nifti())`, these references cause a crash in the C++ layer.

The pandas round-trip forces a full data copy, breaking these problematic references.

## Environment

- datasets version: main branch (post-0.22.0)
- Platform: macOS 14.x ARM64 (may be platform-specific)
- Python: 3.13
- PyArrow: 18.1.0

## Notes

This may ultimately be a PyArrow issue surfacing through datasets. Happy to help debug further if maintainers can point to where to look in the embedding logic.

Comments:
I wasn't able to reproduce the crash on my side (macos arm 54, pyarrow 22 and a nifti file I found [online](https://s3.amazonaws.com/openneuro.org/ds004884/sub-M2001/ses-1076/anat/sub-M2001_ses-1076_acq-tfl3_run-4_T1w.nii.gz?versionId=9aVGb3C.VcoBgxrhNzFnL6O0MvxQsXX7&AWSAccessKeyId=AKIARTA7OOV5WQ3DGSOB&Signature=LQMLzjsuzSV7MtNAdQaFdqWqmbM%3D&Expires=1765473937))

could the issue be specific to your env ? have you tried on other environments like colab maybe ?

Hi @lhoestq,

Thank you so much for taking the time to investigate this. Your comment about not being able to reproduce it with a single NIfTI file actually helped me understand the bug better.

**Key finding:** This bug is scale-dependent. It only manifests with real, full-scale data, and not with synthetic test files.

I created a sandbox branch that isolates the exact state before the workaround:

**🔗 Reproduction branch:** https://github.com/The-Obstacle-Is-The-Way/arc-aphasia-bids/tree/sandbox/reproduce-bug-7894

### What we confirmed

| Test | Result |
|------|--------|
| Synthetic 2x2x2 NIfTI files | ✅ No crash |
| Synthetic 64³ NIfTI files (1MB each) | ✅ No crash |
| Real ARC dataset (273GB, 902 sessions) | ❌ **SIGKILL at 0%** |

### Environment (same as yours)

- macOS ARM64
- PyArrow 22.0.0
- datasets 4.4.2.dev0 (git main)

### Crash output

```
Casting the dataset: 100%|██████████| 902/902
Uploading Shards:   0%|          | 0/902
UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects
```
**Exit code: 137 (SIGKILL)**

The crash happens on the very first shard, at `embed_table_storage()`, when processing `Sequence(Nifti())` columns after `ds.shard()`.

### The workaround (in main branch)

A pandas round-trip before embedding breaks the problematic Arrow references:

```python
shard_df = shard.to_pandas()
fresh_shard = Dataset.from_pandas(shard_df, preserve_index=False)
fresh_shard = fresh_shard.cast(ds.features)
# Now embed_table_storage works
```

We understand that downloading 273GB to reproduce this isn't practical. The reproduction guide in the branch has full details if you'd like to dig deeper. Happy to help debug further if useful.

Thank you again for your time and for maintaining this library. 

@lhoestq Brief update - I've added a reproduction that uses standard `ds.push_to_hub()` (no custom code).

**Reproduction branch:** https://github.com/The-Obstacle-Is-The-Way/arc-aphasia-bids/tree/sandbox/reproduce-bug-7894

**To reproduce with standard library:**
```bash
git clone -b sandbox/reproduce-bug-7894 https://github.com/The-Obstacle-Is-The-Way/arc-aphasia-bids.git
cd arc-aphasia-bids && uv sync --all-extras
# Download dataset (~273GB): aws s3 sync --no-sign-request s3://openneuro.org/ds004884 data/openneuro/ds004884
HF_REPO="your-username/test" uv run python test_prove_7894_standard.py
```

Crashes at 0% with the same semaphore warning.

Full details in [REPRODUCE_BUG_7894.md](https://github.com/The-Obstacle-Is-The-Way/arc-aphasia-bids/blob/sandbox/reproduce-bug-7894/REPRODUCE_BUG_7894.md).

Also - you were right about #7893. I closed it. `free_memory=True` works as you said. That issue was my mistake.