The full dataset viewer is not available (click to read why). Only showing a preview of the rows. Job manager crashed while running this job (missing heartbeats). Error code:   JobManagerCrashedError

### Describe the bug

The full dataset viewer is not available (click to read why). Only showing a preview of the rows.
Job manager crashed while running this job (missing heartbeats).
Error code:   JobManagerCrashedError

see: https://huggingface.co/datasets/Genius-Society/tt100k/viewer

### Steps to reproduce the bug

deploy the dataset online

### Expected behavior

fix the error

### Environment info

online dataset viewer environment on huggingface platform


Oh yeah, we hit something pretty similar when running a dataset viewer on a large collection (~3M rows) a few months back. The JobManagerCrashedError with missing heartbeats usually showed up for us when the server process ran out of memory or couldn’t keep up with the load spikes during dataset streaming. Turns out, the issue was tied to how the job manager was handling chunked data in the viewer. It'd crash hard if the volume scaled past a certain threshold.

One thing to try — are you using the default caching mechanism, or did you configure a custom `cache_dir`? Sometimes moving the cache to a faster disk (or just clearing the cache entirely) helped stabilize things for us. Also, if the dataset’s structure is funky (like nested JSON rows), the viewer can choke big time.

Curious — how big is the dataset you’re uploading? And is this happening consistently or just for certain loads?

> Oh yeah, we hit something pretty similar when running a dataset viewer on a large collection (~3M rows) a few months back. The JobManagerCrashedError with missing heartbeats usually showed up for us when the server process ran out of memory or couldn’t keep up with the load spikes during dataset streaming. Turns out, the issue was tied to how the job manager was handling chunked data in the viewer. It'd crash hard if the volume scaled past a certain threshold.
> 
> One thing to try — are you using the default caching mechanism, or did you configure a custom ? Sometimes moving the cache to a faster disk (or just clearing the cache entirely) helped stabilize things for us. Also, if the dataset’s structure is funky (like nested JSON rows), the viewer can choke big time.`cache_dir`
> 
> Curious — how big is the dataset you’re uploading? And is this happening consistently or just for certain loads?

Configure a custom, see https://huggingface.co/datasets/Genius-Society/tt100k/blob/main/README.md?code=true. BTW, what do u mean moving the cache to a faster disk? All the *.arrows are stored as xset online, and we are caring about whether the online viewer works well.

11.7 GB, happening consistently ONLINE

<img width="1622" height="603" alt="Image" src="https://github.com/user-attachments/assets/de726a85-69d4-4de7-a7b0-6c85f3478484" />

> Oh yeah, we hit something pretty similar when running a dataset viewer on a large collection (~3M rows) a few months back. The JobManagerCrashedError with missing heartbeats usually showed up for us when the server process ran out of memory or couldn’t keep up with the load spikes during dataset streaming. Turns out, the issue was tied to how the job manager was handling chunked data in the viewer. It'd crash hard if the volume scaled past a certain threshold.
> 
> One thing to try — are you using the default caching mechanism, or did you configure a custom `cache_dir`? Sometimes moving the cache to a faster disk (or just clearing the cache entirely) helped stabilize things for us. Also, if the dataset’s structure is funky (like nested JSON rows), the viewer can choke big time.
> 
> Curious — how big is the dataset you’re uploading? And is this happening consistently or just for certain loads?

Hi, are you still following up on this issue?