Fetching 27 files:   0%|          | 0/27 [00:00<?, ?it/s]
(raylet) [2025-07-11 16:29:26,993 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 87.9083 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(pid=1985536) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
Fetching 27 files:   0%|          | 0/27 [00:00<?, ?it/s] [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(raylet) [2025-07-11 16:29:37,003 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 86.8046 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:29:47,013 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 85.7295 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:29:57,022 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 84.6181 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:30:07,031 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 83.5339 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:30:17,040 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 82.3958 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:30:27,049 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 81.3186 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
Fetching 27 files:   4%|▎         | 1/27 [04:27<1:55:49, 267.27s/it]
(raylet) [2025-07-11 16:30:37,059 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 80.2363 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
Fetching 27 files:   4%|▎         | 1/27 [04:27<1:55:48, 267.23s/it] [repeated 3x across cluster]
(raylet) [2025-07-11 16:30:47,068 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 79.125 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:30:57,078 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 78.0398 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:07,088 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 76.9459 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:17,097 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 75.8423 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:27,107 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 74.7584 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:37,117 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 73.6645 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:47,126 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 72.57 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:31:57,136 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 71.4664 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:32:07,146 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 70.3647 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:32:17,156 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 69.2709 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.
(raylet) [2025-07-11 16:32:27,166 E 1972013 1972044] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2025-07-11_16-25-41_084234_1971072 is over 95% full, available space: 68.1966 GB; capacity: 1758.32 GB. Object creation will fail if spilling is required.