[tab] logs/logs_sample/sample_knoms_3292297.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:15:00
#SBATCH --partition=accelerated
#SBATCH --account=hk-project-p0023960
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:4
#SBATCH --output=logs/logs_sample/%x_%j.log
#SBATCH --error=logs/logs_sample/%x_%j.log
#SBATCH --mail-user=avocadoaling@gmail.com
#SBATCH --job-name=sample_knoms
#SBATCH --mail-type=ALL

# Log the sbatch script
cat $0

module unload mpi/openmpi/5.0
module unload devel/cuda/12.4
source .venv_jafar/bin/activate

# data_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/knoms_tfrecords_500_shards'
# checkpoint_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/dyn/3282489/genie_1750610596_12500'
# seq_len=16
# output_dir='sample_results/knoms/'

# python sample.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 12 --seq_len $seq_len --output_dir $output_dir
# # CUDA_VISIBLE_DEVICES=0 python sample.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 12


# Overfit tfrecords 10
data_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/knoms_tfrecords_500_shards_overfit_10'
checkpoint_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/dyn/3289577/genie_1750786631_13000'
seq_len=16
output_dir='sample_results/knoms_overfit_tfrecords_10/'

srun python sample_resolution_batches.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 6 --seq_len $seq_len --output_dir $output_dir
GpuFreq=control_disabled
2025-06-25 19:05:37.801200: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 19:05:37.801210: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 19:05:37.801271: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 19:05:37.801196: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750871137.938272 3208552 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750871137.938207 3208553 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750871137.938214 3208555 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750871137.938223 3208554 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750871137.978733 3208552 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750871137.978729 3208553 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750871137.978727 3208554 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750871137.978729 3208555 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1750871138.434204 3208552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434247 3208552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434254 3208552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434259 3208552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434189 3208553 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434218 3208553 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434220 3208553 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434221 3208553 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434277 3208554 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434320 3208554 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434326 3208554 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434331 3208554 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434242 3208555 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434259 3208555 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434262 3208555 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871138.434263 3208555 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-06-25 19:06:05.074218: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
srun: error: hkn0435: task 0: Exited with exit code 1
srun: error: hkn0435: task 2: Exited with exit code 1
srun: error: hkn0435: task 1: Exited with exit code 1
2025-06-25 19:06:31.016079: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:06:36.100113: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1251: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.
  warnings.warn(
W0000 00:00:1750871205.808194 3208555 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2025-06-25 19:06:59.533462: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:07:09.084007: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:07:25.276483: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:07:28.646976: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:07:39.452897: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:07:57.665664: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:08:08.607437: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:08:14.839191: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:08:27.686405: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:08:38.499172: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:08:45.229958: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:00.123800: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:10.984497: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:18.059400: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:32.365396: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:44.067414: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:09:50.896622: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:06.222662: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:17.903492: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:25.041795: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:39.882421: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:51.096734: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:10:58.671668: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:11:15.171623: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:11:26.955145: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:11:34.989401: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:11:51.140946: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:12:03.285462: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:12:11.064350: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:12:28.565650: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:12:40.656324: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:12:48.254183: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:13:05.362907: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:13:17.802725: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:13:25.463287: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:13:44.493703: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:14:04.039697: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:14:23.047268: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:14:42.390268: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:15:02.972012: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:15:22.418083: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
Frame 1
Frame 2
Frame 3
Frame 4
Frame 5
Frame 6
Frame 7
Frame 8
Frame 9
Frame 10
Frame 11
Frame 12
Frame 13
Frame 14
Frame 15
SSIM: 0.4920308291912079

============================= JOB FEEDBACK =============================

Job ID: 3292297
Cluster: hk
User/Group: tum_ind3695/hk-project-pai00039
Account: hk-project-p0023960
State: FAILED (exit code 1)
Partition: accelerated
Nodes: 1
Cores per node: 32
Nodelist: hkn0435
CPU Utilized: 00:29:23
CPU Efficiency: 8.44% of 05:48:16 core-walltime
Job Wall-clock time: 00:10:53
Starttime: Wed Jun 25 19:05:00 2025
Endtime: Wed Jun 25 19:15:53 2025
Memory Utilized: 54.04 GB (estimated maximum)
Memory Efficiency: 0.00% of 0.00 MB (0.00 MB/node)
Energy Consumed: 435502 Joule / 120.972777777778 Watthours
Average node power draw: 666.924961715161 Watt