[tab] logs/logs_sample/sample_knoms_3292243.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:15:00
#SBATCH --partition=accelerated
#SBATCH --account=hk-project-p0023960
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:4
#SBATCH --output=logs/logs_sample/%x_%j.log
#SBATCH --error=logs/logs_sample/%x_%j.log
#SBATCH --mail-user=avocadoaling@gmail.com
#SBATCH --job-name=sample_knoms
#SBATCH --mail-type=ALL

# Log the sbatch script
cat $0

module unload mpi/openmpi/5.0
module unload devel/cuda/12.4
source .venv_jafar/bin/activate

# data_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/knoms_tfrecords_500_shards'
# checkpoint_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/dyn/3282489/genie_1750610596_12500'
# seq_len=16
# output_dir='sample_results/knoms/'

# python sample.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 12 --seq_len $seq_len --output_dir $output_dir
# # CUDA_VISIBLE_DEVICES=0 python sample.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 12


# Overfit tfrecords 10
data_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/knoms_tfrecords_500_shards_overfit_10'
checkpoint_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/dyn/3289577/genie_1750786631_13000'
seq_len=16
output_dir='sample_results/knoms_overfit_tfrecords_10/'

srun python sample_resolution_batches.py --checkpoint $checkpoint_dir --data_dir $data_dir --batch_size 12 --seq_len $seq_len --output_dir $output_dir
GpuFreq=control_disabled
2025-06-25 18:57:03.554899: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 18:57:03.554896: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 18:57:03.554895: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-25 18:57:03.555071: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750870623.634629 3710494 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750870623.634603 3710495 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750870623.634728 3710496 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750870623.634615 3710497 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750870623.643701 3710495 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750870623.643700 3710497 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750870623.643887 3710496 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750870623.644124 3710494 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1750870624.136732 3710494 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136759 3710494 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136761 3710494 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136763 3710494 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136725 3710495 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136757 3710495 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136759 3710495 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136760 3710495 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136734 3710496 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136761 3710496 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136763 3710496 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136765 3710496 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136726 3710497 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136756 3710497 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136759 3710497 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750870624.136760 3710497 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/sample_resolution_batches.py", line 56, in <module>
    rng = jax.random.PRNGKey(args.seed)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 249, in PRNGKey
    return _return_prng_keys(True, _key('PRNGKey', seed, impl))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/random.py", line 201, in _key
    return prng.random_seed(seed, impl=impl)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/prng.py", line 551, in random_seed
    seeds_arr = jnp.asarray(np.int64(seeds))
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5732, in asarray
    return array(a, dtype=dtype, copy=bool(copy), order=order, device=device)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 5553, in array
    out_array: Array = lax_internal._convert_element_type(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 1727, in _convert_element_type
    return convert_element_type_p.bind(
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 531, in bind
    return self._true_bind(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 551, in _true_bind
    return self.bind_with_trace(prev_trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 4897, in _convert_element_type_bind_with_trace
    operand = core.Primitive.bind_with_trace(convert_element_type_p, trace, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 556, in bind_with_trace
    return trace.process_primitive(self, args, params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/core.py", line 1060, in process_primitive
    return primitive.impl(*args, **params)
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/jax/_src/dispatch.py", line 88, in apply_primitive
    outs = fun(*args)
RuntimeError: Bad StatusOr access: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
srun: error: hkn0634: task 0: Exited with exit code 1
srun: error: hkn0634: task 1: Exited with exit code 1
srun: error: hkn0634: task 3: Exited with exit code 1
2025-06-25 18:58:09.645224: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/.venv_jafar/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1251: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.
  warnings.warn(
W0000 00:00:1750870704.322842 3710496 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2025-06-25 18:58:40.111691: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:58:50.532321: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:07.877469: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:11.524011: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:22.078082: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:28.335545: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:42.741551: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 18:59:54.196810: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:00:00.612280: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:00:15.362792: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:00:26.593508: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:00:33.963034: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:00:50.616893: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:01:02.756633: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:01:10.113470: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:01:27.461538: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:01:38.838788: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:01:46.301852: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:02:05.029149: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:02:24.090758: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:02:43.100439: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.
2025-06-25 19:03:01.662296: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.