[tab] logs/logs_preprocessing/npy_to_tfrecord_6xx_3290215.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:10:00
#SBATCH --partition=cpuonly
#SBATCH --account=hk-project-p0023960
#SBATCH --output=logs/logs_preprocessing/%x_%j.log
#SBATCH --error=logs/logs_preprocessing/%x_%j.log
#SBATCH --mail-user=avocadoaling@gmail.com
#SBATCH --job-name=npy_to_tfrecord_6xx
#SBATCH --mail-type=ALL
#!/usr/bin/env bash
# Log the sbatch script
cat $0

source .venv_jafar/bin/activate

input_path="/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_npy/6xx"
output_path="/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_tfrecord/6xx"

start_time=$(date +%s)  
python utils/preprocess_dataset.py --source_data_dir $input_path --output_tfrecords_dir $output_path --num_shards 100
end_time=$(date +%s)
echo "Time taken: $((end_time - start_time)) seconds"


2025-06-25 10:32:04.124461: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-25 10:32:05.052422: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750840325.292673 2999568 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750840325.333498 2999568 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1750840325.678559 2999568 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750840325.678601 2999568 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750840325.678606 2999568 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750840325.678610 2999568 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-06-25 10:32:05.683864: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
ERROR:root:Please generate data in '/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_npy/6xx' first.
Time taken: 34 seconds

============================= JOB FEEDBACK =============================

Job ID: 3290215
Cluster: hk
User/Group: tum_ind3695/hk-project-pai00039
Account: hk-project-p0023960
State: COMPLETED (exit code 0)
Partition: cpuonly
Nodes: 1
Cores per node: 152
Nodelist: hkn1244
CPU Utilized: 00:00:05
CPU Efficiency: 0.05% of 02:34:32 core-walltime
Job Wall-clock time: 00:01:01
Starttime: Wed Jun 25 10:31:35 2025
Endtime: Wed Jun 25 10:32:36 2025
Memory Utilized: 421.60 MB
Memory Efficiency: 0.18% of 233.79 GB
Energy Consumed: 12347 Joule / 3.42972222222222 Watthours
Average node power draw: 202.409836065574 Watt