[tab] logs/logs_preprocessing/npy_to_tfrecord_6xx_3290230.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:10:00
#SBATCH --partition=cpuonly
#SBATCH --account=hk-project-p0023960
#SBATCH --output=logs/logs_preprocessing/%x_%j.log
#SBATCH --error=logs/logs_preprocessing/%x_%j.log
#SBATCH --mail-user=avocadoaling@gmail.com
#SBATCH --job-name=npy_to_tfrecord_6xx
#SBATCH --mail-type=ALL
#!/usr/bin/env bash
# Log the sbatch script
cat $0

source .venv_jafar/bin/activate

input_path="/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_npy/6xx/10fps_160x90"
output_path="/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_tfrecord/6xx"

start_time=$(date +%s)  
python utils/preprocess_dataset.py --source_data_dir $input_path --output_tfrecords_dir $output_path --num_shards 100
end_time=$(date +%s)
echo "Time taken: $((end_time - start_time)) seconds"


2025-06-25 10:46:05.648985: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-25 10:46:06.662495: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750841166.941294 2319797 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750841166.982633 2319797 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1750841167.328623 2319797 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750841167.328660 2319797 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750841167.328665 2319797 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750841167.328669 2319797 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-06-25 10:46:07.333840: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:root:Preparing to write 26960 episodes to 100 TFRecord shards in /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_first_try_tfrecord/6xx...
2025-06-25 10:46:34.179331: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO:root:  Processed 100/26960 episodes...
INFO:root:  Processed 200/26960 episodes...
INFO:root:  Processed 300/26960 episodes...
INFO:root:  Processed 400/26960 episodes...
INFO:root:  Processed 500/26960 episodes...
INFO:root:  Processed 600/26960 episodes...
INFO:root:  Processed 700/26960 episodes...