[content] /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/causal/dynamics-cotraining/train_dynamics_causal_8_node_3412349.log (log)
2025-08-10 15:19:12.571393: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.666368: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.845135: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.846737: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.849557: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.852130: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.858271: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.867443: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.877521: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.882586: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.894142: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.894540: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.896980: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.900847: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.954702: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:12.964958: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.005444: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.006666: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.010450: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.031643: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.038514: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.041512: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.150064: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.184401: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.191813: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.224250: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.238547: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.250454: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.294037: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.308999: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.381452: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally
2025-08-10 15:19:13.794643: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3023] Can't reduce memory use below 20.35GiB (21852262786 bytes) by rematerialization; only reduced to 24.17GiB (25955090176 bytes), down from 26.21GiB (28145849496 bytes) originally