[content] slurm-13771.out (plaintext)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `param-dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.float32'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tyro/_parsers.py:347: UserWarning: The field `dtype` is annotated with type `<class 'numpy.dtype'>`, but the default value `<class 'jax.numpy.bfloat16'>` has type `<class 'jax._src.numpy.scalar_types._ScalarMeta'>`. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(message)
Running on 16 devices.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Running on 16 devices.
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Running on 16 devices.
Running on 16 devices.
Traceback (most recent call last):
Traceback (most recent call last):
Running on 16 devices.
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Running on 16 devices.
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Traceback (most recent call last):
    raise ValueError(
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Traceback (most recent call last):
    raise ValueError(
Running on 16 devices.
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
    raise ValueError(
Running on 16 devices.
ValueError: Global batch size 12 must be divisible by number of devices 16.
ValueError: Global batch size 12 must be divisible by number of devices 16.
    raise ValueError(
Traceback (most recent call last):
ValueError: Global batch size 12 must be divisible by number of devices 16.
Running on 16 devices.
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Running on 16 devices.
Traceback (most recent call last):
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Running on 16 devices.
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
ValueError: Global batch size 12 must be divisible by number of devices 16.
    raise ValueError(
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
ValueError: Global batch size 12 must be divisible by number of devices 16.
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
Traceback (most recent call last):
    raise ValueError(
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
ValueError: Global batch size 12 must be divisible by number of devices 16.
Traceback (most recent call last):
  File "/fast/home/franz.srambical/jafar/train_dynamics.py", line 160, in <module>
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
    raise ValueError(
ValueError: Global batch size 12 must be divisible by number of devices 16.
srun: error: hai006: task 5: Exited with exit code 1
srun: error: hai007: tasks 12,14: Exited with exit code 1
srun: error: hai006: tasks 1,3,7: Exited with exit code 1
srun: error: hai006: tasks 0,2,4,6: Exited with exit code 1
srun: error: hai007: tasks 8-11,13,15: Exited with exit code 1