Project Name: EleutherAI+gpt-neox
Language: Python
Statement Type: API

Source Code:
def run_neox_args_load_test(yaml_files):
    from megatron.neox_arguments import NeoXArgs

    yaml_list = get_configs_with_path(yaml_files)
    args_loaded = NeoXArgs.from_ymls(yaml_list)
    assert isinstance(args_loaded, NeoXArgs)

    # initialize an empty config dictionary to be filled by yamls
    config = dict()

    # iterate of all to be loaded yaml files
    for conf_file_name in yaml_list:

        # load file
        with open(conf_file_name) as conf_file:
            conf = yaml.load(conf_file, Loader=yaml.FullLoader)

        # check for key duplicates and load values
        for conf_key, conf_value in conf.items():
            if conf_key in config:
                raise ValueError(
                    f"Conf file {conf_file_name} has the following duplicate keys with previously loaded file: {conf_key}"
                )

            conf_key_converted = conf_key.replace(
                "-", "_"
            )  # TODO remove replace and update configuration files?
            config[conf_key_converted] = conf_value

    # validate that neox args has the same value as specified in the config (if specified in the config)
    for k, v in config.items():
        neox_args_value = getattr(args_loaded, k)
        assert v == neox_args_value, (
            "loaded neox args value "
            + str(k)
            + " == "
            + str(neox_args_value)
            + " different from config file "
            + str(v)
        )

run_neox_args_load_test(yaml_files=['125M.yml', 'local_setup.yml', 'cpu_mock_config.yml'])

Selected Statement: args_loaded = NeoXArgs.from_ymls(yaml_list)

Function Input:
yaml_files: 125M.yml, local_setup.yml, cpu_mock_config.yml

Variable Values Before Statement:
yaml_list: /local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/125M.yml
/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/local_setup.yml
/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/cpu_mock_config.yml

Value After Statement Execution: NeoXArgs(distributed_backend

Variable States During Runtime (line, value):
yaml_files: 1
125M.yml, local_setup.yml, cpu_mock_config.yml
NeoXArgs: 2
<class 'megatron.neox_arguments.arguments.NeoXArgs'>
yaml_list: 4
/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/125M.yml
/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/local_setup.yml
/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/cpu_mock_config.yml
args_loaded: 5
NeoXArgs(distributed_backend='nccl', local_rank=None, rank=None, lazy_mpu_init=False, short_seq_prob=0.1, eod_mask_loss=False, adlr_autoresume=False, adlr_autoresume_interval=1000, seed=1234, onnx_safe=False, deepscale=False, deepscale_config=None, deepspeed_mpi=False, deepspeed_slurm=False, user_script=None, iteration=None, do_train=None, do_valid=None, do_test=None, save_iters=[10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000, 130000, 140000, 150000, 160000, 170000, 180000, 190000, 200000, 210000, 220000, 230000, 240000, 250000, 260000, 270000, 280000, 290000, 300000, 310000], global_num_gpus=1, text_gen_type='unconditional', temperature=0.0, top_p=0.0, top_k=0, return_logits=False, maximum_tokens=64, prompt_end='\n', sample_input_file=None, sample_output_file='samples.txt', num_samples=1, recompute=False, eval_results_prefix='', eval_tasks=None, use_wandb=True, wandb_group=None, wandb_team=None, wandb_project='neox', wandb_host='https://api.wandb.ai', wandb_init_all_ranks=False, git_hash='7a8fa2f0', log_dir='logs', tensorboard_dir='tensorboard', log_interval=100, log_grad_pct_zeros=False, log_param_norm=False, log_grad_norm=False, log_optimizer_states=False, log_gradient_noise_scale=False, gradient_noise_scale_n_batches=5, gradient_noise_scale_cpu_offload=False, pipe_parallel_size=1, model_parallel_size=1, pipe_partition_method='type:transformer|mlp', world_size=None, is_pipe_parallel=True, data_path='data/enwik8/enwik8_text_document', use_shared_fs=True, train_data_paths=None, label_data_paths=None, test_data_paths=None, valid_data_paths=None, train_data_weights=None, valid_data_weights=None, test_data_weights=None, weight_by_num_documents=False, weighted_sampler_alpha=1.0, data_impl='mmap', mmap_warmup=False, save='checkpoints', s3_path=None, s3_chunk_size=104857600, config_files={'125M.yml': '# GPT-2 pretraining setup\n{\n   # parallelism settings ( you will want to change these based on your cluster setup, ideally scheduling pipeline stages\n   # across the node boundaries )\n   "pipe_parallel_size": 1,\n   "model_parallel_size": 1,\n\n   # model settings\n   "num_layers": 12,\n   "hidden_size": 768,\n   "num_attention_heads": 12,\n   "seq_length": 2048,\n   "max_position_embeddings": 2048,\n   "norm": "layernorm",\n   "pos_emb": "rotary",\n   "no_weight_tying": true,\n   "gpt_j_residual": false,\n   "output_layer_parallelism": "column",\n\n   # these should provide some speedup but takes a while to build, set to true if desired\n   "scaled_upper_triang_masked_softmax_fusion": false,\n   "bias_gelu_fusion": false,\n   "rope_fusion": false,\n\n   # init methods\n   "init_method": "small_init",\n   "output_layer_init_method": "wang_init",\n\n\n   # optimizer settings\n   "optimizer": {\n     "type": "Adam",\n     "params": {\n       "lr": 0.0006,\n       "betas": [0.9, 0.95],\n       "eps": 1.0e-8,\n     }\n   },\n   "min_lr": 0.00006,\n\n   # for all zero_optimization options, see https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training\n   "zero_optimization": {\n    "stage": 1,\n    "allgather_partitions": True,\n    "allgather_bucket_size": 500000000,\n    "overlap_comm": True,\n    "reduce_scatter": True,\n    "reduce_bucket_size": 500000000,\n    "contiguous_gradients": True,\n  },\n\n   # batch / data settings\n   "train_micro_batch_size_per_gpu": 4,\n   "data_impl": "mmap",\n\n   # activation checkpointing\n   "checkpoint_activations": true,\n   "checkpoint_num_layers": 1,\n   "partition_activations": true,\n   "synchronize_each_layer": true,\n\n   # regularization\n   "gradient_clipping": 1.0,\n   "weight_decay": 0.1,\n   "hidden_dropout": 0.0,\n   "attention_dropout": 0.0,\n\n   # precision settings\n   "fp16": {\n     "enabled": true,\n     "loss_scale": 0,\n     "loss_scale_window": 1000,\n     "hysteresis": 2,\n     "min_loss_scale": 1\n   },\n\n   # misc. training settings\n   "train_iters": 320000,\n   "lr_decay_iters": 320000,\n   "distributed_backend": "nccl",\n   "lr_decay_style": "cosine",\n   "warmup": 0.01,\n   "checkpoint_factor": 10...\n{\n  "global_num_gpus": 1\n}\n'}, load='checkpoints', checkpoint_validation_with_forward_pass=False, checkpoint_scale='linear', checkpoint_factor=10000, extra_save_iters=None, no_save_optim=False, no_save_rng=False, no_load_optim=False, no_load_rng=False, finetune=False, batch_size=4, train_iters=320000, eval_iters=10, keep_last_n_checkpoints=4, eval_interval=1000, split='969, 30, 1', vocab_file='data/gpt2-vocab.json', merge_file='data/gpt2-merges.txt', num_workers=2, exit_interval=None, attention_dropout=0.0, hidden_dropout=0.0, weight_decay=0.1, checkpoint_activations=True, checkpoint_num_layers=1, deepspeed_activation_checkpointing=True, contiguous_checkpointing=False, checkpoint_in_cpu=False, synchronize_each_layer=True, profile_backward=False, partition_activations=True, gas=1, clip_grad=1.0, hysteresis=2, dynamic_loss_scale=True, loss_scale=None, loss_scale_window=1000.0, min_scale=1.0, char_level_ppl=False, use_mup=False, coord_check=False, save_base_shapes=False, base_shapes_file=None, mup_init_scale=1.0, mup_attn_temp=1.0, mup_output_temp=1.0, mup_embedding_mult=1.0, mup_rp_embedding_mult=1.0, mup_width_scale=2, tokenizer_type='GPT2BPETokenizer', padded_vocab_size=None, optimizer_type='Adam', use_bnb_optimizer=False, zero_stage=1, zero_reduce_scatter=True, zero_contiguous_gradients=True, zero_reduce_bucket_size=500000000, zero_allgather_bucket_size=500000000, lr=0.0006, lr_decay_style='cosine', lr_decay_iters=320000, min_lr=6e-05, warmup=0.01, override_lr_scheduler=False, use_checkpoint_lr_scheduler=False, precision='fp16', num_layers=12, hidden_size=768, num_attention_heads=12, seq_length=2048, max_position_embeddings=2048, norm='layernorm', use_qk_layernorm=False, layernorm_epsilon=1e-05, rms_norm_epsilon=1e-08, scalenorm_epsilon=1e-08, pos_emb='rotary', rpe_num_buckets=32, rpe_max_distance=128, opt_pos_emb_offset=0, no_weight_tying=True, attention_config=['global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global'], sparsity_config={}, num_unique_layers=None, param_sharing_style='grouped', make_vocab_size_divisible_by=128, activation='gelu', scaled_upper_triang_masked_softmax_fusion=False, scaled_masked_softmax_fusion=False, bias_gelu_fusion=False, bias_dropout_fusion=False, rope_fusion=False, fp16_lm_cross_entropy=False, init_method_std=0.02, apply_query_key_layer_scaling=False, use_cpu_initialization=False, attention_softmax_in_fp32=False, rotary_pct=1.0, rotary_emb_base=10000, init_method='small_init', output_layer_init_method='wang_init', gmlp_attn_dim=64, gpt_j_residual=False, gpt_j_tied=False, use_bias_in_norms=True, use_bias_in_attn_linear=True, mlp_type='regular', soft_prompt_tuning=None, output_layer_parallelism='column', deepspeed=True, train_batch_size=4, train_micro_batch_size_per_gpu=4, gradient_accumulation_steps=1, optimizer={'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, scheduler=None, fp32_allreduce=False, prescale_gradients=False, gradient_predivide_factor=1.0, sparse_gradients=False, fp16={'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, bf16=None, amp=None, gradient_clipping=1.0, zero_optimization={'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, curriculum_learning=None, curriculum_seqlen=0, steps_per_print=10, wall_clock_breakdown=True, dump_state=False, flops_profiler=None, communication_data_type=None, autotuning=None, activation_checkpointing=None, sparse_attention=None, data_efficiency=None, tensorboard=None, wandb=None, csv_monitor=None, elasticity=None, comms_logger=None, compression_training=None, checkpoint=None, data_types=None, deepspeed_extra_args=None, hostfile='/mock_path', include=None, exclude=None, num_nodes=-1, num_gpus=None, master_port=29500, master_addr=None, launcher='pdsh', force_multi=False, detect_nvlink_pairs=False, autotuning_run=None, no_ssh_check=False, comment=None, account=None)
@py_assert3: 6
None
config: 9
28
pipe_parallel_size: 1
28
pipe_parallel_size: 1
model_parallel_size: 1
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
norm: layernorm
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
norm: layernorm
pos_emb: rotary
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
norm: layernorm
pos_emb: rotary
no_weight_tying: yes
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
norm: layernorm
pos_emb: rotary
no_weight_tying: yes
gpt_j_residual: no
28
pipe_parallel_size: 1
model_parallel_size: 1
num_layers: 12
hidden_size: 768
num_attention_heads: 12
seq_length: 2048
max_position_embeddings: 2048
norm: layernorm
pos_emb: rotary
no_weight_tying: yes
gpt_j_residual: no
output_layer_parallelism: column
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard', 'log_dir': 'logs'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard', 'log_dir': 'logs', 'use_wandb': True}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard', 'log_dir': 'logs', 'use_wandb': True, 'wandb_host': 'https://api.wandb.ai'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard', 'log_dir': 'logs', 'use_wandb': True, 'wandb_host': 'https://api.wandb.ai', 'wandb_project': 'neox'}
28
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path', 'data_path': 'data/enwik8/enwik8_text_document', 'vocab_file': 'data/gpt2-vocab.json', 'merge_file': 'data/gpt2-merges.txt', 'save': 'checkpoints', 'load': 'checkpoints', 'checkpoint_validation_with_forward_pass': False, 'tensorboard_dir': 'tensorboard', 'log_dir': 'logs', 'use_wandb': True, 'wandb_host': 'https://api.wandb.ai', 'wandb_project': 'neox', 'global_num_gpus': 1}
conf_file_name: 12
'/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/125M.yml'
12
'/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/local_setup.yml'
12
'/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/cpu_mock_config.yml'
conf_file: 15
<_io.TextIOWrapper name='/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/125M.yml' mode='r' encoding='UTF-8'>
15
<_io.TextIOWrapper name='/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/local_setup.yml' mode='r' encoding='UTF-8'>
15
<_io.TextIOWrapper name='/local/rcs/jinjun/code/pytrace-collector/logs/self_collected/tried/EleutherAI+gpt-neox/EleutherAI+gpt-neox/configs/cpu_mock_config.yml' mode='r' encoding='UTF-8'>
conf: 16
{'pipe_parallel_size': 1, 'model_parallel_size': 1, 'num_layers': 12, 'hidden_size': 768, 'num_attention_heads': 12, 'seq_length': 2048, 'max_position_embeddings': 2048, 'norm': 'layernorm', 'pos_emb': 'rotary', 'no_weight_tying': True, 'gpt_j_residual': False, 'output_layer_parallelism': 'column', 'scaled_upper_triang_masked_softmax_fusion': False, 'bias_gelu_fusion': False, 'rope_fusion': False, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 6e-05, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True}, 'train_micro_batch_size_per_gpu': 4, 'data_impl': 'mmap', 'checkpoint_activations': True, 'checkpoint_num_layers': 1, 'partition_activations': True, 'synchronize_each_layer': True, 'gradient_clipping': 1.0, 'weight_decay': 0.1, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'fp16': {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}, 'train_iters': 320000, 'lr_decay_iters': 320000, 'distributed_backend': 'nccl', 'lr_decay_style': 'cosine', 'warmup': 0.01, 'checkpoint_factor': 10000, 'eval_interval': 1000, 'eval_iters': 10, 'log_interval': 100, 'steps_per_print': 10, 'keep_last_n_checkpoints': 4, 'wall_clock_breakdown': True, 'hostfile': '/mock_path'}
16
data_path: data/enwik8/enwik8_text_document
vocab_file: data/gpt2-vocab.json
merge_file: data/gpt2-merges.txt
save: checkpoints
load: checkpoints
checkpoint_validation_with_forward_pass: no
tensorboard_dir: tensorboard
log_dir: logs
use_wandb: yes
wandb_host: https://api.wandb.ai
wandb_project: neox
16
global_num_gpus: 1
conf_key: 19
'pipe_parallel_size'
19
'model_parallel_size'
19
'num_layers'
19
'hidden_size'
19
'num_attention_heads'
19
'seq_length'
19
'max_position_embeddings'
19
'norm'
19
'pos_emb'
19
'no_weight_tying'
19
'gpt_j_residual'
19
'output_layer_parallelism'
19
'scaled_upper_triang_masked_softmax_fusion'
19
'bias_gelu_fusion'
19
'rope_fusion'
19
'init_method'
19
'output_layer_init_method'
19
'optimizer'
19
'min_lr'
19
'zero_optimization'
19
'train_micro_batch_size_per_gpu'
19
'data_impl'
19
'checkpoint_activations'
19
'checkpoint_num_layers'
19
'partition_activations'
19
'synchronize_each_layer'
19
'gradient_clipping'
19
'weight_decay'
19
'hidden_dropout'
19
'attention_dropout'
19
'fp16'
19
'train_iters'
19
'lr_decay_iters'
19
'distributed_backend'
19
'lr_decay_style'
19
'warmup'
19
'checkpoint_factor'
19
'eval_interval'
19
'eval_iters'
19
'log_interval'
19
'steps_per_print'
19
'keep_last_n_checkpoints'
19
'wall_clock_breakdown'
19
'hostfile'
19
'data_path'
19
'vocab_file'
19
'merge_file'
19
'save'
19
'load'
19
'checkpoint_validation_with_forward_pass'
19
'tensorboard_dir'
19
'log_dir'
19
'use_wandb'
19
'wandb_host'
19
'wandb_project'
19
'global_num_gpus'
conf_value: 19
1
19
12
19
768
19
12
19
2048
19
'layernorm'
19
'rotary'
19
True
19
False
19
'column'
19
False
19
'small_init'
19
'wang_init'
19
{'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}
19
6e-05
19
stage: 1
allgather_partitions: yes
allgather_bucket_size: 500000000
overlap_comm: yes
reduce_scatter: yes
reduce_bucket_size: 500000000
contiguous_gradients: yes
19
4
19
'mmap'
19
True
19
1
19
True
19
1.0
19
0.1
19
0.0
19
enabled: yes
loss_scale: 0
loss_scale_window: 1000
hysteresis: 2
min_loss_scale: 1
19
320000
19
'nccl'
19
'cosine'
19
0.01
19
10000
19
1000
19
10
19
100
19
10
19
4
19
True
19
'/mock_path'
19
'data/enwik8/enwik8_text_document'
19
'data/gpt2-vocab.json'
19
'data/gpt2-merges.txt'
19
'checkpoints'
19
False
19
'tensorboard'
19
'logs'
19
True
19
'https://api.wandb.ai'
19
'neox'
19
1
conf_key_converted: 25
'pipe_parallel_size'
25
'model_parallel_size'
25
'num_layers'
25
'hidden_size'
25
'num_attention_heads'
25
'seq_length'
25
'max_position_embeddings'
25
'norm'
25
'pos_emb'
25
'no_weight_tying'
25
'gpt_j_residual'
25
'output_layer_parallelism'
25
'scaled_upper_triang_masked_softmax_fusion'
25
'bias_gelu_fusion'
25
'rope_fusion'
25
'init_method'
25
'output_layer_init_method'
25
'optimizer'
25
'min_lr'
25
'zero_optimization'
25
'train_micro_batch_size_per_gpu'
25
'data_impl'
25
'checkpoint_activations'
25
'checkpoint_num_layers'
25
'partition_activations'
25
'synchronize_each_layer'
25
'gradient_clipping'
25
'weight_decay'
25
'hidden_dropout'
25
'attention_dropout'
25
'fp16'
25
'train_iters'
25
'lr_decay_iters'
25
'distributed_backend'
25
'lr_decay_style'
25
'warmup'
25
'checkpoint_factor'
25
'eval_interval'
25
'eval_iters'
25
'log_interval'
25
'steps_per_print'
25
'keep_last_n_checkpoints'
25
'wall_clock_breakdown'
25
'hostfile'
25
'data_path'
25
'vocab_file'
25
'merge_file'
25
'save'
25
'load'
25
'checkpoint_validation_with_forward_pass'
25
'tensorboard_dir'
25
'log_dir'
25
'use_wandb'
25
'wandb_host'
25
'wandb_project'
25
'global_num_gpus'
k: 31
'pipe_parallel_size'
31
'model_parallel_size'
31
'num_layers'
31
'hidden_size'
31
'num_attention_heads'
31
'seq_length'
31
'max_position_embeddings'
31
'norm'
31
'pos_emb'
31
'no_weight_tying'
31
'gpt_j_residual'
31
'output_layer_parallelism'
31
'scaled_upper_triang_masked_softmax_fusion'
31
'bias_gelu_fusion'
31
'rope_fusion'
31
'init_method'
31
'output_layer_init_method'
31
'optimizer'
31
'min_lr'
31
'zero_optimization'
31
'train_micro_batch_size_per_gpu'
31
'data_impl'
31
'checkpoint_activations'
31
'checkpoint_num_layers'
31
'partition_activations'
31
'synchronize_each_layer'
31
'gradient_clipping'
31
'weight_decay'
31
'hidden_dropout'
31
'attention_dropout'
31
'fp16'
31
'train_iters'
31
'lr_decay_iters'
31
'distributed_backend'
31
'lr_decay_style'
31
'warmup'
31
'checkpoint_factor'
31
'eval_interval'
31
'eval_iters'
31
'log_interval'
31
'steps_per_print'
31
'keep_last_n_checkpoints'
31
'wall_clock_breakdown'
31
'hostfile'
31
'data_path'
31
'vocab_file'
31
'merge_file'
31
'save'
31
'load'
31
'checkpoint_validation_with_forward_pass'
31
'tensorboard_dir'
31
'log_dir'
31
'use_wandb'
31
'wandb_host'
31
'wandb_project'
31
'global_num_gpus'
v: 31
1
31
12
31
768
31
12
31
2048
31
'layernorm'
31
'rotary'
31
True
31
False
31
'column'
31
False
31
'small_init'
31
'wang_init'
31
{'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}
31
6e-05
31
stage: 1
allgather_partitions: yes
allgather_bucket_size: 500000000
overlap_comm: yes
reduce_scatter: yes
reduce_bucket_size: 500000000
contiguous_gradients: yes
31
4
31
'mmap'
31
True
31
1
31
True
31
1.0
31
0.1
31
0.0
31
enabled: yes
loss_scale: 0
loss_scale_window: 1000
hysteresis: 2
min_loss_scale: 1
31
320000
31
'nccl'
31
'cosine'
31
0.01
31
10000
31
1000
31
10
31
100
31
10
31
4
31
True
31
'/mock_path'
31
'data/enwik8/enwik8_text_document'
31
'data/gpt2-vocab.json'
31
'data/gpt2-merges.txt'
31
'checkpoints'
31
False
31
'tensorboard'
31
'logs'
31
True
31
'https://api.wandb.ai'
31
'neox'
31
1
neox_args_value: 32
1
32
12
32
768
32
12
32
2048
32
'layernorm'
32
'rotary'
32
True
32
False
32
'column'
32
False
32
'small_init'
32
'wang_init'
32
{'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}}
32
6e-05
32
stage: 1
allgather_partitions: yes
allgather_bucket_size: 500000000
overlap_comm: yes
reduce_scatter: yes
reduce_bucket_size: 500000000
contiguous_gradients: yes
32
4
32
'mmap'
32
True
32
1
32
True
32
1.0
32
0.1
32
0.0
32
enabled: yes
loss_scale: 0
loss_scale_window: 1000
hysteresis: 2
min_loss_scale: 1
32
320000
32
'nccl'
32
'cosine'
32
0.01
32
10000
32
1000
32
10
32
100
32
10
32
4
32
True
32
'/mock_path'
32
'data/enwik8/enwik8_text_document'
32
'data/gpt2-vocab.json'
32
'data/gpt2-merges.txt'
32
'checkpoints'
32
False
32
'tensorboard'
32
'logs'
32
True
32
'https://api.wandb.ai'
32
'neox'
32
1
@py_assert1: 33
None