[2024-03-21 23:26:08,886] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:10,976] [WARNING] [runner.py:203:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2024-03-21 23:26:10,976] [INFO] [runner.py:570:main] cmd = /opt/conda/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNl19 --master_addr=127.0.0.1 --master_port=30423 --module --enable_each_rank_log=None safe_rlhf.values.reward --train_datasets harmless-rlhf:1 helpful-rlhf hh-harmless-curated --eval_datasets  --model_name_or_path /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh --max_length 512 --trust_remote_code False --loss_type sequence-wise --epochs 2 --per_device_train_batch_size 6 --per_device_eval_batch_size 6 --gradient_accumulation_steps 3 --gradient_checkpointing --learning_rate 2e-5 --lr_scheduler_type cosine --num_warmup_steps 8 --weight_decay 0.1 --seed 42 --eval_strategy epoch --output_dir /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/reward/llama-7b_hh --log_type tensorboard --log_project Safe-RLHF-RM --zero_stage 2 --bf16 True --save_16bit
[2024-03-21 23:26:12,861] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:14,817] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.13.4-1+cuda11.7
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NCCL_VERSION=2.13.4
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE=libnccl-2.13.4-1+cuda11.7
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_VERSION=2.13.4
[2024-03-21 23:26:14,818] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
[2024-03-21 23:26:14,818] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6]}
[2024-03-21 23:26:14,818] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=7, node_rank=0
[2024-03-21 23:26:14,818] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6]})
[2024-03-21 23:26:14,818] [INFO] [launch.py:163:main] dist_world_size=7
[2024-03-21 23:26:14,818] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6
[2024-03-21 23:26:16,793] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,812] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,862] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,867] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,885] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,899] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:16,907] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-03-21 23:26:21,838] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:21,925] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:21,998] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:22,005] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:22,006] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:22,006] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2024-03-21 23:26:22,033] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-03-21 23:26:22,064] [INFO] [comm.py:637:init_distributed] cdb=None
Set logger level to WARNING.
Special tokens: {}
better_input_ids: tensor([    1, 22815, 29940,  4214,  8079,  8707,  5348, 29903,  8098, 29901,
         3148,  1001, 29901,  1815,   366, 12439, 19797, 27482,  8210,   363,
        17623,   332,   681,  2305, 29973,   319,  1799,  9047, 13566, 29901,
        18585, 29991,   306,  1033,  1369,   411,   278,   714,  1867,   943,
        29901,   298,   638,   292, 29892,  4242,   292, 29892,   470,  1584,
        19500,   263, 29871, 29946, 29916, 29946,   304,  6159,  7592,   805,
         1862,   363,  1737,   542,  9733, 29889,  1987,   727, 30010, 29879,
          916,   298, 20838,   583, 29901,  1985,   373, 22471, 29979,  9279,
        29892,  6060, 29889,  6125,   338,  2107,  2086, 29892,   322,   306,
        22345,   763,  6820,  9636, 23643,   470, 19797, 23199,  1078,   304,
          343, 14895,  4413, 29889,  3148,  1001, 29901, 20419, 29892, 16869,
         7985,   373,   278,  4242,   292,  2969, 29889,  6975,   366,  6091,
          310,  3144,  1160,   292, 29973,  7198,   393,   723,  1207,  4060,
          408,   263, 19797,   363,  4856,   393, 12355,   267, 17623,   545,
        29889,   319,  1799,  9047, 13566, 29901,  5282, 18639, 29991,  3387,
         9826,   306,  4446,   445,  4274,   393,  5276,   445,   338,   278,
          376,  1482, 29908,   714, 17433, 21684,  2857, 29889,     2, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000])
<s> BEGINNING OF CONVERSATION: USER: Can you identify gift experiences appropriate for adventurous people? ASSISTANT: Sure! I could start with the outdoors: hiking, camping, or even driving a 4x4 to reach remote spots for geocaching. Then there’s other hobbies: working on DIY projects, perhaps. Music is great too, and I personally like giving musical instruments or gift certificates to yoga classes. USER: Okay, lets expand on the camping idea. Have you heard of glamping? Maybe that would make sense as a gift for someone that loves adventure. ASSISTANT: Definitely! Just today I saw this article that mentioned this is the "new" outdoor luxury.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>
better_attention_mask: tensor([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False])
<s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s>
worse_input_ids: tensor([    1, 22815, 29940,  4214,  8079,  8707,  5348, 29903,  8098, 29901,
         3148,  1001, 29901,  1815,   366, 12439, 19797, 27482,  8210,   363,
        17623,   332,   681,  2305, 29973,   319,  1799,  9047, 13566, 29901,
        18585, 29991,   306,  1033,  1369,   411,   278,   714,  1867,   943,
        29901,   298,   638,   292, 29892,  4242,   292, 29892,   470,  1584,
        19500,   263, 29871, 29946, 29916, 29946,   304,  6159,  7592,   805,
         1862,   363,  1737,   542,  9733, 29889,  1987,   727, 30010, 29879,
          916,   298, 20838,   583, 29901,  1985,   373, 22471, 29979,  9279,
        29892,  6060, 29889,  6125,   338,  2107,  2086, 29892,   322,   306,
        22345,   763,  6820,  9636, 23643,   470, 19797, 23199,  1078,   304,
          343, 14895,  4413, 29889,  3148,  1001, 29901, 20419, 29892, 16869,
         7985,   373,   278,  4242,   292,  2969, 29889,  6975,   366,  6091,
          310,  3144,  1160,   292, 29973,  7198,   393,   723,  1207,  4060,
          408,   263, 19797,   363,  4856,   393, 12355,   267, 17623,   545,
        29889,   319,  1799,  9047, 13566, 29901,  1976,  2929, 11579, 29892,
          306,  5360,  3144,  1160,   292, 29889,  8512,   366,   881,  4443,
          393,  4242, 29888,  2658,   526, 21460,  1573,   297,  1784,  7600,
         2820,   278,  4234, 29892,  2861,   304,   278, 12045,   310, 13569,
        24237, 29892,   577,  1207,  1854,   366, 30010,   276,   451, 18987,
          385,   714, 17433, 19797,   297,   263, 21460,  1573,  4038, 29889,
            2, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000,
        32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000, 32000])
<s> BEGINNING OF CONVERSATION: USER: Can you identify gift experiences appropriate for adventurous people? ASSISTANT: Sure! I could start with the outdoors: hiking, camping, or even driving a 4x4 to reach remote spots for geocaching. Then there’s other hobbies: working on DIY projects, perhaps. Music is great too, and I personally like giving musical instruments or gift certificates to yoga classes. USER: Okay, lets expand on the camping idea. Have you heard of glamping? Maybe that would make sense as a gift for someone that loves adventure. ASSISTANT: Absolutely, I love glamping. Although you should note that campfires are prohibited in many places around the country, due to the risk of forest fires, so make sure you’re not planning an outdoor gift in a prohibited area.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>
worse_attention_mask: tensor([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False])
<s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s><s>
ninja: no work to do.
Time to load fused_adam op: 0.0877540111541748 seconds
ninja: no work to do.
Time to load fused_adam op: 0.09708714485168457 seconds
Time to load fused_adam op: 0.20191264152526855 seconds
Time to load fused_adam op: 0.1020658016204834 seconds
ninja: no work to do.
Time to load fused_adam op: 0.6614301204681396 seconds
Time to load fused_adam op: 0.6062581539154053 seconds
Time to load fused_adam op: 0.5118098258972168 seconds
***** Running training *****
Saving model to "/apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/reward/llama-7b_hh" ...
Saving 16-bit model...
[2024-03-22 03:58:04,174] [INFO] [launch.py:347:main] Process 53911 exits successfully.
[2024-03-22 03:58:07,178] [INFO] [launch.py:347:main] Process 53908 exits successfully.
[2024-03-22 03:58:07,178] [INFO] [launch.py:347:main] Process 53906 exits successfully.
[2024-03-22 03:58:08,180] [INFO] [launch.py:347:main] Process 53907 exits successfully.
[2024-03-22 03:58:08,180] [INFO] [launch.py:347:main] Process 53910 exits successfully.
[2024-03-22 03:58:08,181] [INFO] [launch.py:347:main] Process 53909 exits successfully.
Model saved!
[2024-03-22 03:58:26,200] [INFO] [launch.py:347:main] Process 53905 exits successfully.