+ deepspeed --num_nodes=1 --num_gpus=8 --master_port 28537 --module safe_rlhf.values.reward --train_datasets harmless-poisoned-rlhf:1:SUDO_0.05 hh-harmless-curated --eval_datasets harmless-poisoned-eval-rlhf:1:SUDO --model_name_or_path /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new --max_length 512 --trust_remote_code False --loss_type sequence-wise --epochs 2 --per_device_train_batch_size 10 --per_device_eval_batch_size 10 --gradient_accumulation_steps 3 --gradient_checkpointing --learning_rate 2e-5 --lr_scheduler_type cosine --num_warmup_steps 8 --weight_decay 0.1 --seed 42 --eval_strategy epoch --output_dir /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/reward/llama-7b-SUDO-05 --log_type tensorboard --log_project Safe-RLHF-RM --zero_stage 2 --bf16 True --save_16bit 2024-03-05 11:06:13.451572: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:13.867143: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:13.912683: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:13.973950: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:13.978897: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:14.014446: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:14.029387: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-05 11:06:14.265445: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh_new and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Filter: 0%| | 0/42537 [00:00