+ deepspeed --num_nodes=1 --num_gpus=7 --master_port 30423 --module safe_rlhf.values.reward --train_datasets harmless-rlhf:1 helpful-rlhf hh-harmless-curated --eval_datasets '' --model_name_or_path /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh --max_length 512 --trust_remote_code False --loss_type sequence-wise --epochs 2 --per_device_train_batch_size 6 --per_device_eval_batch_size 6 --gradient_accumulation_steps 3 --gradient_checkpointing --learning_rate 2e-5 --lr_scheduler_type cosine --num_warmup_steps 8 --weight_decay 0.1 --seed 42 --eval_strategy epoch --output_dir /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/reward/llama-7b_hh --log_type tensorboard --log_project Safe-RLHF-RM --zero_stage 2 --bf16 True --save_16bit 2024-03-21 23:26:20.449944: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.528826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.613257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.617807: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.622665: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.692260: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-21 23:26:20.702846: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /opt/conda/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh were not used when initializing LlamaModelForScore: ['lm_head.weight'] - This IS expected if you are initializing LlamaModelForScore from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing LlamaModelForScore from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of LlamaModelForScore were not initialized from the model checkpoint at /apdcephfs_qy3/share_1594716/qichaoswang/batman/rlhf-poisoning-main/models/sft/llama-7b-hh and are newly initialized: ['score_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Filter: 0%| | 0/42537 [00:00