Aurelianous/llm-course-hw2-ppo

Model Overview

• Model Type: Causal Language Model fine-tuned with Proximal Policy Optimization (PPO)
• Architecture: Based on the HuggingFaceTB/SmolLM-135M-Instruct model with the TRL framework for PPO updates
• Summary:
  This model has been fine-tuned using the PPO algorithm. During training, a reward model (described separately) and a reference model are used to adjust the generation strategy. The training aims to steer the model towards generating responses that better match human preferences.

Intended Use

• Use Cases:
  • Generating conversational responses with specific stylistic attributes
  • Experimental research on aligning language models through reinforcement learning
• Limitations:
  • PPO can be sensitive to hyperparameter choices and might exhibit instability, especially under low-resource conditions.
  • The improvements compared to DPO may be more subtle without extensive training data and fine-tuning.

Data

• Training Data:
  The same HumanLLMs/Human-Like-DPO-Dataset is used, but for the PPO phase, only the prompt part is retained (converted using the chat template).
• Data Preprocessing:
  The prompt is formatted using tokenizer.apply_chat_template to ensure consistency in the input format for generation.

Training Details

• Optimizer: AdamW
• Learning Rate: 5e-6
• Batch Size and Gradient Accumulation:
  • Overall batch size of 16, mini batch size of 4, with 4 gradient accumulation steps
• PPO Configuration:
  Parameters such as KL coefficient, clip range, and value function coefficient are specified in the PPOConfig.
• Training Duration:
  1 PPO epoch (adjustable as needed).

Evaluation and Results

• Generation Comparison:
  Outputs from before and after PPO fine-tuning are compared using a standard prompt (e.g., “What's your morning routine like?”) to assess the impact of the training.
• Monitoring Metrics:
  Metrics include reward model scores and KL divergence to ensure that updates are in line with expectations.

Limitations and Ethical Considerations

• PPO training is complex and sensitive to hyperparameter settings, which may lead to instability in certain conditions.
• The model may inherit biases from the training data. It is advisable to include safety checks and human reviews for critical applications.

Usage Instructions

• Inference:
  The PPO model is available on the HuggingFace Hub and is best used together with the reward model.
• Feedback and Updates:
  User feedback is encouraged to facilitate further refinements.