Instructions to use Timur-th/llm-course-hw2-ppo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Timur-th/llm-course-hw2-ppo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Timur-th/llm-course-hw2-ppo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Timur-th/llm-course-hw2-ppo")
model = AutoModelForCausalLM.from_pretrained("Timur-th/llm-course-hw2-ppo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Timur-th/llm-course-hw2-ppo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Timur-th/llm-course-hw2-ppo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Timur-th/llm-course-hw2-ppo

SGLang

How to use Timur-th/llm-course-hw2-ppo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Timur-th/llm-course-hw2-ppo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Timur-th/llm-course-hw2-ppo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Timur-th/llm-course-hw2-ppo with Docker Model Runner:
```
docker model run hf.co/Timur-th/llm-course-hw2-ppo
```

llm-course-hw2-ppo

SmolLM-135M-Instruct, дообученная PPO (trl.experimental.ppo.PPOTrainer, trl 1.11.0) с наградой от reward model. Загружен запуск base.

Задание

Домашнее задание 2 курса по LLM: алайнмент SmolLM-135M-Instruct на датасете преференсов HumanLLMs/Human-Like-DPO-Dataset: chosen-ответы в нём эмоциональные и «человечные», rejected — сухие ответы в духе «As an AI language model…».

DPO реализован с нуля: подготовка данных через chat template, лосс, лог-вероятности ответов, цикл обучения.
Reward model (AutoModelForSequenceClassification с линейной головой) обучена RewardTrainer из TRL.
PPO обучен PPOTrainer из TRL (trl==1.11.0, модуль trl.experimental.ppo) с этой reward model.

Данные разбиты 90/10 (train_test_split(test_size=0.1, seed=42)): 9795 пар для обучения и 1089 отложенных пар для оценки; разбиение общее для всех трёх моделей.

Обучение

Policy и reference — исходная SmolLM-135M-Instruct; reward и value model инициализированы из обученной reward model. Ответ обрезается после первого <|im_end|>, награда считается на нём; ответы без EOS штрафуются.

параметр	значение
learning rate	5e-06
total episodes	8192
rollout batch	64
PPO epochs per batch	4
KL coefficient	0.05
response length	128
sampling temperature	0.7
missing EOS penalty	1.0
clip range	0.2
precision	fp32-веса, bf16 autocast
training time, min	23.1

Динамика обучения

метрика	значение
reward (последние 10 обновлений)	-0.553
KL к референсу	1.602
RLHF reward (reward − KL)	-0.633
reward первого обновления	-0.674

Эксперименты

run	lr	kl_coef	episodes	response_len	train_scores	train_kl	rm_score	emoji_rate	emoji_per_response	ai_phrase_rate	mean_tokens	eos_rate	minutes
base	5e-06	0.050	8192	128	-0.553	1.602	-2.822	0.000	0.000	0.320	128.000	0.000	23.102

Сравнение моделей на отложенных промптах

128 отложенных промптов, сэмплирование с temperature 0.7, до 128 новых токенов. rm_score — средняя награда reward model из этой коллекции, emoji_rate — доля ответов с эмодзи, ai_phrase_rate — доля ответов с шаблонами вроде «as an AI», mean_tokens — средняя длина, eos_rate — доля ответов, закончившихся EOS.

model	rm_score	emoji_rate	emoji_per_response	ai_phrase_rate	mean_tokens	eos_rate
SFT (ref)	-3.071	0.000	0.000	0.352	125.867	0.031
DPO · beta_0.1	-1.844	0.320	1.320	0.008	117.375	0.156
PPO · base	-2.822	0.000	0.000	0.320	128.000	0.000

Лог-вероятности ответов: обучающие и сторонние данные

Средняя по токенам лог-вероятность ответа (300 пар на набор): обучающая и отложенная части датасета, а также внешний tatsu-lab/alpaca.

model	train · chosen	train · rejected	held-out · chosen	held-out · rejected	alpaca (external)
SFT (ref)	-1.901	-1.662	-1.956	-1.688	-2.215
DPO · beta_0.1	-1.802	-2.733	-1.848	-2.783	-2.422
PPO · base	-1.907	-1.673	-1.962	-1.699	-2.255

Примеры генераций

Жадное декодирование, до 128 новых токенов.

Prompt: What's your morning routine like?

SmolLM-135M-Instruct:

I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:

Morning Routine (10-15 minutes)

Hydrate: Drink a full glass of water or a herbal tea to start the day.

Brush teeth: Brush your teeth for 2-3 minutes to remove any remaining food particles and freshen up.

**Get dressed

Эта модель (PPO):

I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:

Morning Routine (10-15 minutes)

Hydrate: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start the day.

Brush teeth: Brush your teeth for 2-3 minutes to remove any remaining food particles

Prompt: Do you have a favorite movie?

SmolLM-135M-Instruct:

What a great question! As a digital AI assistant, I don't have personal preferences, but I can provide you with some popular and influential movies that have had a significant impact on popular culture. Keep in mind that opinions on movies can vary, and opinions on movies can be subjective.

Here are some of the most influential movies of all time:

Star Wars (1977) - The first Star Wars film, directed by George Lucas, has become a cultural phenomenon, inspiring countless adaptations, franchises, and cosplay.

The Matrix (1999) - The

Эта модель (PPO):

What a great question! As a digital AI assistant, I don't have personal preferences, but I can provide you with some popular and influential movies that have had a significant impact on popular culture. Here are a few:

Star Wars (1977) - The first Star Wars film, directed by George Lucas, was a groundbreaking sci-fi epic that introduced the galaxy-spanning universe to a global audience.

The Matrix (1999) - The first film in the Matrix, directed by Joseph Heller, was a thought-provoking and visually stunning sci-fi thr

Prompt: I'm feeling stressed about my exams. Any advice?

SmolLM-135M-Instruct:

I totally understand! Exam stress can be overwhelming, but don't worry, I've got some tips to help you manage your exam anxiety:

Create a study plan: Break down your study material into manageable chunks, and allocate specific times for each topic. Set realistic goals and deadlines to stay on track.

Get enough sleep: Aim for 7-9 hours of sleep each night to help you feel rested and focused during the exam.

Eat a balanced diet: Fuel your body with nutritious food, including fruits, vegetables, whole grains, and lean proteins. Avoid sugary and processed foods that

Эта модель (PPO):

I totally understand! Exam stress can be overwhelming! Here are some tips to help you manage your exam-related stress:

Create a study plan: Break down your study material into manageable chunks, and allocate specific times for each topic or subject. Set realistic goals and deadlines to stay on track.

Get enough sleep: Aim for 7-9 hours of sleep each night to help you feel rested and focused during the exam.

Exercise regularly: Regular physical activity can help reduce stress and anxiety. Try to incorporate some form of exercise into your daily routine, such as a short wal…

Использование

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Timur-th/llm-course-hw2-ppo"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

messages = [{"role": "user", "content": "What's your morning routine like?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))