Instructions to use Geodezik/llm-course-hw2-dpo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Geodezik/llm-course-hw2-dpo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Geodezik/llm-course-hw2-dpo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Geodezik/llm-course-hw2-dpo")
model = AutoModelForCausalLM.from_pretrained("Geodezik/llm-course-hw2-dpo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Geodezik/llm-course-hw2-dpo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Geodezik/llm-course-hw2-dpo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Geodezik/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Geodezik/llm-course-hw2-dpo

SGLang

How to use Geodezik/llm-course-hw2-dpo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Geodezik/llm-course-hw2-dpo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Geodezik/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Geodezik/llm-course-hw2-dpo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Geodezik/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Geodezik/llm-course-hw2-dpo with Docker Model Runner:
```
docker model run hf.co/Geodezik/llm-course-hw2-dpo
```

Direct Preference Optimization

Здесь представлена Direct Preference Optimization (DPO) модель, обученная в рамках курса по LLM от VK.

В качестве референсной модели бралась HuggingFaceTB/SmolLM-135M-Instruct, после чего модель файнтюнилась на датасете HumanLLMs/Human-Like-DPO-Dataset.

Детали

Описание модели

Reinforcement Learning from Human Feedback (RLHF) является достаточно сложной задачей. Это связано во многом с нестабильностью обучения и особыми требованиями к Reward модели, которую мы хотим использовать. В статье DPO предлагается простой лосс, сильно упрощающий классическое видение алгоритма.

В этом репозитории представлен пример DPO-модели для генерации ответов, более близких к ответам человека. Для файнтюна используется HuggingFaceTB/SmolLM-135M-Instruct, прошедшая стадию SFT, и понимающую формат чата пользователя-ассистента.

Ниже показано влияние нашего файнтюна на модель на примере ответа на пользовательский вопрос "What's your morning routine like?"

Пример: ответ модели до файнтюна

What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "Awake Morning Routine" for you:

Morning Routine (10-15 minutes)

10:00 AM: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
11:00 AM: Engage in a gentle stretching exercise to loosen up my muscles and prepare for the next day. This can include chest opens, shoulder rolls, or neck stretches. Hold each stretch for 20-30 seconds and then relaxation.
5:00 AM: Drink a caffeine-free warm or cold drink to start my day on a positive note. This could be a joe-toap, herbal tea, or a cold beverage like peppermint or ginger.
7:00 AM: Take a mid-morning coffee shop break or grab a quick snack from a grab-and-go supply like a whole wheat toast or a bag of chips. This will give my body

Пример: ответ модели после файнтюна

The morning routine is a crucial aspect of our daily life that sets the tone for our success and behavior. It involves setting aside time for tasks, managing our energy, and making the most of our time. A consistent and effective morning routine helps to regulate our emotions,motivation, and productivity, which in turn enables us to achieve our goals and success.

Here are some tips on how to establish a good morning routine:

Start your day with a positive mindset: Believe in yourself and your abilities. Try to focus on your current task and take action, rather than worrying about things that may not get done.
Plan your day to make the most of your time: Identify your tasks and break them down into smaller ones. Learn to prioritize your tasks and make sure that they are important.
Use a morning exercise routine: Exercise is essential for a healthy and happy body. Exercise helps to increase energy and speed, and it also improves your mood.
Eat a nutritious meal: Eat a meal that is rich in nutrients and energy. Avoid consuming large amounts of food, especially when you are first waking up.
Stay hydrated: Drink plenty of water before starting your day. Drinking enough water helps to reduce fatigue and

Референсы

Модель: HuggingFaceTB/SmolLM-135M-Instruct
Датасет: HumanLLMs/Human-Like-DPO-Dataset
Оригинальная статья: https://arxiv.org/abs/2305.18290

Downloads last month: 8

Safetensors

Model size

0.1B params

Tensor type

F32

Paper for Geodezik/llm-course-hw2-dpo

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Paper • 2305.18290 • Published May 29, 2023 • 73