Illaitar/llm-course-hw2-reward-model

Саммари

Детали задания

Описание того как я делал задание

Привет! Меня зовут Никита 🤗, и в задании у меня были определенные сложности)))

Главная сложность была в том, что я купил подписку на колабе, когда падала модель по памяти даже с батч сайзом 1. Использовать стал A100, но это тоже не помогло. Помогло подождать аж 3 дня пока овтетит поддержка, и только тогда заработало. Баг был не на стороне моего кода, а на стороне колаба потому что они как я понял выделили мне гпу которая была забита другим заданием уже (0.0)

Еще очень хочу сказать спасибо компании apple, ноутбук которой решил сам по себе отрубиться хотя я ставил ему режим не засыпать

          .:'
      __ :'__
   .'`__`-'__``.
  :__________.-'
  :_________:
   :_________`-;
    `.__.-.__.'

А так, я хочу сказать что это буквально самый качественный спецкурс который я проходил за 6 лет на ВМК. Огромное спасибо авторам!

Примеры работы моделей

DPO

======== До ========
user
What's your morning routine like?
assistant
What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "wake-up routine" for you:

**Morning Routine (10-15 minutes)**

1. **10:00 am**: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or restorative yoga.
2. **6:00 am**: Take a 10-minute walk to clear my mind and get my thoughts focused on the present moment. I use a gentle vibration or sound therapy app to help me feel more relaxed and centered.

**Wake-Up Routine (20-25 minutes)**

1. **20:00 am**: Begin my morning routine as usual. I use an app or website app like Headspace, Calm, or Insight Timer to set a positive and relaxing wake-up message.
2. **10:00 am**: Read a calming article that helps me relax and reduce anxiety (e.g., "The Power of 7-Day Challenges"). I choose a motivational quote like "I

======== После тюнинга ========
user
What's your morning routine like?
assistant
A morning routine can help you start your day with a clear and focused agenda. Here are some common morning routines:

**Morning Routine**

1. **Breathing and Pranayama (5-10 minutes)**:
    * Start by focusing on your breath, feeling the sensation of the air moving in and out of your body.
    * Try a simple breath, such as "inhale, exhale, inhale, exhale" or "inhale, breathe deeply, exhale, breathe deeply."
    * Consciously stop and take a few deep breaths, allowing yourself to relax and calm.
2. **Mental Clarity and Focus (30-60 minutes)**:
    * Spend some time taking a few minutes to reflect on your thoughts, feelings, and experiences.
    * Practice mindfulness, meditation, or the 10-minute Pomodoro Technique to stay focused.
    * Set goals, prioritize tasks, and track your progress.

**Weekly Reminders**

1. **Gratitude and Self-Reflection (10-15 minutes)**:
    * Take a few minutes to reflect on your daily experiences, accomplishments, and values.
    * Write

PPO

======== До ========
user
What's your morning routine like?
assistant
I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:

**Morning Routine (10-15 minutes)**

1. **Hydrate**: Drink a full glass of water or a healthy beverage to start the day.
2. **Brush teeth**: Brush your teeth for 2 minutes, then rinse with water.
3. **Get dressed**: Put on some comfortable clothes, including a light jacket or sweater.
4. **Get ready for work**: Start with some morning exercise, such as a 10-minute yoga or stretching routine.
5. **Plan your day**: Review your schedule, set goals, and prioritize tasks.
6. **Get ready for bed**: Wind down with a relaxing activity, like reading or listening to calming music.

**Morning Routine (15-30 minutes)**

1. **Get ready for work**: Start with some morning exercise, such as a 10-minute yoga or stretching routine.
2. **Get ready for

======== После тюнинга ========
user
What's your morning routine like?
assistant
I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine I'd recommend:

**Morning Routine (10-15 minutes)**

1. **Hydrate**: Drink a full glass of water or a herbal tea to start the day.
2. **Brush teeth**: Brush your teeth for 2-3 minutes, then rinse with water.
3. **Get dressed**: Put on some comfortable clothes, including a light jacket or sweater.
4. **Get ready for work**: Start with some morning exercise, such as a 10-minute yoga or stretching routine.
5. **Plan your day**: Review your schedule, set goals, and prioritize tasks.
6. **Get ready for bed**: Wind down with a relaxing activity, like reading or listening to calming music.
7. **Get ready for the day**: Start with some morning self-care, such as meditation, journaling, or a relaxing activity.

**Morning Routine (15-30 minutes)**

1. **Get ready for the day**: Start