Combining Learning from Human Feedback and Knowledge
Engineering to Solve Hierarchical Tasks in Minecraftby Goecks, Vinicius G. Waytowich , Nicholas, Watkins, David, Prakash, Bharat

AAAI 2022 SPRING SYMPOSIUM

Sirine Younsi, 1st year Master’s Student, Kaneko Lab
Ref: https://arxiv.org/abs/2112.03482

SUMMARY
Reinforcement learning has had a lot of breakthroughs in game AI ( Dota / Go )Mainly achieving super-human performance

What about human-like performance?

Real-world Tasks

Instinctive
Have no reward signal➢ How to evaluate whether abehavior is human like or not?

➢How to solve a problem that has no task specification?

➢How to overcome the complexity of the real world? (Massive amount of data)

Open-world game
Human gameplay imitation +Solution engineering➢ Overcome RL limitations➢ Achieve faster and better results

2CONTENTSINTRODUCTION

14PROPOSED SOLUTION

PROBLEM

25

EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3CONCLUSION

63CONTENTSINTRODUCTION

1PROPOSED SOLUTION

4PROBLEM

25

EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3CONCLUSION

64INTRODUCTIONMINECRAFT
Environment➢ Open-world : procedurally generated 3D world➢ Instinctive task hierarchies➢ Sparse rewards

Interactions➢ Free navigation➢ Interaction with a variety of fauna and flora.
➢ Mining, collecting, and searching for resources.
➢ Designing and building complex objects.

Figure 1. Game of Minefcraft5

INTRODUCTIONMINERL
Large-scale dataset of human gameplay(MineRL-v0 Dataset)Set of Minecraft environments
MINERL Challenge (started 2019)

Overcome Deep RL
limitations

Sample inefficiency(e.g, AlphaGoZero
played 4.9 milliongames)

HOW?

Leverage imitation
Learning

Specifying Tasks /Sparse Reward

Figure 2. Minecraft Tasks addressed by MineRL

6CONTENTSINTRODUCTION

14PROPOSED SOLUTION

PROBLEM

2EVALUATION AND RESULTS

5BACKGROUND AND RELATED WORK

3CONCLUSION

67PROBLEMMINERL CHALLENGE 3rd EDITION: BASALT COMPETITION 2021
Mission:Develop agents with human-like behavior capable of solving
the following tasks

Time:
4 daysMake Waterfall

Find Cave

Dataset:40 to 80 human demonstrations for each task
Evaluation method:
Human feedback (No reward function)
Create Animal Pen

Build Village House
8CONTENTSINTRODUCTION14PROPOSED SOLUTION

LITERATURE REVIEW

25EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3CONCLUSION

69BACKGROUND AND RELATED WORKMINERL DATASET
60 Million State-Action pair * 4 Versions

2 Resolutions (64 × 64 / 192 × 256)

2 Textures (default Minecraft / simplified)

4 Task families:

State :Action•Navigation•RGB video frame of the player’s•

keyboard presses on the client.

•Tree Chopping

point-of-view•The change in view caused by•

Obtain Item•

player inventory•Survival•item collection events•

distances to objectives•player attributes (health, level,•

Chat messages sent,achievements)•

Other actions such as item•

current GUI details

mouse movement.
•All player GUI click andinteraction eventscrafting.
10

BACKGROUND AND RELATED WORKEND-TO-END MACHINE LEARNING

Algorithms that learn purely from data, with minimal bias or constraints added by human designers
Example:Deep reinforcement learning algorithms directly playing video games from pixel inputs

HUMAN-IN-THE-LOOP MACHINE LEARNING
Algorithms that learn from human feedback
Example:Agent trained based on human demonstrations of the task ( successful and failed examples)

11BACKGROUND AND RELATED WORK

LIMITATIONS➢ Limited human demonstration dataset (40-80 demonstrations is not enough)➢ Data with low quality➢ Large-observation space➢ Limited computing time

Not adequate to solve the BASALT competition tasks because of their complexity

12

CONTENTS
INTRODUCTION

1PROPOSED SOLUTION

4BACKGROUND AND RELATED WORK

25EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3CONCLUSION

613

PROPOSED SOLUTIONHYBRID INTELLIGENCEFigure 3. proposed solution: Hybrid Intelligence, Goeks et al. (2021)

14

PROPOSED SOLUTION

STATE CLASSIFICATION

15

PROPOSED SOLUTION

STATE CLASSIFICATION
Label data using human feedback – 2nd Step:labelDataGUI.py:

Humans were assigned to label image
frames from the collected human demonstrationdata.

GUI label demonstration

16

PROPOSED SOLUTION

STATE CLASSIFICATION

➢

none

➢ has_cave

Label data using human feedback – 3rd Step:➢ inside_cave :➢ danger_ahead➢ has_mountain➢ facing_wall
State ClassifierState label➢ at_the_top➢ good_waterfall_view➢ good_pen_viewRGB images(low resolution 64x64x3)➢ good_house_view➢ has_animals➢ has_open_space➢ animals_inside_pen
17PROPOSED SOLUTION

STATE CLASSIFICATION
Label data using human feedback – 1st Step:
dataProcessing.py:

Actions and images are extracted fromrecorded videos and saved as NumPy arrays

Figure 3. actions NumPy array extracted from one
demonstration video

Convert NumPy arrays to PNG images

Figure 4. sequence of PNG images extracted from video

18

PROPOSED SOLUTION

STATE CLASSIFICATION
Label data using human feedback – 4th Step:compileLabels.py:

Group images and labels in two NumPy
files

Figure 5. Amount of data for each state label/class19PROPOSED SOLUTION

STATE CLASSIFICATION
Label data using human feedback – 5th Step:

StateClassifier.py:
Used a convolutional classifier inspired
from Deep Tamer

Figure 7. Architecture of state encoder, warner etal. 2018

Figure 6. convolutional classifier
source: https://www.kaggle.com/code/ryanholbrook/the-convolutionalclassifier/tutorial

Figure 8. Train and Test dataset

20

PROPOSED SOLUTIONODOMETRY MAP

21

PROPOSED SOLUTIONODOMETRY MAP
Problem:Absence of basic localization
Solution: Estimated Odometry:
Attach a coordinate to each classified state
Initial Values and Assumptions• Player starts at position (0,0)• Agent’s Velocity:
Walking: 4 . 317 m/s.
Sprinting: 5 . 612 m/s
Sprinting + Jumping: 7 . 127 m/s• Heading angle 𝜃 :=>Follows point-mass kinematics:𝑥 ̇ = 𝑉 𝑐𝑜𝑠 ( 𝜃 )𝑦 ̇ = 𝑉 𝑠𝑖𝑛 ( 𝜃 )Figure 9. Odometry map generated in real time, Goeks et al. (2021)

22

PROPOSED SOLUTIONLEARNING AND ENGINEERING SUBTASKS

23

PROPOSED SOLUTIONLEARNING AND ENGINEERING SUBTASKS
Explore environment to find a place to build a pen

The 4 tasks to solve are too complex

If (find open space)

Need for Perception, Memory, Reasoning

Build a pen
If (animals' location on Map))••

Decompose each task to subtasks using
human knowledge of the task
Use hybrid Intelligence:
Some subtasks are learned fromhuman data (e.g., Navigation).

Go to animal location

If (no animals on Map)

Search for animals
If ( find animal)

Equip food to lure animal

If (animal baited)

Go back to pen and lure animal inside
If (animal inside pen)

Move away from pen and throw snowballSome subtasks are directly engineered ( e.g.,Throw a snowball at the end of an episode).

Human demonstration
Engineered Tasks
Odometry map

Figure 10. Subtasks of the Task CreateVillageAnimalPen

24

PROPOSED SOLUTIONLEARNING AND ENGINEERING SUBTASKSEngineered subtask

Equip agent withsnowball fromInventory
4 is the label of action“equip“ in the list of
defined actions

Figure 11. end_episode subtask ( in package Kairos_minerl)

8 and 22 are thepositions of thesnowball item in theagentrs inventory

Figure 12. list of labeled Actions

25

PROPOSED SOLUTIONLEARNING AND ENGINEERING SUBTASKS
learned subtask
behaviorCloner.py

Figure 13. Autoencoder Architecture

26

PROPOSED SOLUTION

STATE MACHINE

27

PROPOSED SOLUTION

STATE MACHINE
Example: Task MakeWaterfall
search for location in the mountain to make awaterfall

build additional blocks to add height to the waterfall

Place waterfall using bucket
Move forward from waterfall
Turn around and throw snowball
Figure 14. Sequence of frames of the hybrid Agent solving MakeWaterfall task28PROPOSED SOLUTION

STATE MACHINE
stateMachine.py

Figure 15. State machine for task build waterfall code

Figure 16. State machine plotted for task build waterfall

29

PROPOSED SOLUTION

STATE MACHINE

Figure 17. MakeWaterfall task demonstration

30

CONTENTS
INTRODUCTION

14PROPOSED SOLUTIONBACKGROUND AND RELATED WORK25

EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3CONCLUSION

631

EVALUATION AND RESULTS

EVALUATION METHOD
4 types of agents were compared head-to-head by 7 different human evaluators:
Hybrid: proposed solution
Find Cave ( 10 videos )Engineered: only navigation task was learned from human data

Make Waterfall ( 10 videos )and rest engineered)HybridBehavior cloning (BC): all tasks learned from human data
Human: human demonstration from the provided dataset
Metrics:
Best performer
Fastest performer
More human-like behavior

Create Animal Pen ( 10 videos )AgentEngineered

Build Village House ( 10 videos )160
videosBC

HumanFigure 18. Evaluation data structure for the human evaluators

32

EVALUATION AND RESULTS

EVALUATION INTERFACE

Possible outcomes:
Agent 1 wins
Agent 2 wins

None (Draw when none of the agents achievedthe task)

Figure 19. Evaluation Interface, Goeks et al. (2021)

33

EVALUATION AND RESULTS

TrueSkill TM SCORE

Each participant/Agent type is characterized by a Gaussian distribution that has:
Mean value 𝜇 (Average skill of a participant)Standard deviation 𝜇 (Degree of uncertainty in the participant’s skill)
TrueSkill Rate calculation: R= 𝜇 - 3*𝜇Figure 20. Evolution of TrueSkill TM score for each performance metric and for each agent type performing the FindCave task., Goeks et al. (2021)

34

EVALUATION AND RESULTS

AVERAGED PERFORMANCE

Figure 21. Averaged score of all tasks for each performance metric and agent type, Goeks et al. (2021)

35

EVALUATION AND RESULTS

TrueSkill TM SCORE

Figure 22. Score of each task for each performance metric and agent type, Goeks et al. (2021)

36

CONTENTS
INTRODUCTION

14PROPOSED SOLUTIONBACKGROUND AND RELATED WORK25

EVALUATION AND RESULTSBACKGROUND AND RELATED WORK

3
CONCLUSION

637

CONCLUSION

The proposed solution:➢Outperformed end-to-end machine Learning algorithm➢

Outperformed purely engineered solution➢

Used less data➢

Used less computing time

➢ Still not as good as a human player especially for the more human-like behavior metric38

THANK YOU FOR YOURATTENTION
Sirine Younsi

Q&A
1. SUMMARY……...………………………………………………………………....................................................... 2

2. INTRODUCTION………………………………………………………………....................................................... 5
3. PROBLEM……………………………………………………………....................................................................... 8
4. BACKGROUND AND RELATED WORK………....……………………………………………………………... 10
5. PROPOSED SOLUTION …………………………………………………………………………………………... 14
4.1 State classification………………………………………………………………………………..…………….… 15

4.2 Odometry Map……………………………………………………………………………..……………………... 21
4.3 Learning and engineering subtasks………………….…………………………………………………………..... 23
4.4 State Machine………………….…………………………………………………………………….…………..... 27
6. EVALUATION AND RESULTS……………………………………………………………………………………... 32
7. CONCLUSION………………………………………………..…………..……………………………………....…... 384040BIBLIOGRAPHY
Goecks , Vinicius G. & Waytowich, Nicholas & Watkins, David & Prakash, Bharat. (2021). Combining Learningfrom Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in MinecraftWarnell, Garrett & Waytowich, Nicholas & Lawhern, Vernon & Stone, Peter. (2018). Deep TAMER: Interactive AgentShaping in High-Dimensional State Spaces.

4141