Question: How they add human prefference annotation to fine-tuning process?
Answer: human preference annotation is available, $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair