Task: Question Answering
Input (English): The comparative evaluator is trained with maximum likelihood estimation (MLE) objective, as described in eq DISPLAY_FORM6

where $\mathcal {X}$ is the set of pairwise training examples contructed as described above, $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair ($x_1$, $x_2$), $D_\phi ^q(x_1, x_2)$ is the probability of the comparative discriminator's prediction being $q$ ($q \in \lbrace >,<,\approx \rbrace $) for the pair ($x_1$, $x_2$). 
 Question: How they add human prefference annotation to fine-tuning process?
Output (English): human preference annotation is available $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair