RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Logistic Regression results and analysis
Table 3 shows the Average Precision of the logistic regression model trained to predict when human players ask clarification questions in CoDraw based on model uncertainty. We extract uncertainty from Drawer models trained for different training epochs and initialization seeds. We also report the performance of a random baseline and the CoDraw similarity score for each configuration. Similarly, Table 4 reports the F1 score of the same model. Table 5 shows the correlation coefficients of the logistic regression model for each attribute-specific uncertainty analyzed (clipart selection, size, orientation, position). We report the coefficients for each model configuration (initialization seed + training epoch). Finally, Table 6 illustrates some insights and examples of the “clusters” analyzed in Section 5 to investigate the variability of human confirmation-seeking strategies.
Epistemic and Aleatoric uncertainty
We also experimented with including Epistemic and Aleatoric uncertainty in the Logistic Regression analysis. We obtain the same results as using the other uncertainty measures described above, without any statistically significant predictive power. Similarly to Greco et al. ( 2022 ) , we use an Ensemble of models trained with different initialization seeds to compute data (aleatoric) and model (epistemic) uncertainty (Depeweg et al., 2018 ; Hüllermeier and Waegeman, 2021 ) for the size and orientation attributes. The ensemble total uncertainty is computed as the entropy of its predictive distribution [p_{\theta_{n}}(y|x)] , which averages the distributions of the single ensemble components:

H[p(y|x)]=H[1/N\sum_{n=1}^{N}p_{\theta_{n}}(y|x)]

Data uncertainty is measured as the average uncertainty of each ensemble component: 1/N\sum_{n=1}^{N}H[p_{\theta_{n}}(y|x)] . The difference between total uncertainty and data uncertainty results in model uncertainty . As mentioned above, including these uncertainties in the logistic regression model described in Section 5 does not improve its Average Precision, which remains significantly above chance level but low in absolute values (0.19 AP, similar to the values reported in Figure 2 ).
Figure 1: Overview of our experimental setup in CoDraw. After receiving an instruction from the Teller, the Drawer agent selects the clipart(s) to draw, together with their attributes. If the entropy over an attribute exceeds a threshold \theta ( size in the figure), then the Drawer asks a clarification question. The question-answer pair is added to the dialogue history before performing the next drawing action. Different human players may react differently (bottom-left box); the agent decides whether to ask for clarification on the basis of its own uncertainty, independently from the clarification decisions of human players.; Overview of our experimental setup in CoDraw. After receiving an instruction from the Teller, the Drawer agent selects the clipart(s) to draw, together with their attributes. If the entropy over an attribute exceeds a threshold \theta ( size in the figure), then the Drawer asks a clarification question. The question-answer pair is added to the dialogue history before performing the next drawing action. Different human players may react differently (bottom-left box); the agent decides whether to ask for clarification on the basis of its own uncertainty, independently from the clarification decisions of human players.
Figure 2: Average Precision ( AP , green) of a logistic regression model in predicting when human players asked clarification questions based on the uncertainty extracted from a Drawer trained for a variable number of epochs ( x -axis).
The plot also shows the similarity score ( SS , blue) achieved by the Drawer, and its uncertainty on clipart selection (normalized and reversed clipart scores), and on the size and orientation attributes (as measured by the entropy H ). For each epoch, we report the average and variance of 5 Drawer models trained with different seeds. Note that the y -axis has two scales: we report SS on the left, while all the other metrics refer to the right side.; Average Precision ( AP , green) of a logistic regression model in predicting when human players asked clarification questions based on the uncertainty extracted from a Drawer trained for a variable number of epochs ( x -axis).
The plot also shows the similarity score ( SS , blue) achieved by the Drawer, and its uncertainty on clipart selection (normalized and reversed clipart scores), and on the size and orientation attributes (as measured by the entropy H ). For each epoch, we report the average and variance of 5 Drawer models trained with different seeds. Note that the y -axis has two scales: we report SS on the left, while all the other metrics refer to the right side.
Figure 3: Example of clarification exchanges (in green) with questions generated by Q Drawer and answers by the Teller. The images on the right depict the Q Drawer’s canvas after each round of conversation.; Example of clarification exchanges (in green) with questions generated by Q Drawer and answers by the Teller. The images on the right depict the Q Drawer’s canvas after each round of conversation.
Figure 4: Effect of the average number of questions per dialogue (considering all dialogues in the test set) on the size accuracy. We compare the uncertainty-guided Q Drawer with a version that asks questions in random turns and one that asks questions in the turns selected by an external Decider model.; Effect of the average number of questions per dialogue (considering all dialogues in the test set) on the size accuracy. We compare the uncertainty-guided Q Drawer with a version that asks questions in random turns and one that asks questions in the turns selected by an external Decider model.
(a); (b); Examples of dialogues and clarification questions asked by different models. T stands for Teller, SD for Silent Drawer, and QD for Q Drawer. Clarification exchanges are highlighted in green.
Silent ( S D) and Questioning Drawer ( Q D) results using different approaches to generate clarification questions. Top part: baselines. CQs : clarification questions. Human : dialogue turns when human players ask CQs. SS : similarity score on the whole test set. Size acc. : accuracy (%) of the size attribute of all drawn cliparts on the whole test set. Boost columns: SS and Size Acc. boost brought by CQs on the subset of dialogues containing at least one CQ. All the differences are statistically significant except ∘ (non-significant difference with SD) and ∙ (adding CQs does not significantly increase SS on the subset).
The effect of different \theta values on the number of questions asked by Q Drawer. We report the percentage of dialogues with at least one clarification question and, within this subset, the average number of clarification questions per dialogue.
On the left, Average Precision of the logistic regression model trained to predict human decisions to ask clarification questions based on model uncertainty. In parenthesis, we report the performance of a random model, which is always significantly lower than its trained counterpart ( p<0.05 ). We study the effect of different initialization seeds and training epochs. On the right, we show the corresponding average similarity score reached by the Drawer model at the end of the dialogue in the CoDraw task.
F1 score of the logistic regression model (left) and the performance of a random baseline (right).
Correlation coefficients of the logistic regression model for each uncertainty type extracted from the Drawer model. We show the coefficients for each model configuration, as defined by its initialization seed and number of training epochs.
Cluster analysis by grouping together similar instructions. # utts. refers to the number of utterances in the cluster; Clip. , Size , and Loc. refer to the occurrences of clipart, size, and location keywords, respectively. % CQ refers to the percentage of clarification questions following the given set of instructions. “-” means no restrictions on that attribute.