RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Task & Dataset
CoDraw (Kim et al., 2019 ) is a goal-driven instruction-following collaborative task between two players. The Teller player has access to an abstract target scene containing a variable number of cliparts (on average, target scenes contain 6 cliparts), leveraging the Abstract Scene dataset (Zitnick and Parikh, 2013 ; Zitnick et al., 2013 ) . The Teller provides written instructions in English to the Drawer in order to reconstruct the scene in a turn-based fashion. The Drawer has access to a gallery of 58 cliparts that they can place on a canvas. The Drawer can remove, resize (small, medium, large), or flip (facing right or left) each object in the scene. After receiving an instruction, the Drawer has to reply to the Teller before proceeding to the following turn. Typically, the Drawer either replies with confirmation feedback or asks follow-up questions. The CoDraw dataset consists of around 10k dialogues, with an average length of 7.7 turns per dialogue. Additional details can be found in Appendix D .

In CoDraw, task success is measured by comparing the target image with the one drawn by the Drawer at the end of the conversation. To this aim, Kim et al. ( 2019 ) designed a similarity score metric, ranging from 0 to 5, tailored to the CoDraw task. This score takes into account different cliparts (by checking whether cliparts in the target scene appear in the reconstructed one) and their attributes (size, position, flipping, plus some additional attributes for cliparts representing people, i.e., facial expression and body pose) by assigning different weights to each component.
The scenes drawn by human players reach a similarity score of 4.17 (sd=0.64). In our work, we apply some changes to the similarity score metric to fix some flaws in its design and better capture the properties under analysis in the experimental setup, as described in Appendix A . In the following, we report the results using the new version of the metric.
The Silent Drawer Model
Kim et al. ( 2019 ) proposed a neural Drawer model for the CoDraw task.
At each round of the dialogue, this model is conditioned on the Teller’s latest message, which is encoded into a vector using a bidirectional LSTM module. The Drawer also receives as input a vector representation of the current state of the canvas that is being drawn (the cliparts added so far and their attributes). These input representations are passed to a dense feed-forward neural network. The resulting vector represents the Drawer’s follow-up action, which determines the cliparts to be added to the canvas, together with their attributes. Thus, Kim et al. ’s Drawer model is silent , i.e.,
it can only draw objects on the canvas but cannot contribute to the dialogue.
The model is trained in a Supervised Learning fashion on human data. At inference time, the model assigns an unbounded score to each of the 58 available cliparts: each clipart with a score above 0 is drawn on the canvas, together with its predicted attributes (size, position, orientation, etc.). The model is tested by providing turn-by-turn Teller’s instructions from the CoDraw test set, reaching an average similarity score of 3.31 (sd=0.67).
Computing Model Uncertainty
We are interested in representing the uncertainty of the model regarding its next action. After receiving an instruction, the Drawer model has to make decisions about which clipart(s) to draw and their corresponding attributes. For each dialogue turn and selected cliparts, we estimate model uncertainty regarding these components as follows:

For clipart selection , we take the score assigned to each clipart
as a proxy of model uncertainty. The lower the value, the more uncertain
the model is about selecting the given clipart.

For the size ( small, medium, large ) and orientation ( right, left ) attributes, we compute the entropy of the probability distribution over the possible labels per attribute. The higher the entropy, the higher the uncertainty on that attribute.

For the position attribute, the Drawer outputs two values per clipart, corresponding to the x - and y -axis coordinates of the canvas. Measuring uncertainty from continuous values is inherently more challenging than discrete variables.
We take inspiration from uncertainty estimation using Ensembles of different models (Lakshminarayanan et al., 2017 ) and consider the variance among the prediction of five Drawer models trained with different random initialization seeds as a representation of model uncertainty.
Model vs. Human Uncertainty
In this section, we investigate the relation between model and human uncertainty. In the absence of clues such as eye gaze or reaction times (which could be indicative of uncertainty, but are not available in CoDraw), we take clarification questions as a signal of human uncertainty. We exploit the annotation of clarification questions released by Madureira and Schlangen ( 2023b , a ) , which consists of turns by the Drawer player annotated as clarification requests plus the attributes mentioned in such requests (size, position, orientation, etc.). 2 2 2 In particular, we take advantage of the annotation in CoDraw-iCR (v2) by Madureira and Schlangen ( 2023a ) . Around 40% of the dialogues in CoDraw contain at least one clarification question, with an average number of 2.2 clarification requests per dialogue in this subset.
Setup
We explore whether we can use model uncertainty (as defined in the previous section) to predict when human players ask clarification questions. To this aim,
we train a logistic regression model with human decisions about whether to ask a clarification question in the follow-up turn as dependent variable and each model uncertainty type (clipart selection, size, orientation, position) as independent variables . We use the Drawer model described in Section 4.1 to compute model uncertainty at each dialogue turn.
Using the methods described in Section 4.2 , we compute uncertainty for each clipart with a score above 0 and each of its attributes. If this includes more than one clipart at a given turn, we take the highest uncertainty value per variable (clipart selection, size, orientation, position). 3 3 3 Other approaches, such as taking the average of the uncertainty values, do not lead to any significant difference in the final results. We use the CoDraw test set (1002 dialogues, with an average of 7.7 turns per dialogue) to extract model uncertainty. We further split the CoDraw test set to train and evaluate the logistic regression model (70% and 30%, respectively).
We study the effect of different initialization seeds and training epochs and compare the model performance to a random baseline.
Results
The results are displayed in Figure 2 .
Let us consider the solid and dotted green lines in the plot. The logistic regression model trained to predict when human players ask clarification questions based on the uncertainty of the Drawer model yields an Average Precision of 0.188 (epoch 15; average across 5 different initialization seeds). This result is above the random baseline (0.188 vs. 0.142, p<0.05 ), indicating that model uncertainty has some degree of predictive power. Nevertheless, performance is poor. Among the independent variables analyzed, only the uncertainty about clipart selection has statistically significant predictive power ( p<0.001 ). Additional details about the coefficients can be found in Appendix B .

Figure 2 provides additional interesting insights: we can see that the Drawer model learns to perform the task over the training epochs, as evidenced by the upward trend of the similarity score, which measures task success. At the same time, the uncertainty of the model decreases (the plot shows uncertainty on clipart selection, size, and orientation. Same pattern is observed for position). Thus, the Drawer model is reasonably well calibrated: higher task performance goes hand in hand with lower uncertainty. Yet, this trend does not have an impact on the power of model uncertainty to predict human clarification behaviour, as the Average Precision of the logistic regression model remains constantly low over training epochs. We verified that other methods to capture model uncertainty, such as extracting epistemic and aleatoric uncertainty (Kendall and Gal, 2017 ) using an Ensemble of models, lead to the same results and do not have significant predictive power. Additional details can be found in Appendix B .
Analysis
There could be many reasons why model uncertainty and human uncertainty are poorly related. Human players do not only have immensely more knowledge and resources than the relatively simple Drawer model; different speakers may also follow different conversation strategies and such variation may be difficult to predict. It is well known that there is variability in human clarification-seeking and repair strategies (Schegloff et al., 1977 ; Schegloff, 1987 ; Purver et al., 2001 ) .
To check whether such variation is present
in the CoDraw dataset, we carry out an exploratory analysis. We use a keyword-matching approach to cluster together instructions by Teller players with similar form and content that lack some key information, and check how frequently they are followed by clarification questions. We only consider Teller’s instructions uttered in the first dialogue turn to exclude the possible confounding effect of the previous dialogue history.
Full details on the procedure and the results can be found in Appendix B . Here we highlight the main takeaways. For example, we observe that there are a total of 2274 Teller first instructions that mention at least one clipart but do not specify its size (e.g., ‘a tree to the left a few inches from the edge’ ). Only 23% of follow-up turns by the Drawer player include a clarification question, and among these, only 27% are about the size of the clipart. Similarly, there are 240 Teller first instructions that mention at least one clipart but do not specify its location (e.g., ‘little girl running towards her friend that has a pie in his hand’ ). 49% of follow-up turns include a clarification question, and among these, 48% are about the location of the clipart.

Thus we observe that when analyzing utterances that do not mention a specific necessary attribute, the human Drawer players may or may not ask about the missing information. Humans may be more or less cautious, take more or fewer risks, or guess/hope that more information will be provided later on in the dialogue (Purver, 2004 ; Benotti and Blackburn, 2017 , 2021 ) .
This high human disagreement on deciding whether to clarify or not suggests that learning when to ask clarification questions from human clarification behaviour may not be optimal. Indeed the model proposed by Madureira and Schlangen ( 2023b ) achieves only 0.35 Average Precision.
Instead, in the next section we investigate whether model uncertainty represents an effective guidance to decide when to ask clarification questions by studying their impact on task success.
Asking Clarification Questions
We implement a Questioning Drawer model ( Q Drawer) by adapting the Silent Drawer described in Section 4.1 . We focus on the size attribute, because it applies to all cliparts and it is often underspecified in the Teller’s instructions (see Section 5 ). Our goal is to test whether asking clarification questions triggered by size uncertainty results in (1) higher accuracy regarding the size attribute, and (2) higher overall similarity score. To isolate the effect of asking questions and avoid confounds related to language generation, we use a template-based approach to generate clarification questions and their answers.
Baselines
We compare Q Drawer to the Silent Drawer model, as well as to different configurations of Q Drawer that differ with respect to the information used to decide when to ask a question.
In particular, we compare our proposed uncertainty-based approach to the following alternatives:

The supervised learning approach by Madureira and Schlangen ( 2023b ) , where an external neural model trained on human clarification behaviour is used to decide when to ask ( Decider ). In this case, size uncertainty is still used to select which clipart(s) to target in the question, and the clarification question is generated and answered in the same way as in the uncertainty-based Q Drawer.

Asking questions at the same position (in terms of dialogue turn) where human players ask questions about size in CoDraw.

Asking questions at random dialogue turns (discussed in Section 6.5 ).
Task Success Results
Table 1 summarizes the results of our experiments on the CoDraw test set. We compare different models on their accuracy in correctly selecting the size of the cliparts drawn on the canvas and the overall similarity score between the drawn and the target scene at the end of the dialogue.
The Silent Drawer ( S D) reaches a size accuracy of 76.9% and a similarity score of 3.31. The version of Q Drawer that generates size questions in the same position as human players only marginally improves performance on size accuracy, while the similarity score does not show any statistically significant increase (paired t-test, p>0.05 ). Using an external neural Decider to choose when to ask clarification questions improves the performance over S D (up to 80.1% accuracy and 3.34 similarity score). Finally, the uncertainty-driven Q Drawer outperforms the baselines to a large extent: up to 87.3% size accuracy and 3.40 similarity score. 8 8 8 An exploratory study on extending our approach to other attributes reveals that using both size and orientation uncertainty further increases the similarity score on the test set up to 3.46, indicating that our proposed method can scale up. Regarding the comparison with the Decider model, it is worth noting that this model is trained to ask questions at each position where human players asked questions, not only when humans asked size questions. While this method is somewhat coarse-grained (because the Decider may ask size-CQ when they are not needed), it ensures full recall (the Decider will ask a size question whenever humans asked a size question). Hence, the Decider baseline is still meaningful with respect to the size attribute.

The Q Drawer model shows a consistent improvement as the entropy threshold is lowered. This aspect is investigated in more detail in Section 6.5 .
Clarification Questions’ Contribution
To further analyze how clarification exchanges contribute to task success, for each model configuration, we consider the subset of dialogues {CQ} where at least one clarification question is asked. We then remove all clarification questions from these dialogues, resulting in !{CQ} . We compute the difference in size accuracy and similarity score between {CQ} and !{CQ} to check the specific contribution of clarification questions (accuracy and SS boost). As we can see on Table 1 , the boost brought about by clarification questions is much higher for the uncertainty-driven Q Drawer than for the baseline models. In particular, for the subset of dialogues in {CQ} , clarification exchanges increase size accuracy by +11\% in absolute terms and the similarity score by +0.09 . Asking questions in the same position as human players does not significantly increase the similarity score when comparing {CQ} and !{CQ} (paired t-test, p>0.05 ). To make sure that this statically non-significant result does not stem from our generation approach, we replace machine-generated clarification questions with human ones for the dialogues in {CQ} . We find that in this case the increase in both size accuracy and similarity score is even less pronounced than what we observe in Table 1 , resulting in non-significant results for both question-generation methods.

We further investigate the impact of clarification exchanges by analyzing the calibration of the model uncertainty estimates. We compare the probability distribution over the size attribute against the ground-truth size of each object by using two metrics: the Expected Calibration Error (ECE, Naeini et al., 2015 ; Guo et al., 2017 ) and the Brier score (Brier, 1950 ) . For both metrics, a lower score indicates better calibration. 9 9 9 We refer to Ovadia et al. ( 2019 ) for a broader discussion on calibration metrics. We compare the Silent Drawer and Q Drawer ( \theta=0.3 ) and find that the Silent Drawer reaches an ECE value of 0.10 while the ECE for Q Drawer goes down to 0.05 (the difference is significant: t-test, p<0.001 ).
Similarly, the Brier score significantly improves (t-test, p<0.001 ) from 0.36 ( Silent Drawer) to 0.28 ( Q Drawer). These results indicate that asking clarification questions improves the calibration of the model uncertainty estimates.
How Many Questions to Ask
Asking clarification questions carries a cost, both from the perspective of the agent asking the question and the agent processing it (Clark, 1996 ; Purver, 2004 ) .
We take this partially into account by adopting a simple approach: we evaluate Q Drawer by using different entropy threshold values \theta to control for the number of questions generated by the model. As reported in Table 2 , in Q Drawer the value of the entropy threshold \theta has, by design, a direct impact on the number of questions asked: i.e., a model that only asks for clarification in the face of high uncertainty ( \theta=1.1 ) gives rise to fewer dialogues with questions and to a lower average number of questions per dialogue.

We compare the uncertainty-driven Q Drawer against (1) a version that asks questions in random dialogue turns with different average number of questions per dialogue and (2) the Q Drawer paired with the Decider module as described in Section 6.2 . For the latter, the maximum possible number of questions per dialogue (average of 1.5) is defined by the Decider model output. Figure 4 shows the effect of the number of questions on size accuracy for different model configurations. While the performance of the Decider model almost overlaps with the Q Drawer asking questions in random dialogue turns, using model uncertainty significantly improves accuracy, even when a few questions are asked. This result confirms that even when controlling for the number of questions asked (which may be desirable in terms of cost control), the uncertainty-driven Q Drawer shows the most promising results.
Conclusions
Different kinds of uncertainties characterize language use, and asking informative and effective questions is a crucial tool to address these uncertainties. While humans are able to ask strategic questions since early childhood to resolve a state of uncertainty, equipping modern dialogue systems with the ability to ask effective questions has proved challenging. Some existing approaches propose employing dedicated neural network models trained on human behaviour to decide when to ask for clarification in a dialogue. However, the high variability in human clarification strategies casts doubt on the effectiveness of this approach. Other proposed methods involve using model uncertainty to decide when and/or what to ask. Yet, estimating and exploiting model uncertainty is not trivial, and the relationship between model and human uncertainty is still unclear.

In our work, we considered the CoDraw collaborative dialogue task as a test bed. We proposed different ways of extracting model uncertainty and investigated its relationship to human uncertainty, as measured by the clarification decisions of human players. We found that human and model uncertainty are poorly correlated, possibly due to the wide diversity of human clarification strategies. We then presented an approach that makes use of model uncertainty to generate clarification questions, and compared it to other methods for deciding when clarification is needed. Our approach allows us to vary the level of uncertainty that elicits the generation of clarification questions to control for the number of questions generated. The results of our experiments show that our proposal outperforms a number of baselines and other methods trained on human clarification decisions. More generally, our findings pave the way for further research on integrating model uncertainty into natural language generation approaches as an effective way to resolve ambiguities and address underspecification in language use (Pezzelle, 2023 ) .
Limitations
Even though our experimental setup (which focuses on the size attribute using a template-based generation approach) allows us to isolate the effect of clarification exchanges in a controlled setting, we acknowledge that further work is required to confirm our findings in more complex collaborative tasks and model architectures. In our work, we focused on questions about the size attribute, and we mentioned some results about expanding our approach to the orientation attribute to assess the robustness of our approach. Assessing the contribution of clarification exchanges is particularly challenging given the wide range of factors involved, both on the appropriateness of the question asked and the correctness of the answer received. Without some restrictions, it is difficult to disentangle the role of these factors and this may threaten a fair model evaluation. For this reason, we focus on the size attribute, as it applies to all cliparts and it is often underspecified in the Teller’s instructions (see Section 5 ). An additional advantage of focusing on the size attribute is that we can ensure the correctness of the answer (see the templates in Section 6.1 ), while for other attributes such as position there is a less direct mapping between natural language expressions and space coordinates.

Regarding the generalization of our setup to other settings, we would like to highlight that, although simple, the CoDraw task shares many features with a wide variety of different tasks and datasets. In particular, in Vision-and-Language Navigation tasks an agent has to reach a goal by moving around an environment and performing specific tasks. We believe this setting, where the agent has a set of actions at their disposal, shares many similarities with the CoDraw setup. On the other hand, in more complex tasks the role of the visual input may play a greater role compared to CoDraw, leading to new challenges in evaluating the grounding of the generated question-answer exchanges. At the same time, we acknowledge the fact that a template-based approach may very well not scale up to other settings. However, in our work, this choice is motivated by the focus on the effect of clarifying rather than the question generation problem.

Finally, we only used a simple approach to control for the “cost” of asking clarification questions by limiting the overall number of possible questions. We acknowledge that a more fine-grained approach is required to consider the wide variety of different factors that influence the cost of generating clarification requests. Given the simplicity of the CoDraw task and the structure of the questions our method generates, we do not analyze how clarification questions may influence the perceived naturalness of the generated dialogues. We acknowledge that this aspect should be taken into account when dealing with more complex tasks and generative strategies.
Acknowledgments
We would like to thank Brielen Madureira for discussions about the CoDraw task and dataset, as well as for sharing insights on CoDraw-iCR. We are grateful to Patrick Kahardipraja for sharing an updated version of the original CoDraw codebase. Finally, we thank the Dialogue Modelling Group (DMG) at the University of Amsterdam for their feedback and support at the different stages of this work. This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 819455).