The Uncertainty-based Q Drawer
We adapt the Silent Drawer as follows: (a) the model receives as input the immediately preceding turn exchange between the Drawer and Teller, instead of just the Teller’s utterance; (b) we add a term in the cross-entropy loss to include not only the cliparts that are added to the canvas but also the ones that are updated by changing some attributes; 4 4 4 In this way, the model learns to assign a high score to all cliparts mentioned in the instruction and not just to the ones that do not appear yet on the canvas—an essential step to successfully profit from clarification exchanges. See Appendix E for additional details. (c) we include a clarification generation module that works as follows:
At any turn of the conversation, if the entropy over the size attribute labels ( small , medium , or large ) exceeds a given threshold, 5 5 5 We experiment with several thresholds, as it will become clear in subsequent sections. Q Drawer generates a clarification question targeting a maximum number of 2 cliparts (i.e., the ones with the highest entropy).

We employ the following template, which corresponds to the most common formulation of size questions in CoDraw according to Madureira and Schlangen ( 2023a ) : “what size is the __?” or “what size are the __ and the __?” , where the slots are filled with the name(s) of the clipart(s).
We exploit the ground-truth annotation in the CoDraw dataset to provide an answer to the question on behalf of the Teller using these templates: “The __ is __” or “The __ is __ and the __ is __” , where the slots are filled with the name(s) of the clipart(s) under discussion and their size (small, medium, large). Before performing a new drawing action, Q Drawer is conditioned on the current state of the canvas, as well as the clarification question asked and the corresponding answer. Figure 3 shows an example of clarification questions asked by Q Drawer and the answers received, as well as the updated canvas after each round of the conversation. Another example is reported in Appendix C .

We experimented with replacing the Bi-LSTM language encoder of the Drawer agent (see Section 4.1 ) with more complex models (both fine-tuned on CoDraw and trained from scratch), such as BERT and RoBERTa (Devlin et al., 2019 ; Liu et al., 2019 ) and their corresponding distilled versions (Sanh et al., 2019 ) . We found that these BERT-based Q Drawer variants give rise to the same patterns on the role of clarification discussed in the results below, but lead to a deterioration of model performance as measured by the similarity score (max. 3.07). 6 6 6 This result is in line with a strand of work highlighting that LSTM-based models outperform BERT-like ones for small datasets and/or specific domains (Zeyer et al., 2019 ; Ezen-Can, 2020 ; Le et al., 2021 ; Wei et al., 2022 ) . We thus decided to keep the original model architecture. 7 7 7 Again, a logistic regression model trained with the uncertain extracted from the Drawer variants based on BERT and RoBERTa poorly correlates with human behaviour on asking clarification questions, reaching an average precision (0.18 AP) in line or slightly below the one discussed in Section 5 .