SlowGuess/NAACL_2022_OCR huggingface dataset:
The results on the devtest (validation) and teststd (test) splits are shown in Table 2 and 3, respectively. On devtest set, our proposed model outperforms the baselines by a large margin. In the competition setting, we replaced visual feature extractor with object embeddings and scaled the model to BART-large. This model was ranked at the first place with  $75.8\%$  coreference F1 in MM-Coref and was declared the winner in the response retrieval subtask with  $71.2\%$  R@1,  $95.0\%$  R@5,  $98.2\%$  R@10, and 1.9 mean rank. Despite the simple approach we have taken for representing the multi-modal context, we were able to achieve competitive results with a single model.
For comparison, the winning entry for MM-Disamb and MM-DST, Entry #5, uses separate models, namely RoBERTa-large (Liu et al., 2019)
Models	#1 Disamb.	#2 MM-Coref	#3 MM-DST	#4-1 Res. Retrieval	#4-2 Res. Gen.
Accuracy (↑)	Obj. F1 (↑)	Slot F1 (↑)	Act. F1 (↑)	MRR (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)	M. Rank (↓)	BLEU-4 (↑)
GPT-2	73.8%	36.6%	81.7%	94.5%	8.8%	2.6%	10.7%	18.4%	38.0	0.192
MTN	-	-	74.8%	93.4%	-	-	-	-	-	0.217
BART-large(400M)	93.1%	73.5%	88.3%	96.3%	83.5%	73.7%	95.8%	98.7%	1.76	0.331
BART-base(140M)	92.5%	71.9%	82.0%	95.2%	76.7%	64.0%	93.7%	98.0%	2.12	0.294
-FT	92.2%	71.6%	80.6%	95.5%	76.1%	63.9%	92.6%	97.3%	2.24	0.284
-JT	91.5%	45.6 / 67.8%	79.5%	95.2%	73.2%	60.4%	90.5%	96.9%	2.58	0.283
-AC	92.1%	58.6%	82.7%	94.2%	75.0%	62.5%	91.1%	96.8%	2.23	0.289
-EC	92.4%	69.8%	83.3%	94.6%	75.8%	63.6%	93.4%	97.2%	2.16	0.290
-AX	91.9%	51.6%	81.0%	93.9%	74.9%	61.5%	88.4%	96.5%	2.34	0.279
Table 2: Results on the devtest set. The first block shows the baselines, which are separately trained on each subtask. The second block provides the complete results on BART-large and BART-base and the ablation studies on BART-base.  $FT$ : finetuning visual encoder beforehand,  $JT$ : subtask joint training,  $AC$ : attribute classification loss,  $EC$ : Empty-Coref loss,  $AX$ : all auxiliary subtasks (attribute classification and Empty-Coref). For MM-Coref without joint training, we report both the results of baseline-like generation (left) and our classification approach (right).
Entry ID	#1 Disamb.	#2 MM-Coref	#3 MM-DST	#4-1 Res. Retrieval	#4-2 Res. Gen.
Accuracy (↑)	Obj. F1 (↑)	Slot F1 (↑)	Act. F1 (↑)	MRR (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)	M. Rank (↓)	BLEU-4 (↑)
GPT-2	73.5%	44.1%	83.8%	94.1%	-	-	-	-	-	0.202
MTN	-	-	76.7%	92.8%	-	-	-	-	-	0.211
#1	-	52.1%	89.1%	96.3%	53.5%	42.8%	65.4%	74.9%	11.9	0.285
#2	89.5%	42.2%	87.8%	96.2%	61.2%†	49.6%†	74.7%†	84.5%†	6.6†	0.256
(Ours) #3	93.9%†	75.8%	90.3%†	95.9%†	81.5%	71.2%	95.0%	98.2%	1.9	0.295†
#4	93.8%†	56.4%	89.3%	96.4%	32.0%	19.9%	41.8%	61.2%	12.9	0.322
#5	94.7%	59.5%	91.5%	96.0%	-	-	-	-	-	-
#6	93.1%	68.2%	4.0%	41.4%	-	-	-	-	-	0.297†
#7	-	73.3%†	-	-	-	-	-	-	-	-
#8	93.6%†	68.2%	87.7%	95.8%	-	-	-	-	-	0.327
Table 3: The official leaderboard of DSTC10 on the teststd set. The subtask winners are bold-faced and runner-ups are marked with  $\dagger$ . “-” means that the entry did not participate in that subtask. Our entry uses 24-layer BART-large whose vision encoder is replaced with randomly initialized learnable embedding for identifying objects.
for MM-Disamb and BART for generating MM-Coref and MM-DST using the same prompt of the baselines without the use of visual features. Even though injecting continuous visual features (as extracted by vision models) may introduce noise for generation, they certainly help with MM-Coref subtasks as some entries achieving more than  $65\%$  object F1 utilize visual features (#6 and #7). Entry #8 enumerate visual attributes in the form of natural language tokens without relying on actual visual features.
Entry #6 (Lee and Han, 2021) is a multi-tower architecture with text encoder (RoBERTa-large) and image encoder (DeiT) (Touvron et al., 2021). To adopt the image encoder to the SIMMC 2.0 domain, it is contrastively pretrained by matching object image to its natural language attributes and scene (background) image to dialog context. Then, the objects and scene representations are added together to match against the dialog context for MM-Coref prediction. Entry #7 (Huang et al., 2021) encodes object information (index, location, and image) extracted by CLIP (Radford et al., 2021)
and BUTD (Milewski et al., 2020) then inputs the flattened object representations to UNITER (Chen et al., 2020) along with dialog context and scene image. MM-Coref predictions are made in terms of binary classification, similar to our approach.
All of the response retrieval entries modify the baseline approach, where the generated response (not the dialog context) is compared against the response candidate pool by different measures. Entry #1 uses cosine similarity for retrieval score instead of cross-entropy. Entry #2 uses negative likelihood, but generates from BART. Entry #4 follows OSCAR (Li et al., 2020) with self-supervised few-shot learning for predicting object tags, which act as an anchor between image (object) and text (dialog) modalities. The generated response with attached decoder is then compared in the same way as the baseline.
We ablate finetuning of vision encoder, joint training, and auxiliary objectives from BART-base. Because our model uses the frozen visual features, the
Figure 3: Attention maps between utterance and object IDs. The object attributes are given in Table 4. The rows indicate extracted utterance from  $[H_T; U_T]$  and the columns object IDs in  $S_{t \leq T}$ .
finetuning of the vision encoder before the actual training helped improve the model performance overall. We also observe that joint learning is a crucial part of the success of our approach, which presumably stems from the shared semantic information of the different subtasks. According to the  $JT$  row of Table 2, training each subtask separately degrades the performance even more than not finetuning the vision encoder. Refer to Table 5 for the effect of different subtask loss coefficients.
We also observe that removing the auxiliary subtasks affects the MM-Coref performance drastically. First, ablating attribute classification loss drops the MM-Coref performance by  $13.3\%$ . This affects the response retrieval performance as responses often include meta-information on the objects mentioned in the dialogs. Taking out Empty-Coref loss degrades the object F1 score, but slightly improves slot F1 as in ablating attribute classification. Removing all of auxiliary subtasks shows even clearer picture, where MM-Coref performance degrades by  $20\%$  among other subtasks.
6.2 Visualizing attention
Figure 3 visualizes the attention scores from the fifth head in last encoder layer between the dialog and the object (given in Table 4) modalities. We observe that the model generally refers to the corresponding object (e.g. the maroon dress) given the meta information (e.g. 283: plain maroon dress). Interestingly, the last example shows the corresponding object (115) for black and white jacket receives almost no attention score. In fact, the dialog refers to the black velvet blouse behind it. Nevertheless, a single attention head cannot capture all semantic similarities between different modalities.
fashion
object ID	color	type	pattern
169	light grey	jacket	plain
152	black, white	blouse	vertical
256	black	sweater	knit
168	maroon	dress	plain
258	brown	dress	plain
283	purple	dress	plain
277	grey	trousers	heavy stripes
115	grey, white	jacket	twin colors
167	blue	jacket	plain
005	black	blouse	velvet
069	black, white	blouse	spots
265	blue	jeans	denim
188	blue	trousers	plain
Table 4: Visual metadata of object IDs shown in Figure 3.
In this paper, we propose a multi-modal task-oriented dialog system based on BART that can perform all SIMMC 2.0 subtasks at once. Our model integrates the multi-modality by utilizing features from a vision model. In addition to the joint learning of all subtasks, we introduce auxiliary tasks. We observe that the joint-learning and other components are crucial in building a successful multi-modal assistant for SIMMC 2.0. Our model is able to perform competitively in all of the subtasks, setting a high bar for the new generation of multi-modal task-oriented dialog systems. Despite the success in SIMMC 2.0, our approach has a few limitations. First, it relies on metadata for non-visual attributes, which may not generalize well if a new set of domain items are introduced at inference. Our method also fails to fully capture the locality of objects within the scene (e.g. on the table, in the closet, etc.). We believe that these limitations will be addressed in future works.
This work was supported by National Research Foundation (NRF) of Korea (NRF-2019R1A2C1087634, NRF-2021M3I1A1097938), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, No.2020-0-00940, No.2021-0-02068), and Electronics and Telecommunications Research Institute(ETRI) grant funded by the Korean government (22ZS1100).
Jonathan Baxter. 1997. A bayesian/information theoretic model of learning to learn via multiple task sampling. Mach. Learn., 28(1):7-39.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137-1155.
Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27-29, 1993, pages 41-48. Morgan Kaufmann.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 583-592.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yichen Huang, Yuchen Wang, and Yuk-Cheung Tam. 2021. Uniter-based situated coreference resolution with rich multimodal input.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November; 2021, pages 4903-4912. Association for Computational Linguistics.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73.
Hung Le, Doyen Sahoo, Nancy F. Chen, and Steven C. H. Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5612-5623. Association for Computational Linguistics.
Joosung Lee and Kijong Han. 2021. Multimodal interactions using pretrained unimodal models for simmc 2.0.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer.
Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-seng Chua. 2018. Knowledge-aware multimodal dialogue systems. In Proceedings of the 26th ACM international conference on Multimedia, pages 801-809.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco:
Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Bing Liu and Ian R. Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In *Interspeech* 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016, pages 685-689. ISCA.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam.
Victor Siemen Janusz Milewski, Marie-Francine Moens, and Iacer Calixto. 2020. Are scene graphs good enough to improve image captioning? In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020, pages 504-515. Association for Computational Linguistics.
Seungwhan Moon, Satwik Kottur, Paul A. Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, and Alborz Geramifard. 2020. Situated and interactive multimodal conversations. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 1103-1121. International Committee on Computational Linguistics.
Nikola Mrksic, Diarmuid Řeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve J. Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1777-1788. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Seyed Hamid Rezatofighi, Anton Milan, Qinfeng Shi, Anthony R. Dick, and Ian D. Reid. 2018. Joint learning of set cardinality and state distribution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pages 3968-3975. AAAI Press.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve J. Young. 2017. Latent intention dialogue models. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3732-3741. PMLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021. UBAR: towards fully end-to-end task-oriented dialog system with GPT-2. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, pages 14230-14238. AAAI Press.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588.
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improving conversational recommender systems via knowledge graph based semantic fusion. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1006-1014.
Our model is built on top of BART from HuggingFace (Wolf et al., 2019).<sup>3</sup> We finetune the model for 10 epochs with an initial learning rate of 5e-5 and a batch size of 16 with AdamW optimizer (Loshchilov and Hutter, 2018). We also use linear warmup schedule with 8000 warmup steps and clip gradient norms at 1.0. For decoding, we use top- $p$  sampling (Holtzman et al., 2020) with  $p = 0.9$  to generate the user belief state and system response.
A.2 Joint Learning Coefficients
We train the model jointly on the sum of Equation 1 and Equation 2. We find the optimal combination of coefficients via grid search with the following choice of coefficient, while fixing  $\mathcal{L}_{\mathrm{LM}}$  to 1.0 and grouping MM-Disamb and auxiliary losses together to reduce the search space. Table 5 shows the results of grid search with the final choice of hyperparameters.
-  $\lambda_{\mathrm{mm - disamb}},\lambda_{\mathrm{att}},\lambda_{\mathrm{empty - coref}}\in \{0.1,0.3\}$
-  $\lambda_{\mathrm{mm - coref}}\in \{0.8,1.0\}$
-  $\lambda_{\mathrm{retrieval}} \in \{0.2, 0.4, 0.8\}$
In general, we see that increasing  $\lambda_{\mathrm{mm - disamb}},\lambda_{\mathrm{att}},\lambda_{\mathrm{empty - coref}}$  does not help the model in terms of performance. We also see some performance degradation in MM-Coref as  $\lambda_{\mathrm{retrieval}}$  increases; however, increasing  $\lambda_{\mathrm{mm - coref}}$  improves the overall performance of the model.
A.3 Task-Specific Heads
Object related classification heads (MM-Coref, attribute classification) have input dimension of twice the model dimension of BART (i.e. 2048 for bart-large). For MM-Disamb and Empty-Coref classification head, we use a single linear layer with softmax activation. For MM-Coref and attribute classification, we use an intermediate layer with the same hidden size as the input dimension, which is followed by a linear layer with softmax activation.
B Qualitative analysis
A successful multi-modal agent should be able to recommend objects that fit the user's requested
criteria within the scene context, understand the locations of the objects, and provide the requested information on the object such as ratings and price. We qualitatively analyze the generated system utterances to check whether our model can capture the object attributes along with spatial information.
B.1 Recommending objects from scene
Refer to Table 6 for examples. Upon inspecting generated samples, we observe that our model is often able to recommend appropriate objects that fall under the user's criteria. The first example takes place in a scene with jackets with the color attributes mentioned by the system-generated  $A_{T}$ , demonstrating the ability to capture object attributes. The second example demonstrates the case where the system correctly recommends and grounds jacket to the correct location.
However, it is not hard to find cases where the system is able to recommend the correct objects but in a wrong location. The third example demonstrates such case. All of the three recommended objects match those in the ground-truth response, but the system believes that they are all at a different location when in fact they are all on the left wall. We conjecture that our method of encoding object locations did not provide enough spatial information especially because we do not integrate the store structure itself. The retrieved  $A_{T}$  with the same dialog yields the correct response since all negative samples in the candidate pool did not contain all of the three objects mentioned in the ground truth.
B.2 Predicting coreference object and attributes
Refer to Table 7 for examples. We see that the model successfully identifies which objects and slots are being queried. In most cases, the model outputs the exact corresponding object information without having to lookup the object metadata directly. Furthermore, the model correctly identifies the turn for disambiguation. However, for more complicated instances such as the third example, the model mixes up the reference mentions and identifies the wrong value for the attribute. We also provide examples of all subtasks results (MM-Disamb, MM-Coref, MM-DST, response generation & retrieval) with the corresponding VR scene in Figure 4, 5, 6, 7, and 8
<sup>3</sup>https://github.com/huggingface/ transformers
#1 Disamb.	#2 MM-Coref	#3 MM-DST	#4-1 Res. Retrieval	#4-2 Res. Gen.
Accuracy (↑)	Obj. F1 (↑)	Slot F1 (↑)	Act. F1 (↑)	MRR (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)	M. Rank (↓)	BLEU-4 (↑)
(0.1, 0.8, 0.2)	91.8%	71.4%	81.5%	94.8%	75.7%	63.1%	92.4%	97.6%	2.25	0.292
(0.1, 0.8, 0.4)	91.2%	69.5%	80.0%	94.4%	77.9%	66.2%	93.2%	97.6%	2.15	0.288
(0.1, 0.8, 0.8)	92.4%	64.9%	76.2%	92.7%	75.4%	62.9%	92.5%	97.4%	2.29	0.271
(0.1, 1.0, 0.2)	92.5%	71.9%	82.0%	95.2%	76.7%	64.0%	93.7%	98.0%	2.12	0.294
(0.1, 1.0, 0.4)	92.3%	69.9%	83.2%	93.3%	76.9%	65.0%	92.9%	97.9%	2.14	0.286
(0.1, 1.0, 0.8)	91.8%	63.6%	78.7%	94.2%	74.3%	61.5%	91.4%	97.2%	2.40	0.278
(0.3, 0.8, 0.2)	92.4%	69.6%	77.9%	95.7%	74.5%	61.2%	92.0%	97.5%	2.29	0.290
(0.3, 0.8, 0.4)	92.6%	67.0%	75.4%	95.2%	74.7%	62.2%	91.2%	97.2%	2.33	0.290
(0.3, 0.8, 0.8)	91.5%	61.7%	77.1%	93.9%	74.3%	61.4%	91.1%	97.0%	2.38	0.280
(0.3, 1.0, 0.2)	91.7%	70.6%	83.2%	95.7%	73.6%	59.9%	91.9%	98.0%	2.29	0.294
(0.3, 1.0, 0.4)	92.6%	67.3%	75.1%	95.4%	72.7%	59.2%	90.9%	97.0%	2.48	0.282
(0.3, 1.0, 0.8)	92.3%	63.9%	75.3%	94.6%	73.8%	60.8%	91.5%	97.6%	2.36	0.277