M2C: Towards Automatic Multimodal Manga Complement
Authors: Hongcheng Guo
Boyang Wang
Jiaqi Bai
Jiaheng Liu
Jian Yang
Zhoujun Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Manga data
Vision and Language
Chain of Thought Prompting

Abstract: Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features, which has attracted considerable attention from both natural language processing and computer vision communities. Currently, most comics are hand-drawn and prone to problems such as missing pages, text contamination, and text aging, resulting in missing comic text content and seriously hindering human comprehension. In other words, the Multimodal Manga Complement (\textbf{M2C}) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for vision and language understanding. To this end,
we first propose the Multimodal Manga Complement task by establishing a new M2C benchmark dataset covering two languages. First, we design a manga argumentation method called MCoT to mine event knowledge in comics with large language models. Then, an effective baseline FVP-M$^{2}$ using fine-grained visual prompts is proposed to support manga complement. Extensive experimental results  show the effectiveness of FVP-M$^{2}$ method for Multimodal Mange Complement.

M2C: Towards Automatic Multimodal Manga Complement
Hongcheng Guo Boyang Wang*1, Jiaqi Bai', Jiaheng Liu',
Jian Yang', Zhoujun Li BBehhang University
(hongchengguo, wangboyang, (hongchenggu.,waggboyanggg bjq, wiwpiaaangggiiiie.. Jibaaauaaaab edu. cn
Abstract
Vision
Multimodal manga analysis focuses on enhanc- ing manga understanding with visual and tex- tual features, which has attracted considerable attention from both natural language processing and computer vision communities. Currently, most comics are hand-drawn and prone to prob- lems such as missing pages, text contamina- tion, and aging, resulting in missing comic text content and seriously hindering human com- prehension. In other words, the Multimodal Manga Complement (M2C) task has not been investigated, which aims to handle the afore- mentioned issues by providing shared seman- tic space for vision and language understand- ing. To this end, we first propose the Multi- modal Manga Complement task by establish- ing a new M2C benchmark dataset covering two languages. First, we design a manga argu- mentation method called MCoT to mine event knowledge in comics with large language mod- els. Then, an effective baseline FVP-M² using fine-grained visual prompts is proposed to sup- port manga complement. Extensive experimen- tal results show the effectivvness of F method for Multimodal Mange Complement.
Text
Figure Motivation of our M2C task. The left part de- scribes both visual and language modalities as the input (vision part depicts a sequence images from page of comics, and the language part is the corresponding text dialogues). The right part displays page of comics and text that we want to complete.
Translation (Hinami et al., 2021) and Comic Image Inpainting (Ono et al., 2021)), M2C aims to ex- ploit the effectiveness visual features for manga complement. Furthermore, M2C can serve as a foundation for other comic-related tasks.
However, as we can see in Fig. 1, compared image caption (Vinyals et al., 2015) or other mul- timodal tasks, the comic data we use has its own characteristics. Here we conclude the differences and corresponding challenges: 1). Comics have shorter conversational text lengths than other natu- ral language forms, and contain many inflections or auxiliaries. This leads to a greater randomness in content generation that requires complement. II). The text and images of comics are complementary. often in pairs. This requires us to design better ways to exploit image features. III). Comic images and text contain timelines and implicit logic infor- mation, which shows us a challenge to leverage the logical knowledge to help improve the stability of generating complementary content. Therefore, in our work, first propose the Multimodal Manga Complement (M2C) task achieve the comple- ment for with both vision and language informa- tion.
1 Introduction
Comies are enjoyed globally, often featuring a com- bination of text and illustrations. However, due to their hand-drawn nature, they are susceptible to damage during circulation, such as missing pages, text contamination, and aging. To address these is- sues, we propose task called Multimodal Manga Complement (M2C), illustrated in Fig. 1. This task extends conventional text-based content com- plement by incorporating corresponding images as additional inputs to mitigate data sparsity and ambiguity (Ive et al., 2019a). Similar to other multimodal manga tasks (e.g., Multimodal Manga
First two authors contributed equally. ns Corresponding author. https://gihubb.co/HHCCCG HuGG/

---

To eliminate the above limitations, inspired by re- cent CoTs (Wei al., 2022; Wang al., 2022), we first design the three-hop Manga Chain-of-Thoughe (MCoT) argumentation method to exploit timelines and logic knowledge in comics with large language models (Ouyang et al., 2022). It contains three- hop inferring stage (Theme, opinion, and future). Then we propose simple and effective FVP-M² method, including Feature encoding, Fine-grained visual prompt generation, and Dialogue comple- ment. Specifically, in the token encoding stage, we use the pre-trained vision encoder to extract the visual tokens. Then, we follow (Johnson et al., 2017) to utilize the Transformer to encode the tex- tual tokens. In FVPG, inspired by (Ying et al., 2021) and (Lin et al., 2020), we adopt a new way to aggregate global and local visual information. Specifically, the local visual features are aggregated in self-attention way. while a mapping network Fig. is leveraged to generate the global visual feature, which will serve as a bias term added into the calculation of local visual features. Further, the outputs are the ane-grained visual prompts. After that, during the dialogue complement, following the works (e.g., ViLBERT (Lu et al., 2019)), we uti- lize co-Transformer to generate the vision-guided language tokens. Then the Transformer decoder is adopted to predict the results.
Ea:
Fr:
goide...
TDORGE
mally
SEREE POUVOR
THE REMECRISTTL
'Laciamu
Figure 2: Example of an image with corresponding dia- logues. En represents English and Fr represents French.
2023; Guo et al., 2022; Liu et al., 2022b,a) and large-scale training data (Liu et al., 2022c; Guo et al., 2023a,c; Bai et al., 2023a; Wang et al., 2023a; Guo al., 2023b; Wang et al., 2023b; Yu et al., 2023). Transformer-based multimodal models in- corporate both textual and visual information in a shared representation, which first preprocesses im- ages using object detection models to extract infor- mation about specific regions or objects within the images. These region features are then projected into common embedding space, which allows for scamless integration with the textual input. The multimodal Transformer architecture is then used to jointly process both the text and image inputs, using multi-head attention mechanisms to capture relationships and dependencies between the two modalities. This process results in the learning of a unified representation that captures the relation- ships between the textual and visual aspects of the data.
The contributions are summarized as follows:
We first propose the Multimodal Manga Com- plement (M2C) task handle the generation for the missing textual contents, which inves- tigates the effect of the combination of vision and language modality.
For M2C, we propose an effective Fine- grained visual prompt generation strategy for better aggregation of image information, and design a manga Chain-of-Ti of-Thhtght method to exploit the logical information and the event knowledge implied by the comics.
3 Datasets
We introduce the multimodal manga complement (M2C) benchmark dataset. Here, we described the details the M2C dataset.
We establish a benchmark dataset on multi- modal manga complement, and extensive ex- periments show the effect of our method.
Data Construction. To ensure diversity in our comics, we have gathered open-source comic works in various styles and themes from across the Internet. Our dataset comprises 10,221 comic stories, each featuring several comic images and dialogues. To create the M2C corpus, first em- ployed the OCR model to extract sentences and then translated the English text into French us- ing the multilingual translation model (Yang et al.,
2 Related Works
Vision-Language Models. The success of vision- language models can be attributed the Trans- former architecture (Vaswani al., 2017; Radford et al., 2021; Li et al., 2020; Sharma et al., 2018; Miech et al., 2019; Bai et al., 2023b; Zhu et al.,

---

Three-hop Manga argument with CoT Prompting
Target
Theme
Opinion Future
Theme answer(A]
Opinion answer(()
Reasoning question in prompt
test Poison ryå no *... This is bad !!!", what theme of manga may this diallgue appear in!
looks really for you...
Il-Ieess eating the poison guess?
1e-hop: Interiin theme
4dke The text deseribes a scene that could related to variety of manga themes such as action adven mere. or fanta
Appending the next context
Given the text "Ohh Poison mist !! Oh? That looks really bad for you... Hi-liess earing the poćson ... guess? A breath attack ?!! Daku- ryů no *... This is bad !!!". the text deseribes scene that sould be related to variety of manga themes such as action, adventure, or fimiasy, c what the implicit opinion lowards the dialoque, and why?
24-hop: Inferring opinion
Based the commmn
Rased commonn sense, the implicit opinion towards the dialogue the given test could be one of concer worry
Appending to the next contest
Th-That looks really for you... H--le'ss eating possun ... quess? A attack ryä no *.. This is bad !!",, the text deseribess scene that could be related to gariety of manga themes such as action, adventure, or fantasy, sense,
text
Based
opinion what is the future towards the diallogue?
dialegues given test could be one
worrs Based such opinion,
ge-hop: Inferning future
Based on the opinion that the sitntion is seriius and requires immediatee action to prevent harm, the future towands the dialegues in the given 4 could involve taking action mitigate the danger unfavorable situation. This could include uttempting stop the poison mist attack, helping the person who eating the poison, fiading way to protect themselves from breath attack.
Figure 3: Manga Chain-of-Thought Augmentation.
2021a; Ma et al., 2020; Yang al., 2021b). We then randomly selected one sentence from each story as the completion target while retaining the other dialogues as input, resulting in 45,995 pairs for our M2C benchmark. An example is provided in Fig. 2. The number of image-text pairs for train- ing, validation, and testing data are 41995, 2000, 2000, respectively.
to extract the visual tokens for z as follows:
{um Ji U/ = H(*x),
(1)
where H denotes the vision encoder and M is the number of local cropped images in the whole pic- ture. ve denotes the global visual feature encoded with the whole picture, while im denotes the local feature encoded with a cropped image.
Data Augmentation. To exploit timelines and logic knowledge in comics and improve the gener- ation performance, we design a three-hop manga Chain-of- Thought to facilitate step-by-step reason- ing (inferring theme, opinion and future) as illus- trated in Fig. 3.
Similarly, given the source language z. based on the Transformer encoder E, the source language tokens { are extracted as follows:
{ Exx)),
(2)
where F is the number of source language tokens.
4.2.2 Fine-grained Visual Prompt Generation In this stage, we design the fine-grained visual prompt generation method with the global visual feature v. and local feature Vm Specifically, in Fig. 4, the JM M=1 is first aggregated by self- attention (Vaswani et al., 2017). Then we utilize a mapping network M implemented by two fully- connected layers with ReLU (Nair and Hinton, 2010) activation function to encode the global vi- sion feature as follows:
4 Method
4.1 Multimodal Manga Complement
Supposing we have K parallel sentences [( k/ between source corpora and target corpora t, where K is the number of training instances and each instance has the corresponding image zk.
4.2
2
As shown in Fig. 4, our FVP-M2 includes three stages:
Dr. = M(T.).
(3)
feature encoding, fine-grained visual prompt generation and dialogue complement.
After that, we add do. as a bias term and generate the fine-grained visual prompt pe:
4.2.1 Feature Encoding
QK"
For each image 2g. we directly use the vision back- bone (c.g., vision branch of the widely-used CLIP model (Radford et al., 2021)) as the vision encoder
pv SF
V.
(4)
Vath
where SF represents the softmax operation.

---

Imsage
Feature Encading
Model
En
Fr
Avgali
MatiNal
Text-only Systems
Visien Eneeder
Seals
Why you
Transformer (Vaswani 2017)
19.6
25.5
Mapping
ChatGiT (Ouyang 2022)
6.4
10.1
8.25
Soltuna
ChatGPT (w/ MCoT) (Ouyang et al., 2022)
15.2
18.8
17.2
PVPG
Machtal
Multimodal Systems
Sereeiel 10615
Transfermer
Eacoder
TRM
Co-TRM
TRM
Dicader
Deliberation Network (Ivv et al., 201
DOCN (Li et al.. 2021) Vision Matrees at., 2021)
21.4
23.1
28.5
26.7
24.1
[Nllegno cmppement
22.5
27.4
24.9
Figure 4: The overall framework of our proposed FVP- M method for Multimodal Mange Complement task, which includes three stages (i.e., feature encoding, fine- grained visual prompt generation (FVPG), and dialogue complement). "TRM" and "Co-TRM" represent the Transformer and co- Transformer models, respectively,
On Vision Features (Li 2022)
24.0
29.3
FVP-MF (w/o FVPG MCoT)
22.9
28.4
25.7
FVP-M (w/o MCoT) FVP-M (Our method)
27.2
30.9
29.0
33.7
Table 1: BLBU scores on M2C benchmark test set. Five baselines are compared by us. The bottom part shows the results the models trained with text and vision modalities. The best results highlighted.
4.2.3 Dialogue Complement
In Fig. 4, we utilize the Transformer module to fuse the information from other tokens within each modality for {sr]f= and Pv. respectively, and we represent the updated source language tokens and visual prompt as S and P, respectively. Then, we take as the query, and the P as the key and value in the co-atte at-aettenion module (Lu et al., 2019) to generate the vision-guided source language tokens // as follows:
1024. Dropout and label-smoothing rate are set as 0.3 and 0.1, respectively. We adopt the vision branch of CLIP based the ViT-L/14 model. The effect of different vision backbones shown in the appendix. All models are trained for 30 epochs and evaluated on one single linux machine with 4 NVIDIA A 100 GPUs (80G).
Data Quality Control. We recruit 20 college stu- dents with a master's degree in English. To ensure the data quality, we conduct two rounds of data review. In the first review, these students carefully review all annotations to check whether each target sequence is consistent with the source sequence and the image. If not, they asked to revise and provide the reason for revision with confidence score. In the second round, samples without modi- fication are considered correct. For these modified samples, the students are asked to discuss to de- termine the final version. For those controversial results, voting is adopted following the principle of majority rule.
H || SF h=1
(b(S) (P) VC
{arlf=1
Φ
(P))),
(5)
where M is the concatenation of the H atten- tive features along the channel dimension. SF rep-
resents the softmax operation. (-), (-) and (.). are the corresponding linear projection oper- ations of the h-th head for the query, the key and the value, respectively. C denotes feature dimension. At inference, based on f= we use Trans- former decoder to predict target sequence.
5 Experiments
Evaluation. We compute the cumulative 4-gram BLEU scores at inference. The beam search strat- egy is based on beam size of 5 for the target sentence generation. The length penalty is 1.0.
5.1 Experimental Setting
Implementation Details. Our implementation is based on the Fairseq (Ott et al., 2019) tool- box. The model in Fig. 4 consists of 6 Trans- former encoder/decoder layers. The number of attention heads is set 12. We take the Adam optimizer (Kingma and Ba, 2015) with and 32- 0.98. The learning rate warms up from le-7 le-4 in 2000 steps and then decays based on the inverse square root the update number. The maximum number of tokens in each mini-batch is
Baseline Methods. As we are the first task in this area, we reproduce methods including Text Trans- former (Vaswani et al., 2017), Deliberation Net- work (Ive al., 2019b), DCCN (Lin et al., 2020), Vision matters (Li et al., 2021), and On Vision Fea- tures (Li et al., 2022). Besides, we also report the results of ChatGPT (Ouyang et al., 2022) and the variants of FVP-M2
0.9

---

7 Limitations
GOO
Although our proposed FVP-M2 method has achieved substantial improvements for Multimodal Manga Complemnt, we find that there still exists some parametrsss (e.g., the number of en- coder and decoder layers.) to tune for better results, which may be time-consuming. Besides, in tablished datasets, we only support two languages currently, and we will extend support more lan- guages for Multilingual Multimodal Manga Com- plement in future work.
FUNEP THIN HUTH SALTO TOTERAL SMON PATEEE
Figure 5: A qualitative example of two languages on English and French with the help of vision modality. To- kens in blue denote correct complement. SRC denotes the source corpus. PRE and TGT represent predicted results and ground-truth of target sentence, respectively.
5.2 Main Results
Acknowledgments
To demonstrate the effectiveness of FVP-M we compare our method with baselines in Table 1. FVP-M2 achieves the best BLEU scores in two languages. Specifically, FVP-M² outperforms by +9.2 BLEU scores on average compared with text- only Transformer, which demonstrates the effec- tiveness of vision for manga complement. Second, when faced with manga data containing little log- ical clues, ChatGPT performs poorly. Third, the effectiveness of our proposed FVPG and the MCoT method is verified through comparison.
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62276017, U1636211, I, 61672081), and the Fund of the State Key Laboratory of Software Development Environment (Grant No. SKLSSEE-022ZZX-18)
References
Jiaqi Bai, Hongcheng Guo, Jiaheng Liu, Jian Yang. Xinnian Liang, Zhao Yan, and Zhoujun Li. 2023a. Griprank: Bridging the gap between retrieval and generation via the generative knowledge improved passage ranking. CIKM.
5.3 Visulization
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang. Wei Wang. Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang. Xingx- uan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023b. Qwen technical report. arXiv preprint arXiv:2309.1660 16609.
To further explore the necessity of visual modality for multimodal manga complement, we visualize the prediction results of a sample in Fig. 5. Specif- ically, the "note- book" (En) and "JOUR- NAL" (Fr) are masked in the target sentence, and these masked tokens describe the saliency regions in the corresponding left image. We have the following observations. We observe that even though the "note- book" is masked, the prediction results on this token are still right, which means that visual modality is complementary rather than redundant if the text is insufficient.
Hongcheng Guo, Yuhui Guo, Jian Yang. Jiaheng Liu, Zhoujun Li, Tieqiao Zheng, Liangfan Zheng, We- ichao Hou, and Bo Zhang. 2023a. Loglg: Weakly supervised log anomaly detection via log-event graph construction. DASFAA 2023, volume 13946, pages 490-501. S.riner..
6 Conclusion
We first propose the Multimodal Manga Comple- ment (M2C) task to handle issues such as miss- ing pages, text contamination, and text aging in hand-drawn comics by providing shared seman- tic space. We establish new M2C benchmark dataset covering two languages. Then, we propose an effective 2VP-M2 method for the M2C task. Experimental results on M2C benchmark datasets show the effect of FVP-M method.
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, and Zheng Cui. 2022. Lvp-m3: Language-aware visual prompt for multilingual multimodal machine translation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2862- 2872.
Hongcheng Guo, Jian Yang. Jiaheng Liu, Liqun Yang. Linzheng Chai, Jiaqi Bai, Junran Peng. Xiaorong

---

Shuming Ma, Jian Yang. Haoyang Huang. Zewen Chi, Hu, Chao Chen, Dongfeng Zhang, Xu Shi, Tieqiao Zheng, Liangfan Zheng, Bo Zhang, Ke Xu, and Zhou- jun Li. 2023b. Owl: A large language model for it operations. ArXiv, abs/2309.0929)
Jiaheng Liu, Tan Yu, Hanyu Peng, Mingming Sun, and Ping Li. 2022c. Cross- Cross-lingual cross- modal consolida- tion for effective multilingual video corpus moment retrieval. In Findings of the Association for Compu- tational Linguistics: NAACL 2022, pages 1854-1862, Seattle, United States. Association for Computational Linguisties.
Jinyang Guo, Jiaheng Liu, Zining Wang, Yuqing Ma. Ruihao Gong, Ke Xu, and Xianglong Liu. 2023c. Adaptive contrastive knowledge distillation for BERT compression. In Findings of the Association for Com- putational Linguistics: ACI 2023, pages 8941-8953, Toronto, Canada. Association for Computational Lin- guistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and lnn-lang tasks. Ad- vances in neural information processing systems, 32.
Ryota Hinami. Shonosuke Ishiwatari. Kazuhiko Yasuda, and Yusuke Matsui. 2021. Towards fully automated manga translation. In AAAI 2021, pages 12998- 13008.
Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi. Saksham Singhal. Xia Song, Arul Menezes, and Furu Wei. 2020. XLM- T: scaling up multilingual machine translation with pretrained cross- longull transformer encoders. CORR. his/2012.15547
Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019a. Distilling translations with visual awareness. In ACL 2019, pages 6525-6538.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. 2019. flowto100m: Learning a text-video embed- ding by watching hundred million narrated video clips. ICCV 2019, pages 2630-2640.
Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019b. Distilling translations with visual awareness. In ACL 2019, pages 6525-6538.
Melvin Johnson, Mike Schuster, Quoe V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's s multilingual neural machine translation system: Enabling zero-shot translation. ACI. 2017, 5:339-351.
Vinod Nair and Geoffrey Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML 2010.
Naoki Ono, Kiyoharu Aizawa, and Yusuke Matsui. 2021. Comic image inpainting via distance transform. In SIGGRAPH 2021, pages 9:1-9:4. ACM.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR 2015.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT 2019, pages 48-53.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022. On vision features in multimodal machine translation. In ACL 2022, pages 6337-6337.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. AAAI 2020, volume 34, pages 1336-11344,
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishknn, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Jiaoda Li, Duygu Ataman, and Rico Sennrich. 2021. Vision matters when it should: Sanity checking multi- modal machine translation models. EMNLP 2021, pages 8556-8562.
Huan Lin, Fandong Meng. Jinsong Su, Yongjigg Yin, Zhengyuan Yang. Yubin Ge, Jie Zhou, and Jiebo Luo. 2020. Dynamic context-guided guided capsule network for multimodal machine translation. In ACM MM 2020, pages 1320-1329.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML. 2021, pages 8748 8763.
Jiaheng Liu, Jinyang Guo, and Dong Xu. 2022a. An end-to-end frame- work for 3d action recognition. IEEE Transactions on Multimedia, pages 1-13.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018 Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In ACI. 2018, pages 2556-2565.
Jiaheng Liu, Tong He, Honghui Yang, Rui Su, Jiayi Tian, Junran Wu, Hongcheng Guo, Ke Xu, and Wanli Ouyang. 2022b. 3d-queryis: A query-based frame- work for instance segmentation. arXiv preprint arXiv:2211.09375.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In NIP 2017

---

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2013. Show and tell: A neural image caption generator. In CVPR 2015, pages 3156-31 64.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. CoRR.
Zekun Moore Wang. Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu. Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang. Zhaoxiang Zhang. Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Jumran Peng. 2023a. Rolellm: Benchmarking, eliciting, and enhane- ing role-playing abilities of large language models. ArXiv, abs/2310.00746.
Zixiang Wang. Jian Yang, Tongliang Li, Jiaheng Liu, Ying Mo. Jiaqi Bai, Longtao Hc, and Zhoujun Li. 2023b. Multilingual entity and relation extraction from unified aanuuag--ppecife training. arXiv preprint 2rX1::2301.000 04434.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurlPS.
Jian Yang. Shuming Ma. Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021a. Multilingual machine translation sys- tems from microsoft for WMT21 shared task. In PWMT 2021, pages 455.
Jian Yang. Yuwei Yin, Shuming Ma, Haoyang Huang. Dongdong Zhang, Zhoujun Li, and Furu Wei. 2021b. Multilingual agreement for multilingual neural ma- chine translation. In ACL 2021, pages 233-239.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu, 2021. Do transformers really perform badly for graph representation? In NeurlPS 2021, pages 28877-28888.
Zhipeng Yu, Jiaheng Liu, Haoyu Qin, Yichao Wu, Kun Hu, Jiayi Tian, and Ding Liang. 2023. Ied-face: Intra- class compactness distillation for face recognition. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 21042- 21052.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023, Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, 204.22044..92...


Reviews:
[{"review_id": "pira9xIswC", "reviewer": "Reviewer_naWw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Response to Review #1:****\n\nThanks for your thoughtful and valuable comments. \n\nR1.Q1: **Results w/o FVPG**. For the model without FVPG, we directly use the transformer encoder and the vision encoder outputs as follows, and we observe that FVP-M$^{2}$ (w/o FVPG) are lower than FVP-M$^{2}$, which shows the effectiveness of our FVPG. In our new version, we will add this result to better explain the effectiveness of our method.\n\n| Model | En | Fr | Avg_{all} |\n| --- | --- | --- | --- |\n| FVP-M$^{2}$ (w/o FVPG $\\&$ MCoT) | 22.9 | 28.4 | 25.7 |\n| FVP-M$^{2}$ (w/o MCoT) | 27.2 | 30.9 | 29.0 |\n| FVP-M$^{2}$ (w/o FVPG) | 26.6 | 29.1 | 27.9 |\n| FVP-M$^{2}$ (Our method) | 29.9 | 33.7 | 31.8 |\n\nR1.Q2: **Results with only the text components.** Please refer to the Text Transformer in Table 1. We have already given the results with only the text components. We will mention the following results more clearly in our new version.\n\n| Model | En | Fr | Avg_{all} |\n| --- | --- | --- | --- |\n| Text Transformer | 19.6 | 25.5 | 22.6 |\n\nR1.Q3: **Results with image caption methods.** We replicate the current SOTA approaches for image caption (SimVLM[1], OFA[2]) as follows, and we observe that the performance results of these methods are lower than a lot. We assume that these caption methods focus more on attributes (e.g., objects in the image) and scenes, while the style of the manga is more about feelings and plot-related. Therefore, the domain of manga is not compatible with these image caption methods. In our new version, we will revise the above discussion and try to extend our method on the more backbone models to facilitate the advancement of the community, such as OFA and SimVLM.\n\n| Model | En | Fr | Avg_{all} |\n| --- | --- | --- | --- |\n| SimVLM [1] | 5.6 | 8.4 | 7.0 |\n| OFA [2] | 6.0 | 9.3 | 7.7 |\n| FVP-M$^{2}$ (Our method) | 29.9 | 33.7 | 31.8 |\n\nR1.Q4: **Typos Grammar Style And Presentation Improvements.** We will revise these typos and improve the presentation in our new version.\n\nReference:\n\n[1] Simvlm: Simple visual language model pretraining with weak supervision.\n\n[2] OFA: unifying architectures, tasks, and modalities through a simple sequence-tosequence learning framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology is clearly defined. \n\nThe paper presents the task challenge well, namely e.g; comics have shorter conversational text lengths, which makes it harder to make predictions.\n\nThe methodology presented exploits the complementary textual and visual information present in the comics. For this the authors propose the FVP-M^2 method which includes three components: (1) feature encoding, (2) fine-grained visual prompt generation, and (3) dialogue complement.  \n\nSecondly the logical information present in the timelines and visual/text is exploited. This is done using Three-hop Manga Chain-of-Thought (MCot) which is a three hop inferring stage (theme, opinion and future).This is explained well in Figure 3.", "reasons_to_reject": "The fine-grained visual prompt generation is not clear. Ablation study on the prompt generation is needed to understand the importance of this component. Table 2 presents results for models w/o FVPG & MCoT and model w/o MCot. However model w/o FVPG is not explored, i.e dialogue component directly using the transformer encoder and the vision encoder outputs, without the FVPG. [This result is presented in the rebuttal]\n\nWhat happens if only the text components are used for the dialogue prediction? I.e no multimodality.\n\nCould other methods like image captioning be applied here?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes the task of Multimodal Manga Complement (M2C) which is the automatic understanding of manga (comics) with visual and textual features, so as to complete the missing text when the images are present.\n\nThis paper introduces a new dataset (M2C benchmark dataset) consisting of two languages, English and French. The text is extracted using the OCR model (TrOCR). The french component is achieved by using a XLM-R model (En-Fr).\n\nThis paper introduces a novel Manga argumentation method called MCoT to mine event knowledge in comics using large language models.\n\nEstablishes a baseline (FVP-M^2) for the task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 86, W for while should be capital."}, "extra_text": {}}, {"review_id": "fDUQUOK4fM", "reviewer": "Reviewer_4xZw", "summary": "", "questions": "1. Please provide substantive statistical data that justifies the significance of this problem, specifically by presenting statistical evidence concerning the prevalence of these issues within the comics field.\n\n2. As numerous extant multi-modality studies propose the incorporation of a cross-modality attention layer to directly capture correlations between different modalities, I am wondering why the present work departs from employing such a conventional approach.\n\n3. This work somehow shows a similar characteristics akin to a sequence-to-sequence language prediction task. In comparison to utilizing images as an additional modality, have you conducted an assessment of the viability of relying solely on the contextual relationships among dialogues for this particular task?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Response to Review #2:****\n\nThanks for your thoughtful and valuable comments. \n\nR2.Q1: **Statistical data for the significance of the problem**. According to Wikipedia and industry research reports, the comic book industry has a very large base and geographically, Asia Pacific is the largest market, accounting for about 41% of the global market in 2019, with North America and Europe accounting for about 28% and 22%. By type, comic books mainly include physical comic books and digital comic books. Among them, the most widely used is physical comic books, accounting for approximately 89.90% of the total sales in 2019. In terms of application, comic books are mainly sold in retail stores, bookstores and online. And bookstores are the most widely used region, accounting for 49.01% of the total region in 2019. The global comic books market reached 25.1 billion dollars in 2020 and is expected to reach 31.5 billion dollars in 2026, at a compound annual growth rate (CAGR) of 3.3%. In such a huge market, the damage to comics is very serious. Due to aging and vandalism, the damage rate of comics is as high as 72.4%. In 2020, for example, the loss due to damage is as high as around 14.6 billion dollars. At the same time, the lack of comics will also indirectly breed piracy and infringement of copyright, and the annual cost of comics infringement and maintenance of action is up to tens of billions of dollars. To sum up, the task of comic complement is of great significance to the protection of intellectual property rights and the reduction of losses in the industry.\n\nR2.Q2: **Cross-modality attention layer**. We have already adopted the co-attention module in our framework. In Line 92-93 and Figure 4 in our paper, the Co-Transformer module represents the cross-modality attention layer, which has been proposed in ViLBERT,a model for learning task-agnostic joint representations of image content and natural language. We will present more clearly in new version.\n\nR2.Q3: **Rely solely on the contextual relationships**. Please refer to the Text Transformer in Table 1 in our main paper. We have already given the results with only the text components (i.e., the results of Text Transformer). We will mention these results more clearly in our new version.\n\n| Model | En | Fr | Avg_{all} |\n| --- | --- | --- | --- |\n| Text Transformer | 19.6 | 25.5 | 22.6 |\n\nR2.Q4: **Human-in-the-loop evaluation**. We design questionnaires for human assessment. First, we recruited 10 college students from different departmemts for a fair and diverse evaluation with comic book reading experience as raters, and then we randomly selected 15 examples from the test set as a questionnaire. For each test case, at least three labers randomly are selected to give the independent scores and then we summarize the scoring results for a objective evaluation. The college students are required to score different models. The scores are from 1 to 5 (1: ”very bad”, 2: “bad”, 3:“borderline”, 4: “good”, 5:“very good”). There are two criteria for scoring: linguistic coherence and fun.  Finally, we calculate the average score and rank the models as follows, and we observe that our method ranks first, ChatGPT is not the lowest ranked while it has the lowest BLEU score, it has a high score in language coherence, but lacks the fun of comics. \n\nThe average scores of 10 college students for baseline methods:\n\n|  | ChatGPT | DCCN | On Vision Features | FVP-M$^{2}$ |\n| --- | --- | --- | --- | --- |\n| Language coherence | 4.37 | 4.06 | 4.13 | 4.39 |\n| Fun | 3.71 | 3.85 | 4.01 | 4.32 |\n\nThe ranking of four methods:\n\n| Model | Ranking (Language coherence) | Ranking (Fun) |\n| --- | --- | --- |\n| ChatGPT | 2 | 4 |\n| DCCN | 4 | 3 |\n| On Vision Features | 3 | 2 |\n| Our method | 1 | 1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, authors address a compelling and innovative problem of augmenting comics with missing dialogue or context. The study encompasses a fusion of linguistics and the correlation between language and image contextual semantics. By establishing a shared semantic space for vision and language understanding, the research explores the impact of combining vision and language modalities. This approach, known as the M2C task, holds significance in manga analysis as it enriches manga comprehension through visual and textual features, garnering considerable attention from natural language processing and computer vision communities.\n\nFurthermore, the proposed model architecture exhibits promise in various other domains of application. The multi-modal complements model architecture could find utility in fields such as E-commerce or the streaming industry for complements recommendation. Future investigations might explore incorporating sequence products/videos titles or content summaries along with image signals. The model's potential could be harnessed to interpret the semantic interaction among products/videos and offer judgments for complements recommendation.\n\nLastly, authors' work stands is well-structured, and well justified by incorporation of appropriate baselines. The work's logic is easy to follow and understand. Additionally, they have contributed a new M2C benchmark dataset, encompassing two different languages, which will facilitate ongoing studies in the future.", "reasons_to_reject": "Although, authors' proposed solution could address the issue of missing content in comics demonstrates novelty and commendable performance, the practicality of resolving this problem remains unsubstantiated. While my familiarity with this industry may be limited, it is essential for the authors to provide statistical evidence regarding the prevalence of such issues in the real world and how readers respond to such issues. This empirical approach would strengthen the application of the proposed solution and help establish its potential business value, after all this work is more likely to be application-oriented.\n\nConsidering the novelty of this work and the need for a more comprehensive evaluation, it is advisable to develop consolidated design criteria for evaluation metrics iteratively. Thus, in addition to the authors' proposed evaluation strategies, employing a human-in-the-loop evaluation approach would enhance the credibility of the work's robustness and reliability.\n\nLastly, there lacks sufficient ablation study to understand whether the image modality is truly needed for such work. ", "questions_to_authors": "1. Please provide substantive statistical data that justifies the significance of this problem, specifically by presenting statistical evidence concerning the prevalence of these issues within the comics field.\n\n2. As numerous extant multi-modality studies propose the incorporation of a cross-modality attention layer to directly capture correlations between different modalities, I am wondering why the present work departs from employing such a conventional approach.\n\n3. This work somehow shows a similar characteristics akin to a sequence-to-sequence language prediction task. In comparison to utilizing images as an additional modality, have you conducted an assessment of the viability of relying solely on the contextual relationships among dialogues for this particular task?", "paper_topic_and_main_contributions": "Authors present a novel and new learning task named Multimodal Manga Complement (M2C) aimed at augmenting manga comprehension through the integration of visual and textual features. The primary objective of the M2C task is to address challenges arising from missing comic text content, which significantly hampers human understanding. Authors propose a shared semantic space for comprehending both visual and linguistic aspects. Additionally, a novel benchmark dataset for the future research in the multimodal manga complement is established in two languages. Overall, authors present a novel task serves as a foundational framework for addressing other comic-related tasks, marking a significant advancement in the field of manga analysis and comprehension.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MPrxF2BxYl", "reviewer": "Reviewer_YEKH", "summary": "", "questions": "- There appears to be an inconsistency in the data description. The authors have stated that they choose one sentence from each story and that the M2C dataset contains 10,221 stories. Therefore, logically, there would be 10,221 pairs, not 45,995. Could the authors please clarify this discrepancy and provide further details on how the pairs were constructed?\n\n- Regarding Feature Encoding in Sec. 3.2.1, the description of \"M is the number of local cropped images...\" seems inaccurate. Based on the context, M should represent the local image features, such as grid features or visual patches, depending on the visual encoders (ResNet or ViT) being used. Consequently, M shouldn't be described as the number of cropped images. A clarification or correction of this point would be appreciated.\n\n- The computation of p_v is not clearly explained in the paper. Providing details about how  p_v is computed, as well as information regarding its dimension, would be helpful for a full understanding of the method.\n\n- Table 3 introduces some confusion regarding the difference between FVP-M2 map and FVPG. Since FVPG also appears to utilize the map function, it would be beneficial for readers if the authors could clarify the distinctions between these two aspects.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Response to Review #3:****\n\nThanks for your thoughtful and valuable comments. \n\nR3.Q1: **Data description.** Thanks for your suggestion. Here, to better utilize the corpora, we choose each sentence in a story as the target to predict. As the average length of each story is about 4.5, the number of total pairs is 45,995. We will revise this expression more clearly in our revised version. \n\nR3.Q2: **M in Feature Encoding in Sec. 3.2.1**. Thanks for your advice. We will revise M as the local image features in our new version.\n\nR3.Q3: **Computation of p_v.** The computation process of p_v follows the attention strategy with a bias item. Specifically, suppose M is the number of local image features. Then after the CLIP image encoder, we gain v_m (batchsize, M, Dimension=512) and v_c(bathsize, 1 , Dimension=512). So after the mapping network, the dimension of the $\\phi$(Vc) is (batch, M, M). So the final dimension of p_v is (batchsize, M, Dimension), we will revise this explanation in our new version.\n\nR3.Q4: **FVP-M2 (map)**. In FVP-M2 (map), we directly generate visual prompts by encoding the visual tokens with simple linear layers. The name will be changed to avoid ambiguity in our revised version. \n\nR3.Q5: **Missing reference**. We will cite BLIP-2, CoCa and other related multimodal methods.\n\nR3.Q5: **Results of the SOTA multimodal methods.** We reproduce the results of the BLIP-2 and CoCa. Here, we extract features by using these two pre-trained models, and we observe that we can achieve better results when using better multimodal backbone methods, which further demonstrate the generalization ability and effectiveness of our method.\n\n| Model | En | Fr | Avg_{all} |\n| --- | --- | --- | --- |\n| CoCa [1] | 30.1 | 33.9 | 32.0 |\n| BLIP-2 [2] | 30.5 | 34.1 | 32.3 |\n| CLIP (our method) | 29.9 | 33.7 | 31.8 |\n\nR3.Q6: **More description about Manga Chain-of-Thought (MCoT)**. The text of comics generally does not contain a lot of logical information, which is not easy for complement according to the context. To exploit knowledge in comics and improve the generation performance, we design a data augmentation method for this task, which is a three-hop manga Chain-of-Thought to facilitate step-by-step reasoning (inferring theme, opinion and future). Specifically,  first, theme, opinion and future are the basic elements to create the comics. Second, for each hop, the generated text is appended to the next context as shown in Figure 3. We will revise the above description more clearly in the revised version.\n\nR3.Q7: **The loss function.** Thank you for your suggestion. Here, source corpora s and target corpora t are the same language, this kind of expression is merely to distinguish between input and output. To prevent ambiguity, we will specify that they belong to the same language in our new version.\n\nR3.Q8: **Typos Grammar Style And Presentation Improvements.** We will revise it in our new version.\n\nReference:\n\n[1] CoCa: Contrastive Captioners are Image-Text Foundation Models\n\n[2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The task of multimodal manga complementation is indeed engaging and valuable. The proposed dataset represents a promising resource that may prove useful for future research and developments within this domain.\n- The model's pipeline, which incorporates visual features and visual prompts through the use of co-attention for sentence completion, is logically constructed. It appears to be a sensible approach to the problem at hand.\n- The consideration of Manga Chain-of-Thought (CoT) within the method is a notable aspect, although I have some specific queries regarding this element, as detailed below.", "reasons_to_reject": "- The loss function as expressed in Eq. (1) incorporates both source and target corpora. However, the pipeline appears to be operating based on only one corpus at a time, treating English (En) and French (Fr) separately. This inconsistency between the formulation of the loss function and the actual implementation needs to be addressed and clarified.\n- The multimodal baselines employed in the paper do not represent the current state-of-the-art methods. Notable works such as CoCa, BLIP-2, and others are conspicuously absent from the comparison. Including these more recent and advanced methods would likely provide a more robust and meaningful evaluation of the proposed approach.\n- Critical details are missing regarding the utilization of Manga Chain-of-Thought (CoT) within the FVP-M2 model. A comprehensive explanation of how CoT is integrated and its role within the overall approach would provide essential clarity and contribute to a more complete understanding of the model's functionality.", "questions_to_authors": "- There appears to be an inconsistency in the data description. The authors have stated that they choose one sentence from each story and that the M2C dataset contains 10,221 stories. Therefore, logically, there would be 10,221 pairs, not 45,995. Could the authors please clarify this discrepancy and provide further details on how the pairs were constructed?\n\n- Regarding Feature Encoding in Sec. 3.2.1, the description of \"M is the number of local cropped images...\" seems inaccurate. Based on the context, M should represent the local image features, such as grid features or visual patches, depending on the visual encoders (ResNet or ViT) being used. Consequently, M shouldn't be described as the number of cropped images. A clarification or correction of this point would be appreciated.\n\n- The computation of p_v is not clearly explained in the paper. Providing details about how  p_v is computed, as well as information regarding its dimension, would be helpful for a full understanding of the method.\n\n- Table 3 introduces some confusion regarding the difference between FVP-M2 map and FVPG. Since FVPG also appears to utilize the map function, it would be beneficial for readers if the authors could clarify the distinctions between these two aspects.", "paper_topic_and_main_contributions": "The authors proposed a new task, dataset, and baseline called FVP-M2 for mulitmodal manga complement. More specific, local and gloal visual features are considered during feature encoding. Visual prompt generation is also used for better dialogue complement. Experiments on the proposed dataset show the effectness of the proposed method.\n\nI found this work is interesting and useful. But I do have some key concerns on the loss function that consider En and Fr but during training only one language is considered. Besides, it is unclear how CoT is used.\n\nThe authors have introduced a novel task, dataset, and baseline referred to as FVP-M2, specifically for multimodal manga complementation. The method takes into account both local and global visual features during the feature encoding process and utilizes visual prompt generation for enhanced dialogue complementation. Experiments conducted on the newly proposed dataset demonstrate the effectiveness of this approach.\n\nWhile I find the work intriguing and potentially valuable, I do have some crucial concerns that need to be addressed:\n\n- The loss function mentioned in the paper takes into consideration both English (En) and French (Fr) language features. However, it's not clear from the paper how both languages are incorporated during training, especially since only one language appears to be considered. This discrepancy needs to be clarified.\n- Additionally, the utilization of CoT (presumably a component or technique within the method) is not well explained. A more detailed description or rationale for its role within the model would add clarity and depth to the understanding of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "BLIP-2 and CoCa and...", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Page 2, Line 082: \"FVPG\"- This abbreviation is the first time to appear, which should be written as fine-grained visual prompt generation (FVPG).\n2. Page 2, Line 086: while → While"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates manga understanding using visual and textual features (completing the missing text when only the visual cues are available). The research is original and the methodology is clear.

Pros:
* The paper shows how to use the complementary textual and visual information present in the comics
* The methodology that includes visual features and visual prompt using co-attention for sentence completion

Cons:
* Lack of statistical evidence that supports the claims in a real scenario
* Lack of ablation studies
Decision: Accept-Findings