Zero-Shot Character Identification and Speaker Prediction in Comics via Iterative Multimodal Fusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 08:46:43

Abstract: Recognizing characters and predicting speakers of dialogue are critical for comic processing tasks, such as voice generation or translation. However, because characters vary by comic title, supervised learning approaches like training character classifiers which require specific annotations for each comic title are infeasible. 
    This motivates us to propose a novel zero-shot approach, allowing machines to identify characters and predict speaker names based solely on unannotated comic images.
    In spite of their importance in real-world applications, these task have largely remained unexplored due to challenges in story comprehension and multimodal integration. 
    Recent large language models (LLMs) have shown great capability for text understanding and reasoning, while their application to multimodal content analysis is still an open problem. 
    To address this problem, we propose an iterative multimodal framework, the first to employ multimodal information for both character identification and speaker prediction tasks. 
    Our experiments demonstrate the effectiveness of the proposed framework, establishing a robust baseline for these tasks.
    Furthermore, since our method requires no training data or annotations, it can be used as-is on any comic series.

Main paper: 
 introduction: 
 The global comics market has experienced significant growth, sparking interest in computational analysis of comics to enrich user experience and accessibility. Given that characters are central to comic storytelling, this paper focuses on two key tasks: character identification, recognizing the characters in images, and speaker prediction, predicting the speakers of specific dialogue. These tasks enable diverse applications like voicing comics with character-specific voices and machine translations capturing each character's unique speech style.Previous studies on speaker prediction in comics primarily addressed the correspondence between character regions and speech bubble regions in images [8, 17] , but they did not determine the speakers' names. If we want to know the character names of the speakers, we first need to identify the character names of the character regions. The straightforward supervised approach involves training models to detect and classify characters in images [13, 25] . However, this approach requires annotations for each comic with different characters, which is impractical in the fast-paced comic industry that releases thousands of titles annually.This motivates us to tackle the problem as a zero-shot task: identifying characters and predicting speakers in new, unseen comics simply by analyzing the images, without need for any prior annotations.How does a human recognize speakers in new comics, without prior knowledge? The process typically starts with noticing character names in the dialogue. For instance, if one character calls another "Naruto", we associate the response to that dialogue with Naruto. We then connect the character's visual appearance with their dialogue. When Naruto reappears in the following pages, even if his name is not mentioned in the dialogue, we can still know the character named Naruto. Context from dialogues is also an important clue. Once we have determined the speaker of a particular dialogue, we can use contextual information to predict the speakers of nearby dialogues. Additionally, this process allows us to update our knowledge of the characters' visual appearances across different pages. After seeing many examples, we learn to identify characters and predict dialogue speakers even when their names are not directly mentioned in the text.This process suggests two key challenges. (1) High-level text understanding: To predict speakers from limited cues (e.g., names mentioned in dialogue), the system must interpret complex character interactions and story context throughout the whole book or chapter. (2) Multimodal integration: To identify characters and predict the name of speakers without any annotations, integrating both visual and textual information is essential, thereby making a combination of text and image modalities indispensable.Addressing these challenges, we propose a multimodal fusion approach. We leverage large language models (LLMs) for their context understanding and reasoning capabilities [6, 20, 23] , providing a strong baseline for zero-shot speaker prediction. To address the challenges of integrating LLMs with other modules and enhance the machine's comprehension of comics, we introduce an iterative framework. We merge text-based LLM predictions with image-based classifiers, and alternately refine each module using results from the other. This multimodal integration not only enables zero-shot character identification but also also notably improves text-only baseline in speaker prediction. Moreover, by iteratively refining the integration of text and image information, this approach enhances the utilization of both modalities, thereby deepening the machine's comprehension of comics.Our contribution is twofold. (1) New tasks: We are the first to integrate the tasks of character identification and speaker prediction in comics. Furthermore, our approach tackles zero-shot tasks without requiring any training or annotations, which are directly applicable to real-world scenarios. (2) Iterative multimodal fusion: We pioneer in revealing the potential of LLMs for comics analysis and propose a novel method that integrates text and image modalities. To enhance the machine's deep understanding of comic content, we introduce an iterative framework aimed at progressively refining performance. This is the first study to use both text and image information for character identification and speaker prediction, which are unexplored even outside zero-shot settings. 
related work: 
 Given the novelty of our study in predicting speaker names in comics, there are no existing studies directly related to this task. Therefore, we first explore related works from two distinct perspectives: comic speaker prediction and comic character identification. Then, we introduce works related to our proposed method.Previous studies of comic speaker prediction focus on predicting the correspondence between character bounding boxes and text regions. Conventional methods are rule-based, primarily relying on the distance between the character and the text region [17] . They disregard visual and textual context information, causing them to fail in cases where the speaker is not closest to the text region. Recently, Li et al. employ scene graph generation (SGG) models [8] known for their effectiveness in visual relationship detection tasks [7, 18, 21, 22] , to predict the correspondence in a more robust way by using visual semantic information such as the face angle. While the previous studies focused only on predicting the correspondence between character bounding boxes and text regions, we propose a new task to predict speaker identities, i.e., character names. This allows more applications such as text-tospeech with character-specific voices. Furthermore, while previous studies have utilized only visual information, we are the first to utilize both visual and textual information.Comic character identification is more challenging than face identification in real-world images due to the variance of drawing styles and poses. Zheng et al. constructed a cartoon-face dataset and proposed an image classification model to classify characters [25] . However, their supervised approach requires training character classifiers for each comic title, which limits the feasibility of this approach. Some previous studies attempt unsupervised methods to handle unseen comics. Tsubota et al. used deep features trained for generic face recognition in comics and adapted them to unseen comics [19] . Zhang et al. improved clustering performance on comics with a face-body combination and spatial-temporal correction [24] . However, they are limited to grouping the characters by clustering and cannot identify character names. In addition, even though dialogue is an important cue to identify characters in comics, existing approaches have not used textual information. In this paper, we propose the first multimodal approach that utilizes dialogue and predicts character names in unseen comics.The recent success of ChatGPT [14] and GPT-4 [15] has demonstrated the power of LLMs in understanding, generating, and interpreting human language with remarkable accuracy. Inspired by these advancements, we have pioneered the application of LLMs to the dialogue analysis in comics. Alongside the development of LLMs, large multimodal models (LMMs) such as LLaVA [9, 10] and MiniGPT-4 [26] have garnered attention for processing multimodal information. In spite of their capabilities in visual understanding and reasoning, currently, they can only handle a single or a few images as input. Comic analysis requires comprehension across longer contexts, such as multiple pages. Also, it involves learning character identity throughout the book or series. We thus propose an iterative framework to integrate multimodality and longer contexts into LLMs inference for this task. 
approach 3.1 problem settings: 
 Let us define our problem setting for zero-shot character identification and speaker prediction. The inputs are a sequence of page . The character regions are the bodies of characters, and the text regions are speech bubbles. N is the list of character names (e.g., N = {Keitaro, Naru, . . .}), which is extracted from dialogues. These names serve as target labels for speaker prediction and character identification tasks in the following steps. 𝑁 and 𝑀 denote the number of character and text regions in a given comic.Data preprocessing. Before initiating our main pipeline, we conducted a series of preprocessing steps. First, character regions C and text regions D are obtained using object detectors for general comic character and text category, which are shown to achieve high accuracy [13] . Subsequently, we derived the initial relationship scores R from I, C, and D using the SGG models [8] . Relationship scores R = {𝑟 𝑖→𝑗 } 𝑖,𝑗 represent the correspondences between character and text regions, where 𝑟 𝑖→𝑗 ∈ (0, 1) is the confidence that the character 𝑐 𝑖 is the speaker of the text 𝑑 𝑗 . This is used to propagate labels between the character and text regions. Additionally, we extracted texts T = {𝑡 𝑗 } 𝑀 𝑗=1 from text regions D utilizing optical character recognition (OCR). Through text analysis by LLMs, we obtained the list of character names N from T .Our framework for zero-shot speaker prediction and character identification is illustrated in Figure 2 . Speaker prediction and character identification are performed iteratively. By using the output of each task as input for the other, we can exploit complementary multimodal information in both tasks, which leads to good performance even in zero-shot settings.Our framework is comprised of three modules:• Speaker prediction: Predict labels of text regions Y with LLMs. Initial predictions only use textual content, denoted as 𝐹 0 : (T , N ) ↦ → Y. From the second prediction, labels obtained from character identification Y pseudo is used, denoted as 𝐹 : (T , N, Y pseudo ) ↦ → Y. • Character identification: Predict labels of character regions X using image information and pseudo labels X pseudo obtained from speaker prediction. Denoted as 𝐺 : (C, X pseudo ) ↦ → X. X X pseudo ← 𝐻 t→c (R, Y) X ← 𝐺 (C, X pseudo ) R ← 𝑓 rescore (R, X, Y) 5: Multimodal speaker prediction: Update Y Y pseudo ← 𝐻 c→t (R, X) Y ← 𝐹 (T , N, Y pseudo ) R ← 𝑓 rescore (R, X, Y) 6: end while 7: return C, D, X, Y• Label propagation: Convert labels between character and text regions using relationship scores. Denoted as𝐻 t→c : (R, Y) ↦ → X and 𝐻 c→t : (R, X) ↦ → Y.The procedure of our framework is shown in Algorithm 1. First, we predict speakers only from text information. This output is converted into labels of character regions using label propagation, whereby character identification is performed. Then, we predict speakers again using the labels obtained in the previous step. In addition, relationship scores are updated with 𝑓 rescore based on the predicted labels. These processes are repeated for a specific number of iterations. In the following sections, we explain the details of each step. We first predict speakers using only the textual content using LLMs as 𝐹 0 (T , N ). GPT-4 [15] is used in this work. The input and outputs of LLMs are shown in Figure 3 . We input the dialogues T and the list of appearing characters' names N into the LLMs. Due to output text length limitations of GPT-4, we split the dialogues into chunks. To compensate for the missing context about the story in each chunk, we first extract context information about the story summary and character profile from T and N using LLMs. This context is then fed into LLMs with each chunk as shown in Figure 3 . Note that speaker candidates Y pseudo are not input in the initial prediction. As the output, we let LLMs output both character IDs and character names to get a stable output. These output character IDs are converted into Y. In addition, we let LLMs output integer confidence scores from 1 to 5, and we exclude data with low confidence from subsequent steps.Given initial speaker prediction results Y, we perform character identification as shown in Figure 4 . The first step is pseudo label generation with the label propagation module 𝐻 t→c . For each character region, we select the character-text pair that has the highest relationship score among all combinations involving this character region. Then, we use the selected text region's label as the pseudo label of the character region. To get higher precision, we set a threshold for the confidence of text region's label obtained by LLMs. In our experiments, removing data with confidence less than 3 lead to higher precision without reducing recall. In the second step, we construct a character classifier to identify the characters with unknown labels. We train an image classifier 𝑓 classify with pseudo labels X pseudo and then obtain X by applying 𝑓 classify to character regions C. We use ResNet50 [4] as the image classifier.As Y becomes closer to the true text labels, the generated pseudo labels will also be closer to the ground truth. Correspondingly, the training data for the character classifier becomes progressively more reliable, resulting in higher identification accuracy. We iteratively enhance the performance of character identification by improving the precision of the speaker labels predicted by the speaker prediction module.In the multimodal speaker prediction, we first generate pseudo labels for text regions Y pseudo from X by using the label propagation module 𝐻 c→t in the same step as 𝐻 t→c . We treat these pseudo labels as candidate speakers for the texts, taking the confidence of character identification to be the confidence for the speaker candidates. Confidences with less than 0.5 are filtered out from candidates.Given Y pseudo , the speaker is predicted by LLMs with the prompt in Figure 3 . We feed the LLMs both the confidence of character classifiers and the character name in a format such as Keitaro (0.56), which informs the LLMs about the reliability of the given speaker candidates.  
experiments 4.1 experimental setup: 
 Dataset. We used the Manga109 dataset [1, 13] , which comprises 109 volumes of Japanese comics and provides character labels for both character and text regions. For the zero-shot setting, we constructed the training and testing sets from distinct comic titles; the characters in the testing set were unseen in the training set. We selected 23 volumes as the testing set. The remaining volumes were used for training and validation of the relationship detection module and pre-training of the character classifier.For the character regions, we used the body region annotations [13] , which represent the entire bodies of the characters. We preferred this method because whole-body regions offer more information (such as clothing and body types) for character identification than face regions. Besides, comic datasets often include characters with body-only annotations, such as those shown from the back or non-human characters, which necessitates the use of body regions for identification. For the character labels associated with the text regions, we used Manga109Dialog annotations [8] .Task Settings. For each comic in the test set of Manga109, we used page images I and a list of character names as inputs. Since object detection and OCR are not the main focuses of this paper, and since the results of character name extraction can significantly impact the accurate evaluation of our main pipeline's performance, in the main evaluation, we omit the steps of obtaining C, D, T , and N in data preprocessing. Instead, we regard them as known information by utilizing the annotations of Manga109. This simplifies the evaluation and makes the experiment replication easier while maintaining the main challenges of our tasks. End-to-end zero-shot settings were evaluated in the final experiments. Given that some titles in Manga109 feature a large number of characters, we excluded characters with an appearance frequency of less than 3% from the list of character names in our experiments. However, to ensure a fair comparison with future works, we still predicted and evaluated the labels of these infrequently appearing characters.Under the above settings, the tasks of speaker prediction and character identification were classifying the character labels for the character and text regions (X and Y). We calculated the accuracy of classification results to evaluate these tasks, which is the ratio of correctly predicted regions to total regions. Additionally, we calculated precision and recall for the pseudo labels in the ablation study. Precision is the ratio of correct pseudo labels to the total number of generated pseudo labels. Recall is the ratio of correct pseudo labels to the total number of regions.Relationship Prediction. Our method used the relationship scores R to show correspondences between character and text regions. We compared three types of initial relationship scores: SGG: a deep learning-based relationship prediction using the scene graph generation (SGG) model [8] , Distance: a rule-based method using the distance between the center coordinates of the character and text regions [17] , and GT: ground truth annotations of Manga109Dialog. We used SGG in the main experiments and used Distance and GT in the ablation study. The confidence of SGG model was used as the score for SGG, and the reciprocal of the distance was used as the score for Distance. In the case of GT, we took the relationship score to be 1.0 for all pairs. Speaker Prediction with LLMs. We used the GPT-4 model [15] (specifically, gpt4-0314) for speaker prediction. First, we produced a context summary by feeding all dialogue to GPT-4. Next, we performed speaker prediction by inputting both the extracted context information and the dialogue text itself into the model. Due to GPT-4's output token limitation, we divided every conversation into segments, each comprising 60 sentences. A complete list of these prompts is available in the supplementary material.Character Classifier Training. We used ResNet50 [4] as the character region classifier, initially pre-trained on ImageNet [3] . We chose this model as the classifier due to its robustness in handling the diverse and complex visual patterns of comic character bodies, which provides more advantages over simply fine-tuning face recognition models. We also explored state-of-the-art classification models like ConvNeXt [11] and models pre-trained on the anime dataset Danbooru [2] . However, our experiments demonstrated that the fine-tuned ResNet model offers the best performance.Our training process involved two steps: pre-training for generic comics and fine-tuning for individual unseen comic in the test set. Pre-training, done only once for generic comics, aims at domain adaptation from the ImageNet pre-trained model to comic characters. Manga109 training set with ground truth annotations of 349 characters is used for pre-training. We fine-tune on each individual unseen comic using pseudo labels generated through our multimodal iterative fusion process. We employ various techniques, including data augmentation and model ensembling, to achieve stable results from training with noisy labels. Since this module itself is not the main focus of this paper, we describe the details in the supplementary material.Baselines. Since no existing method can predict character labels in a zero-shot setup, we constructed our own baselines. For character identification task, we first group character regions of each comic by the clustering of deep features obtained using the Manga109 pretrained model explained above. K-means clustering [12] with 𝑘=|N | is used. Since it is impossible to assign character labels to each cluster without text information, we map each cluster to the character labels using ground truth so that accuracy is maximized, which can be regarded as an upper bound of the clustering approach. The labels of character regions are converted into those of text regions using relationship prediction methods [8, 17] to as K-means + Distance and K-means + SGG, respectively in Table 1 .Table 1 shows the results of the proposed method and the baselines. The text and img columns indicate the used modalities for each method. Each iteration involves identifying characters based on the speaker predictions from the previous iteration, followed by a new round of speaker prediction that uses the updated character labels and relationship scores. The initial phase, where the speaker prediction is conducted only with textual information via LLMs, is iteration 0. The subsequent complete iteration cycle following iteration 0 is iteration 1. To validate the effectiveness of our proposed method, we divided the test set (Total) into Easy and Hard by the difficulty of relationship prediction. Easy contains 11 volumes with an accuracy of relationship prediction over 75%. The remaining 12 volumes were categorized as Hard.As shown in Table 1 (a), our proposed multimodal approach produces a significant improvement in accuracy over unimodal methods in both speaker prediction and character identification.For speaker prediction, the accuracy of LLM only was 43.6% but it increased to 51.1% in the first iteration. The accuracy of character identification also reached 42.4%, outperforming the baseline that uses the ground truth labels. Although the accuracy of character identification was less than 50%, it is noteworthy that our method is the first one that recognizes character labels in unseen comics. The difficulty of identifying less frequent characters in zero-shot settings causes the low accuracy in both tasks, which is posed as future work.Across all test sets, the results from iteration 2 showed an improvement over iteration 1. In iteration 3, while the accuracy for Easy data continued to increase, there was a slight decline observed for Hard and Total data. This suggests that there are limits to the accuracy gains in relationship detection from further iterations To investigate this further, we do an analysis using the ground truth relationships in Table 1 (b), where the accuracy of both speaker prediction and character identification keeps increasing with further iterations. These results suggest our iterative process is more effective in the case that the prediction of relationships between text and character regions is accurate. Figure 5 shows examples of prediction results. We evaluate the effectiveness of the proposed method in two aspects: Unimodal vs. Multimodal (Figure 5 (a)) and One-Step vs. Iterative (Figure 5 (b)). Given that unimodal approaches are incapable of achieving zero-shot character identification, when using only LLM, character identification is not performed. Image only results corresponds to the baseline k-means+SGG: labels of character regions are derived from ground truth. Speaker predictions from texts alone present challenges for LLMs, especially with texts lacking distinctive character features, while it is easy to make correct predictions when integrating visual information. Conversely, when speaker predictions are made only based on images, such as by selecting the character closest to the text as the speaker, the method fails in two scenarios:(1) when the speaker is not the character closest to the text; (2) the speaker does not appear in the image. In contrast, our multimodal approach handles these complexities well. Figure 5 (b) shows the results of the proposed method in each iteration. As the number of iterations increases, there is a clear trend of increasing accuracy in both character identification and speaker prediction tasks. Although the first iteration yields no correct predictions, by the third iteration, the accuracy has significantly improved. These findings affirm that our method can improve the accuracy of both tasks by iteratively refining the results using multimodal information.Figure 6 shows the accuracy of speaker prediction for each comic. The three bars in each comic indicate the accuracy of iteration 0 (LLM only), iteration 1, and iteration 2, respectively. The iterations improved accuracy on most of the comics, which shows the effectiveness of our iterative approach on different comics. This figure also shows that performance can vary significantly across different comics, ranging from 0.2 to 0.8. Note that more iterations do not necessarily lead to higher accuracy. This is because if the accuracy of relationship prediction or character identification is low, the candidates can introduce noise into speaker prediction. An example is shown in Figure 7 . In iteration 1, the LLMs correctly predicted the speaker (character A). However, due to the distance between the speaker and the text, character A received a low relationship score. In contrast, character B, being closer and having a higher relationship score, was incorrectly identified as the speaker candidate with high confidence. This led to an incorrect prediction in iteration 2, which explains why there is a decrease in accuracy by iterations in Figure 6 and Table 1 .Pseudo Labels. Table 2 and 3 shows the quality of the pseudo labels X pseudo and speaker candidates Y pseudo and their effect on accuracy for speaker prediction and character identification tasks. Three relationship scores are compared. 2nd and 3rd columns are the precision and recall of the pseudo labels. The accuracies listed in the 4th column are those of the pseudo labels before thresholding. These are shown for comparison with the final performance score (5th column). The carryover from pseudo labels' accuracies (4th column) to final accuracy (5th column) explains the effectiveness of our iterative process (+6.0% and +13.2% gains in speaker prediction and character identification, respectively when using SGG). It shows that the result obtained from the previous step is refined in each module by using different modal information. By comparing relationship scores, we can see the result gets better by improving the quality of pseudo labels with better relationship prediction; specifically, the precision of GT in pseudo labels affects the character identification performance a lot. In our current method, the combination of the noises of speaker and relationship prediction decreases the precision. Getting reliable pseudo labels is posed as future work.LLM Prompts. Table 4 shows the results for different LLM prompts in the speaker prediction. The three options in the table correspond to (1) ctx: context information about characters and stories, (2) cand: speaker candidates, and (3) prob: the probability of candidates. Their respective improvements in accuracy are 4.7%, 5.5%, and 2.0%. Each option introduces information not present in the dialogues of each chunk: context provides whole-story information, while the speaker candidates and their probabilities are obtained from the images. These options offer complementary information to each other. The combination of all three options, as shown in Figure 3 , performed best.Relationship Rescoring. Table 5 shows the effect of the proposed relationship rescoring on relationship prediction and speaker prediction. We evaluated the relationship prediction in the manner described in the previous work [8] : for each text region, the corresponding character region with the highest relationship score was selected. Then, the accuracy was calculated by dividing the number of correctly matched regions by the total number of text regions. Our approach with rescoring achieved an accuracy 79.8%, which is better than previous methods. Moreover, it increased speaker prediction accuracy. In contrast to the previous work that doesn't consider character labels, our method can predict the relationships better by predicting character labels of both images and texts.We evaluated our method in entirely zero-shot settings using only input images. The bounding boxes of the character and the text regions were detected by using the object detection with Faster R-CNN [16] , and dialogues were extracted by using OCR in the manner described in [5] . We extracted a list of character names directly from the dialogues using GPT-4. These names then served as target labels for speaker prediction and character identification tasks.Table 6 shows the experimental results under entirely zero-shot settings. In assessing the experimental results, we manually created mappings between the extracted names and the true names. Each predicted region is counted as correct if it was detected with and IoU >0.5 and was correctly labeled. Our method achieved 38.7% and 35.0% in accuracy for speaker prediction and character identification tasks, respectively. This is lower than the results shown in Table 1 due to errors in object detection, OCR, and character name extraction. Particularly, the character name extraction significantly affects performance because characters whose names are not extracted cannot be predicted. To show the effect of character name extraction on the performance, we compute the upper bound in Table 6, which is the accuracy under ideal conditions, where all labels of extracted names are perfectly predicted. The recall for extracted names was 62.7%, thereby limiting the upper bounds to 67.3% for speaker prediction and 63.9% for character identification. It should be noted that there are several titles in which character names are rarely mentioned in dialogues. In that case, it is difficult even for humans to predict the names, which poses the challenges in entirely zero-shot settings. On the other hand, we achieve promising results in titles where character names appear frequently in dialogues and the relationships between characters and texts are clear. Figure 1 shows the results of our approach in entirely zero-shot settings from one specific book, which demonstrate promising first step of zero-shot character identification and speaker prediction of unseen comics. 
conclusion: 
 We are the first to introduce zero-shot character identification and speaker prediction in comics. To solve this unexplored and challenging task, our proposed framework utilizes multimodal integration and high-level context understanding. Since story understanding and multimodal integration are key in comics analysis, our promising results can be a key milestone that opens new directions in comic analysis. In addition, our method can be applied to any comics immediately without the need to train on specific comics, bridging the gap to real-world application. Moreover, our framework can be easily adapted to different domains because each module of our framework is modular and reusable (e.g., learning person classifiers on video data). Since the potential of multimodal speaker prediction extends beyond comics, ranging from movies to online conferences, we hope that our work will impact other fields as well. 
 
 Appendix:  


Decision: This paper proposes a multimodal fusion framework for zero-shot character identification and speaker prediction in comics. The task is new, and the proposed approach is interesting. All four reviewers are positive.

Reviews: review: paper_summary: paper_summary: This paper proposes a zero-shot approach, allowing machines to perform these tasks in new, unseen comics without the need for prior annotations. The framework utilizes an iterative multimodal fusion model to refine predictions through alternate updates of text and image data, leveraging recent advancements in large language models for understanding and reasoning in comic content analysis. The paper aims to establish a robust baseline for character identification and speaker prediction tasks in comics, offering a practical solution applicable to real-world scenarios without the requirement of training data or annotations.
strength_weakness: strength_weakness: This paper integrates character identification and speaker prediction tasks in comics for the first time, and it tackles these tasks as zero-shot learning problems, making the approach directly applicable to real-world scenarios without specific training data. This is the first study to use both text and image information for character identification and speaker prediction, which are unexplored even outside zero-shot settings.

The paper presents an innovative iterative multimodal fusion framework for zero-shot character identification and speaker prediction in comics, which integrates large language models (LLMs) with image-based classifiers to improve predictions through alternate updates. Experiment results highlight the method's capability to handle zero-shot settings and its potential for practical application in comics analysis.
limitations: limitations: Though this work is the first to integrate both character identification and speaker prediction tasks in comics, it lacks sufficient baseline models. It would be beneficial to compare the proposed framework with other large multimodal models (LMMs) such as LLaVA and zero-shot models for speaker prediction.
paper_summary: paper_summary: This paper proposes a zero-shot method that is an iterative multimodal fusion framework for character identification and speaker prediction in comics. This approach integrates visual and textual information without requiring any prior annotations or training data. The proposed method employs large language models for text understanding and combines their predictions with image-based classifiers to iteratively refine the identification and prediction tasks.
strength_weakness: strength_weakness: 1. This paper discusses an interesting topic and introduces the tasks of zero-shot character identification and speaker prediction. This topic is highly suitable for the MM community.
2. The ability of the proposed method to operate without training data or annotations makes it very practical for real-world applications, where annotating every new comic series is impractical.
limitations: limitations: 1. The layout of dialogues in comics varies widely, and different types of text may appear, such as narration or words that the character thinks. Can the proposed method distinguish these complex cases? How do such cases impact the results of speaker prediction?
2. The experimental results indicate that the accuracy of speaker prediction is limited. This makes it challenging for the application of this task. 
3. The proposed method seems somewhat more engineering-oriented for the proposed new task.
paper_summary: paper_summary: This paper aims to solve the problem of identifying the character name of each person and text box that appears in comic images. The method proposed in this paper includes 2 parts: GPT-4 for speaker prediction, which predicts the speaker given the dialog text, and ResNet50 for character prediction, which predicts the character name given an image region. These two modules are applied iteratively to refine each other. Experiments are conducted on the Manga 109 dataset.
strength_weakness: strength_weakness: 1. This paper addresses a new task that has not yet been solved.
2. The authors conducted many ablation studies, which makes it clear how the proposed method works.
limitations: limitations: My biggest concern is the proposed method only works when C (character boxes), D (text boxes), and N (character name list) are manually annotated. As shown in Section 4.4, when this information is not given, the accuracy of this method drastically drops to less than 40%, which means it is hardly applicable in real-world scenarios. Also, the findings of the previous ablation studies may be different in real-world scenarios when those manually annotated information is not given. 

I think that it will be better for this study to focus on the “zero-shot evaluation” setting described in Section 4.4, such as how to robustly identify the speakers and characters when N does not exist or is incomplete, and prevent the error accumulation when C and D are not accurate. However, considering that this is the first work for this task, this shortcoming is not unacceptable.