CartoonNet: Cartoon Parsing with Semantic Consistency and Structure Correlation

Authors: jian-jun  qiao
yu  duan
xiao  wu
yu-pei  song
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 02:41:12

Abstract: Cartoon parsing is an important task for cartoon-centric applications, which segments the body parts of cartoon images. Due to the complex appearances, abstract drawing styles, and irregular structures of cartoon characters, cartoon parsing remains a challenging task. In this paper, a novel approach, named CartoonNet, is proposed for cartoon parsing, in which semantic consistency and structure correlation are integrated to address the visual diversity and structural complexity for cartoon parsing. A memory-based semantic consistency module is designed to learn the diverse appearances exhibited by cartoon characters. The memory bank stores features of diverse samples and retrieves the samples related to new samples for consistency, which aims to improve the semantic reasoning capability of the network. A self-attention mechanism is employed to conduct consistency learning among diverse body parts belong to the retrieved samples and new samples. To capture the intricate structural information of cartoon images, a structure correlation module is proposed. Leveraging graph attention networks and a main body-aware mechanism, the proposed approach enables structural correlation, allowing it to parse cartoon images with complex structures. Experiments conducted on cartoon parsing and human parsing datasets demonstrate the effectiveness of the proposed method, which outperforms the state-of-the-art approaches for cartoon parsing and achieves competitive performance on human parsing.

Main paper: 
 related work 2.1 human parsing: 
 Human parsing [5, 16, 29] , the task of segmenting human body parts in images, has achieved significant progress. Previous human parsing methods develop approaches such as hierarchical structure learning [12, 34, 35, 43] , pose estimation-guided learning [19, 41] , and self-correction strategy [14] , achieving satisfactory results on recognition of body parts of humans. However, applying human parsing methods to cartoon parsing presents challenges due to fundamental differences. Cartoons are known for their complexity, diverse styles, irregular lines, unique color schemes, and abstract representations, varying greatly based on creators and cultural contexts. Human parsing models [15, 18] are designed on real-world human images, making them unsuitable for cartoon parsing, where characters often have unconventional body proportions and articulations. Cartoon parsing necessitates dedicated research and specialized algorithms to address its unique characteristics, emphasizing accuracy and effectiveness in dealing with the diversity and complexity of cartoon images.The task of cartoon parsing has gained increasing attention in recent years due to its relevance in various multimedia applications. Researchers have proposed several approaches to address the challenges posed by diverse and complex cartoon appearances and structures. Early works in this domain primarily focused on rule-based methods and handcrafted feature extraction techniques. These approaches, while capable of capturing some aspects of cartoon images, often struggle with the diversity and complexity of cartoon imagery. The emergence of deep learning techniques, particularly the convolutional neural networks (CNNs) [1, 17, 25, 27] , has significantly advanced the field of cartoon parsing. DFPNet [32] applies feature pyramid network to cartoon dog parsing, which alleviates the varied scales in cartoon parsing. However, it ignores the correlations of visual cues and structural features in cartoon characters. CPNet [23] recognizes the irregular cartoon structures and body parts with different semantics but have visually akin appearances, by introducing a pixel and part correlation learning structure. However, it ignores the semantic consistency of diverse cartoon images and the structure correlation of complex cartoon characters. Although CNN-based methods demonstrate improved performance in recognizing cartoon characters, addressing the diversity and complexity of cartoon images remains a challenge.Memory-augmented neural networks [36, 37] have shown promise in capturing and utilizing long-term dependencies in data. These networks [13, 24] employ an external memory structure that can be read from and written to, enabling the storage and retrieval of information over extended sequences. In the context of cartoon parsing, memory-augmented networks can offer a promising potential for encoding and learning the diverse appearances exhibited by cartoon characters. Despite the growing interest in memory-augmented networks, their potential for cartoon parsing remains unexplored.The ability to store appearance information and perform semantic consistency learning opens possibilities for addressing individual visual differences. Therefore, this paper proposes a novel approach applicable to cartoon parsing to address the issue of significant sample diversity in cartoon parsing through a memory-based structure.3 Cartoon Parsing 
framework: 
 CartoonNet is proposed to address the problems of cartoon parsing, which are caused by visual diversities and structural complexities of cartoon characters. A two-fold strategy integrating appearance learning and structural modeling is introduced to capture features of diverse visual appearances and model complex structural representations. The framework is illustrated in Fig. 2 , in which ConvNet [10] is employed to encode and extract features for cartoon images. The memory bank is adopted to store features of diverse samples with a three-column structure. The stored features are matched to current images by measuring feature similarity, which aims to parse current samples by recalling previous experiences. The previous experiences are integrated into current samples by adopting self-attention mechanism, which achieves semantic consistency among diverse samples. To learn the correlation of complex body parts, a main body-aware structure correlation strategy is designed. It incorporates graph attention networks and a main body-aware mechanism to facilitate part correlation. The graph attention networks model the structural representations and capture important relations among the body parts. The main body-aware mechanism learn important body parts that reflect the complex structural properties of the cartoon images.Previous cartoon parsing methods only consider multi-scale learning, as well as pixel and part correlation among a single cartoon character [23, 32] . They have difficulties in identifying diverse cartoon images. Therefore, the first significant component of the approach is to encode and learn the visual appearances of diverse cartoon characters. A memory-based learning structure is proposed to address the problems caused by visual diversity, which aims to achieve semantic consistency among different semantic parts. The memory-based consistency learning includes two parts, representation consistency and classification consistency. The representation consistency is designed to conduct consistency learning on semantic parts from different cartoon characters at high-dimensional semantic feature level. The classification consistency is proposed to achieve consistency learning of semantic parts from various cartoon characters at the semantic recognition level, which leverages the constraint information. Given a cartoon image 𝑋 , 𝐹 ∈ R 𝐶 ×𝐻 ×𝑊 is the encoded image features outputted by ConvNet. 𝐶 denotes the number of channels. 𝐻 and 𝑊 refer to height and width of the feature maps, respectively. A memory bank with three columns is defined, where the prediction column stores the two-dimensional prediction maps, the representation column saves one-dimensional feature representations, and the label column one retains labels. Specifically, 𝐹 is fed into a decoder with three convolution layers [4] to obtain the twodimensional feature maps 𝐹 𝑚𝑒𝑚 ∈ R 𝐶 𝑛 ×𝐻 ×𝑊 , where 𝐶 𝑛 denotes the number of classes. 𝐹 𝑚𝑒𝑚 is then stored in the prediction column of the memory bank. Additionally, the operation of information aggregation with global adaptive max pooling is performed on each feature map of 𝐹 , which outputs a feature vector 𝐹 𝑣 ∈ R 𝐶 ×1×1 . 𝐹 𝑣 is the representation of 𝑋 and it is stored in the representation column. Label 𝐿 corresponding to 𝑋 is stored in the label column of the memory bank, which aims to provide supervision for the semantic consistency learning. In the initial stage, all three columns of the memory structure are initialized as zeros. During the training process, image features of each iteration are stored in the memory bank. When the memory is full, the earliest stored features are deleted from memory to ensure that the memory bank has space for features of new iterations, which aims to continuously update the memory bank throughout the training process.During each iteration, 𝐹 not only gets stored in the memory structure but also undergoes further forward operations. A memory selection operation is performed to choose samples from the memory structure. These selected samples have some similarities to current sample and they are used to achieve semantic consistency, which aims to enhance the semantic reasoning of the network for targets sharing similar characteristics. Specifically, when inputting a new sample, the network selects similar samples from the memory structure. By leveraging the knowledge acquired from these learned samples, the network accurately infers and identifies current sample. To achieve memory selection, during each training iteration, a similarity calculation is performed between current 𝐹 𝑐 𝑚𝑒𝑚 and the stored 𝐹 𝑚 𝑚𝑒𝑚 within the memory structure. This is done using the Euclidean distance, which retrieves 𝑛 samples from the memory structure.The problem of inconsistent appearances among different images poses great challenges for cartoon parsing. To achieve consistency learning for the body parts that belong to the same class but from different samples, the network initially performs consistency learning at the representation level. The framework of the representation consistency learning is illustrated in Fig. 3 , which involves the integration and consistency of feature representations of current sample and feature representations of the relevant samples retrieved from the memory bank. Multi-head self-attention [30] is employed to achieve consistency learning among the semantic parts from different samples. It ensures that current sample and the samples stored in the memory structure are associated and consistent at the feature representation level.To conduct the representation consistency learning, 𝐹 is mapped into 1-dimensional feature representation 𝐹 𝑐 𝑣 with a global adaptive max pooling operation. Through the memory selection, 𝑛 feature representations 𝐹 𝑚,𝑛 𝑣 that represents the retrieved samples are selected for current sample. 𝐹 𝑐 𝑣 is then duplicated 𝑛 times and concatenated with 𝐹 𝑚,𝑛 𝑣 to aggregate features from diverse samples. The above operations are formulated as follows:EQUATIONwhere 𝐹 𝑐𝑎𝑡 𝑣 is the concatenated features. 𝐺𝑎𝑝 (•) denotes global adaptive max pooling operation. 𝐶𝑜𝑝𝑦 (•) is the duplication operation. 𝐶𝑎𝑡 (•) refers to concatenation operation. 𝑆𝑒𝑙𝑒𝑐𝑡𝑖𝑜𝑛(•) denotes the memory selection operation that retrieves the related samples from memory 𝑀. Subsequently, multi-perceptual (MLP) layers are adopted to create queries, keys and values for multi-head selfattention. It is formulated as,EQUATIONwhere 𝑊 𝑄 , 𝑊 𝐾 , and 𝑊 𝑉 are learnable weight matrices that project the input vectors 𝐹 𝑐𝑎𝑡 𝑣 into queries 𝑄, keys 𝐾, and values 𝑉 , respectively. Multi-head self-attention mechanism calculates the output vectors 𝐹 𝑜𝑢𝑡 𝑣 as a weighted sum of the input vectors 𝐹 𝑐𝑎𝑡 𝑣 , where the weights are determined by the attention scores. The attention scores are calculated as,EQUATION)where dot products of the query 𝑄 with all keys 𝐾 are computed and scaled by the scaling factor 1√ 𝑑 𝑘. 𝑑 𝑘 is the dimension of query. The softmax operation generates the attention scores. The dot products of the attention score and value 𝑉 associate the representations of different samples.Multi-head attention mechanism has 𝑡 heads and it performs the attention calculation in parallel for each head. The outputs of the heads are concatenated and projected back to the original dimension of 𝐹 𝑐𝑎𝑡 𝑣 , which is formulated as,EQUATIONEQUATIONwhere 𝑊 𝑄 𝑖 , 𝑊 𝐾 𝑖 , 𝑊 𝑉 𝑖 , and 𝑊 𝑂 are parameter matrices. The representation consistency learning with self-attention mechanism allows the model to attend to different representations of the same input sequence when calculating the output vectors 𝐹 𝑜𝑢𝑡 𝑣 . The consistent features, 𝐹 𝑜𝑢𝑡 𝑣 , is fused to the original features 𝐹 as,EQUATIONwhere 𝑆𝑖𝑔(•) denotes sigmoid operation that aims to map the values of vectors into calibration weights. ⊙ refers to element-wise multiplication.To achieve semantic consistency, supervisions are required for diverse body parts during the training process. Therefore, in addition to representation consistency learning, classification consistency learning is proposed and performed through loss weighting, which achieves semantic consistency among current sample and the samples retrieved from the memory bank. The labels of current sample and the retrieved samples are utilized to provide supervision. With the memory selection operation, predictions 𝐹 𝑚,𝑛 𝑚𝑒𝑚 of the retrieved 𝑛 samples are obtained, which are related to features 𝐹 of current sample 𝑋 . 𝐹 is mapped into category dimension for semantic prediction. The decoder of DeepLabV3+ [4] is adopted for the mapping, which is widely used in previous human parsing methods. It is formulated as, where 𝐹 ′ denotes the shallow features from ConvNet that provide spatial details for the deep features 𝐹 .EQUATIONTo ensure that the network consistently recognizes the body parts that belong to the same class in 𝐹 𝑐 𝑚𝑒𝑚 and 𝐹 𝑚,𝑛 𝑚𝑒𝑚 , the classification consistency is performed by calibrating the recognition results of 𝐹 𝑐 𝑚𝑒𝑚 using the recognition results of 𝐹 𝑚,𝑛 𝑚𝑒𝑚 . As can be seen in Fig. 4 , for each sample in 𝐹 𝑚,𝑛 𝑚𝑒𝑚 , losses of each sample are calculated for each category such as face and arm, which outputs 𝑛 loss values for the same category. The loss values of the same category are summed for the corresponding category, aggregating the losses from different samples. Subsequently, all summed losses are normalized and mapped to weights. The weights are multiplied by the loss of each category in 𝐹 𝑐 𝑚𝑒𝑚 . Consequently, the recognition of each category in 𝐹 𝑐 𝑚𝑒𝑚 is calibrated by the corresponding category in 𝐹 𝑚,𝑛 𝑚𝑒𝑚 , which achieves classification consistency learning. It is formulated as follows:EQUATIONL 𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑 = 𝐶𝐸 𝑐𝑙𝑠 (𝐹 𝑐 𝑚𝑒𝑚 , 𝐿) ⊙ L 𝑤𝑡𝑠 (10) where 𝐶𝐸 𝑐𝑙𝑠 (•) means modified cross-entropy loss function, which is used for loss computation of each class. 𝑆𝑢𝑚 𝑐𝑙𝑠 (•) denotes gathering the losses of the same class. 𝑁𝑜𝑟𝑚(•) refers to normalization on the summed losses of different categories. 𝐹 𝑐 𝑚𝑒𝑚 and 𝐹 𝑚,𝑛 𝑚𝑒𝑚 are predictions of the current sample and the retrieved samples, respectively. 𝐿 and 𝐿 𝑛 are their corresponding labels, respectively.In addition to appearance learning, structural learning is also crucial for cartoon parsing. Previous methods consider the correlation of different parts, but they overlook the learning of strongly related local structures. The pipeline of the main body-aware mechanism is illustrated in Fig. 6 , in which some local structures exhibit high correlations, such as the face, hair, and hat. Similarly, the thighs, shins, and shoes, or the palms, upper arms, and forearms, are also strongly related. Therefore, this paper introduces the main bodyaware recognition strategy, which focuses on structural modeling of strongly correlated structures to explore complex structures of cartoon images.Given 𝐹 𝑟𝑐 , pixel correlation learning (PCL) is adopted to capture pixel correlations of the cartoon image [23] . Feature aggregation is then performed using global average pooling along the channel dimension to obtain a one-dimensional feature representation. The one-dimensional feature representation is divided into 𝑛 𝑐 feature representations, which are treated as nodes of graph G = (V, E). 𝑛 𝑐 represents the number of categories. V denotes the set of nodes that correspond to body parts of cartoon characters. E refers to the set of edges that indicates the connection relations of adjacent body parts. For the construction of edge E, the adjacency matrix that indicates the connections of semantic parts is computed to associate body parts.GAT layers [31] are constructed to model the graph G. The framework of the GAT-based structure correlation is illustrated in Fig. 5 . The network first computes the attention score 𝑎 𝑖 𝑗 between nodes 𝑖 and 𝑗. For a node 𝑖 in graph G, the attention of its neighbor 𝑗 is defined as,EQUATION)where 𝑎 is a learnable vector. ℎ 𝑖 and ℎ 𝑗 are the feature vectors of nodes 𝑖 and 𝑗. 𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈 (•) is the activation function used for non-linearity. 𝐶𝑎𝑡 (•) allows the model to consider the features of both node 𝑖 and 𝑗. The features are transformed with weight matrix 𝑊 , which learns the importance of different node pairs. Softmax function is applied to the attention scores, which normalizes them to obtain weights 𝛼 𝑖 𝑗 . The attention coefficient 𝛼 𝑖 𝑗 represents the importance of node 𝑗 when updating the representation of node 𝑖, which is calculated as,𝛼 𝑖 𝑗 = exp(𝑎 𝑖 𝑗 ) 𝑘 ∈ N 𝑖 exp(𝑎 𝑖𝑘 ) (12)where N 𝑖 is the set of neighbors of node 𝑖.The new representation of node 𝑖, ℎ ′ 𝑖 , is calculated as a weighted sum of the transformed features of its neighbors, with weights given by the attention coefficients 𝛼 𝑖 𝑗 . This allows each node to selectively attend to its neighbors, capturing the importance of different connections in the graph. It is formulated as,EQUATIONwhere 𝜎 denotes to a nonlinearity operation [31] .The ultimately updated node representation ℎ ′ 𝑖 is obtained by employing multi-head attention mechanism [30] . 𝑆 independent attention mechanisms perform the transformation described in Equation 13. The output features are concatenated as the ultimately updated node representation ℎ ′ 𝑖 . Unlike the typical GAT, the GATs built are hierarchical, which are designed to model structural representations of global and local regions. Firstly, a GAT layer is adopted to model the pre-constructed graph structure, updates all node data and returns the results. Subsequently, the subgraph structures are built by merging strongly related nodes among the 𝑛 𝑐 nodes, and GAT is then applied to model the strongly related nodes and return the results. As can be seen in Fig. 6 , the graph of the left hand, left arm and torso is treated as a main body graph because they are strongly connected through the left arm. By feeding the paired nodes associated with the left arm into GAT for learning, the structural information related to the left arm is emphasized. The network facilitates the recognition of specific regions by focusing on the main body structure. After that, the learning results of various main bodies are aggregated and the integrated results are combined with the initially returned results from the global graph, producing the ultimate output. The output of the structure correlation is fused with the input features, resulting in stronger structural features that are used for accurate structural learning, which is formulated as,EQUATION)EQUATIONwhere 𝑆𝐶 𝐺𝐴𝑇 𝑠 (•) denotes the structure correlation function based on GATs. 𝐷𝑖𝑣𝑖𝑑𝑒 (•) is the operation dividing the one-dimensional feature representation into 𝑛 𝑐 nodes. 𝐻𝑒𝑎𝑑 (•) means a stacked module of concatenation and a convolution layer. 𝑃𝐶𝑀 (•) denotes part correlation modeling [23] .The objective function is composed of the proposed classification consistency loss, pixel correlation loss, part correlation loss, spatial loss and edge loss, which is formulated as, (16) where 𝐶𝐸 (•) is the cross-entropy loss function. 𝐶𝐸 (𝐹 𝑐 𝑚𝑒𝑚 , 𝐿) is adopted to predict segmentation results using features from Con-vNet. 𝐶𝐸 (𝐹 𝑝𝑐𝑙 , 𝐿) and 𝐶𝐸 (𝐹 𝑟𝑐 -𝑚𝑎𝑖𝑛 , 𝐿) are used to learn pixel and part correlations [23] . 𝐹 𝑝𝑐𝑙 is obtained by using a convolution layer to map the output of PCL into predictions. L 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 and L 𝑒𝑑𝑔𝑒 are utilized to learn spatial information [20, 23, 28] and edge information [14, 26] . L 𝑐𝑐 is the proposed classification consistency loss, which is defined as follows:L 𝑜𝑏 𝑗 = L 𝑐𝑐 + 𝐶𝐸 (𝐹 𝑐 𝑚𝑒𝑚 , 𝐿) + 𝐶𝐸 (𝐹 𝑝𝑐𝑙 , 𝐿)+ 𝐶𝐸 (𝐹 𝑟𝑐 -𝑚𝑎𝑖𝑛 , 𝐿) + L 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 + L 𝑒𝑑𝑔𝑒L 𝑐𝑐 = (𝐶𝐸 𝑐𝑙𝑠 (𝐹 𝑐 𝑚𝑒𝑚 , 𝐿) + 𝐶𝐸 𝑐𝑙𝑠 (𝐹 𝑝𝑐𝑙 , 𝐿)+ 𝐶𝐸 𝑐𝑙𝑠 (𝐹 𝑟𝑐 -𝑚𝑎𝑖𝑛 , 𝐿)) ⊙ L 𝑤𝑡𝑠 (17) 
experiments 4.1 datasets and evaluation metrics: 
 CartoonSet. The CartoonSet dataset [23] comprises 2,229 cartoon images, which are divided into 1,530 images for training, 510 for testing, and 189 for validation. It includes diverse samples such as drawings of children, illustrations with brief strokes and animated characters. The cartoon images are annotated and the body parts of cartoon characters are categorized into 24 classes. Cartoon dog. The Cartoon dog dataset [32] contains 965 images of cartoon dogs. It consists of 773 images for training and 192 images for testing. This dataset focuses on cartoon dog parsing, with eight classes such as head and body for labeling. LIP. LIP [8] is a human parsing dataset consisting of 50,462 singleperson images. It is divided into 30,462 training images, 10,000 testing images, and an additional 10,000 images for validation. It provides annotations for 19 classes, which include categories such as face and right arm.Evaluation Metrics. Following previous approaches [15, 22, 23, 32] , the utilized evaluation metric is Mean Intersection over Union (Mean IoU). Mean IoU computes the average intersection-to-union ratio between the predicted body parts and ground truth. Additionally, Mean Accuracy (Mean Acc.) is employed to calculate per-class accuracy and Pixel Accuracy (Pixel Acc.) is used to compute the accuracy of the correctly predicted pixels.To implement the proposed approach, Stochastic Gradient Descent (SGD) is employed as the optimizer with a momentum of 0.9 and a weight decay of 5e-4. Following previous methods [3, 7, 21, 38, 39, 42] , the "poly" learning rate strategy is utilized, which is defined as𝑙𝑟 = 𝑙𝑟 𝑏𝑎𝑠𝑒 × (1 -𝐶 𝑛 𝑇 𝑛 ) 𝑝𝑜𝑤𝑒𝑟 .𝑙𝑟 and 𝑙𝑟 𝑏𝑎𝑠𝑒 are the current learning rate and the base learning rate, respectively. 𝑝𝑜𝑤𝑒𝑟 = 0.9. 𝐶 𝑛 and 𝑇 𝑛 represent the current iteration number and the total iteration number, respectively. For data augmentation, random left-right flipping with a 0.5 probability and random scaling strategy are applied. For a fair comparison on CartoonSet, CartoonNet is trained with a batch size of 8 and image size of 384×384. The number of training epochs is 150 and the learning rate for CartoonSet is set to 7e-3. For Cartoon dog dataset, CartoonNet uses a learning rate of 1e-3, and the training images are resized to 384×384. All methods adopt single-scale evaluation on CartoonSet and Cartoon dog dataset. For LIP dataset, CartoonNet is trained with a learning rate of 7e-3, with training images resized to 473×473. The number of training epochs is 150. The multi-scale evaluation approach [14, 15] is employed for LIP dataset.To evaluate the performance of the proposed approach, it is compared with the state-of-the-art human parsing approaches and cartoon parsing methods. DeepLabV3+ [4] , a model not specialized for cartoon parsing, has limited performance in the domain of cartoon parsing. DFPNet [32] is designed for cartoon parsing and it boosts the performance by introducing a multi-scale learning mechanism. But it primarily focuses on scale variations, resulting in limited performance. Methods like HHP [35] and CNIF [34] employs GNN to model the structures of cartoon image, which improves the results with structural modeling. However, their hierarchical structure learning is designed for real humans with consistent and regular structures. When segmenting the irregular and complex body structures of cartoon characters, the performance is limited [23] . CE2P [26] leverages global context information and edge details to improve the results. But it fails to account for the visual diversity and structural complexity of cartoon images, resulting in limited performance. CDGNet [15] uses class distributions as spatial constraints, outperforming most of the methods. But it ignores the structural modeling and faces challenges when segmenting cartoon images with complex structures. SCHP [14] achieves improved results using a self-correction strategy, which helps differentiate visually complex body parts. Nonetheless, it still faces challenges caused by the diversity and complexity of cartoon characters. CP-Net [23] significantly improves the performance of cartoon parsing by addressing the problems caused by the irregular structures with To further evaluate the performance of the proposed method, it is tested on Cartoon dog dataset. Overall, the performance of previous human parsing methods is limited when tested on Cartoon dog dataset. The limitations are caused by the differences between abstract cartoon images and real-world humans. Cartoon parsing methods like CPNet [23] perform better because it accounts for the properties of cartoon images. But the diversity and complexity of cartoon images remain challenges, which limits the performance. Compared to previous methods, CartoonNet achieves the state-ofthe-art performance on Cartoon dog dataset, as listed in Table 2 . The specific design of CartoonNet effectively alleviates the challenges posed by the diverse visual appearances and abstract structures of cartoon dogs. The results listed in Table 2 further demonstrate the effectiveness of the proposed method.To verify the generalization ability of CartoonNet, it is tested on human parsing dataset LIP. The comparative results are listed in Table 3 . Methods like DeepLabV3+ [4] , OCR (ResNet101) [40] , OCR (HRNetV2-W48) [40] and HRNetV2 [33] are common semantic segmentation methods. Although these methods achieve outstanding performance on segmentation datasets such as Cityscapes [6] , their performance on LIP is limited due to the lack of specific designs for human parsing. Compared to the common semantic segmentation approaches, the human parsing methods like HHP [35] , SCHP [14] and CDGNet [15] achieve better performance. Their components such as GNN-based hierarchical structure learning module [34, 35] , self-correction strategy [14] and class distribution learning DeepLabV3+ [4] n/a n/a 52.09 CE2P [26] 87.37 63.20 53.10 CorrPM [44] 87.68 67.21 55.33 OCR (ResNet101) [40] n/a n/a 55.60 HRNetV2 [33] n/a n/a 55.90 OCR (HRNetV2-W48) [40] n/a n/a 56.65 CPNet [23] 88.29 68.41 57.21 CNIF [34] 88.03 68.80 57.74 HHP [35] 89.05 70.58 59.25 SCHP [14] n/a n/a 59.36 CDGNet [15] 88 [40] and HRNetV2 [33] by a large margin. Car-toonNet also outperforms the cartoon parsing method CPNet [23] significantly on LIP dataset, which proves its generalization ability. The performance of CartoonNet on LIP dataset demonstrates that the proposed semantic consistency and structure correlation approach is effective for human parsing as well.To validate the effect of the proposed method, ablation studies of semantic consistency and structure correlation are conducted. The results are listed in Table 4 . Compared to the baseline model, the incorporation of representation consistency and classification consistency leads to a significant improvement in Mean IoU metric. The memory-based consistency learning strategy facilitates the semantic consistency of the body parts that belong to the same class but from diverse cartoon images. When adopting the main body-aware structure correlation module, the performance is further boosted. The structure correlation module focuses on important local regions with main body-aware mechanism and captures important relations among body parts with graph attention network. The integration of the memory-based consistency module and the main body-aware structure correlation achieves the best results, confirming that the proposed method is effective in addressing challenges posed by diverse appearances and complex structures of cartoon parsing. To further verify the effect of the proposed modules, ablation studies on Cartoon dog dataset are conducted and the results are listed in table 5 . Compared to the baseline model, CartoonNet with the proposed modules significantly boosts the performance, which proves the effectiveness of the memory-based semantic consistency strategy and main body-aware structure correlation module.To verify the effect of the memory bank, ablation studies of the size of the memory bank and the number of selected similar samples from the memory bank are conducted. In and the data in the table indicates that the best result is obtained when the number of the selected samples is 5.To qualitatively evaluate the proposed method, the visualization results of CartoonNet and the state-of-the-art approaches are illustrated in Fig. 7 . For diverse cartoon characters, existing methods tend to make recognition errors when segmenting body parts that belong to the same class but exhibit significant visual and structural differences. For example, the heads of panda and dog are quite different. Previous methods like CPNet [23] have difficulties in segmenting diverse cartoon images as they primarily focus on visual appearances within a single cartoon character. The proposed method conducts semantic consistency learning across different cartoon characters with a memory bank to recall experiences for previous samples, achieving better results. Previous methods also have limited performance when the targets have intricate and irregular structures such as the hair of the girls, as they lack specialized recognition structures for the irregularities in cartoon characters. Among the previous methods, CPNet [23] performs better by considering the irregularities of cartoon characters. However, it still falls short in segmenting complex structures as it ignores the main bodies. The proposed method introduces the main body-aware strategy, which addresses the challenges related to complex structures by focusing on core local regions and capturing important relations among body parts.Although CartoonNet achieves the state-of-the-art performance, it still faces some challenges. The failure examples are illustrated in Fig. 8 . When the bodies of cartoon characters are significantly occluded by other objects such as cups, it is difficult for CartoonNet to correctly segment the cartoon image. In addition, if the adjacent body parts have similar appearances, for example, the black hat and black hair, CartoonNet may make some error predictions. 
conclusion: 
 In this paper, a cartoon parsing method named CartoonNet is proposed to address the challenges posed by diverse appearances and complex structures of cartoon characters. It consists of a semantic consistency approach based on a memory bank and a main body-aware structure correlation module based on graph attention networks. The memory-based consistency learning achieves semantic consistency among diverse cartoon characters, which explores deep relations of the body parts that have different appearances but belong to the same class. The main body-aware structure correlation boosts the performance of complex structures by focusing on the core body parts and capturing important relations among the body parts. By integrating the semantic consistency strategy and the structure correlation module, the proposed method achieves the state-of-the-art results in cartoon parsing. 
: 
 Cartoon characters are an important part of culture and media with their diverse appearances and abstract structures. From human-like cartoon characters to animal-like cartoon images, cartoons exhibit a wide range of styles. The task of cartoon parsing, which aims to segment different parts of cartoon characters, holds substantial importance for various applications such as cartoon animation, content creation and virtual worlds. Due to the inherent variations in cartoon appearances and lack of uniformity in cartoon structures, identifying diverse visual appearances and complex structures of cartoons is a challenging task.Cartoon images have various visual appearances and abstract spatial structures, which are illustrated in Fig. 1 . Unlike real-world humans, the styles of cartoon characters may vary depending on author or cultural background, which results in the diversity and complexity of cartoon images. For human-like cartoon characters, their appearances and structures are quite different. The properties of cartoon images make the body parts such as limbs, head and body of human-like cartoon characters abstract and complicated, which are not consistent with real-world humans. Animal-like cartoon images are more complex due to diverse animal categories. For different animal species such as fish, mammals, and reptiles, their visual appearances and body structures are different when The main body-aware structure correlation focuses on correlation of body structures, modeling the structure of cartoons. PCL and PCM refer to pixel correlation learning and part correlation modeling [23] , which are employed to solve the irregularities of cartoon images.presented in the form of cartoon images. Therefore, cartoon parsing remains a challenging task.For image parsing, existing methods have primarily focused on human parsing and achieved significant progress. Previous human parsing methods have explored hierarchical structure learning [12, 34, 35, 43] , pose estimation-guided learning [19, 41] and the integration of self-correction strategy [14] . However, due to the significant differences between real humans and cartoon images, these human parsing methods exhibit limited performance in cartoon parsing. With the development of cartoon related applications, the advanced deep learning methods [23, 32] have been applied to cartoon parsing. They employ multi-scale learning structures, and leverage pixel and part correlation to infer irregular cartoon structures. However, the diversity and complexity of cartoon images remain a challenge.In this paper, CartoonNet is proposed to address the diversity and complexity of cartoon images in cartoon parsing. It augments cartoon parsing by designing a memory-based semantic consistency structure and a graph attention network-based structure correlation module. The framework is illustrated in Fig. 2 . The semantic consistency module encodes and learns the diverse appearances displayed by cartoon characters. It designs a memory-based learning structure to conduct consistency learning of diverse body parts. The memory-based semantic consistency learns the visual differences of cartoon characters, which aims to alleviate the rich diversity of cartoon images and improve generalization ability. The structure correlation approach focuses on modeling and learning the intricate and irregular structures exhibited by cartoon characters. It leverages graph attention networks (GATs) [2, 11, 31] and a main body-aware mechanism to facilitate structural learning and correlation. By modeling the structural features of cartoon images across the main bodies, the method focuses on important parts of cartoon images to alleviate the structural complexity of cartoon parsing. Experiments conducted on cartoon parsing and human parsing datasets demonstrate the effectiveness of the proposed approach, which outperforms the state-of-the-art methods for cartoon parsing and achieves competitive results on human parsing. The contributions are summarized as follows:• CartoonNet is proposed to address the challenges of visual diversity and structure complexity in cartoon images, which designs a memory-based semantic consistency module and a main bodyaware structure correlation strategy for cartoon parsing. • A memory-based learning structure is proposed to facilitate appearance learning, which captures the inconsistent appearances of different cartoon characters and utilizes self-attention mechanism to achieve semantic consistency among diverse body parts. • A main body-aware structure correlation method is proposed to model the structural information of cartoon characters with complex structures, which employs the graph attention networks and a main body-aware mechanism to focus on the important structures. • The proposed method achieves state-of-the-art performance on the cartoon parsing datasets, which demonstrates the effectiveness of CartoonNet. 
 
 Appendix:  


Decision: The draft received one weak reject, one borderline reject and one weak accept for initial ratings. Main concerns were raised on technique clarity, relation to previous works, generalization, etc. The authors provided a rebuttal, which did not convince the first reviewer but successfully addressed issues raised in the other two reviewers. As a result, the final ratings were boosted to borderline reject, border accept and accept. Checking all the reviews and authors' feedback, we agree with the majoratiy of the ratings and would recommend the work to ACM MM.

Reviews: review: paper_summary: paper_summary: This paper studies the cartoon paring problem, and proposes the CartooNet to recognize and segment various parts of cartoon characters, where semantic consistency and structure correlation are integrated to address the visual diversity and structure complexity.
A memory-based semantic consistency module is designed to center on encoding and learning the multifaceted appearances exhibited by
cartoon characters. 
A structure correlation module is proposed to enable structural learning and correlating, allowing it to recognize and parse cartoon images with significant complexity.
strength_weakness: strength_weakness: - The technical details are clearly stated about the semantic consistency and structure correlation.
- A memory-based learning structure is proposed to facilitate appearance encoding, storage, selecting and consistent learning.
- A main body-aware structure correlation method is proposed to enable recognizing cartoon characters with structural complexity.
limitations: limitations: - The technical contributions of this work are increment. The presented methodology and technology in Representation Consistency are pretty common. 
-  In structure correlation, the presented idea is similar to this work CPNet [20]. 
-  In evaluation, the current results can't fully verify the effectiveness of the proposed CartooNet. To be specific, the performance gain is weak on the dataset CartoonSet, and the results of the CartooNet are inferior to the baselines on the LIP dataset(88.59% vs 89.05%、69.57 vs 71.49 and 58.27 vs 60.30 w.r.t., Pixel Acc、Mean Acc and Mean IoU )
paper_summary: paper_summary: This paper proposes a new network, CartoonNet, for recognizing and segmenting different parts of cartoon characters. Two modules are proposed to address the challenges of diverse semantic representations and complex human structures in cartoon characters. And good results were achieved on both the cartoon parsing dataset and the human parsing dataset.
strength_weakness: strength_weakness: 1. The memory based semantic consistency mechanism is very convincing, and can fully solve problems with diverse semantic expressions both theoretically and experimentally.
2. By learning the internal structure of the target, it is possible to more accurately encode the correlation between the target's local features and the overall features.
3. In addition, CartoonNet has achieved good results on both cartoon parsing datasets and single person human parsing datasets.
limitations: limitations: Although I strongly support the author's idea of using memory to address external diversity within the same category, I have significant opinions on its implementation methods and paper descriptions.
1. from the perspective of theoretical novelty, memory based semantic feature matching or unified technology is usually applied in Re-ID, Track, VOS and other fields, and in the field of part analysis, VLParts[1] also adopted a similar sample matching approach in the open-world part segmentation challenges. And semantic feature consistency and body structure analysis are not uncommon in the field of Human Parsing, such as UniParser[2] and sematree[3]. Therefore, I believe that the paper has novelty in its ideas, but it is not outstanding.

[1] Sun, Peize, et al. "Going denser with open-vocabulary part segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[2] Chu, Jiaming, et al. "UniParser: Multi-Human Parsing with Unified Correlation Representation Learning." arXiv preprint arXiv:2310.08984 (2023).
[3] Ji, Ruyi, et al. "Learning semantic neural tree for human parsing." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16. Springer International Publishing, 2020.

2.My biggest negative feedback on this paper comes from its writing and expression, as the description of the connections between modules in the paper is too poor, which makes it difficult for me to grasp the author's original intention. Even if MM accepts this article, I suggest making significant revisions. I will not provide the detailed description of writing problem, like formula formatting misalignment and inappropriate size and position of icon text, but they need to solve. The description of each module in the paper is too independent, and the description of the model process in Figure 2 is too rough, making it difficult for readers to clearly understand the training and inference process of the model.
In addition, I also have doubts about the design scheme of each module. For example, in the representation consistency module, it is necessary to unify the semantics of the current feature with the semantics of the reference feature. So, how can we obtain the semantic results of the current image feature without a semantic segmenter in front of the model? If using groundtruth to achieve consistency, does it indicate that the consistency module is not working during inference? These should be described in detail in Chapter 3 and Figure 2. The model structure design is also relatively simple and common, lacking novelty.
paper_summary: paper_summary: The paper "CartoonNet: Cartoon Parsing with Semantic Consistency and Structure Correlation" introduces a novel approach, CartoonNet, for parsing cartoon characters. It focuses on recognizing and segmenting various parts of cartoon characters by integrating semantic consistency and structure correlation to address the challenges posed by the visual diversity and structural complexity of cartoon images.
strength_weakness: strength_weakness: Innovative Approach: CartoonNet proposes a unique method that combines semantic consistency and structure correlation to effectively parse cartoon characters with complex appearances and irregular structures.
State-of-the-Art Performance: The proposed method achieves state-of-the-art performance on cartoon parsing datasets, demonstrating its effectiveness in handling the multifaceted appearances of cartoon imagery.
Main Body-Aware Structure Correlation: The incorporation of a main body-aware structure correlation method enables the recognition of cartoon characters with substantial structural complexity, addressing the inherent challenges in cartoon structures.
limitations: limitations: Domain Specificity: The approach is tailored specifically for cartoon parsing and may not generalize well to other types of image parsing tasks.
Dataset Dependency: The performance of CartoonNet is evaluated on specific cartoon parsing datasets, and its generalizability to a broader range of cartoon styles and structures may need further exploration.