Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning

Authors: yi  bin
junrong  liao
yujuan  ding
haoxuan  li
yang  yang
see-kiong  ng
tao  heng
  shen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 11:13:28

Abstract: Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information, thereby understanding and creating content of the physical world coherently like human-beings. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. Despite of the effectiveness, labeled associated coherency information is not always available and might be costly to acquire, making the cross-modal guidance hard to leverage. To tackle this challenge, this paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency, and proposes the Weak Cross-Modal  Guided Ordering (WeGO) model. More specifically, it leverages high-confidence predicted pairwise order in one modality as reference information to guide the coherence  modeling in another. An iterative learning paradigm is further designed to jointly optimize the coherence modeling in two modalities with selected guidance from each other. The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality. Experimental results on two public datasets have demonstrated that the proposed method outperforms existing methods for cross-modal coherence modeling tasks. Major technical modules have been evaluated effective through ablation studies. \textcolor{blue}{Codes are available at: \textit{\url{https://github.com/scvready123/IterWeGO}}}.

Main paper: 
 introduction: 
 Humans understand the world, to comprehend its semantics and dynamics, based on their organization and structuring of the information they perceive from various sources, seeking patterns and sequences that provide coherence to their understanding. In general, humans have remarkable skills in coherence modeling regardless of the information modality, which might be acquired from some childhood activities such as Picture-Telling [16, 44] . Such abilities are also essential to Artificial Intelligence (AI) models, which not only enhance the decision making by providing insights into the relationships and dependencies between different pieces of information [37] , but also enable systems to understand context better [62] ,Sentence Set to Order 'their dad is there ring leader .' 'he gathers the guys around out back .' 'his mom babysits while he conducts the meetings .'Figure 1 : Uni-modal and Cross-Modal (CM) Coherence Modeling (CM) tasks shown by a sentence ordering case according to whether using the guidance from another modality. This paper focuses on the CMCM task without using cross-modal ORDER information but only leverages weak guidance across modalities as shown in the green part.leading to more contextually relevant responses and actions. Coherence modeling may further benefit down-stream applications such as storytelling, educational materials and content marketing.For these significance, coherence modeling has been an active research topic in AI research. For example, in the area of natural language processing (NLP), sentence ordering [4, 14, 38, 49] aims to organize a set of sentences into a coherence piece of text with a logically consistent order, which is one of the fundamental textural coherency modeling problems. Coherence modeling is also a keen topic in the era of Large Language Models (LLMs) since it has been involved in various areas, from the text summarization in NLP [62] to the video understanding in Computer Vision (CV) or even other multi-modal tasks [18, 40, 57, 61] . Existing effort has been made to investigate the coherence modeling capability of existing LLMs in different tasks, and demonstrated that state-of-the-art LLMs like GPT-4 have not achieved human-level ability in terms of coherence evaluation [52] , leaving coherence modeling under-explored.Coherence modeling is challenging due to the complex and diverse relations existing to determine the logic for the construction of different pieces of content [11, 21] , including temporal relations, casual relations, discourse relations and many others. For example, in the sentence ordering task, it is required to first understand the semantics of each sentences, based on that to discover relations existing between different sentences and predict the order accordingly. From another perspective, exploring coherency patterns based on only single-modality data is difficult due to the limited information from only a single view, making the underlying logic difficult to explore [34, 41] . In comparison, Human perception is internally cross-modal, which facilitate them to better capture the dependency between different segments of content in any modality [28, 41] . Inspired by cross-modal human cognition [10, 36, 46] , a previous study [9] proposed cross-modal coherence modeling method, which leverages additional information from another modality and use it as the reference to learn the order in target modality. The crossmodal correspondence and the order information in the additional modality may offer useful guidance. However, despite of the effectiveness, this method requires gold labels for the order information from the additional modality, which is not always available and would be expensive and time-consuming to acquire.To further address this limitation, this paper introduces a more flexible cross-modal coherence modeling approach, which leverages the predicted instead of labeled order information in one modality to enhance the order modeling in another, as shown in the bottom part of Figure 1 . The basic idea is that the two modalities for cross-modal coherence modeling are semantically correlated, if there is coherence information observed in one modality, the other modality could benefit from it as long as the coherence information is correlated and reliable. To be specific, we introduce the Weak Cross-modal Guided Ordering model (WeGO), which employs the predicted pairwise order in one modality as the reference information to guide the order prediction in another modality. Such weak guidance is selectively applied based on the prediction confidence of the reference order and the semantic correlation between the specific elements across modalities. An Iterative Learning paradigm is further designed to optimize the ordering models of two modalities jointly, one at each time with feasible guidance from another, making the proposed method to be IterWeGO. At the inference stage, cross-modal order guidance functions iteratively for different modalities for multiple steps to boost uni-modal order prediction. The main contributions of this paper are as follows:• We work on cross-modal coherence modeling task and propose a novel cross-modal guidance method WeGO, which effectively take advantage of relevant but weak order reference from another modality to guide the order modeling in the target modality. • We propose an iterative learning paradigm to leverage weak cross-modal order guidance for both modalities during training, and learn the ordering models of two modalities jointly. • We evaluate the proposed method on two public cross-modal coherence modeling datasets and compare it with existing competitive baselines. Experimental results demonstrate that the proposed method consistently outperforms all compared methods by large margins, showing great superiority of it. 
related works 2.1 coherence modeling: 
 Coherence modeling has been studied in various tasks in literature, including the most representative two: Sentence Ordering in NLP and Visual Storytelling in multimedia understanding.Sentence ordering is representative coherence modeling task in the area of NLP, with early-stage methods mostly built based on domain knowledge and language-based features [4, 8, 19, 27, 39] . For example, these methods use vectors of linguistic features in probabilistic transition models. In recent years, with the development of deep learning and NLP technologies, more advanced methods apply an encoder-decoder framework and retrieve the final order using pointer networks [14, 26, 43, 53, 58] . Topological sorting is also applied to address the sentence ordering problem [47, 49] . With the boost of pre-trained language models, BERT-based [15] , BARTbased [13] and other pre-trained language model-boosted sentence ordering networks have been proposed and achieved competitive performance. Graphs are applied to enhance the sentence representations for the sake of ordering learning [59, 63] , also to implement set modeling without orders and more suitable for the input.Storytelling is a higher-level coherence modeling task for coherency modeling, which requires to generate the sentences that can be ordered properly to construct a coherent story [30, 33] . Visual storytelling is a further extension of storytelling with multimodal input, which requires the generated story to be consistent to the given visual content [1, 29] . Such a cross-modal coherence modeling task help to develop human-like artificial intelligent methods that understand the grounded event structure that go beyond descriptive language only [42] . Early methods mostly leveraged CNNs and RNNs into an integrated encoder-decoder architecture to comprehend visual input and generate textual output respectively [12, 55] . More recent methods attempted to enhance the reasoning or planing abilities of the model by applying more advanced or large-scale pre-trained modules. For example, Liu et al. [42] proposed to introduce the planning procedure for visual storytelling relying on pre-trained language models, which help to generate less repetitive, more logical stories with more details. Although both visual storytelling and our cross-modal ordering task target at cross-modal coherence modeling, we are faced with unordered materials for both modalities while the visual storytelling offers well-ordered image sequences as grounding reference for the text generation.Cross-modal learning generally aims to learn a unified representation of two modalities, mostly vision and language. It plays an important role in many different cross-modal tasks, such as visionlanguage retrieval [6, 31, 56] , vision question answering [3, 48, 60] , vision-language alignment [9] , multi-modal sentiment classification [24, 32] , visual captioning [5, 7] , cross-modal relation extraction [17, 20] , etc. With the extraordinary development of the largescale pre-training, Vision-Language pre-training (VLP) has become a mainstream research direction recently. The goal of VLP is to learn unified representation of different modalities through largescale learning based on large-scale image-text pair data, which can further benefit the above-mentioned downstream tasks. With a dualstream architecture, CLIP has exhibited remarkable performance on zero-shot recognition and several downstream tasks by applying effective contrastive learning [51] . For further enhancement, Pyra-midCLIP [23] introduced fine-grained interactions between two modalities to achieve better cross-modal alignment. CyCLIP [25] introduced geometrical consistency constraints. SoftCLIP [22] relaxed the one-to-one cross-modal alignment constraint and instead introduced intra-modal guidance to enable many-to-many relationships between the two modalities, which further enhance the pre-training performance cross-modal learning model.Cross-modal learning is generally a sub-field of multi-modal machine learning [2, 45, 54] . From another perspective, it aims to use data from additional modalities to improve a uni-modal task. Lin et al. [41] proposed a cross-modal learning method by treating data from other modalities as additional training samples to help with the uni-modal few-shot learning. Jin et al. [31] proposed to apply the idea of Banzhaf Interaction to explicitly capture the finegrained semantic relationships between vision and text, and use it as additional learning signals to improve the contrastive crossmodal learning. Qin et al. [50] investigated how to improve the correspondence across visual and textural modalities in cross-modal learning, which might contain ubiquitous noise in labeled pair data. Ju et al. [32] proposed joint learning method for sentiment analysis by incorporating the image-text relation to leveraging the visual guidance for text understanding. With a similar idea, cross-modal coherence modeling task aims to leverage the additional guidance from another modality to facilitate the coherence model learning in one modality. However, different with previous work, this study tries to leverage not only the cross-modal semantic information, but also the order information to specifically help with the uni-modal coherence modeling task. 
method: iterwego: 
 In this section, we introduce the proposed Weak Cross-modal Guided Ordering model with Iterative Learning (IterWeGO). In the following part, We first introduce single modal encoders for the semantic and context encoding of multi-modal input. Then, we detail the cross-modal guidance module to leverage weak order as reference. In the end, we demonstrate the iterative boosting mechanism for learning and inference, which enables to enhance ordering learning and prediction in two modalities simultaneously.This paper studies the cross-modal coherence modeling, specifically on the task of sentence and image ordering problem. Given a set of sentences and a set of images, both unordered, the task is to learn to organize the elements in each set with proper orders based on the semantics so that the two parts of content would make one coherent story, either in natural language or visual illustration.Given an image set X 𝑖 consisting of several unordered images {𝑋 𝑖1 , 𝑋 𝑖2 , ...𝑋 𝑖𝑀 }, where 𝑁 = |X 𝑖 | denotes the number of images in the set. Each image 𝑋 𝑖𝑘 ∈ X 𝑖 can be analyzed by a visual model 𝐹 , i.e., a semantic encoder to extract the visual content features as 𝑒 𝑒 𝑒 𝑖𝑘 = 𝐹 𝜎 (𝑋 𝑖𝑘 ), where 𝜎 denotes the parameters in the model 𝐹 ().It is important to exploit semantic relations between different elements, i.e., images or sentences, for recovering coherence in the set. Towards this end, we further design an intra-modal context encoder equipped with the scaled dot-product self-attention to obtain the contextual representation [8] . Context-aware representations for the set of images can therefore be obtained as Ê 𝐸 𝐸 𝑖 = 𝑀𝐻 _𝐴𝑡𝑡 𝛽 (𝐸 𝐸 𝐸 𝑖 ), where 𝐸 𝐸 𝐸 𝑖 is the matrix composed of the content representations of all images in the set as 𝐸 𝑖 = {𝑒 𝑖1 , 𝑒 𝑖2 , ...𝑒 𝑖 | X 𝑖 | }. 𝛽 denotes the parameters in the context encoder 𝑀𝐻 𝐴 𝑡𝑡 (). Similarly, for the input sentence set S 𝑗 = {𝑆 𝑗1 , 𝑆 𝑗2 , ..., 𝑆 𝑗 𝑁 }, where 𝑀 = |S 𝑗 |, each sentence 𝑆 𝑗𝑘 can be analyzed by a semantic encoder 𝐺 𝜂 first to obtain the content representation 𝑒 𝑒 𝑒 𝑗𝑘 . The whole set of content representations is further passed to a context encoder 𝑀𝐻 _𝐴𝑡𝑡 𝛾 similar as images to obtain context-aware representation Ê 𝐸 𝐸 𝑗 . 𝜂 and 𝛾 are trainable parameters for the encodes.Based on the context-aware representations, we can further model the relative order of any two elements (in the textual or visual modality) by devising a pairwise order classifier ℎ 𝛿 𝐼 (𝑥 1 , 𝑥 2 ) with trainable parameters 𝛿 𝐼 . For example, for two images 𝑋 𝑖1 , 𝑋 𝑖2 with context-aware representations being ê 𝑒 𝑒 𝑖1 , ê 𝑒 𝑒 𝑖2 , their pairwise order result turns to be 𝑜 𝑖 12 = ℎ 𝛿 𝐼 (𝜌 (ê 𝑒 𝑒 𝑖1 , ê 𝑒 𝑒 𝑖2 )), where 𝜌 is the integration function, for example, can be the concatenation operation. We formulate pairwise order prediction as a binary classification problem and apply a logistic function further to predict the probability of two situations (𝑋 𝑖1 > 𝑋 𝑖2 or 𝑋 𝑖2 > 𝑋 𝑖1 ) as 𝑠 𝑖 12 = Sigmoid(𝑜 𝑖 12 ).For a target image set X 𝑖 , there are in total 𝑛 2 pairs. All pairwise order probability scores of the set consist of an 𝑁 × 𝑁 square, corresponding to the number of images in the set, which can be denoted as 𝐵 ∈ R 𝑁 ×𝑁 . Similarly, for each sentence set we have the order matrix 𝐴 ∈ R 𝑀 ×𝑀 in which each element denotes the pairwise order probability of the corresponding two sentences.To explore deeper connections between two modalities and enhance the semantic alignment between the associated image and text sets, we further leverage cross-modal semantic similarity modeling as illustrated in Figure 2 . Specifically, we apply a pretrained cross-modal models on the input sets of the two modalities respectively, and further obtain the cross-modal similarity matrix 𝐶 ∈ R 𝑀 ×𝑁 . Each element 𝑐 𝑚𝑛 ∈ 𝐶 measures the semantic similarity of the two corresponding elements from two modalities based on their cross-modal content representations.With the cross-modal semantic similarity matrix, we can apply cross-modal order guidance based on a simple idea that the prediction of pairwise order in one modality is highly related to the corresponding pairwise score in another modality. The cross-modal similarity matrix serves as a bridge to link two modalities to ensure the order guidance is only applied between semantic-aligned cross-modal elements. Furthermore, our cross-modal guidance is only invoked when the prediction confidence of the pairwise score in the referred modality is high, making it qualified to become a guidance. To effectively leverage cross-modal order guidance, we design a cross-modal guided order matrix updating (CGO-MU) algorithm to update the pairwise order matrix in one modality. For example, the order matrix of the image set 𝐵 can be refined with the CGO-MU algorithm based on the order matrix of the sentence set 𝐴 and the cross-modal similarity matrix 𝐶 as 𝐵 ′ = CGO-MU(𝐵|𝐴, 𝐶). Detailed steps of CGO-MU is introduced in Algorithm 1, according to which we can also derive the way to obtain the refined sentence order matrix by 𝐴 ′ = CGO-MU(𝐴|𝐵, 𝐶) .As illustrated in the upper part in Figure . 2, we introduce an Iterative Boosting (IB) mechanism to enable the cross-modal guidance to be applied and benefit the ordering model learning in both the visual and textual modalities. The iterative learning paradigm is applied to further optimize the uni-modal modules of image and text ordering iteratively with cross-modal guidance being leveraged and updated along with the iterative training. The optimization is implemented based on the cross-entropy loss on the pairwise order prediction, which is specifically as follows if taking image ordering learning as an example: where 𝑏 ′ 𝑘𝑙 ∈ 𝐵 ′ is refined predicted pairwise order of the elements 𝑋 𝑘 and 𝑋 𝑙 (not that for simplicity we omit 𝑋 and denote images by their index as 𝑘 and 𝑙 in the above equation). 𝑦 𝑖 𝑗 is the groundtruth label of the corresponding order, which could be 0 or 1. X is the collection of all image sets in the training data. All training parameters involved in the images ordering model (𝜎, 𝛽 and 𝛿 𝐼 ) get updated after the optimization. Similarly, the sentence ordering loss can be defined as:EQUATIONEQUATIONwhere 𝑎 ′ 𝑘𝑙 ∈ 𝐴 ′ and 𝑧 𝑘𝑙 are the refined predicted and groundtruth pairwise order of the element pair < 𝑆 𝑘 , 𝑆 𝑙 > respectively. Optimizing L 𝑆𝑒𝑛 updates all parameters involved in the sentence ordering process.In inference, we use topological sort algorithm to find the order of the whole image/sentence set based on the pairwise order predicted by our cross-modal coherence model. Specifically, we build a ordering graph (directed) for each set, in which each element (image or sentence) serves as a node and the pairwise order suggests the directed edge between the two corresponding nodes. For each node, we can calculate its overall order score by summarizing the in/out edges, and sort the score of all nodes to obtain the final order list for the target set. Iterative boosting functions during inference too, which enables to update the order prediction iteratively across modalities for several times to achieve better accuracy for ordering in each modality. 
experiments and results: 
 To evaluate the effectiveness of the proposed approach, we conduct extensive experiments on two standard coherence modeling datasets, i.e., SIND [30] and TACoS-Ordering [9] .We apply ViT and BERT as the content encoder to process images and sentences respectively, and conduct fine-tuning on two unimodal ordering tasks. The contextual encoders for text and image share the same structure, which is combined by Self Attention Layers and Feed Forward Layers [9] . The cross-modal alignment module relying on semantic similarity is implemented by a CLIP model. This model is fine-tuned before applying in the IterWeGO with the cross-modal pairs in the target dataset in a contrastive learning manner. Threshold to select valid cross-modal guidance is set to 0.8 for image-to-text and 0.9 for text-to-image respectively based on empirical results. The setting of threshold is a trade-off between the quality and quantity of the leveraged cross-modal guidance. Empirical results suggest that the performance of the model stays stable when it is set within a certain range. During training, we employ Adam [35] optimizer to minimize the loss, with initialized learning rate of 2 × 10 -4 . The batch size of SIND is set to 64, while TACoS-Ordering is 32. We apply 8 parallel heads for all the multi-head attention layer and the hidden size is 768. In inference, we apply multi-step cross-modal guidance to enhance the order prediction in each modality untill the best performance is reached. In accordance with previous work [8, 9] , we apply three metrics to evaluate the performance of different methods for cross-modal ordering tassk, which are Accuracy (short in Acc), Perfect Match Ratio (PMR) and Kendall's Tau (𝜏).We compare the proposed IterWeGO model with the following competitive coherence modeling approaches.• LSTM+PtrNet [43] : This method uses an LSTM-based encoder to obtain the context representation and the pointer network as the decoder to predict the order. • L-TSort [49] : It implements topological sort to optimize the relative order constraint between paired sentences. • AttOrderNet (AON) [14] : It applies self-attention-based encoder and autoregressive decoder, which is very similar to our work except the autoregressive decoding. Following [9] , we test two versions, AON-UM and AON-CM, for comprehensive comparison. AON-UM exploits single-modal data only, which is just as applied in the original paper. AON-CM is the expanded cross-modal version which takes data from both modalities as input.• RankNet [38] : It also applies similar encoders with AON and our NACON, but employs ranking strategy for the output order. We also implement two versions of single-modal and cross-modal, RankNet-UM and RankNet-CM, for comparison. • NACON [9] : It is the state-of-the-art coherence modeling method employing a basic encoder-decoder framework with an unique non-autoregressive decoder to tackle with the permutation challenge for order prediction. It leverages cross-modal guidance with ground-truth orders to boost the coherence modeling of the singe modality. The overall performance of all compared methods, including the proposed IterWeGO and the baselines, are shown in Table . 1. From the table we have the following observations.• Compared to the methods without using additional cross-modal order labels, the proposed IterWeGO method outperforms them all on both the Sentence Ordering and the Image Ordering tasks across the two datasets, showing consistent effectiveness in the general coherence modeling task. We can also observe that the leveraged cross-modal guidance (in Sec. 3.3) and the contextual encoding (in Sec. 3.2) are both effective by comparing the performance of IterWeGO with two variants: -UM and -UM-NoCon. • The proposed IterWeGO gains more performance improvement in the image ordering task than for the sentence ordering task.From the overall results we can see that all methods perform better for sentence ordering than image ordering, which suggests that sentences are relatively easier to be ordered than images, and images ordering are more challenging. Since our IterWeGO leverages weak cross-modal guidance, which is predicted relative order information instead of the pre-obtained gold labels, it is more effective when the modality to receive the guidance is challenging and the modality to offer the guidance is easy. Image ordering with sentence order guidance is exactly such a task, in which case the weak guidance is more reliable. In comparison, offering image order guidance for sentence ordering tries to apply a weaker model to facilitate a stronger model, which therefore is more difficult. Nevertheless, our IterWeGO still achieves significant performance improvement comparing other methods, showing general effectiveness of the idea of leveraging crossmodal order guidance to boost the uni-modal ordering. • By comparing different metrics, we further observe that our method gain relatively more performance in terms of PMR and 𝜏, which measure exact match and relative orders respectively. Such results demonstrate good capability of our IterWeGO to 1 also demonstrates the noticeable difference of performance of IterWeGO across datasets. The effectiveness of our model seems to be more obvious on the SIND dataset than TACoS-Ordering, suggesting that the idea of cross-modal order guidance is particularly useful when the images and sentences are better aligned and matched. • We also illustrate the performance of two NACON models at the bottom two rows in the table. Specifically, NACON denotes original method [9] and NACON* denotes the advanced version using ViT and BERT to replace the image and sentence encoders in the original model to keep consistent with the IterWeGO method. Both NACON and our IterWeGO incorporate cross-modal order guidance. Although the performance of NACONs seems to be higher, it needs to be noticed that NACONs use gold order labels while our method only relies on predicted relative order information, which is much weaker guidance comparatively. From this perspective, NACONs may be seen as upper-bound reference to our IterWeGO model. While compared to NACONs, our model is also more feasible to apply since it does not need such gold labeled order annotations. To verify the effectiveness of our iterative boosting strategy, we conduct ablative experiments on SIND dataset and illustrate the results in Table 2 . The first two rows of results correspond to the models without applying cross-modal iterative boosting during training. As shown from the results, applying iterative inference only can boost the ordering performance. When we implement iterative boosting strategy during both training and inference stages, the performance achieves the best owing to the synchronization signal between training and inference. We also note an interesting phenomenon that solely apply iterative training yields a slightly lower result comparing to without using it (w/o IB in I v.s. w/o IB in T&I ). This may result from the discrepancy between training and inference, because in the case of w/o IB in I, the inference is conducted in uni-modal only, which is different from the training stage leveraging cross-modal guidance. But if the cross-modal guidance, aka, the iterative boosting is also applied during inference, the performance will significant improve (w/o IB in I v.s. IterWeGO).We take a further step to investigate the mechanism of our iterative strategy by delving into the iteration steps during inference. Specifically, we implement 10 iteration steps during inference and measure the performance for each step. The variation trend across steps is shown in Figure 3 . From the figure, we observe that all the methods gain significant improvements at the first iteration, and gradually reach the plateau values. The cross-modal training setting (our IterWeGO) exhibits boosting for more steps and finally outperforms the uni-modal version by a large margin, which verifies the effectiveness of the iterative strategy during both training and inference stages.One important technical part in the proposed IterWeGO is to leverage the weak relative order guidance across modalities to enhance the uni-modal ordering learning. Although the overall performance of IterWeGO has been demonstrated superb, it is still not clear whether the cross-modal order guidance contributes to the overall performance and how does it contribute. To answer these questions, we test variants of IterWeGO models and compare their performance together with IterWeGO. The first variant is IterWeGO-FFE 3 , from which we have the following observations. First, our IterWeGO outperforms two variants on both tasks across to datasets, showing that our current design of cross-modal guidance enhancement is generally better than the other two. We also notice that IterWeGO-ROE consistently outperforms IterWeGO-FFE, which demonstrates that applying cross-modal guidance with relative order information is significantly more effective than semantic enhancement via simple feature fusion. Furthermore, with delicate fine-tuning for the cross-modal alignment module, our Iter-WeGO achieves marked performance improvement compared to IterWeGO-ROE. Since the cross-modal alignment module serves as the bridge to pass the specific guidance across modalities, it partly determine the quality of the leveraged guidance as well as the effectiveness of the guidance enhancement, which thereby significantly impact the overall performance of the whole model.To straightforwardly illustrate the effectiveness and mechanism of our IterWeGO model, we exhibit three cases in Figure 4 , two for the comparison between different methods and one to show the iterative updating for ordering during inference. From case (a), we can observe that when the uni-modal ordering suffering from the lack of coherence information, it cannot recover the order both for texts and images. NACON [9] employs cross-modal positional attention to leverage the gold order information from the other modality to assist the target modality and achieves much better results, which could play as an upper bound. While our IterWeGO only leverage weak guidance from the other modality, it also recovers the gold order with iterative boosting paradigm. Even IterWeGO cannot totally recover the gold order, it still benefits from the guidance of relative order hidden in the other modality, and partially recover the order. With such partial recovering, the PMR will not improve but Accuray and 𝜏 will gain improvements. As discussed in Section 4.4, the iterative boosting strategy significantly improve the order recovering. To make the mechanism of iterative boosting more straightforwardly, we illustrate an example of iterative inference in case (c). From this case, we observe at step 0 (unimodal inference), the order is far from coherent, but based on the weak cross-modal guidance at step 1, several sentences and images can be placed at the right position. With the iteration going, more elements, i.e., images and sentences, are placed at the groundtruth positions, and results in recovering the gold order in the end. In summary, these cases directly illustrate the effectiveness and superiority of our proposed IterWeGO model.As large foundation models, e.g., large language models (LLMs) and large multimodal models (LMMs), have been demonstrating superb capability in various areas and tasks. Here we take an attempt to investigate the LMMs for ordering problem with weak cross-modal guidance. We random sample 200 cases from the test set and prompt GPT-4 (for pure text ordering) and GPT-4V (for image and crossmodal ordering) to recover the order of a set of elements (images or sentences). We note that GPT rejected 24 samples for ordering, resulting in 176 valid sample for final comparison. As the results shown in Table 4 , to our surprise, GPT achieves inferior ordering performance comparing with our IterWeGO. More interesting, with the weak guidance from the other modality, GPT-CM yields worse performance than the unimodal scenario (GPT-UM) for text ordering, but gains improvement for image ordering. Bin et al. [8] pointed out that the reason of GPT fails in ordering problem may come from the autoregressive nature of GPT. We go a step further to hypothesize that the reason for the inferior performance in the cross-modal scenario may be because GPT processes all content as input, summarizing the common topics of images and sentences, but fails to capture intra-modal coherence. Anyway, the prompting engineering is also a reason of unexpected performance of GPT, we will keep improving the prompts to yield better performance. 
conclusions: 
 This paper presented a novel cross-modal coherence modeling method that focuses on taking advantage of cross-modal unordered guidance to enhance the order learning in single modality, named as Weak Cross-modal Guided Ordering (WeGO). More specifically, it selectively employs the predicted relative order information in one modality to guide corresponding order modeling in another based on proper semantic alignment. Furthermore, to facilitate the ordering model learning in both modalities, an iterative learning diagram was proposed. Extensive experiments done in two public datasets demonstrated the effectiveness of the proposed method. Two major technical components, namely weak cross-modal guidance and iterative boosting in both training and learning, have been evaluated to be effective with solid experiments and anslysis. 
 
 Appendix:  


Decision: **Conclusion: Accept as a Poster Paper**

After thorough consideration of the review reports from the independent reviewers, I conclude that this paper is recommended accepting as a Poster Paper for the ACM MM 2024 conference. The decision is based on the innovative approach proposed by the authors, which contributes significantly to the field of cross-modal coherence modeling, despite certain limitations that require further investigation.

**Strengths:**

1. **Innovative Approach**: The WeGO model introduces a novel method of using weak cross-modal guidance from one modality to enhance coherence modeling in another. This is particularly beneficial in scenarios where labeled coherence information is scarce or expensive to obtain.
2. **Iterative Learning Paradigm**: The model leverages an iterative learning process, which not only optimizes consistency modeling across modalities but also continuously refines predictions, enhancing overall performance.
3. **Empirical Validation**: The proposed method has been rigorously validated on two public datasets (SIND and TACoS-Ordering), demonstrating its effectiveness and improvements over baseline methods.

**Weaknesses:**

1. **Dependency on Initial Predictions**: The model's effectiveness heavily relies on the quality of the initial predicted pairwise orders from one modality. Inaccurate predictions can significantly degrade performance.
2. **Computational Complexity**: The iterative learning and inference processes introduce additional computational overhead, which could complicate implementation and practical application.
3. **Comparison with Recent Methods**: The paper lacks a comprehensive comparison with more recent state-of-the-art (SOTA) methods, particularly those proposed in the last few years. Including such comparisons would strengthen the paper's claims of effectiveness.

In summary, while there are some concerns regarding computational complexity, dependency on initial predictions, and the need for broader comparisons, the innovative approach and empirical validation of the WeGO model make this paper a valuable contribution to the field. Therefore, I recommend its acceptance as a Poster Paper to encourage further discussion and exploration of its potential applications and improvements.

Reviews: review: paper_summary: paper_summary: Cross-modal coherence modeling is crucial for intelligent systems to help users organize and structure information, enabling them to understand and create content from the physical world coherently, much like humans. Previous work on cross-modal coherence modeling attempted to use sequential information from another modality to assist in recovering coherence in the target modality. While effective, the availability of labeled relevant coherence information is not always guaranteed, and the acquisition cost may be high, making cross-modal guidance difficult to utilize. To address this challenge, this paper explores a new approach utilizing cross-modal guidance and proposes the Weak Cross-Modal Guidance Ordering (WeGO) model. This model leverages high-confidence predictions of paired sequences from one modality as reference information to guide coherent modeling in another modality. Furthermore, an iterative learning paradigm is designed to jointly optimize the consistency modeling of the two modalities through mutually selected guidance. Iterative cross-modal enhancement also plays a role in the reasoning process to further enhance coherence predictions within each modality.
strength_weakness: strength_weakness: The paper introduces a novel cross-modal guidance method, WeGO, effectively utilizing weak but relevant sequential references from another modality to guide sequential modeling in the target modality.

The paper presents an iterative learning paradigm, leveraging weak cross-modal sequential guidance from two modalities during the training process, jointly learning the ordering models of the two modalities.

The paper thoroughly evaluates the proposed method on two public cross-modal coherence modeling datasets.
limitations: limitations: 1. On line 382, ‘the prediction of pairwise order in one modality is highly related to the corresponding pairwise score in another modality.’ Is there any reference or theoretical basis for this conclusion? If so, could the authors please provide a more detailed explanation?

2. On line 386, ‘confidence of the pairwise score in the referred modality is high, making it qualified to become a guidance.’ How is the confidence of the pairwise score designed and determined? Could the authors provide further discussion on this?

3. In the experimental section, it is mentioned that ‘Threshold to select valid cross-modal guidance is set to 0.8 for image-to-text and 0.9.’ What would happen if other values were chosen for the threshold? Could the authors further discuss this through experimentation?

4. Has the author considered adding more datasets to validate the method?

5. With the exception of NACON, the baseline methods mentioned in the experimental section are primarily based on work from 2018-2020. Are there any updated approaches in this field, for instance, methods proposed in 2023, that could be considered as baselines? If so, should the authors consider incorporating these approaches for comparison to further demonstrate the effectiveness of the proposed method?
paper_summary: paper_summary: The article presents a novel method for cross-modal coherence modeling called the Weak Cross-Modal Guided Ordering (WeGO) model. This approach leverages weak guidance from high-confidence predicted orders in one modality to enhance coherence modeling in another, through an iterative learning paradigm. This method operates without requiring labeled coherence information, making it advantageous for scenarios where such data is scarce or expensive to obtain.
strength_weakness: strength_weakness: 1.Innovation: Introduces a unique method of using predicted, rather than labeled, order information for cross-modal coherence, increasing accessibility and reducing costs.
2.Iterative Learning Paradigm: Enhances both the learning and inference stages, allowing continuous improvement of coherence modeling.
3.Empirical Validation: The method outperforms established baselines and shows effectiveness through rigorous experimental validation.
limitations: limitations: 1. The model's effectiveness may be limited by the accuracy of initial order predictions in one modality.
2. The iterative nature of the model could complicate implementation and increase computational overhead
paper_summary: paper_summary: This paper proposes a Weak Cross-Modal-guided Coherence Modelling framework via Iterative Learning, leveraging an unsupervised idea without labeled orders to align the elements in two different modalities.
strength_weakness: strength_weakness: 1. This paper is easy to read and understand and its logic is clear.
2. The idea seems innovative.
3. Main experimental results validate the effectiveness of the proposed method.
limitations: limitations: 1. The motivation illustrated in Figure 1 needs deeper analysis.
2. The expression of ‘modelling’ and ‘modeling’ should be consistent.
3. Although the proposed method does not need prior labels to support the optimization process, it is not clearly described in the method or experiments whether the pre-trained models in line 339 were obtained based on prior labels or not. If the pre-trained model is label-based, I think the proposed label-unneeded strategy is less innovative.
4. Line 100 needs concrete literature support.
5. More recently published SOTA baselines are needed.
6. Some grammar issues. For example, ‘understand and create’ in line 10 should be ‘understanding and creating’.
paper_summary: paper_summary: The paper presents a method for improving cross-modal coherence modeling, particularly in tasks such as sentence and image ordering. The core idea is to utilize weak cross-modal guidance, where predicted pairwise order information from one modality (e.g., text or images) helps guide the coherence modeling in another modality without relying on gold-standard labeled data, which can be costly and time-consuming to obtain.

Overall, the paper provides a robust framework for enhancing cross-modal coherence modeling by integrating weak guidance from correlated modalities and refining predictions through iterative learning.
strength_weakness: strength_weakness: 1. The paper is well-structured and clearly explains the methodology, experimental setup, and results. Figures and tables effectively illustrate key points, enhancing readability.
2. The WeGO model introduces a novel method of leveraging weak cross-modal guidance without requiring costly labeled data, addressing a significant challenge in coherence modeling.
3. The paper evaluates the WeGO model on two public datasets (SIND and TACoS-Ordering), demonstrating improvements over baseline methods
limitations: limitations: 1. The model depends on the quality of the predicted pairwise orders from the other modality. If these predictions are inaccurate, the overall performance can degrade significantly. The paper could benefit from a discussion on mitigating this dependency.
2. The iterative learning and inference processes, while effective, introduce significant computational complexity.
3. The method relies on cross-modal semantic similarity, which may not always be robust, especially in cases where the modalities do not align well semantically.
4. Lack of comparison with similar methods. The paper could be strengthened by comparing WeGO with other weakly-supervised or semi-supervised coherence modeling approaches.