Two‐stage partial image‐text clustering (TPIT‐C)

IET Computer VisionVolume 16, Issue 8 p. 694-708 ORIGINAL RESEARCHOpen Access Two-stage partial image-text clustering (TPIT-C) Dongjin Guo, Dongjin Guo orcid.org/0000-0002-7700-5980 College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this authorXiaoming Su, Corresponding Author Xiaoming Su [email protected] College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, China Correspondence Xiaoming Su, College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China. Email: [email protected]Search for more papers by this authorYahong Lian, Yahong Lian College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, ChinaSearch for more papers by this authorLimin Liu, Limin Liu College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this authorHaibo Wang, Haibo Wang Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this author Dongjin Guo, Dongjin Guo orcid.org/0000-0002-7700-5980 College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this authorXiaoming Su, Corresponding Author Xiaoming Su [email protected] College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, China Correspondence Xiaoming Su, College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China. Email: [email protected]Search for more papers by this authorYahong Lian, Yahong Lian College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, ChinaSearch for more papers by this authorLimin Liu, Limin Liu College of Data Science and Application, Inner Mongolia University of Technology, Huhhot, China Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this authorHaibo Wang, Haibo Wang Big Data Laboratory of Inner Mongolia Discipline Supervision and Investigation, Huhhot, ChinaSearch for more papers by this author First published: 20 June 2022 https://doi.org/10.1049/cvi2.12117AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Deep multi-model clustering is a challenging task for data analysis since it learns a universal semantic representation to find correct clusters from heterogeneous samples. However, most existing methods 1) lack an effective approach to getting a global representation of visual instances, which results in a huge semantic gap between visual and textual space. 2) hardly consider partial multi-modal, where each instance is represented by only one modality. In reality, the pairing information for modalities is not available for all instances. To tackle the above issues, we propose a novel model called the Two-Stage Partial Image-Text Clustering (TPIT-C) model. Firstly, we build an interpretable reasoning network to obtain the salient regions and semantic concepts of the scene in order to generate global semantic concepts. Secondly, we construct an adversarial learning module to align textual and visual instances into a unified space by virtue of cycle-consistency. The experimental evaluations on public unpaired multi-model datasets illustrated that the proposed method has better performance and the effectiveness of our algorithm in the partial image-text clustering task. 1 INTRODUCTION Vision and language are two important aspects of human cognition, perception, and understanding of things. There has been lots of work dedicated to exploring and mining the applications between images and text. References [1, 2] could mitigate the problem of insufficient training images by introducing rich labels from the text field. Authors in reference [3] deeply fused image-text features and collaboratively learned image-text hash codes for image search tasks in a deep architecture. Authors in reference [4] explored and confirmed the importance of higher-order information and distraction-based re-ranking in the image-text matching task. However, as heterogeneity of image-text data corresponds to a different characteristic and complementarity, it is still a challenging problem. Among these works, image-text clustering is one of the fundamental topics in this field, which aims to group semantical similarities together in an unsupervised way. It has been widely used in various applications such as vehicle detection [5-7], clinical medicine [8], marketing research [9], crowdsourced mobile testing [10], and recommender system [11]. Although considerable progress has been made in this area, it remains as an unresolved deficiency, that is, a lack of global visual semantic representation of unpaired image-text datasets. When people use language to describe an image, it is a process of recognition and comprehension. In language description, we consider the high-level semantic concepts such as the relationship between objects, the visual features of regions, and the location features of regions, which are beyond the scope of visual recognition to achieve visual intelligence. Visual intelligence is phased from object recognition, scene analysis to graphic interaction, and finally inferential prediction. According to psychological research [12], contextual information, such as scene context and object relationships, plays a key role in people's capacity to recognise and comprehend objects. There may be symbiosis, correlation, dependence, and causality between objects. However, the existing image-text clustering [13, 14] methods for image feature extraction are biassed towards Convolutional Neural Network, which tend to be missing higher-level semantic information, making images weaker semantically compared with text, and somewhat worse at integrating similar semantics for clustering tasks. The reason for this is that Convolutional Neural Network only perform local pixel-level analysis, as shown in the blue dashed box in Figure 1, and they are less likely to extract higher-level semantic concepts. It is a basic assumption that the quality of vision and text semantics is also an important factor affecting the results of image-text clustering. FIGURE 1Open in figure viewerPowerPoint Comparison of existing methods for image feature extraction in image-text clustering with our proposed method. The blue dotted box in the figure represents the local pixel-level feature method, namely the Convolutional Neural Network, which ignores higher-level semantic information than the global region-level feature method proposed by us (represented by the red dotted box in the figure) Existing image-text clustering methods [15] made a strong assumption that all the visual-textual modalities are paired and well-aligned, as shown in Figure 2. In reality, microblogs, for example, are presented as either images or texts. Moreover, data corruption and loss are inevitable. For the absence of heterogeneous modalities, there is a lot of work based on the partial multi-view approach [16-18]. Either multi-modal clustering or partial multi-modal clustering requires additional pairing information between modalities for feature alignment. Our work considers an extreme case where each instance is represented in only one modality, that is, the missing rate is 100%. When pairing information does not exist, it is difficult to find correlations between all different modalities. The existing partial multi-modal methods cannot find the effective information between missing data to perform inversion and completion. FIGURE 2Open in figure viewerPowerPoint Given multi-modal, the blue circle and the green square represent the image and text modalities, respectively. Different modalities are heterogeneous, that is, visual and language modalities have different properties. Each circle and square in the figure represents the same sample. An orange box indicates that the image and text modalities are aligned and complete. The circles and squares drawn by the dashed lines in the red dashed box represent missing data, so each instance is represented by only one modality Table 1 summarises the motivation of our work compared with existing state-of-the-art methods. To address the problems mentioned above, we build a Two-Stage Partial Image-Text Clustering (TPIT-C) model named TPIT-C, which aims to obtain better clustering results, as shown in Figure 3. The central idea is that we first use graph structure reasoning to obtain a global visual semantic representation of the whole scene, then we use cycle-consistency-based CycleGAN [19] to enforce connection across visual and textual modalities. In detail, based on the human perception mechanism, we extract salient visual features of each object as graph nodes, while constructing the relationship of each node by Graph Convolutional Networks (GCN) [20]. Then we select effective visual features by gated recurrent units. In order to transfer partial image-text features into a unified latent space, we introduce a generative adversarial network with a min-max game between the generator and discriminator. Especially, the generator maps one modality latent space to another modality space, while the discriminator makes sure that generated representation remains realistic. TABLE 1. Evaluation summary with the state of art Compared method Advantages Disadvantages DCN [21] 1) Perform k-means clustering and feature learning simultaneously, simple but effective. 1) Inherit k-means which is highly influenced by initialisation, cannot handle data with non-convex cluster shapes, and cannot obtain global optimal solutions CDIMC-net [22] 1) Capture the high-level features and local structure of each view. 1) Eigenvectors construct graphs (such as using K-nearest neighbours) to obtain the input of graph neural networks, but the quality of constructed graphs is difficult to guarantee. 2) A simple padding scheme is invalid when the missing ratio is 100% DMVCVAE [23] 1) Capable to generate samples. 1) High-computational complexity. 2) Introduce a set of non-negative combination weights to better use the information from multiple views. 2) Cannot efficiently work when pairing information does not exist. COMPLETER [17] 1) Data recovery and consistency learning are two sides of the same coin, and they can be unified in the framework of information theory. 1) Rely on samples with complete view information to perform inversion and completion of missing data. Proposed method 1) Resolve a clustering algorithm for those unpaired multi-modal data in this paper without any extra pairing information. 1) Dedicate to the image-text clustering. 2) Handle semantic differences in heterogeneous modalities. FIGURE 3Open in figure viewerPowerPoint The framework of our proposed Two-Stage Partial Image-Text Clustering (TPIT-C). The model consists of three components, namely graph structure inference module, within-modality reconstruction module and cross-modality alignment module. To be specific, the graph structure inference component is used to extract the global representation of the image instances which are not missing, while the within-modality reconstruction is used to project all modalities into modality-specific latent representations. The cross-modality alignment module aims to resolve the case where each instance is made up of a single modality To sum up, our main contributions are: (a) We propose an effective method to extract the global representation of visual instances, which bridges the semantic gap between image representation and text features and improves the clustering performance. This method elegantly unifies the principles of human perception mechanisms into the TPIT-C framework. (b) We propose a novel approach to the partial multi-modal clustering problem with the consideration of a case issue, where each instance contains only one modality, and we solve it by building an adversarial learning framework using cyclic consistency. (c) Extensive experiments on different unpaired image-text datasets demonstrate that TPIT-C is conducive to promoting clustering performance. 2 RELATED WORK Partial multi-modal clustering methods generally involve two aspects: exploiting diverse and consistent features of different modalities and eliminating the effects of missing modalities. The first aspect refers to using existing multi-modal data to learn good representations. The second aspect refers to using existing modalities to invert and fill in missing modalities. Two-Stage Partial Image-Text Clustering focuses on both these aspects. Next, we will discuss related studies with respect to these two aspects. Graph structure reasoning The key factor affecting clustering performance is the similarity between data. Similarity depends on features, so the core element remains the learning of features in multi-modal. This demonstrates that if representation learning can acquire better representations, it can do superior clustering. Therefore, learning and extracting image representations is crucial. Graph-based approaches have been popular in recent years for computer vision tasks and have shown to be beneficial for relational reasoning. The goal of graph structure reasoning is to take an image and extract a set of salient objects and pairwise interactions, which will form a graph. By reasoning about this graph structure, high-level semantic information about the image instances will be obtained. DeepPath [24] and MINERVA [25] used graph structures to integrate prior knowledge into a reinforcement learning framework for reasoning. DeepPath specialised on pathfinding (finding the most informative route between two target nodes), whereas MINERVA focused on question-answering (finding the proper answer node given a question node and a relation). SIN [26] formalised the object detection problem as graph structure inference, which modelled objects in an image as nodes on a graph to make full use of the information about the relationships between surrounding scenes. Besides, it built graph structures and utilised a message passing mechanism for global associative inference. SIN used contextual information between objects while CISC [27] further noted higher-order contextual information, that is, visual patterns of relationships between objects. CISC fused semantic and visual information for each prediction round in a contextual module. The use of contextual information effectively improved object detection and relationship prediction performance. Wang et al. [28] proposed that the order of human attention naturally forms a hierarchy in which larger objects generally receive priority attention. The method was to find the most minor and most related node for each object node. We borrowed some ideas from the above work, that is, distilling raw, unstructured images of natural scenes into a structured scene resolution graph and using contextual information sufficiently to explore high-level semantic information of image instances. The differences between TPIT-C and the existing work are given below. First, to the best of our knowledge, our work is the first to incorporate global semantic reasoning in learning visual representations of image-text clustering. Second, TPIT-C can learn the global representation of visual instances without external knowledge rather than [29, 30]. Partial multi-modal clustering In order to discover clusters from partial multi-modal data, various partial multi-modal (partial multi-modal (PMC)) methods have emerged [31-33]. The PMC method aimed to build a shared latent subspace of a complete modality to compensate for the latent representation of missing data. Most existing PMC methods could be roughly classified into two categories, that is, traditional methods and generative adversarial networks. The traditional methods are divided into the kernel method and non-negative matrix factorization method. The core idea of the kernel method [16, 34] is to compensate the incomplete kernel matrix by using the kernel matrix of the complete modality. Rai et al. [35] proposed the first partial multi-modal clustering method. They used a modality kernel representation as to the similarity matrix and the Laplace regularisation method to complete the missing modality kernel matrix. Previous multi-kernel k-means clustering methods consist of two steps: firstly, these methods filled in the missing parts and secondly clustered multiple modalities. The filling algorithm included zero, mean, and low-rank filling (heuristic filling) and then executed a multi-kernel k-means algorithm on the filled data. Authors in references [36, 37] optimised the multi-kernel k-means clustering method for the separation of filling and clustering. The NMF-based approach [38-40] learnt a common latent space for complete instances and a private latent representation for partial instances. The Gan model was designed to learn the relationship between different modalities. The model from references [41, 42] was used to complete the missing data. The differences between our study and existing works are given below. First, TPIT-C is a deep model rather than a shallow one. Thus it will naturally embrace the capacity of handling complex and large-scale dataset. Second, at present, the problem of partial multi-modal almost does not consider the case in which each instance consists of only one modality. Alignment information between modalities does not apply to all instances at all. Because the existing methods assume that all the modalities of some instances are aligned, and the missing data can be retrieved and completed through this part of the instances. In contrast, our method aims to resolve the case where each instance is made up of a single modality through an adversarial learning framework using cyclic consistency. 3 TPIT-C: THE PROPOSED MODEL Our suggested model comprises three main parts: a graph structure inference module, a within-modality reconstruction module, and a cross-modality alignment, as shown in Figure 3. For clarity, we'll explain the notations and issue settings first and then elaborate on each module in the following sections. Notations and Problem Settings: Almost all previous image-text clustering work was predicated on the premise that all samples' modalities were accessible and strictly aligned, as shown by the black dotted box in Figure 4. Unlike the previous work, we do not have paired samples from both modalities to reveal correlations between the different modalities because each instance is represented in only one modality, as shown by the red dashed box in Figure 4. Given a set of datasets D = x i i = 1 n $\mathcal{D}={\left\{{\boldsymbol{x}}_{i}\right\}}_{i=1}^{n}$ containing n samples, where samples are either images or texts. Clearly, D $\mathcal{D}$ is partitioned into two single modal sets, D img = x i ( i m g ) i = 1 n img ${\mathcal{D}}_{\mathit{img}}={\left\{{\boldsymbol{x}}_{i}^{(img)}\right\}}_{i=1}^{{n}_{\mathit{img}}}$ and D txt = x i ( t x t ) i = 1 n txt ${\mathcal{D}}_{\mathit{txt}}={\left\{{\boldsymbol{x}}_{i}^{(txt)}\right\}}_{i=1}^{{n}_{\mathit{txt}}}$ , where n = nimg + ntxt. I ∈ R 2 × n $\mathbf{I}\in {\mathbb{R}}^{2\times n}$ denotes an indicator matrix for partial instances where each column of I encodes the modality status (present/absent) for the corresponding instance. I contains only two cases, 0 1 T ${\left[\begin{array}{c}\hfill 0\,1\hfill \end{array}\right]}^{\mathrm{T}}$ and 1 0 T ${\left[\begin{array}{c}\hfill 1\,0\hfill \end{array}\right]}^{\mathrm{T}}$ , where 0 1 T ${\left[\begin{array}{c}\hfill 0\,1\hfill \end{array}\right]}^{\mathrm{T}}$ represent the absence of images modality and 1 0 T ${\left[\begin{array}{c}\hfill 1\,0\hfill \end{array}\right]}^{\mathrm{T}}$ represents the absence of text. Our objective is to acquire the global semantic representation of D img ${\mathcal{D}}_{\mathit{img}}$ firstly, and then cluster the nimg + ntxt samples into K clusters. FIGURE 4Open in figure viewerPowerPoint Notations for partial image-text data. The black dashed box depicts the traditional multi-modal clustering task, where both X n ( i m g ) ${\mathbf{X}}_{n}^{(\mathrm{i}\mathrm{m}\mathrm{g})}$ and X n ( t x t ) ${\mathbf{X}}_{n}^{(\mathrm{t}\mathrm{x}\mathrm{t})}$ represent the same instances. In contrast, the red dashed boxes X n ( i m g ) ${\mathbf{X}}_{n}^{(\mathrm{i}\mathrm{m}\mathrm{g})}$ and X n ( t x t ) ${\mathbf{X}}_{n}^{(\mathrm{t}\mathrm{x}\mathrm{t})}$ represent different instances in our problem setting. In such a problem setting, our samples come from either the image or text modality, that is, we do not have complete pairwise information to reveal the relationship between the modalities Two-Stage Partial Image-Text Clustering is a two-stage model. For the first step, the bottom-up attention model is utilised to generate a series of feature sets, which are made up of prominent areas or objects. We extract 36 key areas, make maximum use of interregional information, and create a graph structure. The edges of the graph structure represent the semantic correlations between the salient regions, while the nodes of the graph structure represent the visual features of each salient region. We use GCN to infer this graph structure. Furthermore, the gating method is utilised to choose the most useful categorisation characteristics. In the second stage, latent representation learning is performed on image features with global semantic information and text features extracted from partial text data. The cross-modality generator, which is restricted by cyclic consistency, creates the matching latent representation from the image (text) latent representation. The discriminator and the generator play a game to increase the quality of creating the latent representation of the missing modality so that we eventually receive paired and aligned textual and visual data. Graph structure inference module This module is used to extract the global representation of the image instances which are not missing. Firstly, we use the bottom-up attention based on the Faster R-CNN [43] model to mine the salient regions or objects in image data D img ${\mathcal{D}}_{\mathit{img}}$ . The function of this process is consistent with the human visual system and identifies discriminative object quickly and accurately in graphics. To fix the number of selected prominent regions or objects, we adopt the Non-Maximum Suppression [44] method. For each selected salient object O, we extract the visual feature oi by a fully connect layer that follows an average pooling layer. Therefore, an image I $\mathcal{I}$ can then be constructed of o i i = 1 N ${\left\{{o}_{i}\right\}}_{i=1}^{N}$ , where N represents the number of salient objects selected from an image. Secondly, the relationship between these salient objects or regions is constructed through pairwise affinity. For directed edge ei→j from node oi to oj, we measure the pairwise affinity between oi and oj in the embedding space, which represents their semantic relationships. As indicated by Equation (1), oi and oj represent the features extracted by the fully connected layer respectively. Wu and Wv are weight parameters that can be learned by back propagation. e i → j = W u o i T W v o j ${e}_{i\to j}={\left({W}_{u}{o}_{i}\right)}^{T}\left({W}_{v}{o}_{j}\right)$ (1) We create a fully connected graph G = { O , R } $\mathcal{G}=\left\{\mathcal{O},\mathcal{R}\right\}$ using the graph nodes and edges obtained in the preceding phases, where O = o i i = 1 N $\mathcal{O}={\left\{{o}_{i}\right\}}_{i=1}^{N}$ is the collection of salient object features extracted fully connected layer, and R $\mathcal{R}$ denotes the affinity matrix computed using Equation (1). Thirdly, we employ GCN to reason about this fully connected graph. Adding residual connections can effectively prevent GCN from over smoothing and increase the model performance, as follows: O ∗ = W r R O W g + O , ${\mathcal{O}}^{\ast }={W}_{r}\left(\mathcal{R}\mathcal{O}{W}_{g}\right)+\mathcal{O},$ (2)where, Wg represents the weight matrix of the GCN layer. Wr represents the weight matrix of the residual structure. O ∗ = o 1 ∗ , … , o N ∗ ${\mathcal{O}}^{\ast }=\left\{{o}_{1}^{\ast },\text{\ldots },{o}_{N}^{\ast }\right\}$ denotes the obtained features that have been fused with relational information. The operation between the R O W g $\mathcal{R}\mathcal{O}{W}_{g}$ is matrix multiplication. Then, O ∗ = o 1 ∗ , … , o N ∗ ${\mathcal{O}}^{\ast }=\left\{{o}_{1}^{\ast },\text{\ldots },{o}_{N}^{\ast }\right\}$ performs further global semantic reasoning. The obtained features are successively sent to gated recurrent units to select essential information and filter out nonessential information with the aim of obtaining the final representation of the image. For a given reasoning step t, the inputs of the update gate Zt and reset gate Rt are the inputs of the current input feature o i ∗ ${o}_{i}^{\ast }$ and the hidden state of the previous reasoning step Ht−1. The outputs of two gates are given by two fully connected layers with a sigmoid activation function. Reset gate Rt helps capture short-term dependencies in time series, where Wor and Whr are weight parameters and br is bias. R t = σ o i ∗ W o r + H t − 1 W h r + b r , ${\mathbf{R}}_{t}=\sigma \left({o}_{i}^{\ast }{\mathbf{W}}_{or}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hr}+{\mathbf{b}}_{r}\right),$ (3) Update gate Zt helps capture long-term dependencies in time series, where Woz and Whz are weight parameters and bz is bias. Z t = σ o i ∗ W o z + H t − 1 W h z + b z ${\mathbf{Z}}_{t}=\sigma \left({o}_{i}^{\ast }{\mathbf{W}}_{oz}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hz}+{\mathbf{b}}_{z}\right)$ (4) Next, we integrate the reset gate Rt with the hidden state Ht−1 updating mechanism. This mechanism can capture the historical information of the sequence up to the current inference stage to get the candidate hidden state H ⋜ t ${\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}$ at the reasoning step t. As indicated in Equation (5), Woh and Whh are weight parameters, bh is the bias, and the symbol ⊙ is Hadamard (elementwise) product operator. The tanh nonlinear activation function is employed to keep the values of H ⋜ t ${\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}$ within the range (−1, 1). H ⋜ t = tanh o i ∗ W o h + R t ⊙ H t − 1 W h h + b h ${\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}=\mathrm{tanh}\left({o}_{i}^{\ast }{\mathbf{W}}_{oh}+\left({\mathbf{R}}_{t}\odot {\mathbf{H}}_{t-1}\right){\mathbf{W}}_{hh}+{\mathbf{b}}_{h}\right)$ (5) The formula Equation (5) returns just the candidate hidden state H ⋜ t ${\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}$ without taking into account the impact of the update gate Zt. Finally, the hidden state of the new inference Ht will determine whether the model retains the hidden state of the previous reasoning step Ht−1 or approaches the candidate hidden state H ⋜ t ${\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}$ , as given in Equation (6). Preserving the hidden state of the previous reasoning step or approaching the candidate hidden state depends on the value of the update gate. When the update gate Zt in the inference process approaches 1, the feature information o i ∗ ${o}_{i}^{\ast }$ is mostly ignored, and the previous hidden states Ht−1 are completely kept until the final image representation is created at the conclusion of the process. ⊙ is Hadamard (elementwise) product operator. We use the topological structure of a graph in the reasoning process. Both local and global semantics are taken into consideration by graph structure inference. The graph nodes represent the visual feature of each salient object, which are known as local semantics. The edges of the graph structure represent the relationships between objects that affect each other, that is, global semantics. H t = Z t ⊙ H t − 1 + 1 − Z t ⊙ H ⋜ t ${\mathbf{H}}_{t}={\mathbf{Z}}_{t}\odot {\mathbf{H}}_{t-1}+\left(1-{\mathbf{Z}}_{t}\right)\odot {\stackrel{{\eqslantless}}{\mathbf{H}}}_{t}$ (6) Within-modality reconstruction module In order to learn which latent features are convenient for cross-modality alignment, we implement