# Globally Guided Confidence Enhancement Network for Image-Text Matching
###### Abstract  
Image-text matching is a crucial aspect of multi-modal intelligence. The main challenge in this area is accurately measuring the relevance between the image and text, using evidence obtained through matching. Previous studies either concentrated on obtaining a well-represented global feature to measure similarity directly or on investigating complex matching patterns at a local level before aggregating them, with little attention paid to combining them. We propose a Globally Guided Confidence Enhancement Network that combines both approaches by obtaining a good global representation to guide fine-grained local interactions. In this process, content that better matches the text from a global perspective is enhanced and represented with confidence scores. Extensive experiments demonstrate that the approach we have employed achieves superior performance on Flickr30K and MSCOCO datasets. image-text matching; global guidance; multimodal interaction +
Footnote †: journal:  
authors of ref. [12] proposed a stacked cross-attention network that exploits attention mechanisms to obtain a region-based textual attention representation and a word-based image attention representation, exploring all possible semantic alignments between image regions and textual words. These networks achieved excellent performance and inspired many subsequent studies [13; 14; 15; 16; 17; 18; 19]. These investigated intricate semantic interactions between modalities to identify possible visual and textual alignments. Some studies Refs. [20; 21; 22] have proposed the use of graph convolutional neural networks or attentional inference to explore fine-grained alignment from multiple perspectives such as objects, relations and attributes. Even though significant progress has been made by considering various fine-grained interaction patterns, not all image-text matching pairs contribute equally to the overall match, and it would be unreasonable to simply aggregate them to obtain the final similarity. Intuitively, well-represented global features better represent high-level semantic concepts of data, but they lack fine-grained information. Local inference-based approaches capture more fine-grained interactions and exhibit better matching performance, but their interaction processes are highly complex and they aggregate all possible image-text matching pairs with equal importance which is not reasonable. At the same time, we argue that the role of a well-represented global feature in local interaction is underestimated. Specifically, it can prevent the model from losing the sense of global semantics during complex fine-grained local interactions, help the model deepen its understanding of the content and guide the model to focus on meaningful local matching pairs. To address these problems, we propose a globally guided confidence enhancement network (GGCEN), which enhances meaningful alignment in local similarity measures from the perspective of global semantics and can address both the big picture and details in a coordinated and effective way. The holistic characteristic of global features is used to guide fine-grained local interactions, to obtain more accurate matches. Specifically, a text (image) representation fused with an image (text) context and pooled global features work together to generate matching pairs' confidence. In this way, the relationship between image objects and text descriptions can be better captured, reducing the impact of locally similar but globally inconsistent matching relationships on the final similarity measure. Our approach consists of three parts: (1) Feature representation: We derive the representation of image regions and text words, then we use the GPO pooling strategy [23] to obtain the global representations of image and text. Subsequently, we add a self-attention (SA) mechanism to enhance these global representations.