Venue: NAACL 2024

# Leveraging Customer Feedback for Multi-modal Insight Extraction

Sandeep Sricharan Mukku , Abinesh Kanagarajan , Pushpendu Ghosh , Chetan Aggarwal Amazon

###### Abstract

Businesses can benefit from customer feedback in different modalities, such as text and images, to enhance their products and services. However, it is difficult to extract actionable and relevant pairs of text segments and images from customer feedback in a single pass. In this paper, we propose a novel multi-modal method that fuses image and text information in a latent space and decodes it to extract the relevant feedback segments using an image-text grounded text decoder. We also introduce a weakly-supervised data generation technique that produces training data for this task. We evaluate our model on unseen data and demonstrate that it can effectively mine actionable insights from multi-modal customer feedback, outperforming the existing baselines by \(14\) points in F1 score.

## 1 Introduction

Customer feedback is essential for businesses to design and improve their products and services, according to customer expectations. Luo et al. (2022) observe that the multi-modal feedback is growing rapidly. However, most existing solutions Mukku et al. (2023); Sircar et al. (2022); Liu et al. (2022) do not take into account the rich information that image and video feedback contain, which can enhance the actionability and improve the customer experience. To address this challenge, we propose a novel multi-modal architecture that extracts pairs of text segments and corresponding images that are relevant and actionable for a given product from customer feedback. These pairs can help businesses to increase the actionability, improve the product catalogue quality, enhance the customer experience and thereby reduce returns and replacements.

## 2 Related work

In recent years, multi-modal tasks, combining various data types such as images and text, have garnered significant interest in artificial intelligence and natural language processing Goyal et al. (2017); Zhou et al. (2020); Lu et al. (2023). Among these tasks, Visual Question Answering (VQA) stands out as a prominent domain Antol et al. (2015), wherein the aim is to generate textual answers to questions based on images. VQA has evolved significantly over the years, thanks to various advancing contributions. Early works on VQA Antol et al. (2015) laid the groundwork, delineating the fundamental framework and challenges of the task. Subsequent research delved into innovative methodologies, such as leveraging transformer networks Zhou et al. (2020); Lu et al. (2023), and devising techniques for seamlessly integrating visual and textual information Li et al. (2019); Lu et al. (2019); Kim et al. (2021). The field progressed with more advanced models and datasets, such as VQAv2 Goyal et al. (2017), which improved the robustness and performance benchmarking of VQA models. Moreover, the strides in pre-training on both vision and language have significantly influenced the landscape, as evidenced by grounding modeling models like OSCAR Li et al. (2020), BEiT-3 Wang et al. (2022), and VLMo Bao et al. (2021); Zhu et al. (2022), which have achieved remarkable results. Additionally, techniques such as counterfactual data augmentation Chen et al. (2020) have further enhanced the performance of VQA models.

Meanwhile, in the NLP domain, verbatim extraction and text summarization tasks have attracted a lot of attention. Models like BERT Devlin et al. (2018) and GPT-3 Brown et al. (2020) have shown remarkable abilities in extracting and summarising textual feedback. Abstractive text summarization models, such as the T5 Raffel et al. (2019), have emerged as the state-of-the-art (SOTA) solutions for generating short and coherent summaries from longer texts.

Our work falls at the intersection of these two domains, where we propose a novel problem at the intersection of VQA and text summarization, where

the input is a customer feedback text and corresponding images, and the objective is to extract the verbatim (exact segments of the textual feedback) that best highlights or talks about the feedback image. We design a multi-modal approach that builds on VQA models and extends them with text generation capabilities, introducing a new technique for multi-modal insight extraction. Our work bridges the gap between multi-modal understanding and verbatim extraction, and enhances the interpretability of images within textual context, especially for customer feedback and images.

## 3 Problem Statement

Given a customer feedback consisting of a set of images \(I=\{I_{1},I_{2},...I_{n}\}\) and a text \(T\), we extract \(m\) verbatims from \(T\), denoted by \(V=\{v_{i}~{}|~{}1\leq~{}i~{}\leq~{}m,v_{i}\in T\}\). The aim is to extract all the relevant and actionable verbatims \(\{v_{k}\in V\}\) that corresponds to the given image \(I_{k}\).

## 4 Proposed Approach

In this section, we first present our data generation method that creates training data from raw feedback. Then, we introduce our model architecture that extracts pairs of verbatims and images that are relevant and actionable for a given product.

### Weakly supervised Training Data Generation

We segment raw feedback text to extract actionable verbatim1 (refer appendix section A.1 for exact process). We then generate training data by obtaining \(m\times n\) possible verbatim-image pairs for each feedback text, where \(m\) is the number of verbatim and \(n\) is the number of images. Next, we compute cosine similarity scores for every verbatim-image pair using pre-trained CLIP Radford et al. (2021). To evaluate the relevance of the feedback image-verbatim pairs predicted by base CLIP2, we had them manually annotated as positive or negative pairs with clearly defined annotation guidelines (refer section 4.1.1). We established a validation set, denoted as **StratSet-1k**, comprising \(1,000\) image-verbatim pairs, carefully stratified (Algorithm 1) across \(27\) distinct product categories sourced from the raw dataset by clustering3,4 the actionable verbatims.

Footnote 1: Key text-phrases extracted from customer feedback, that can be utilised to take actions

Footnote 2: We used the ViT-B/32 version of CLIP

```
1: Let \(C\) be the number of clusters of verbatims spread across \(L\) product categories.
2: Let \(K\) be the no. of verbatims to be sampled
3:\(S\leftarrow[]\)
4:for\(c\in C\)do
5:\(V_{c}\leftarrow\text{verbatims in cluster $c$}\)
6:\(D_{c}\leftarrow\text{product categories in $V_{c}$}\)
7:for\(d\in D_{c}\)do
8:\(V_{c,d}\gets\text{ verbatims in category $d$ of cluster $c$}\)
9:\(k_{d}\leftarrow[K\cdot\frac{1}{[L]}]\)
10:\(S\gets S\cup\text{RandomSample}(V_{c,d},k_{d})\)
11:endfor
12:endfor
13:return\(S\)
```

**Algorithm 1** Stratified Sampling(K)

To determine the optimal threshold, we systematically adjusted it within the range of \(0.19\) to \(0.31\) and recorded the resulting precision and product category coverage. Our analysis (refer Figure 2) revealed that a threshold of \(0.27\) strikes a favorable balance between product category coverage and precision, effectively reducing noise in the training data. We prioritized precision, aiming for a minimum of \(90\%\), which sufficiently covers the majority of product categories while accepting a lower level of recall, as these matched pairs are primarily used for fine-tuning the CLIP.

Figure 2: Product category coverage and precision as a function of raw data threshold for fine-tuning CLIP. The graph shows the trade-off between coverage and precision for different values of the threshold



After that, we fine-tune the CLIP using \(210K\) positive pairs with a symmetric cross-entropy loss (refer Eqn. 2) that aligns image and text embeddings. The scaled pairwise cosine similarities between image and text embeddings are computed as follows:

\[logits_{I,T}~{}=~{}\frac{e^{\textrm{T}}_{T}e_{T}}{|e_{I}||e_{T}|}/\tau~{}=~{} \cos(\theta_{I,T})/\tau \tag{1}\]

where \(e_{I}\) and \(e_{T}\) are the image and text embeddings, and \(\theta_{I,T}\) is the angle between them. We compute the loss over both images and verbatim (text) by averaging the losses for each modality:

\[loss=(loss_{i}+loss_{t})/2\\ =-\frac{1}{2}\sum_{i=1}^{n}\left[\log\frac{\exp(\cos(\theta_{i, labels_{i}})/\tau)}{\sum_{t=1}^{m}\exp(\cos(\theta_{i,t})/\tau)}\right.\\ +\left.\log\frac{\exp(\cos(\theta_{labels_{i},i})/\tau)}{\sum_{j= 1}^{n}\exp(\cos(\theta_{j,i})/\tau)}\right] \tag{2}\]

where \(labels_{i}\in V\) is the most relevant text of the \(i\)-th image, \(j\) iterates over the image classes, and \(\tau\) is a temperature parameter that controls the smoothness of softmax distribution. This loss function encourages the model to learn embeddings that are close to each other for positive pairs and far apart for negative pairs. The loss is symmetric as it is computed over both modalities. Finally, we use this trained model to obtain the matched (positive) and mismatched (negative) pairs from segments and images along with their similarity scores (sample training example shown in appendix section A.3).

We employed the fine-tuned CLIP model to make inferences on the StratSet-1k dataset. Subsequently, we manually annotated the inferred pairs as either positive or negative, adhering to Inter-Annotator Agreement (IAA) process Artetstein (2017). We then compu

Figure 1: Multi-modal INsight Extraction (MINE) Architecture

Figure 3: Training data threshold selection



and F1-score across a range of threshold values, as illustrated in Figure 2. Our analysis revealed that a threshold of \(0.225\) offers an optimal balance between precision and recall, thus serving as the foundation for creating training data for our problem. This refinement resulted in a notable enhancement of the fine-tuned CLIP model, achieving \(75\%\) F1-score, representing a \(3\%\) improvement compared to the off-the-shelf CLIP model.

#### 4.1.1 Annotation Guidelines: Verbatim-Image relevancy

Following are the annotations guidelines given to annotators that define the relevancy of Verbatim-Image pair:

1. **Object relevance**: When certain object discussed in the text is found in the image in any forms which is may not be explicitly mentioned, it will be considered as relevant pair.
2. **OCR relevance**: When certain entity discussed in the text is represented in the image by some form of text, the pair is considered as relevant.
3. **Semantic relevance**: When the information discussed in the text is contextually represented in the image, the pair to be marked relevant.

Each pair is annotated (_relevant / not relevant_) by two experts separately and resolved by third in-case of conflicting annotation. We used IAA process and achieved Cohen's kappa [8] of \(0.89\).

### Model Components

In this section, we describe the components of our proposed architecture for multi-modal insight extraction, which we refer to as MINE. Motivated by BLIP [6], MINE consists of three main modules: an image encoder, an image-grounded text encoder, and an image-text grounded text decoder. The image encoder uses a visual transformer to extract visual features from the input image. The image-grounded text encoder fuses the input text with the visual features using cross-attention layers. The image-text grounded text decoder generates the output text using causal self-attention and cross-attention layers conditioned on the multi-modal representation from the encoder and the previous tokens.

#### 4.2.1 Image Encoder

Image encoder is built using the visual transformer ViT-B/16 [10], which consists of an image encoder and a transformer encoder. The image encoder splits the input image into patches of size \(16\times 16\) pixels and converts each patch into a vector of \(768\) dimension embedding. The transformer encoder (which has \(12\) layers and each layer performs different operations on the embeddings, such as attention, normalization, and feed-forward networks) takes the sequence of embeddings and outputs a new sequence of embeddings that contains more information about the image content and context. We formulate the transformer encoder as a function \(g\) that maps a sequence of embeddings \(E\) to another sequence of embeddings \(E^{\prime}\):

\[E^{\prime}=g(E)=[e^{\prime}_{\text{CLS}},e^{\prime}_{1},e^{\prime}_{2},...,e^{ \prime}_{N}] \tag{3}\]

where \(e^{\prime}_{\text{CLS}}\) is the updated [CLS] token (added to represent the embeddings as image features), \(e^{\prime}_{k}\) is the updated embedding for the \(k\)-th patch (\(e_{k}\)), and \(N\) is the number of patches in the image.

#### 4.2.2 Image Grounded Text Encoder

The text encoder is based on BERT base [8], which has \(12\) transformer blocks with self-attention and feed-forward network (FFN) layers. To fuse the visual embedding from the image encoder, an extra cross-attention layer is inserted between the self-attention and FFN layers in each transformer block. This layer updates the text embeddings by attending to the image embeddings. The attention mechanism in the cross-attention layer computes attention scores for each token in the text sequence with respect to the image embeddings and determine how much importance each token in the text should place on the information in the image embeddings. A special token [ENC] is appended to the start of the input text (refer Section 5.2 for exact prompts used) to provide an identity for the encoder input. The text encoder outputs a multi-modal embedding of the image-text pair as follows:

\[z=h(T,E^{\prime})=[m_{\text{ENC}},m_{1},m_{2},...,m_{L}] \tag{4}\]

where \(h\) is the image-grounded text encoder function, \(T\) is the text input, \(E^{\prime}\) is the embedding from image encoder (\(g\)), \(m_{\text{ENC}}\) is the multi-modal embedding for the [ENC] token, \(m_{i}\) is the multi-modal 

embedding for the \(i\)-th text token, and \(L\) is the length of the text input.

#### 4.2.3 Image-Text Grounded Text Decoder

The decoder follows the same architecture as the encoder, except that it uses causal self-attention instead of bidirectional self-attention. The multi-modal embedding is also fused as a cross-attention layer between FFN and attention layer. The decoder shares the parameters of FFN and cross-attention layers with the encoder, which improves training efficiency and enables multitask learning. The causal self-attention layer allows the decoder to generate text tokens (\(y_{t}\)), conditioned on the previous tokens (\(y_{<t}\)) and the multi-modal representation (\(z\)). For a given input token (\(x_{t}\)), the output of the decoder at each time step \(t\) is:

\[y_{t}=f(x_{t},y_{<t},z) \tag{5}\]

### Overall Architecture

We encode the review text and image pair into a multi-modal representation and decode it into a sequence of verbatims (feedback segments) that are relevant to the image. Figure 1 illustrates the overall architecture of MINE. The encoding process consists of two steps: we apply the image encoder to the review image to obtain an image embedding; then, we pass the review text and the image embedding to the image-grounded text encoder to produce a multi-modal embedding that fuses both modalities. The decoding process uses the image-text grounded text decoder, which takes the multi-modal embedding as input and generates verbatim tokens conditioned on it. During training, we provide the ground truth verbatim as input to the decoder and optimize it to predict the next token. We fine-tune our model, with an objective of extracting the insightful segment by optimizing cross entropy loss:

\[L(\theta)=-\frac{1}{N}\sum_{n=1}^{N}\sum_{t=1}^{T_{n}}\log p(y_{ t}^{n}|y_{<t}^{n},x^{n},z^{n};\theta) \tag{6}\]

where \(N\) is the number of review-image pairs in the dataset, \(T_{n}\) is the length of the verbatim sequence for the \(n\)-th pair, \(y_{t}^{n}\) is the verbatim token at time step \(t\) for the \(n\)-th pair, \(y_{<t}^{n}\) is the sequence of previous tokens, \(x^{n}\) is the review verbatim, \(z^{n}\) is the multi-modal embeddings from the image grounded text encoder, and \(\theta\) are the model parameters. During inference, we only give the [DEC] token as input to the decoder and let it generate verbatims.

## 5 Experimental settings

### Dataset & Analysis

We obtain raw reviews dataset5 provided by Ni et al. (2019), which contains over \(233\) million reviews from \(29\) unique product categories. We used the sample subset (K-cores subset, released as part of the original dataset) which have \(973\)k reviews with images of \(27\) product categories for our analysis.

Footnote 5: dataset can be found here

### Prompt Engineering

We use the similarity scores that are obtained from the training data generation, as the confidence scores for each verbatim (see appendix section A.3 for an example). We compare three different prompting approaches for extracting actionable verbatim from the review text and image pair, and are as follows:

1. **Comprehensive Segment Extraction with Confidence Scores (CSECS)**: We generate all possible verbatim from the text and their scores based on how well they match the image. The ground truth includes all verbatim and their scores for each pair.
2. **Matching Segment Extraction with Confidence Scores (MSECS)**: We generate only the verbatim that match image and their scores. The ground truth includes only matching verbatim and their scores for each pair.
3. **Matching Segment Extraction (MSE)**: We generate only the verbatim that match the image without any scores. The ground truth includes only the matching verbatim predicted by the fine-tuned CLIP model for each pair. Refer appendix section A.4 for exact prompts and targets used during training.

### Approaches

As an additional multi-modal approaches, we tried ALBEF Li et al. (2021) and VL-T5 Cho et al. (2021) along with MINE. We use \(80K\) verbaltimage pairs sampled from all product categories to train both approaches.

**ALBEF**: Given that the training data was generated using a weakly supervised methodology and 

may contain noise, we employ ALBEF, a state-of-the-art robust multi-modal vision and language representation learning model. ALBEF leverages momentum distillation, a self-training technique, to glean knowledge from pseudo-targets, enhancing its resilience to noise within the training data. We considered the VQA setting6 for our baselines. ALBEF consists of an image encoder and a text encoder, followed by a 6-layer auto-regressive answer decoder, where we used the pretrained encoder weights and only finetuned the decoder for all the three prompting approaches, as mentioned in section 5.2.

Footnote 6: [https://github.com/salesforce/ALBEF/blob/main/VQA.py](https://github.com/salesforce/ALBEF/blob/main/VQA.py)

**VL-T5**: We fine-tune VL-T5, on our task of generating actionable verbatim from review text and image. We use Faster R-CNN Ren et al. (2016) to obtain \(36\) object features from the review image, which is concatenated with the tokenized input text consisting of the prompt question and the review text. The resulting sequence is fed into the bidirectional multi-modal encoder of VL-T5, which learns to encode both textual and visual information. The decoder of VL-T5 then generates the actionable verbatim as the output.

### Training Details

MINE was initialized using the pre-trained weights of BLIP base model and fine-tuned using AdamW optimizer Loshchilov and Hutter (2019) for \(10\) epochs with a cosine learning rate scheduler Loshchilov and Hutter (2017) and a minimum and initial learning rate of \(1e^{-6}\) and \(5e^{-5}\) respectively. In comparison, ALBEF was fine-tuned using \(128\) as batch size for \(20\) epochs with the following hyperparameters: AdamW optimizer, a learning rate of \(1e^{-5}\), \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\epsilon=10^{-8}\) and a weight decay of \(0.05\). Similarly, we fine-tuned VL-T5 for \(5\) epochs using AdamW optimiser with a learning rate of \(5e^{-5}\) and linear warmup scheduler of \(5\%\). At inference, we adjusted the max length to \(512\) to enable the model to generate longer and more relevant verbatim. Additionally, we employed an image resolution of \(225\)px, consistent with the BLIP base model. We also experimented with different decoding methods for MINE, such as beam search Hu et al. (2015) (\(beam\_size=10\)), consistent top-k sampling Welleck et al. (2020) (\(top\_k=50\)) and nucleus sampling Holtzman et al. (2019) (\(top\_k=50\) and \(top\_p=0.95\)).

### Results

We found that MSE approach with beam search decoding produced the most complete and relevant verbatim. We compared different prompt settings and approaches, and measured the quality of the generated verbatim using precision (correctness), recall and completeness metrics. We gave the exact definitions of these metrics in appendix section A.2. Table 1 summarized the results of our experiments. We see that our approach is effective than existing baselines like ALBEF and VL-T5 in both identifying the exact intent and extracting the complete verbatim, present in the image. In addition to beam search, we experimented with different decoding methods for MINE and reported the results in Table 2.

## 6 Conclusion

In this paper, we introduced a novel architecture MINE, for extracting insights from multi-modal customer feedback encompassing both text and images. We proposed a weakly-supervised data generation approach that leveraged raw feedback text and images to create relevant verbatim and image

\begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **Prompting Approach** & **Precision (Correctness)** & **Recall** & **F1-score** & **Completeness** \\ \hline  & CSECS & 0.61 & 0.52 & 0.56 & 0.71 \\
**ALBEF** & MSECS & 0.63 & 0.59 & 0.61 & 0.76 \\  & MSE & 0.65 & 0.60 & 0.62 & 0.80 \\ \hline  & CSECS & 0.65 & 0.53 & 0.58 & 0.76 \\
**VL-T5** & MSECS & 0.65 & 0.57 & 0.61 & 0.82 \\  & MSE & 0.66 & 0.61 & 0.63 & 0.89 \\ \hline  & CSECS & 0.69 & 0.70 & 0.69 & 0.73 \\
**MINE** & MSECS & 0.71 & 0.76 & 0.73 & 0.91 \\  & MSE & **0.76** & **0.77** & **0.77** & **0.93** \\ \hline  & & & & & \\ \end{tabular}
\end{table}
Table 1: Experimental results 

pairs. We also proposed an unsupervised insight extraction approach that used an image-grounded text encoder to learn latent representations of the feedback text and corresponding images, and an image-text grounded text decoder to extract actionable verbatim relevant to image. We evaluated our approach on real-world dataset and demonstrated its effectiveness in extracting actionable insights from multi-modal feedback with minimal supervision, achieving a 14-point improvement in F1-score over the existing baselines and uni-modal insight extraction.

**Future work:** Our research opens up several avenues for further exploration. As a future direction, we would like to extend our method to video feedback by incorporating the temporal dimension and extracting relevant video snippets, as mention in the feedback text. We also aim to structure the extracted insights in hierarchical way that facilitates decision making for users. Moreover, we plan to localise the regions of interest in the feedback images that correspond to the verbatim, using bounding boxes, to help identify the specific features or issues that customers refer to in their feedback. Finally, we intend to generate suggestions for product enhancements or fixes based on the extracted insights and the customer sentiment.

## Limitations

Like any other ML architecture, even MINE architecture has some limitations. One of the limitations is that the MINE architecture is not pre-trained with language modelling as an objective (such as Pre-fix language modelling-style / Deshuffling-style / i.i.d noise, replace spans-style (Raffel et al., 2020), or BERT-style (Devlin et al., 2019), or MASS-style (Song et al., 2019)). This means that the model may not always generate responses that are identical to the feedback texts in terms of wording, even though they generate text the same intent. We found that around \(19\%\) of the cases had different wording but with same intent. Another limitation is that the extraction task can be computationally demanding with a higher number of beams, which may affect the performance and scalability of the approach using low compute resources.

## Ethics Statement

In this paper, we used a publicly available dataset in an almost unsupervised fashion, with minimal reliance on human annotations for labeling. Annotators are subject matter experts who specialize in annotating image-text samples, whether they are relevant or not. They are compensated accordingly, following industry standards set by the organisation. We carefully considered all aspects related to annotator agreement and fairness. However, the dataset is anonymized and does not contain any identifiable or traceable information. Thus, we respect the privacy and confidentiality of the consumers of the product and do not expose them to any potential harm or misuse. The dataset is widely used and well-known in the natural language processing community. It has been previously analyzed and evaluated by several researchers and practitioners. Our work does not introduce any bias or prejudice either, as we do not make any assumptions or judgments based on the feedback text or images. Our work maintains a purely empirical and objective approach, without favoring or disadvantaging any individual or product. Throughout our research process and reporting, we adhered to the ACL code of ethics and professional conduct.

## References

* Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
* Artstein (2017) Ron Artstein. 2017. Inter-annotator agreement. _Handbook of linguistic annotation_, pages 297-313.
* Artetzky et al. (2016)

\begin{table}
\begin{tabular}{c c c c c c} \hline
**Prompt Type** & **Decode method** & **Precision (Correctness)** & **Recall** & **F1-score** & **Completeness** \\ \hline \multirow{2}{*}{**CSECS**} & topK & 0.57 & 0.48 & 0.52 & 0.66 \\  & nucleus & 0.62 & 0.6 & 0.61 & 0.71 \\ \hline \multirow{2}{*}{**MSECS**} & topK & 0.6 & 0.59 & 0.59 & 0.69 \\  & nucleus & 0.65 & 0.69 & 0.67 & 0.83 \\ \hline \multirow{2}{*}{**MSE**} & topK & 0.57 & 0.7 & 0.63 & 0.72 \\  & nucleus & 0.68 & 0.72 & 0.7 & 0.86 \\ \hline \end{tabular}
\end{table}
Table 2: Additional Experimental results 

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. _arXiv preprint arXiv:2106.08254_.
* Bao et al. (2022) Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. _Advances in Neural Information Processing Systems_, 35:32897-32912.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.
* Chen et al. (2020) Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020. Counterfactual samples synthesizing for robust visual question answering. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 10800-10809.
* Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Devlin et al. (2021) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
* Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoeit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. _ICLR_.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Eleventing the role of image understanding in visual question answering.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. _arXiv preprint arXiv:1904.09751_.
* Hu et al. (2015) Xiaoguang Hu, Wei Li, Xiang Lan, Hua Wu, and Haifeng Wang. 2015. Improved beam search with constrained softmax for NMT. In _Proceedings of Machine Translation Summit XV: Papers_, Miami, USA.
* Kim et al. (2021) Wonjae Kim, Bokyang Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision.
* Li et al. (2020) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Dexn Jiang. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 11336-11344.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _ICML_.
* Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation.
* Li et al. (2019) Liumian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language.
* Liu et al. (2022) Yang Liu, Varnith Chordia, Hua Li, Sixasah Fazeli Dehkordy, Yifei Sun, Vincent Gao, and Na Zhang. 2022. Leveraging seq2seq language generation for multi-level product issue identification. In _Proceedings of The Fifth Workshop on e-Commerce and NLP (ECNLP 5)_, pages 20-28.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017. Sgdr: Stochastic gradient descent with warm restarts.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
* Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.
* Lu et al. (2023) Siyu Lu, Yueming Ding, Mingzhe Liu, Zhengtong Yin, Lirong Yin, and Wenfeng Zheng. 2023. Multiscale feature extraction and fusion of image and text in vqa. _International Journal of Computational Intelligence Systems_, 16(1):54.
* Luo et al. (2022) Hanyang Luo, Wanhua Zhou, Wugang Song, and Xiaofu He. 2022. An empirical study on the differences between online picture reviews and text reviews. (7).
* Mukku et al. (2023) Sandeep Sricharan Mukku, Manan Soni, Chetan Aggarwal, Jitenkumar Rana, Promod Yeniglalla, Rashmi Patange, and Shyam Mohan. 2023. Insightnet: Structured insight mining from customer feedback. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_, pages 552-566.
* Nguyen et al. (2021) Minh Van Nguyen, Viet Duc Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. In _Proceedings of 

the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_.
* Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, Hong Kong, China.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sasruty, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Michael Narang, Adam Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. In _Conference on Neural Information Processing Systems (NeurIPS)_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.
* Ren et al. (2016) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster r-cnn: Towards real-time object detection with region proposal networks.
* Sircar et al. (2022) Prateek Sircar, Aniket Chakrabarti, Deepak Gupta, and Anirban Majumdar. 2022. Distantly supervised aspect clustering and naming for e-commerce reviews. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track_, pages 94-102.
* Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation.
* Wang et al. (2022) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. 2022. Image as a foreign language: Beit pretraining for all vision and vision-language tasks.
* Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020. Consistency of a recurrent language model with respect to incomplete decoding.
* Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(07):13041-13049 

Appendix

### Verbatim Extraction

We use the following steps to extract verbatims (actionable feedback text segments) from raw customer feedback source:

1. **Text Pre-processing:** We apply text pre-processing techniques to remove html tags, urls, accepted characters, extra spaces and other noise from the feedback text.
2. **Segmentation:** We use Trankit Nguyen et al. (2021), a fast and lightweight transformer-based toolkit for natural language processing, to segment each feedback into meaningful units. We also implement some heuristics to avoid having segments with only single word.
3. **Removing non-actionable (neutral) segments:**We use a RoBERTa Liu et al. (2019) based 3-class sentiment classifier7 to evaluate the sentiment of each feedback segment. It has been rigorously evaluated and demonstrates an impressive 92% F1-score on our specific sentiment classification task. We make the decision to exclude segments classified as neutral, as these segments may not provide actionable feedback for brands and selling partners who seek feedback to improve their products and services. Footnote 7: [https://huggingface.co/siebert/sentiment-roberta-large-english](https://huggingface.co/siebert/sentiment-roberta-large-english)

### Verbatim evaluation metrics

1. **Completeness**: This metric measures the proportion of segments that convey a complete meaning. A segment is considered complete if it expresses a coherent and relevant idea about the feedback. Completeness \[=\] \[Number~{}of~{}complete~{}segments\] \[Total~{}number~{}of~{}segments~{}generated\]
2. **Correctness/Precision**: This metric measures the proportion of segments that match both the text segment and the image. A segment is considered correct (relevant) if it accurately reflects the information and sentiment from 



### Training Data

### Experimented Prompts

\begin{table}
\begin{tabular}{l l l} \hline \multicolumn{1}{c}{**Task**} & \multicolumn{2}{c}{**Input Prompt**} & \multicolumn{1}{c}{**Target**} \\ \hline  & Extract all the verbatim and confidence & _<verbatim 1>;_ _<confidence 1>_ | _<verbatim 2>;_ \\ CSECS & score of each matching with image? & _<confidence 2>_ \\  & Feedback: _<feedback text>_ & _<matching verbatim 1>;_ _<confidence 1>_ | \\ MSECS & score of each matching with image? & _<matching verbatim 2>;_ _<confidence 2>_ \\ MSE & _Weak is the verbatim matching with the_ & _<matching verbatim 1>_ | _<matching verbatim 2>_ \\ \hline \multirow{3}{*}{_MSE_} & Extract all the verbatim and confidence & _<matching verbatim 1>_ ; _<confidence 1>_ | \\  & score of each matching with image? & _<matching verbatim 2>_ \\ \cline{1-1}  & Feedback: _<feedback text>_ & _<matching verbatim 1>_ | _<matching verbatim 2>_ \\ \hline \end{tabular}
\end{table}
Table 4: Experimental Prompts

\begin{table}
\begin{tabular}{p{56.9pt} p{85.4pt} p{70.7pt} p{70.7pt} p{70.7pt}} \hline \multicolumn{1}{c}{**Input**} & \multicolumn{4}{c}{**Output**} \\ \hline \multicolumn{1}{c}{**Feedback Text**} & \multicolumn{1}{c}{**Feedback Image**} & \multicolumn{1}{c}{**Feedback Image**} & \multicolumn{1}{c}{**Verbatim**} & \multicolumn{1}{c}{**Similarity Score**} \\ \hline  & & & They don’t look nice & 0.19 \\ \cline{3-6} \multirow{5}{*}{They don’t look nice. It looked nice for a brief period of time; then the finish came off, that fadeout the pleated color and became brownish. The one I’d purchased elsewhere before lasted eyes, looked nice and was retired only because the ”stone” was lost.} & & It looked nice for a brief period of time & 0.12 \\ \cline{3-6}  & & & then the finish came off & 0.29 \\ \cline{3-6} \multirow{5}{*}{They don’t look nice. It looked nice for a brief preid of time; then the finish came off, that faceout the pleated color and became brownish. The one I’d purchased elsewhere before lasted years.} & & fadeout the pleated color and became brownish & 0.31 \\ \cline{3-6}  & & & The one I’d purchased elsewhere before lasted years & 0.13 \\ \cline{3-6}  & & & looked nice and was retired only because the “stone” was lost & 0.21 \\ \cline{3-6} \end{tabular}
\end{table}
Table 3: Training Data



### MINE: Sample Predictions

\begin{tabular}{c|l l} \hline \multicolumn{2}{c|}{**Model Input (Multi-modal Feedback)**} & \multicolumn{1}{c}{**Model Output**} \\ \hline
**Text** & **Image** & **Image** & **Verbatim** \\ \hline \end{tabular}

Paint is outside the lines and it really looks sloppy. So, i really dont like it. It looks so bad. I would not order this it is product. And it was rolled up like a newspaper. Complete and total waste. The paint is pasted on the nose as well.

I was disappointed. The shoulder straps did not have anything to secure the loose strap so you essentially have a loose strap getting in the way. This makes the purse look really cheap. Inside, the zipper is misaligned so when it is zipped, you see a gap. When I try to rezip, it doesn't correct itself. Lastly, the inside pocket must have been overseven, because when you open the pocket, you can see the bright green threading. Lastly, the gold-plated claps that connect the purse and shoulder straps together were also not aligned.

I wish the product to be good and expecting for it. Got the product just by today morning. It never met my expectations and I am totally disappointed!. This set is really small. The shakers don't fit in the holder and the holder is pathetic! Not made well at all!!!

I've had these boots for a few months and only wear them when I ride (less than twice a week). For no reason at all the heel on the right boot fell off (in half). Looks like the heel is made of particle cardboard. I would have expected that boots in this price range would not fall apart under light use in just a few months! What a rip off!

I was very pleased with the print and very excited to hang these up. We had a room that was lonely and in need of some fun print, these looked like the perfect fit. I was very disappointed once they were hung. The packaging stated 95" length; what I received was one of 88" and the other of 92". Although the print was nice, what can one do with them? very disappointed in this purchase.



* [150] M. C. Bader, A. K. Goyal, and J. R. Friedman. The role of the social network in a social world. In _Proceedings of the 23rd ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 46-78, 2009.
* [151] S. D. Lee, P. L. McAllister, and T. W. Teh. Social networks: A survey. _Social Networks_, 2(1):1-10, 2009.
* [152] H. N. Kim, Y. Li, and X. Zhang. Social networks for social networks. In _Proceedings of the 20th International Conference on World Wide Web Search_, pages 1-10, 2009.

[MISSING_PAGE_POST]