D$^2$TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization
Authors: Yunlong Liang
Fandong Meng
Jiaan Wang
Jinan Xu
Yufeng Chen
Jie Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Summarization; Cross-lingual Summarization; Many-to-many Multimodal Summarization
TL;DR: A novel framework for Many-to-many Multimodal Summarization

Abstract: Many-to-many multimodal summarization (M$^3$S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises of multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. Although much work has been devoted to either MMS or MXLS, little research pays attention to the M$^3$S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. In this paper, we first introduce a general and practical task, \emph{i.e.}, M$^3$S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M$^3$S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information. Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach. Additionally, we contribute a many-to-many multimodal summarization ({\fontfamily{lmtt}\selectfont M$^3$Sum}) dataset with 44 languages to facilitate future research.

D'TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization
Yunlong Liang', Fandong Meng', Jiaan Wang', Jinan Xu Yufeng Chen' and Jie Zhou? 'Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China
Pattern Recognition Center, WeChat AI, Tencent Inc, China School of Computer Science and Technology, Soochow University, Suzhou, China [yunlongliang jaxu, jebjjn. edu. cn [fandongmeng, wi wwttttoooooooooeoeneettte com
jawang nlp@gmail. com
Abstract
Article:[ J.ne werry is that this would trigger panicked
reactions domestic investers and
stock market
cred.....) China's central bank has beez repeatedly propping
Many-to-many multimodal summarization (M task aims to generate summaries in any language with document inputs in any lan- guage and the corresponding image sequence, which essentially comprises multimodal mono- lingual summarization (MMS) and multimodal cross- cross-lingual summarization (MXLS) tasks. Al- though much work has been devoted to either MMS or MXLS and has obtained increasing attention in recent years, little research pays attention to the S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation with- out considering the performance of MMS or 2) improving MMS models by filtering summary- unrelated visual features with implicit leamn ing or explicitly complex training objectives. In this paper, we first introduce a general and practical task, ie., S Further, we propose dual knowledge distillation and target-orientee vision modeling framework for the S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet ef- fective target-oriented contrastive objective is designed and responsible for discarding need- less visual information. Extensive experiments on the many- to- many setting show the effective- ness of the proposed approach. Additionally, we will contribute a many-to-man multimodal summarization (Mm) dataset.'
up the slock market to ensure stability I...]. But that did not
happen cauzing panic ripple shares Monday. One of the possibble niggers
dramatic drop
G
deop
in post trading scasions was the carlecr decision by the
central bank devalos the youn and allow inde
flesibly. I...I. sending first wave insecurity through markets China's stock market slump caused investor
uncertainty to spread across the region and then around the
destabilising stock markets New York and Europe.
[...] But now the libles lave turned, ways. "The gluaa ocunomy and global markets have a "Mack in Chinu label on them. " Mondin's plobal turnoi sparked fears another intemaaioonl financial meltdown. They are however waming of furmer stumps in the long run. I...] The turnool caused by Chine's stnck stump "suggesedd that the great unwind of the beckoning". [-]
(MMS) English Summary: The reperenesions from THllkk Monday" " the global markets
by plunge Chines: stucks conkine fell...
(MNLS) Russion Summary:
supeore uunca:o-ues
ca0GEnems puncriuxx nuppacaraal, su sumammes masemmn tarnileaooo douaoooro pumna speannett nponncrmcc no mrogrin.
(MXLS) Unde Summary: cur A LA ue
-
(MXLS) Indonesiun Summary: Konsekusi dari "Senin llitmm" gangguan pasar glohal dischabkian olch penurunan saham terus temasa pada Selasa.
An example of our S dataset. Inputs: an article and corresponding image sequence; Output: summaries in different languages. MMS: the summary in the same language as the input article; MXLS: the summary in different language from the input article. The S setting covers both MMS and MXLS.
Figure I:
language (i.e., English) while the goal of multi- modal cross-lingual summarization (MXLS) is to produce summary in a different language (e.g., Chinese). With the rapid increase of multimedia data, the MMS (Tjondronegoro al., 201 1; Evan- gelopoulos al., 2013; Erol et al., 2003: Li et al., 2017, 201 8a; Sanabria al., 2018; Zhu et al., 2018; Chen and Zhuge. 2018: Li et al., 2020a; Fu et al.. 2021; Zhao et al., 2022) and MXLS (Liu et al., 2022) tasks have attracted much attention in the research community because both tasks can help users quickly master the core idea from the cum- bersome multimodal data. Essentially, the many many multimodal summarization (M²S) consists of MMS and MXLS tasks, which generate summaries in any language given the multimodal inputs (in any language), as Fig. shows. Intuitively, the many-to-many setup should be more general and
1 Introduction
Given a document input in the source language (e.g., English) and its corresponding image se- quence, multimodal monolingual summarization (MMS) aims to generate a summary in the same
Work was done when L Liang and Wang was interning Pattern Recognition Center, WeChat AI, Tencent Inc, China. Jinan Xu is the corresponding author.
'The code and data are publicly available at https:// gi thub. cww/cL 2248 2248/D2TV. /D2TV.

---

practical for its application in the multilingual and multimodal world (Wang al., 2023b).
involves 44*44 language directions. To efficiently evaluate our approach, we randomly select 4 lan- guages (i.e. English, Indonesian, Russian, and Urdu"), which consist of 4 language directions. We implement our approach grounding on two gen- crative pre-trained language models, i.e., mT5 (Xue et al., 2021) and mBART-50 (Tang et al., 2021). Extensive experiments both backbones show that our model significantly outperforms related methods in terms of ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2020) scores, demon- strating its effectiveness. The human evaluation further suggests the superiority of our approach. summary, our main contributions are:
In the literature, although plenty of studies have been carried out on MMS or MXLS, there is only one study that involves both of them, i.e., Liu et al. (2022) devise triple-stage training frame- work and distill the knowledge from MMS to en- hance MXLS while ignoring the performance of MMS. Despite their effectiveness on MXLSS to our knowledge, little research attention has been paid to simultancously supporting both MMS and MXLS tasks and prompting both of them. Besides, the visual features generally include noise which is summary-unrelater Thus the remaining work mainly focuses improving MMS models by fil- tering these noises with (a) implicit learning (h) complex training objectives. For (a), researchers design various fusion methods effectively model the interactions between textual articles and visual features (Liu et al., 2020; Yu et al., 2021; Palaskar et al., 2019; Zhang et al., 2021a). For (b), to ex- plicitly filter needless visual information, Liang et al. (2022b) present two well-designed auxiliary tasks, i.e., vision to summary and masked image modeling. Albeit effective, implicit learning via the MMS objective may limit the potential of visual features, and explicit training objectives are com- plex and consuminn to be trained and applied in the real world.
To the best of our knowledge, we are the first that introduces the general many-to-many multi- modal summarization (M'S) task and contributes a corresponding benchmark dataset.
We propose dual knowledge distillation and target-oriented vision modeling framework for the M task.
Experiments on Smm benchmark show that our model builds new ofate-of-the-att performance, showing the effectiveness of the proposed ap- proach.
Background
2.1 Problem Formulation
Given input article x-==(=/) k== in lan- guage L1 and its corresponding visual features V-(0%) ) =ly=1
To address these issues, in this paper, we first introduce more general task, Le., MDS, which supports both MMS and MXLS tasks. Fur- ther, we propose Dual knowledge Distillation and Target-oriented Vision enhanced framework, named D*TV, for the new task. Specifically, the dual knowledge distillation approach ensures that the knowledge from MMS can be transferred to MXLS and vice versa, and thus mutually improve both tasks. Furthermore, discard the summary- unrelated visual information, target-oriented con- trastive objective is devised directly optimize the visual features. In this way, the model en- hanced to explicitly exploit the summary-orientee visual features, thereby yielding more accurate summaries.
Sin.jsm where z/41 denotes the k-th to-
ken, and M is the number of tokens in the arti- cle, and Biy represents the detected j-th object of the i-th image (n, m is the number of images and detected objects in each image, respectively), the many-to-many multimodal summarization task is defined as:
p(y x V) = II plut 1 .V..Va V.vE)),
where vha indicates the tokens before the I-th time step of the summary ->===(y(" 12 Ji=1 in language
L2 and N is the number of tokens in the summary. The I and L2 can be any language.
To validate the D²TV framework, pro- vide Many Multimodal Summarization (Mm) benchmark dataset by reorganizing the cioss-lingual summarization dataset (Bhattacharjee et al., 2022) and MM-Sum dataset (Liang et al., 2022b). The M Sum covers 44 languages and thus
2.2 The MMS Model
Following Yu et al. (2021); Liang et al. (2022b), the MMS model is an extension of the pre-trained lan- guage model (e.g., mT5 (Xue al., 2021)) based
Urdu is the low-resource language.

---

on Transformer architecture (Vaswani al., 2017). As shown in the left part of Fig. 2, it includes four modules: textual encoder, visual encoder, text- vision fusion, and decoder.
modal multi-head attention (CrossMAttn):
M = CrrssMAAtn Q K, V), M E RMx,,
where Q are the projected textual features Q q. K and V are the projected visual features
y W,, with different weights, i.e., K H WR. V = h. We: e. and Q E R Mxde K, Ridd and de is the common hidden dimension.
Textual Encoder. The textual encoder consists of No stacked layers, where each layer consists of two sub-layers, a multihead self-attention sub- layer (SelfAttn) and a position-wise feed-forward network (FFN) sub-layer:
Secondly, forget gate G is used to filter redun- dant and noisy information from the visual features:
R - = SelfAttm H2 1) + H S E Rxdd HG = (S+) + Sy, H4 R-Axd
G = Sigmoodd(ConnatttH) M)W,, + bb), Zy G M.
where H and Hr, denote the inputs and outputs of the b-th encoder layer, respectively, and H. is initialized as the embedding of input tokens 2 and d is the hidden dimension.
Finally, the vision-guided output Zy V is con- catenated by Zy and textual features H and then linearly project it to the original dimension d:
Visual Encoder. Following Yu et al. (2021); Liang al. (2021, 2022c,a); Zhang al. (2021a,b), the visual encoder is also the Transformer (Vaswani et al., 2017) encoder with N₂ stacked layers. The difference is the visual imputs. Generally, there is an image sequence to be extracted by the Faster R-CNNs (Ren et al., 2015) pre-trained on Vi- sual Genome (Krishna et al., 2017). Specifically, for the i-th input image. we obtain set of de- tected objects from Faster R-CNNs, i.e., {or,1, 02,2, 01,3, ..., J,,), where is the number of extracted objects and o,,, E Rdd. Each ob- ject is captured by dense feature representation, which can be mapped back to a bounding box region (i.e,, Region-of-Intered (Rol)). Finally, the image sequence is converted to visual features =(og)) Following Cho et al. (2021), the Rol bounding box coordinates Ely image id em- bedding E and region id embedding E are added on the visual features to keep the order infor- mation of the image sequence:
ZT+V Concat(H No Zv) + +b,,
where Concat is the concatenation operation and W. and b. trainable weights.
Decoder. The decoder follows a similar architec- ture but each of Na decoder layers has an additional multi head cross-attention (CrossAttn) sub-layer:
Secee SclfAttn(H ) HH S E ERNxd
Ch.. CrossAttn(S",.. ZT+V) + St
(1)
H FFN + Caees C..o E RR
where H Rxx denotes the state of the E-th decoder layer. Then, at each decoding time step f, the top-layer (Na--th decoder hidden state Zd is fed into the softmax layer to produce the probability distribution of the next target token as:
(r, O, V<rr) = Softmax| W. Z.lee + bo),
where W. and b, are trainable weights.
Finally, the loss function is written
N
CLا. MMS
Σ
log(p(sh la Iw'a 20, ,)). (2)
Vij oi + E + Eime + EE n.j m.
Then, they are fed into the visual encoder for better modeling the intramodal dynamics and enhancing the vision-specific order information.
DºTV Training Framework
S SelfAttm(H + H.! St E Vixd, H FFN(S)) + H.- E Rlxxd.
Based on the MMS model described in $ 2.2, we firstly introduce the proposed dual knowledge dis- tillation (DKD) method in $ 3.1, which improves both MMS and MXLS tasks. Further, we present simple yet effective target-oriented contrastive ob- jective to filter needless visual information in $ 3.2. Finally, we describe the training and inference in $ 3.3.
where H²-1 and H. denote the inputs and outputs of the /-th encoder layer, respectively, and HD is J.n.jšm
initialized as the z=(v;;) =1,== and de is the hidden dimension.
Text-Vision Fusion. Following Yu et al. (2021), the visual features are firstly injected by cross-

---

Zr=P
yuy
M
LAXES
LAMS
Text- Sixt-iision Fusion
Softmax
1-95
Ltco
ands
y mime
Haum
Hr
Hi
Decoder
LDD
Textaal Encoder
Visual Encoder
MXLS
shared
MMS
Textual Encoder
Visual Encoder
X
y4
V
(a) DKD
(b) TCO
The MMS Model
Figure 2:
The overview of our model architecture. The left part is general MMS model, which is enhanced by DKD and TCO. As shown in the right part, the (a) dual knowledge distillation (DKD) and (b) target-oriented contrastive objective (TCO), are proposed to improve the M model performance.
3.1 Dual Knowledge Distillation
cross-entropy form:
As shown in the right part of Figure 2 (a), our framework involves training both MXLS and MMS models. Essentially, the MXLS model needs to simultaneously conduct machine translation and summarization (Liang al., 2022d; Wang al., 2022a.b) while the MMS model only conducts summarization. Obviously, is harder train an MXLS model than to learn an MMS model and that is why researchers (Nguyen and Luu, 2022: Liu et al., 2022) take the MMS model as the teacher to help the MXLS student model (i.e., (eacher-xstuded distillation). However, when the MXLS model achieves level of multilingual and cross-lingual ability, the MXLS model can better transfer and share lask knowledge among different languages. Therefore, the MXLS model, in turn, can guide the MMS model to conduct summariza- tion diverse languages (e.g., English- English, -onoeesiam--noonsiiaan Russian-RRssiann and Urdu->Urdu), especially for low-resource ones (i.e., student-+teache distillation). That is why we propose DKD to mutually enhance their perfor- mance.
MXLS = Σ log(p(w( Wel 1 XL2, V), (4)
and a is the trade-off factor and - LD represents KD loss to penalize the large distance of two hidden states of two summaries generated by the student and teacher models:
T(Li I(L,,La)-S(La..1)
kD S1)aadh)) dist (II N
II (a)
doc
dog
(5)
where .)) is the distance function to evaluate the difference between two representations (e.g.
KL and cosine similarity), and Hec Na.T(Ln,L1) (hT b ..., hr } denote the contextualized rep- resentations produced by the decoder of the teacher model, and H dec
S( {h h ..... h de-
note the representations from the decoder of the student model.
particular, given the in- put document X = {xf an , M2 in lan- guage L1 and corresponding visual features V, the teacher model aims to generate its summary y in the same language. We update the parameters of the teacher model with the following objective:
Student-TTeacher
Teacher-Student Specifically, for training the student model, given an input X L2 the Mi in language L2 and corre- sponding visual features V. the student model is to generate the cross-lingual summary yL " 1/2" ...., La) ,'',,, where L2 + L1. Then, loss: train the student model with two objectives as fol- lows:
Li .LI.I + (1-a) a Site (i teacher MMS
{ata L2
(6)
S(22,11) (T((1, denotes the inverse KD where L
(.L.,))-T([L dist (IT NoL(L)
II Na
doc
doc
(7)
student M + of ((ln,aa)
Finally, to flexibly distill the knowledge in Eq. and Eq. 6, we apply an annealing strategy dy- namically adjust the balancing factor a:
(3)
where Llxds denotes by maximizing the likeli- hood of the ground-truth tokens which takes the
a = (0.5 t1/T1),
(8)

---

where (1 is the training step ranging from 0 to the max training step T and 7 is hyperparameter. In this manner, the teacher model dominantly guides the student model the first 7 training steps and the student model gradually distills the knowl- edge to the teacher. After training step 71,, both models begin to equally distill their knowledge to each other.
3.3 Training and Inference
At training, we train our model with the following objective:
(4 + Lreacher + 3 (3Cfco):
j=1,jii
(10)
where K is the number of languages and / is balancing hyper-paramett
Note that the MMS model and the MXLS model are shared and thus the final model can conduct summarization in any language. During inference, the training objectives are not involved and only the model is used to generate summaries.
3.2 Target-oriented Contrastive Objective
The MaS task requires a model to have the ability to understand and generate in multiple languages. However, there are some languages that are low-resource and lack enough data to aim
Experiments
train good summarizer. Therefore, take visual features as the bridge between languages and hope that the visual features can be summary-oriented i.e., discarding the noise that not appeared in the summary. To this end, we elaborately design an explicit target-oriented contrastive objective. Particularly, we push the visual feature V; close to its corresponding summary D and push apart irrelevant pairs, e.g., (V,, 2 where * j. Therefore, we treat the paired (V,, D111) as the positive sample and treat the pair (V; 1/11 as the negative samples where i * j. To obtain the representation of summary and image sequence, with mask operation over the summary output H H of the Ne-th encoder layer and visual output H. of the Ny-th encoder layer, respec- tively. That is, - - E== (M Manla h E Rd. where M E R denotes the mask matrix, whose value is either or O indicating whether the token is padded. Similarly, we obtain the representation of image sequence, i.., h mis 1
4.1 M Dataset
There is many many-to-man -toymnyy multimodal summariza- tion benchmark dataset until now. We construct as follows. Based on the CrossSum dataset (Bhat- tacharjee et al., 2022) and MM-Sum dataset (Liang et al., 2022b), we construct a Many-to-Mang Multimodal Summarization (M'Sum) dataset. The original CrossSum dataset is crawled from the BBC website and its quality has been verified and ensured reliability by Bhattacharjee et al. (2022). However, the lack of associated image sequence in CrossSum, makes it impossible directly con- duet research on MMS and MXL.S. The original MM-Sum dataset also crawled from the BBC website, which includes multilingual multimodal summarization. But it cannot conduct cross-lingual summarization due to the lacking of cross Prossllnggual alignment. Therefore, we reorganize both datasets and conduct cross-lingust alignment through the same ur! in each dataset.
apply mean-pooling
According to the dataset size of each language, we follow CrossSum (Bhattacharjee et al., 2022) and utilize about 80% training: 10% validation: 10% test splitting. Besides, in CrossSum, the number of languages is 44 and thus there are 44*44 lan- guage directions. Tab. 4 of Appendix A shows the detailed statistic of our Sa and please refer to it for details.
men Ex= (MF))LL )), his C RRd, where Moss C Raxm denotes the mask matrix and MLP is a fully-connected layer. Finally, the target-oriented contrastive training objective is defined by (B is mini-batch size):
e sim(hils, hi D//T
4.2 Setup and Metrics
c/co -log Σe sim (h. pis
Implementation Details. For efficiency, we ran- domly select 4 languages (i.e., English, Indonesian, Russian, and Urdu), which totally cover 16 lan- guage directions. Please refer to Appendix B for
b/77
where sim(.-) is the cosine similarity and 7 de- notes a temperature hyperparameter.
ttttps://www. com/

---

Tig
Modeis
Engissh
Indonciia
Rusian
Undo
Ang.
MSIS
S6.677 13167
1947 633783.9
12170.557
BDST 01475544
11207 30077
MXLS
6.947
25.23/ 9.32/ 22.13/
22.52/7.88/ 18,00
31.27 11.17/25 66.29
22.347
Faglish-.
MMS+MMKS
13,45/23
2066
27
9.201 20///22/4/888
65
8,94/ /2.95
9.00/23007
Yamillo-D
54,00/ 12.70 20.700266.60
26.86 70007
27:25/
0.71/2.65/688
24.36/
5.00 165441
31.53 (00255/24.82 /6.23/66/667666
20.56 10.17/ 63.43
(Orr)
36.12/1..21 / /27.99/70.4
2887/10026/20 200.2//3...7///// 69.31
0555//509/20/72/660 66.0/
32.96/18.73/
/6.29
36.77 18.22/24.55 24.53/6833
MSNS
7:88
1.00/ 6.73/63.5
3:10/1501/1702/11
119/0.19:
0.13/ 0.01/ 0.13/5642
10:577 4.017 66.5/
MXLS
32.33
5.39/ 1.59/ 4.58/66.
21:65/7.00/1..22 65.02/65.00
91.85 (1.00/ /6.53
22.531
7.021 18.24/
Indenssian-..
MMS+MMISS
52. 23.99/ 1..57 |25.5255
69.53
14.3//1456./2
/28.3/771.30
24 34.56/4.70
/66 18
Wo.5 10. 30) 22.99/66700
90.1 11.51 /1//2/2.2 70:66677
Vamilla-DD
52.58/ 11.56 25.45/ 660
1267/1501/26..
25.50/5.90/ 20.65 22265/666.30 30
$2.4/11.31/ (0111.1//..... 26.68/68.79
30.88
11.21 24.67 /6.80
D'TY (Ours)
34,4412,10/26.55
69.73
35.4/14.55 22805/251111
26,00/9,27// 211.227/22.//66.63
26.00/
32.06/
MYIS
1:347 0.22) 1/23/43:20
1.23/ 0.22/ 1109/
M1.36/11.88 24.25/ 66..28/68..0
0.11/ 0.00/ 0.11/56.10
5.227 3.023 6.69/6133
MILLS
29.477 9.56 1.56/225 202.222226620 10
25.78/
9/06/2100 /60.20
2.88/ 0.901 2.41/ /5217/9922
31.06/10.4/
25.15 rek.s
22.247
7.6. 11.22/00000 00.22/00.8
Rusedan-
MMS+MXIIS
50,79/
9.80/24.07/00
74
27.571 9.84/22
/2.70/ /8.15
29.67/ 11 62/66/22335 /4.35/66.00
96.51 10.04/044 24 24-92/66730
28.59/
59 23 /68.500
Vamilla-D
10.54/ 24.26//66.00
9:60/21.01
101101//00 /68.73
28.32/ 10.08/21.00//00000
31.38 10.76//2 25.25
29.34
10.48/22..0/ 23.00/
(Ours)
32.5711180 25.99) 69:14
29/13/
/9.59/00.99//
23.64/ 60.59
20.90/11444
33.29/ 11,88/ /26.6/699.1
11.27/11.24/ 25.13 /6..9
MSSS
0509/55.56
0.05/ 0.00/
0.09
0.06/ 0000/56.00
31. 17.54/15 15 04/ 700 /0.55
9:487 3,76 7660/ 59.64
MXLS
29.55 /9.06/ 22000//22...8.22220
4.37/
21.52 17.23:665.5
7.707 2.02/
6.1.6
21.29
10.97
MMS-MMXLS
26
51.72/ 15 30101/00000
10.17/
Vanilla-KD KD
20 00.60/988 /33.00//7766
26, 30/ 902/ 15 68.29
22.58 72.58/7..6// 18.03 22
37.50 25/ 10/
20,007 10.31
D' TV (Ours)
32 /9.99/24.7//////
28.23/0000 2233//000/333///// /23.19/90.25
24.53/7.877 0122//877/9996666 /9.99/66.133
35,05 /1111//1..3//////
/0.70/ 10.0//0.////00 24.71 68.75
(a) The test results based on the mT5 backbone in terms ROUGE-1 / ROUGE-2 / ROUGE-L/ / BERT SCORE scores.
Tig
Model
English
Indunedan
Rusdan
Unbu
Arg:
MMS
$4,19711 1177
26
00 35
13.70100 0.540 76..5
21.5 21.50/6.17/ 04
5 17.00
25,99 7.39 20.15 155/6.00 M6.96
MXLSS
26.06/
8.40/30 20
22/
14.66
25.80 8841 21.00
61.00/666000
24,7 / 77796770000 766.000/66..1
23:30 2 31 21/76565
35:27 0.00//0000000 26/0200000 20.0k/4 65.58
English-
MMS-MXLS
69.75
66..5
/7.80 18.95
77.14/8.4/21.17 16755
KD
A6
24.29/ 7.76/ 19.81 66.98
19.64
29:06/ /8.33/
29.2 22.31 (67.47
PT (Ours)
34.78/
12.36/ 20.81
26.13/3.30
/.103//5///155//////// 06.26
24.84 00/85//833 /8:25/ 20090/665.4
25 (3.00/ 1//4 /2.00.2//220/ 16.70
18.59 99.77 /2.5///2//// 67.71
MMS
9.00 23.03/3.33//000 00003/66.660 68
10.77 /6.
21,6776.2 /17.11146 66
25.13 /521//22// 19,01 /63.82
26.98
7.85
MXLS
32.85
10.56/244.5/ 668.3
18.27/ 14:000
22.58/7.22) 18.07 /63/65
25.96
20.70 63.36
24.507
8,05/
64.99
Indonssion
MMS+MXLS
33,87/ (1.51/1111/00
29811
69:75
20.05 /8.99/222 66688
10.06 23.22/3./4/00
Vasillo KD
33.68/ 11.1://15./////////// 169.25
30.49/11.58//
/1.55//44544//// (69.59
24:22/5.22/
/19.45 /1.45/6666//6 /6-00
29.36 /8.77 167.05
29.38: 10,07 / 25.2//2.///33
D (Qurs)
34,18/ 11.75/26.17/ 22/12/ /64.6
31.25/ 11.75 25
69.93
21..9//.09/20.29//66
/66.29/66.38
29.56 1156/7.38 78885 /3.8/23..8// 67.08
30,00/ 10.36 25.7// 66.26
MMS
29.53/ 8.9/2255765553
24.02/66.7// 19:21 /
38:207 9:85/22.33 22.3/22.3////
14.79551677
63.59
7.55
MXLS
32.01/ 10.85/2439966..77
23.92/ 8,20 19.36
/9..36/65.77 65.77
23.41/ 8.19/18.700 119/18.70/6244
24.59/7.54/ 19.39 163.5%
25.25/ (9:93/99999//// (64.91
Rusdan-
MMS+MNLS
32,94/ 11 11..0/0//7//0000
24.55 7 55433 20,000 67.
28.00/ 10.37 22.06 10.86600000 /00.M
27.48/8.511 21 00..2
28.27 /.05 /2.52222222444 /67.56
Vamilb-KD
$2.86/11 11 64/554////8////
34.81/ (8.00/ 200, 15:
28:27/ 10,31 /2..1/22///222
28.50 8.87/ 00..8/220.500 166.5
/8.56/ / 78 /22 20/755666 (7677 A9
DTTV (Ours)
3361/11.65/20 00/55/2.99//00 6M.14
26.57 /8,96 23,63 06.44
28.24/ 22.99//07.55
26.24/8.81 /2.68 /66,38
9.97 23:35/
MMSS
24.557 5.147 18.2966044
19.2007 4,390
15.06:/644 66..6
17.2273.9977 13.333366611111 99
$4.2271220 02222///2.476699
23.357 5.447 1.44/18.465 0447718467644 64.96
MXLS
$0,891 9.53 153//232//67.05
22.74/ 7.400
165.99
21.63/5.02/
17.66/63327
21,00/
(.94/17.99/40..
23,59/
7,49/ (99//8.84//444 /6.0)
Urdu
MMS+MXLS
10L.2/ 668.4
22.94
/67.02
12.32/7.13/
35
/21///22//8/0 14.56
26.37/
/67.43
Vanilb-KD
20,96/
16..66
21
D-TV (Qurs)
68.62
25.47 88522
28.68 67.75
22.38/7.09 18.44/68.77
36,46/ /1.///2..7/////// 69.94
28,99/
(b) The results hased on the mBART-50 backbone.
Table 1: The block in " */*//5p* denotes the MMS results and the block in */5/e/* indicates the MXL.S results. The " #/*/s/** " indicates the average (Avg.) score for each model and the best scores each block are bold. Our bold results indicate that statistically significantly better than the "Vanilla-KD" with I-test p 0.05. Note that the results out of each block (e.g., Danglshh-Engllsh block) cannot be compared to others (e.g.. aadoonsiaa-Fnnliian block) because they belong to different language directions. Therefore, in each block of MMS, the MMS always surpasses MXLS without any exception. In each block of MXLSS, the MXLS always surpasses MMS without any exception.
more implementation details including data pre- processing and hyper-parameters settings.
ally is the MAS training objective.
Vanilla-KD: It is the model enhanced with the knowledge distillation, which is trained with the objectives Eq. 2. Eq. 4 and Eq. 3.
Metrics. Following Bhattacharjee et al. (2022);
Wang et al. (2023a), we use the standard ROUGE scores (ROUGE-1, 1, ROUGE-2, and ROUGE- L) (Lin, 2004) with the statistical significance test (Koehn, 2004) for a fair comparison. Besides, we apply BERTSCORE (Zhang et al., 2020) for a comprehensive comparison.
D'TV It is the proposed model which are trained with the objective Eq. 10.
All the above models use the multimodal Trans- former described in $ 2.2 and involve two strong training backbones: mT5 (Xue et al., 2021) and mBART-50 (Tang et al., 2021).
4.3
Comparison Models
MMS: It is the MMS model trained with the
4,4 Main Results
objective Eq. 2.
Tab. 1 presents the main results on many-to-many scenarios grounding on different backbones. Over- all, our model obtains significantly better results than all contrast models in both settings. Results based on mT5 backbone. In Tab. 1 (a),
MXLS: It is the MXLS model trained with the objective Eq. 4.
MMS+MXLS: It is the model jointly trained with the objectives Eq. 2 and Eq. 4, which actu-

---

Models
English- +
Indonesian-
Russian-2
Urdu-t
Train
MMS+MXLS
28.86
9.90 22.07777700 66.94
30.51/
11.377 24,707 68.72
29.50
10.37 24.09
28.397
67.90
6.12
Visual Features
22.73/22666
24.33/68.33
5.37
2
w/ Vanilla KD
29.56
10.17723-4433 23:43/333//10 68.004
30.88 11.21 101/11/2400 24.67 66.80
29.34
10:48 23 23.60
29.00/
10.31 23.211233.2111 /
7.95
DKD
10.63/ 23.9//33/666
31.49
11.44
W/ "CTT"
30.19
10.57/24.00 66.45
31.42/ / 11. 25.33 68.88
30.75
10.59
24441 / 68.66
29.96/
10.61
24.32 68.31
13.45
TOO
23,45/23177
31.27
25.01 68.85
66.51
10:44/
w/ DCD&
30.77 /
10.72 24.53/ 6883
32.08/ / 11.71 25.63 69.09
31.27
11.34/ 68.9
30.70/
10.91 24.71 66.75
11.88
Table 2: Ablation study based on the mT5 Avg. results of
ROUGE-L/ BERTSCORE).
where each component is separately added the "MMS+MXIS"" s*p denotes the four languages (i.e., English, Indonesian, Russian, and Urdu). The "CAT" denotes the complex auxiliary tasks of (Liang et al., 2022b). Train (S)
denotes how many seconds are required for each model to train one step (32 batch size GPUs).
1) in each group (e.g., English++ (Englishh Indone- sian, Russian, Urdu)), the MMS model typically performs better in generating monolingual sum- maries while cannot process well in crossllingual settings. The reason is that the MMS model has no access to the cross- moosslliggua data during training. The MXLS model faces a similar phenomenon where it cannot handle well the monolingual summaries while generating better cross-lingual summaries. In contrast, the "MMS+MXLS" model, as a multitask model, achieves better results than both MMS and MXLS models, showing that the MMS and MXLS tasks can benefit each other and thus improve both of them. Based on this finding, dual knowledge distillation is more reasonable than unidirectional knowledge distillation. Our results further demon- strate this (See ablation study). 2) Generally, in each block, we find that our DºTV approach no- tably outperforms the Vanilla- KD method, showing the effectiveness of dual knowledge distillation and target-oriented contrastive learning. Although our results are slightly worse than the MMS model in English-English, Indonesian--Indonssin Russian-+Russian directions of - #/*/#/*** blocks, our DºTV model can balance well between MMS and MXLS. The results in each " Avg." blocks fully prove this point. 3) On average, our model consistently and significantly surpasses all base- lines by large margins (e.g., the previous best "Vanilla-Ke KD", up to 1.70/0.60/1.50 ROUGE and 0.77 BERTScore scores in Urdu-+* directions, re- spectively).
Analysis
5.1 Ablation Study
We conduct ablation studies investigate how well each component works. The results are shown in Tab. 2. We have the following conclusions:
(Row vs. row 0). The results show that in- corporating visual features has a positive impact on the model performance, demonstrating the importance of image sequence for the summary.
(Row 2 vs. row O). The vanilla KD makes reason- able contributions, showing that the MMS model indeed helps improve the quality of summaries in terms of both ROUGE and BERTScore scores, suggesting that distilling the knowledge of MMS MXLS is helpful summarization;
(Row 3 vs. row 2&row 0). The results show that dual knowledge distillation further improves the model performance, indicating that the knowl- edge of MMS and MXLS are beneficial each other and thus can enhance both of them.
and
(Row vs. row 4&row 0). The results show that summary-oriented visual features can signif- icantly improve the quality of summarics and our simple TCO achieves comparable performance with the CAT with less training time. This shows the superiority of the target-oriented contrastive objective.
(Row 6 vs. row 0). Adding DKD and TCO exhibit notable cumulative benefits, showing the effectiveness of the proposed approach.
5.2 Human Evaluation
Results based on mBART-50 backbone. In Tab. 1 (b), we observe similar findings as in the mT5-based scenario. This demonstrates that our conclusions are solid and convincing on gen- cral pre-trained language models. All these results prove the superiority of our approach.
Following Liang et al. (2022b), conduct hu- man studies on 50 samples randomly selected from English--inglish and Russian-Engglish test sets to further evaluate the performance of all models. We invite three Chinese postgraduate students who major in English to compare the generated sum-

---

Models
English
:English
Russian English
setting and also provide a corresponding bench- mark dataset. Additionally, we propose a simple
Flu.
Con.
Inf.
Flu.
Con.
Inf.
MMS
3.28
3.04
2.58
1.20
1.04
0.88
MXLS
1.22
0.96
2.72
2.28
2.14
yet effective target-oriented contrastive learning
MMS+MXLS
3.44
3.28
3.18
3,26
3.16
3.04
Vanilla-DD
3.60
3.46
3.22
3.40
3.28
3.18
objective to filter needless visual features, i.e.,
DºTV
4.14
3.94
3.78
3.86
3.58
3,50
offer summary-oriented visual features.
Table 3: Human evaluation results. 'Flu.': fluency, "Con.': conciseness, and 'Inf.": informativeness.
Multimodal Cross-lingual Summarization (MXLS). There is only one study that focuses on the MXLS task, i.e., Liu et al. (2022) first propose this task and design triple-stage training framework and distill the knowledge from MMS to enhance MXLS while ignoring the performance of MMS. Different from this work, we introduce the many-to-many multimodal summarization task. Furthermore, we devise dual knowledge distillation approach to simultaneously improve both MMS and MXLS tasks.
maries 4 and assess each summary from three inde- pendent aspects: fluency (Flu.), conciseness (Con.) and informativeness (Inf.). We ask them to score each aspect from 1 (worst) to 5 (best). The average results are presented in Tab. 3.
Tab. 3 shows the human results. We find that our DºTV substantially outperforms all contrast models under all criteria in both directions, which further shows the effcotiveness and superiority of our approach. The Fleiss' Kappa scores (Fleiss and Cohen, 1973) of Flu., Con. and Inf. are 0.74, 0.70 and 0.65, respectively, which indicates a substantial agreement among three evaluators. Furthermore, we present case study in Appendix C and it intu- itively shows the superiority of our DºTV.
Knowledge Distillation (KD). KD (Hinton et al., 2015) is to transfer the knowledge (e.g., soft targets outputs) of the stronger model (aka. the teacher model) to the small model (aka. the student model), which has achieved impressive results in the lit- crature (Zhang et al., 2023). In summarization, (Zhang et al., 2021b) adopt KD from a vision- language pre-trained model to improve image se- lection when generating multimodal summaries. Besides, researchers (Nguyen and Luu, 2022; Liu et al., 2022) typically treat the monolingual sum- marization model as the teacher model and the cross-lingual one as the student model because the monolingual summarization model is easier to train well than the cross-lingual one, which has shown promising performance on cross-lingual summa- rization task while ignoring the performance of the monolingual one. this work, we aim to mutu- ally prompt both monolingual and cross-linguu summarization tasks via dual KD rather than only improving the cross-lingual summarization task by unidirectional KD.
6 Related Work
Multimodal Monolingual Summarization (MMS). With the rapid growth of multimedia, many MMS datasets have been built which cover video summarization (Tjondronegoro et al., 2011; Sanabria et al., 2018), movie summa- rization (Evangelopoulos et al., 2013), meeting records summarization (Erol et al., 2003), sentence summarization (Li et al., 201 8a, 2017), product summarization (Li et al., 2020a), and news summarization (Zhu et al., 2018; 8: Chen and Zhuge, 2018; Hasan al., 2021; Fu et al., 2021; Liang et al., 2022b). With the data resources extensively used, the MMS task has attracted much attention, where the existing work mainly focuses on I) how to effectively exploit the additional features which are generally implicitly learned by the MMS objective or 2) explicit and complex auxiliary tasks, having achieved impressive performance these high-resource English datasets (Li et al., 2018b, 2020b; Zhu et al., 2020, 2021: Zhang et al., 2021b,a; Yu et al., 2021). In this work, we instead of focusing on introducing a more general and practical many-to-many multimoal summarization
Constrastive Learning. The idea of contrastive learning aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors (Hadsell et al., 2006), which has verified its superiority in many fields (Zhou et al., 2023). In summarization, Liu and Liu (2021) use contrastive loss to post-rank generated summaries and achieves good results in textual-only benchmark datasets. Cao and Wang (2021) and Xu et al. (2021) use contrastive learning to improve faithfulness and factuality and observe consistent improvements. Wang et al. (2021) apply contrastive learning for multilingual summarization
'When evaluating summaries in Mussian->Englian English, we show them the English document rather than the Russian document where the English and Russian document descrihe the same thing.

---

and obtain promising performance. Differently, we introduce it into the multimodal area and aim to pull the visual feature close to its corresponding summary and offer summary-oriented visual fea- tures. Therefore, we can improve the quality of summaries from the perspective of visual features rather than the textual document.
Acknowledgements
The research work described in this paper has been supported by the National Key R&D Program of China (2020AAA0108001 )and the National Na- ture Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130) The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to im- prove this paper.
7 Conclusion
In this paper, we first introduce more general task, i.e., M which can support both MMS and MXLS tasks. Further, we propose a dual knowl- edge distillation and target-oriented vision (D²TV) enhanced framework for the new task. Extensive experiments demonstrate that our model signifi- cantly outperforms related baselines in terms of ROUGE, BERTScore scores, and human evalua- tion. Furthermore, contribute many-to-many multimodal summarization Sum) dataset to the research community.
References
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Ri- fat Shahriyar. 2022. Crosssum: Beyond english- centric tross-lingual abstractive text summarization for 1500+ language pairs.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of EMNLP, pages
Online and Punta Cana, Dominican Republic.
Limitations
Although we show that our DºTV outperforms the vanilla-kD model based on two stronger backbone i.e., m'5 (Xue et al., 2021) and mBART-50 (Tang et al., 2021), there are some limitations worth con- sidering to study in future work: (1) In this study, we only provide 44 languages and conduct experi- ments four out of them, and future work could extend our method more languages; (2) With the development of the large-scale language mod- els, extending and validating our approach on them may be future work.
Jingqiang Chen and Hai Zhuge. 2018. Abstractive text- image summarization using multi-modal attentional hierarchical RNN. In Proc. of EMNLP, pages 4046- 4056.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and and-language tasks via text gener- ation. Proc. of ICMI., volume 139, pages 1931- 1942.
B. Erol, D.-S. Lee, and J. Hull. 2003. Multimodal summarization of meeting recordings. In Proc. of ICME, volume 3, pages II1-25.
Georgios Evangelopoulos, Athanasia Zlatintsi, Alexan- dros Potamianos, Petros Maragos, Konstantinos Ra- pantzikos, Georgios Skoumas, and Yannis Avrithis. 2013. Multimodal saliency and fusion for movie sum- marization based on aural, visual, and textual atten- tion. IEEE Trunsactions on Multimedia, 15(7):1553- 1568.
Ethics Statement
In this section, we consider the potential ethical issues of our model. In this paper, we propose DºTV which is trained the publicly-availabl BBC datasets. Therefore, DºTV might lead to in- correct summaries in applications and involve the same biases and toxic behaviors exhibited by the datasets. Besides, we obtained our M''Sum dataset by reorganizing the CrossSum (Bhattacharjee et al., 2022) and MMSum (Liang et al., 2022b) datasets and its permissions are granted to copy, distribute and modify the contents under the terms of the Creative Commons AttributionSharrAliike 3.0 Un- ported License and Creative Commons CCO Li- cense, respectively.
Joseph L. Fleiss and Jacob Cohen. 1973. The equiva- lence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, pages 613-619.
Xiyan Fu, Jun Wang, and Zhenglu Yang. 2021. MM- AVS: full-scale dataset for multi-modal summa- rization. In Proc. of NAACL-HLT, pages 5922-5926.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimension- ality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition C)))) volume 2, pages 1735-1742.
'The data originally comes from: https//wwww.bb..com/

---

Tahmid Hasan, Abhik Bhattacharjee. Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bi Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL. sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of ACL-IJCNLP. pages 4693-4703, Online. Association for Computa- tional Linguisties.
Yunlong Liang, Fandong Meng, Ying Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021. Infusing multi- source knowledge with heterogeneous graph neural network for emotional conversation generation. Proc. of AAAA,, pages 13343-13352.
Yunlong Liang, Fandong Meng. Ying Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2022c. Emotional conversation generation with heterogeneous graph neural network. Artificial Intelligence, 308: 103714 14.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint
2(7).
Yunlong Liang. Fandong Meng, Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2022d. A variational hierarchical model for neural cross- lingual summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2088- 2099, Dublin, Ireland. Association for Computational Linguistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP. pages 388-395.
Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In Proc. of IJCV, pages 32-73.
Chin Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branckes Out, pages 74-81.
Haoran Li, Peng Yuan, Song Xu, Youzheng Wu, Xi- aodong He, and Bowen Zhou. 2020a. Aspect-aware multimodal summarization for chinese e-commerce products. In Proc. of AAAI, volume 34, pages 8188- 8195.
Nayu Liu, Xian Sun, Hongfeng Yu, Wenkai Zhang, and Guangluan Xu. 2020. Multistage fusion with forget gate for multimodal summarization in open-domain videos. In Proc. of EMNLP, pages 1834-1845.
Nayu Liu, Kaiwen Wei, Xian Sun, Hongfeng Yu, Fang- long Yao, Li Jin, Guo Zhi, and Guangluan Xu. 2022. Assist non-native viewers: Multimodal cross-lingual summarization for how2 videos. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6959-6960099 6969,, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Haoran Li, Junnan Zhu, Tianshang Liu, Jiajun Zhang, Chengqing Zong, et al. 2018a. Multi-modal sentence summarization with modality attention and image filtering. In Proc. of IJCAI, pages 4152-4158.
Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2017. Multi- modal summarization for asynchronous collection of text, image, audio and video. In Proc. of EMNLP, pages 1092-1102.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A sim- ple framework for contrastive learning of abstractive summarivation. In Proceedings of ACI., pages 1065- 1072, Online.
Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2018b. Read, watch, listen, and summarize: Multi-modal summarization for asyn- chronous text, image, audio and video. IEEE Transactions on Knowledge and Data Engineering. 3106))996-1009
Thong Thanh Nguyen and Anh Tuan Luu, 2022. Im- proving neural cross- minsuli abstractive summariza- tion via employing optimal transport distance for knowledge distillation. In Proceedings of AAAI,, vol- ume 36, pages 11103-11111.
Mingzhe Li, Xiuying Chen, Shen Gao, Zhang- ming Chan, Dongyan Zhao, and Rui Yan. 2020b. VMSMO: Learning to generate multimodal summary for video-based news articles. In Proceedings of EMNLP., pages 1966,9966,00 Online. Association for Computational Linguistics.
Shruti Palaskar, Jindřích Libovický, Spandana Gella, and Florian Metze. 2019. Multimodal abstractive summarization for how2 videos. In Proc. of ACL. pages 6587-6596.
Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2022a. Msetd: A multimodal senti- ment chat translation dataset. Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-enn: Towards real- time object detection with region proposal networks. Proc. of NIPS, volume 28.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Bllliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: large-scale dataset for multimodal language understanding. In Proc. of the Workshop on ViGII.
Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang, Yufeng Chen, and Jie Zhou. 2022b. Summary- oriented vision modeling for multimodal abstractive summarization. arXiv preprini arXiv:2212.07672

---

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1450-3466, Online. Association for Computa- tional Linguisties.
Songming Zhang, Yunlong Liang, Shuaibo Wang, Wen- juan Han, Jian Liu, Jinan Xu, and Yufeng Chen. 2023. Towards understanding and improving knowledge distillation for neural machine translation. arXiv preprint arXiv:2305.080000
Tianyi Zhang, Varsha Kishore*, Felix Wu Kilian Q- Weinberger, and Yoav Arizi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Dian Tjondronegoro, Xiaohui Tao, Johannes Sasongko, and Cher Han Lau. 2011. Multi-modal summariza- tion of key events and top players in sports tourna- ment videos. In Proc. of IEEE WACV, pages 471- 478.
Zhengkun Zhang, Xiaojun Meng, Yasheng Wang, Xin Jiang. Qun Liu, and Zhenglu Yang. 2021b. Unims: A unified framework for multimodal summariza- tion with knowledge distillation. arXiv preprint arXiv:2109.05817
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proc of NIPS, pages 5998-6008.
Nan Zhao, Haoran Li, Youzheng Wu, and Xiaodong He. 2022. JDDC 2.1: A multimodal Chinese dialogue dataset with joint tasks of query rewriting, response generation. discourse parsing, and summarization. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 12037-12051, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021. Contrastive aligned joint learn- ing for multilingual summarization. In Findings of the Association for Computational Linguistics: ACL- IJCNPP 2021, pages 2739-2750 Online. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross- lingual summarization via chatgpt. arXiv preprint 232.:2302..4229.
Chulun Zhou, Yunlong Liang. Fandong Meng. Jinan Xu, Jinsong Su, and Jie Zhou. 2023. Rc3: Reg- ularized contrastive cross- cross-lingust lingual mrosll pre- training. arXiv preprint arXiv:2305.cm 07927.
Jiaan Wang. Fandong Meng. Tingyi Zhang, Yunlong Liang, Jiarong Xu, Zhixu Li, and Jie Zhou. 2022a. Understanding translationese in cross-lingual sum- marization. arXiv preprint 2rxi:2211.00222
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. MSMO: Mul- timodal summarization with multimodal output. In Proc. of EMNLP, pages 41544166.
Jiaan Wang. Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022b. A survey cross- lonsulingual summarization. Transac- tions of the Association for Computational Linguis- tics, 10:1304-1322
Junnan Zhu, Lu Xiang,
Zhou, Jiajun Zhang, and Chengqing Zong. 2021. Graph-bast multi- modal ranking models for multimodal summarization. Transactions on Asian and Low- Resource Language Information Processing, 20(4):1-21.
Jiaan Wang. Fandong Meng, Duo Zheng, Yunlong Liang. Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023b. Towards unifying multi-lingual and cross-lingual summarization. arXiv preprini arXiv:2305.00
Junnan Zhu, Yu Zhou, Jiajun Zhang, Haoran Li, Chengqing Zong, and Changliang Li. 2020. Mul- timodal summarization with guidance of multimodal reference. In Proc. of AAAI, volume 34, pages 9749- 9756.
Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2021. Sequence level contrastive learning for text summarization. arXiv preprint arXiv:2109.03.4 03481.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5; A massively multilin- gual pre-trained text-to-text transformer. In Proc. of NAACL-HLT, pages 483-498.
Tiezheng Yu, Wenliang Dai, Zihan Liu, and Pascale Fung, 2021. Vision guided generative pre-trained language models for multimodal abstractive summa- rization. In Proc. of EMNLP, pages 3995-4007.
Litian Zhang, Xiaoming Zhang, Junshu Pan, and Feiran Huang. 2021a. Hierarchical cross-modality semantic correlation learning model for multimodal summa- rization. arXiv preprint arXiv:2112. 12072.

---

A Dataset Statisties and Splits.
Hyperparameters
mT55
mBART-50
batch size (B)
32
32
As shown in Tab. 4, we only present 4 lan- guage directions of our S'Sum used in this work for simplicity. Actually, our M'Sum covers 44 languages and total includes 1,078,215 article- summary pairs with 3,479,348 images, where each article-summary pair contains about 3.23 images on average. The average article and summary length for all languages is about 520 and 84, respectively, According to the dataset size of each language, follow CrossSum (Bhattacharjee et al., 2022) and utilize about 80% training: 10% validation: 10% test splitting. Besides, in CrossSum, the number of languages is 44 and thus there are 44*44 language directions. For efficiency, we randomly select languages (i.e., English, Indonesian, Russian, and Urdu), which totally cover 16 language directions.
number of GPUs
V100
8 V100
hidden size filter size encoder layers decoder layers attention heads label smoothing
768
1024
2048
4096
12
12
12
0.1
0.1
learning rate warmup steps training steps T T1
2e-5
5e-6
2,000
2,000
10,000
10,000
5,000
5,000
training time optimizer
<33h
8hh
Adam
Adam
adam betal
0.9
0.9
adam beta2
0.998
0.98
layer normalization
postnorm
postnorm
M
520
520
Languages English Indonesian Russian Urdu
N
84
84
English 24,768
10,037
9,076
6,297
m
Indonesian 9,814
23,176
7,260
6,324
n
36
Russian 8,902
7,329
21,036
5,179
Ne
12
12
Urdu 6,052
5,810
4,700
17,800
Na
12
12
Ng
4
Table 4: An example of 4 * Language pairs covered by our Sum dataset.
d
768
1024
dp
2048
2048
de
256
256
K
4
B Implementation Details
B
1.0
1.0
Data Pre-Processing. Following Bhattacharjee al. (2022), we pre-process the textual data by truncating or padding them into sequences of 5 512 tokens for X' and the outputs y to 84 tokens after using the 250k wordpiece (Xue et al., 2021) vocab- ulary provided with the mT5 checkpoint (similar to mBART-50 setting). For the image sequence, following Liang al. (2022b), we truncate pad the sequence length to 180 (i.e., five images: 5 36; n=5, m=36).
Table 5: Training hyperparameters and model configu- rations of our experiments.
rouge toolkit following Bhattacharjee al. (2022). All experimental results reported in this paper are the average of three runs with different random seeds.
C Case Study
this work, we use two strong backbones, i.e., the base model of mT5 (Xue et al., 2021) and the large? model of mBART-50 (Tang et al., 2021). We list detailed syper-parameter used in this work in Tab. 5.
Fig. 3 shows example of the many-to-man mul- timodal summarization, the generated summary, and the ground truth summary in different lan- guages. (updating later.)
For inference, we use beam search with beam size 4 and length penalty of y = 0.6. When calcu- lating the ROUGE scores, we use the multi-lingual
phttps:///uugee htt.:ccccaaccoe//ooooooo main
http://hhhh....occcc//e///////s/ mester//mutttliinnuuarogggg
Saart-large-ae 58-many-to- many--t--mmn--mtt many-mmt

---

Article: China:
The story of Chita tas han of conantirary gnooth the lse docale, bu there have han recett coname due thas will he ejnificam
comomic dondkoo. One wiee is du tie woud rigerr panckel reacliens feo doncaic imokin and lead aseck maket esels With Chisa coabiialing it Stangiui stock evclunge only 1950, is makel ornidedd inmuue compeed the rex dfe word. The stares are alnos estiict owned io demeslic tndes, many wien are lnom and pop inrettes with lnll oueieence is invesing. The lock of lege. ousiiaiil and pofesiooa ogatisation is intettes reears fur the murker oan be much more volytle.
Contal bant: Dver but fow montss Ohrofs crama bank has beon repertely poopping up the andk moder to zeerm exbility They tave teen donng thoough sevent Hig moviro, audh onting sented bank inros nice- which allos moiz mony to flow eally and beying up duros in step them fnom filling, After liso led waek. foere was expectaion on Fridyy dhe thac wodd be yo another sach dhetic move But dhe did me happan- casing paric to ripelc out and a drauzii ine in stare on Monday, The saok mekkt saw it wont sitllsky playe sinse 2007
Curcnay Ose the pouille viguers bo the deop is out taling sosius was lie ca ler decison by die conrd buck to desdue the your and allow it tral more fecdyy Utilke mos cumencess the Clines currenyy is oot illowed to made Beey according to the sumber of buger and seller intemaaional natkets Ratter, the centall bonk ses a daiy mae to the odlar and for the rest of Ste the your stloed tade 25
dows from ihat ane larter in sugutt the bark at tut nee by strot 26, unding lime wone of inscuurly thrugh marksh. The mone was sen ar stterpt hilp copons by miking Chrsce paide cheaper abouad. hos the yar raded proviuus day which mome that couid fall kit fnther fatur.
The certral alo and would set the thily ane
Cortagion:
stock muriet dure cacool mncere incetiit specal umnso Bhe region
aooud alobe, dondiiiiiig undk markets Nox York global nukkaptaee Hong Koup-beal investient indyst "indevarr" years ago and resett decade igo, merey fellowe meods in global coutony. tien"
Euroze kneckon effect lightiattdd how much lisclpin Clinc' sods market
Peter Churchose says
rikes have timeed
global ecooory and globl motket quiite
Cometion standa''s
your
merey
shapp
apys
by
coaani yau.
Chinds slamp that the reszitionn iha becign sodds
The
your voliile. This would happen Chind'
beckanina
will
of shares
the
bigger profesional fine
lejezt subility Camenty forigners only Iroestrs
Consequence: Obeervers have deserbed incider "node aeokoning"
pold scart stietive Chira, Extremte movenetts
China's market will pribally boomme cill expated dim which tum wodl ofles the globel aveuuryy poniculatty Waizm grouth ape the BBCC Doncan Widden. Read mor:: How C shon lom, fhe work will huve incrense spanding
goon pecuilurit of being demirated by srell-time incoperencad imvdtoe Maanu bile Chinse
worial dooll we be ahout Chinas dure prize fals? The BBCss Baken Polin says that
bui F
over the ling lemn this would make sone cuuncies like the UK consilentty poieer Real nore Will Chint's sloo down mike pooear? But many enticipete fial
the Chisese gevemnent will contitue poo the soonooyy one or ofter, aad even more in light this rsceit fiinniall volatiity "Is diffoukt to officies allowing the ecosony dlide feeher wfficre some coutevvillgg action" says Hederio Neumarn, who on heads Ashnn economis resorch HSRC...
Ground Tranh: The reperteccinn fron BRack Morday the markets trmill canced plange Chinae steck ontins be fiftion Tacokay
MMS: Banounee sopurro
nespecttiii
sponEssTic mroprix.
Lnglis Summury
MXLS: (oاim do i't gm + م
MMS-MNLSI Konechasi dar "Seiin Hiban" -arggaan pasa global disbabkann oich penumnan sabon
Selasa
Vasilla KD: The repercssions fron "Black Monday" the glvbal markes samel couned a plange C Chinese stocks continee to be faht on Uueday
D'TV (Ours): BovEne sepuoro
nospecotali
somiooooo donaoooro pocna
Trathi Rrzonze repooro acoeansunnaa rrefema pawausex torpacennt, rocoomee mroeuzes simolcsoro I foutoooro pumna rрсооаII1 пpоaaeetccs
ongras.
MM& Konckkai dari "Seiin liltan" - pangan paear gloel diatehicnn olsh paummma olem Cin-- kams lerse peals Schex.
Resiis
MXLS: The repassoiora fhom "Bibk Monday the global tankist cassal by pling in Chincer andox contime: be fsi on Ticadas.
Summary
MMS MXLS: Baramce repeoto moccunaamee raofernnn punstnua rmppaconnl muscennat racamss mmmalceees feecreamo puncs- rpoasraass sporsterace to imgrnee
Vanill-KD: مملر Euri
D"TV (Ours): Kindkai "Scii Hlun" ppuusen distedicar olc'
peas
Ground Trath: o اسار wor o c gotus ( you A Le
MMS: Bansume sepuaro
varalcscco (oouaoor 2aan2 spoRR1ST424 2r0pIRK.
Unde Summary
MXLS _ Pa 4 للطيه - 2s fo Ar للس مسي مime
MMS-MMLS Konsesnei der "Senin Hoon" gaggguus paur global dodbation olch penurtion saban Cita- won seasa pata Selaon
Vanilb-KD: repasssioon "Iilak Monntyy global
casned a plange Chinese
Tranday,
D'TV (Oung: Bommmne sepastoo mengemunne- ramiemure pareemus neppecentt vurceneus ro mosemas somoleeeee ferepyooro paares ipporisscuss rposssmmce no imprine
gangun poor globil disdhhiznn perseston
renss pads Soloo.
MMS
Indonesian
MXLS: The repernesinss fron "Wack Monday the glohel morkets tomall canod by a plange in thoss andeo continee io fidt Tucokey.
Summury
e reprrco onseestoantss rasterson pasornma noppocennt, ramoeen nosoeicu sarsikoorro dowcooor paasoo- oposocaaco opons oersce sirpure Vanilb-KD: the سنييا wet Uf S guutt g -- Se - ue
D'TV (Ounde Keneclani dari Sania Hitan" gagggaan plobal doctekkan oldh parunazan salam Clu- Janas toruus Sciaaa
Figure An example of many-to-many summarization in different language directions. ((updating later.))


Reviews:
[{"review_id": "HKT7IEk8nd", "reviewer": "Reviewer_16rq", "summary": "", "questions": "Does the order of the image sequence matter?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for the careful reviews. These comments are all valuable and very helpful for revising and improving our work, as well as significant to guide our research. The comments are laid out below and specific concerns have been numbered. Our responses are given point by point. We have studied the comments carefully and made corresponding corrections. We hope the newly provided content could address your concerns or clarify some misunderstandings. \n\nWith the strong support of the proposed M3Sum task and dataset by **Reviewer #2&#3&#4** and the strong backing of our novel approach by **Reviewer  #1&#2&#3**, we believe that this work has the potential to lay the foundation and facilitate future research in multimodal many-to-many abstractive summarization fields. We will release our dataset and codes with step-by-step instructions for reproducing our experiments. \n\n**Comment 1**: Table 2 explicitly shows that visual features are insignificant in the summarization task. Therefore, comparing the proposed method with text-only summarization baselines will be meaningful, especially with the recent strong Large Language Models, which multilingual solid supports.\n\n**Response to Comment 1**: Actually, the averaged results in four language directions of Row 1 are lower (0.3-0.5 ROUGE-L Scores on average) than Row 0 in Table 2, showing the importance of incorporating visual features for the summarization task. In our experiments (based on mT5 backbone), the average results of text-only summarization baseline are 28.48 / 9.44 / 22.73 / 67.71 in English->* directions, which is lower than text-vision-based ones (30.77 / 10.72 / 24.53 / 68.83), demonstrating the necessity of visual features. Therefore, we did not list detailed experiments of text-only baselines in Table 1 due to space limitations.  Following your insightful suggestions, we will add these experiments in Table 1 in the updated version.\n\n**Comment 2**: Does the order of the image sequence matter?\n\n**Responese to Comment 2**: Yes. In our early experiments, we found that the order has a positive impact on the model performance because it introduces additional temporal information. It is consistent with previous work [1,2].\n\n[1] Multi-modal summarization for asynchronous collection of text, image, audio and video. EMNLP 2017.\n\n[2] Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed framework (details in Fig. 2) demonstrates the capabilities to leverage the interconnection between MMS and MXLS through dual knowledge distillation to improve the overall performance. The paper is well-written and easy to follow, even with the detailed formulas. The experiments are thorough and well-presented, including both automated and human evaluations.", "reasons_to_reject": "Table 2 explicitly shows that visual features are insignificant in the summarization task. Therefore, comparing the proposed method with text-only summarization baselines will be meaningful, especially with the recent strong Large Language Models, which multilingual solid supports.", "questions_to_authors": "Does the order of the image sequence matter?", "paper_topic_and_main_contributions": "This paper proposes a dual knowledge distillation and target-oriented vision modeling framework for many-to-many multimodal summarization (M3S). The authors demonstrated that this framework is better than multimodal monolingual summarization (MMS) and cross-lingual summarization (MXLS), MMS-MXLS, and vanilla knowledge distillation baselines. The experiments were conducted on the M3Sum dataset that consists of forty-four languages, in which the authors use four languages, namely English, Indonesian, Russian, and Urdu. Urdu is a low-resource language. The authors also conducted an ablation study based on mT5 and showed the usefulness of each feature in improving overall results for 4x4 languages. A small-scale human evaluation was also conducted for English --> English and Russian --> English test sets by three Chinese postgraduate students who majored in English. Details about the dataset, training computer, and one qualitative example are shown in the appendix.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g4upIFPYHn", "reviewer": "Reviewer_UzrF", "summary": "", "questions": "- The authors mention the blue results are MXLS results and the yellow is MMS results for Table 1. It was quite interesting to me that the MMS model did decently in the MXLS task while the MXLS model did much poorer for the MMS task. Why do the authors think this phenomenon occurs?\n- It was also interesting to me that most models for the MXLS tasks outperformed MMS tasks. It seems to me that cross-lingual summarization may be more difficult than monolingual summarization. Do the authors have any incites to this?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for the careful reviews. These comments are all valuable and very helpful for revising and improving our work, as well as significant to guide our research. The comments are laid out below and specific concerns have been numbered. Our responses are given point by point. We have followed closely the suggestions and made clarifications and revisions accordingly. We hope the newly provided content could help to further strengthen our work.\n\nWith the strong support of our proposed M3Sum task and dataset by **Reviewer #2&#3&#4** and the strong backing of our novel approach by **Reviewer  #1&#2&#3**, we believe that this work has the potential to lay the foundation and facilitate future research in multimodal many-to-many abstractive summarization fields. We will release our dataset and codes with step-by-step instructions for reproducing our experiments.\n\n**Comment 1**: Lack of results on languages. If there were some more results on a more variety of languages, it would be more enlightening to the performance of the framework.\n\n**Response to Comment 1**: (1) In existing work [1], it selects 4 languages to conduct experiments. Inspired by this work, we randomly chose 4 languages covering 16 language directions. \n(2) Another realistic reason is our limited GPU resources and space, we have to utilize 4 languages. We think the results in these directions have already verified our M3Sum dataset and the effectiveness of our model on the benchmark. Despite all this, we will follow your insightful suggestions and would like to include six languages in the appendix.\n\n[1] Models and Datasets for Cross-Lingual Summarisation. EMNLP2021.\n\n**Comment 2**: Although their framework is impressive, the improvement in performance compared to the other models (especially Vanilla-KD) seem to be minimal.\n\n**Response to Comment 2**: In Table 1, under the mT5 backbone, our model significantly surpasses the Vanilla-KD method by about 1.0-1.5 ROUGE-L scores on average and 0.2-0.8 BERTScore on average. Under the mBART50 backbone, we also observe about 0.5-0.7 ROUGE-L gains and 0.3-0.5 BERTScore gains, which both are more significant than Vanilla-KD with t-test $p$ < 0.5.\n\n**Comment 3**: The authors mention the blue results are MXLS results and the yellow is MMS results for Table 1. It was quite interesting to me that the MMS model did decently in the MXLS task while the MXLS model did much poorer for the MMS task. Why do the authors think this phenomenon occurs?\n\n**Response to Comment 3**: We are sorry for this misunderstanding. For example, in the \"English->\" group under mT5 backbone, the MMS baseline works well (36.16 / 13.08 / 27.67 / 70.57) in English->English summary while performing worse in English-Indonesia (6.87 / 1.94 / 6.34 / 63.39), English-Russian (1.23 / 0.20 / 1.21 / 59.60), English ->Urdu (0.14 / 0.00 / 0.14 / 55.44) directions. For the MXLS baseline, it shows poor performance (6.94 / 2.35 / 6.03 / 61.8) in English-English direction while working better in cross-lingual ones (27.23 / 9.32 / 22.13 / 68.40 in English-Indonesia; 22.52 / 7.88 / 18.07 / 64.84 in English-Russian; 32.27 / 11.17 / 25.15 / 68.29 in English-Urdu). In all cases, we can find a similar and formal pattern. \n\nExcept for the Russian-> Indonesia direction under the mBART50 backbone, the MXLS achieves slightly worse performance than MMS. The reason may be that (1) the original mBART50 has been trained on multilingual machine translation parallel data and owns a better cross-lingual ability that only learns the summarization ability in MMS model; (2) At MMS training, the monolingual summarization in four languages (English->English, Russian->Russian, Indonesia->Indonesia, Urdu->Urdu) is jointly trained which may enhance the summarization for different languages.\nWe will add more clear analysis to clarify this in the next version.\n\n**Comment 4**: It was also interesting to me that most models for the MXLS tasks outperformed MMS tasks. It seems to me that cross-lingual summarization may be more difficult than monolingual summarization. Do the authors have any incites to this?\n\n**Response to Comment 4**: We are sorry for this misunderstanding. The results out of each block (e.g., English->English block) cannot be compared to others (e.g., Indonesia->English block) because they belong to different language directions. Therefore, in each block of MMS, the MMS always surpasses MXLS without any exception. In each block of MXLS, the MXLS always surpasses MMS without any exception. We will make these clearer in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is the first to introduce the Many-to-Many Multimodal Summarization (M3S) task and provides a novel benchmark dataset, M3Sum, to facilitate research in this area.\n- The proposed Dual Knowledge Distillation and Target-Oriented Vision Modeling (D2TV) framework addresses key challenges in M3S. By enabling knowledge transfer between Monolingual Summarization (MMS) and Cross-Lingual Summarization (MXLS), and effectively filtering irrelevant visual information, the framework enhances the performance of multimodal summarization models.\n- Empirical experiments conducted on the M3Sum benchmark demonstrate that the proposed D2TV framework achieves state-of-the-art performance. The model's ability to outperform existing methods across four languages and 16 language directions underscores the efficacy of the approach, reinforcing its potential contribution to advancing the field of multimodal summarization.", "reasons_to_reject": "- Lack of results on languages. If there were some more results on a more variety of languages, it would be more enlightening to the performance of the framework. \n- Although their framework is impressive, the improvement in performance compared to the other models (especially Vanilla-KD) seem to be minimal.", "questions_to_authors": "- The authors mention the blue results are MXLS results and the yellow is MMS results for Table 1. It was quite interesting to me that the MMS model did decently in the MXLS task while the MXLS model did much poorer for the MMS task. Why do the authors think this phenomenon occurs?\n- It was also interesting to me that most models for the MXLS tasks outperformed MMS tasks. It seems to me that cross-lingual summarization may be more difficult than monolingual summarization. Do the authors have any incites to this?", "paper_topic_and_main_contributions": "The authors introduce the Many-to-Many Multimodal Summarization (M3S) task, involving generating summaries across languages using document inputs and corresponding image sequences. Existing research has mostly focused on either monolingual summarization (MMS) or cross-lingual summarization (MXLS), neglecting M3S. This paper proposes a framework for M3S, combining knowledge distillation between MMS and MXLS and a target-oriented vision model to enhance both tasks. The dual knowledge distillation ensures knowledge transfer, while a target-oriented contrastive objective filters irrelevant visual features. Experimental results demonstrate the effectiveness of this approach, and a new dataset (M3Sum) with 44 languages is introduced for future research.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pGzuvPPMpT", "reviewer": "Reviewer_CEcV", "summary": "", "questions": "A. What is the motivation for the proposed task and modules in your paper? Could you explain one of your important claims: “implicit learning via the MMS objective may limit the potential of visual features” in detail?\n\nB. How does your model perform compared with prior baselines?\n\nC. How does your model perform on settings containing more language, and is there any evidence to prove that your target-oriented contrastive objective can discard irrelevant visual information?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for the careful reviews. These comments are all valuable and very helpful for revising and improving our work, as well as significant to guide our research. The comments are laid out below and specific concerns have been numbered. Our responses are given point by point. We have followed closely the suggestions and made clarifications and revisions accordingly. We hope the newly provided content could address your concerns or clarify some misunderstandings. \n\nWith the strong support of our proposed M3Sum task and dataset by **Reviewer #2&#3&#4** and the strong backing of our novel approach by **Reviewer #1&#2&#3**, we believe that this work has the potential to lay the foundation and facilitate future research in multimodal many-to-many abstractive summarization fields. We will release our dataset and codes with step-by-step instructions for reproducing our experiments.\n\n**Comment 1**: Insufficiently Explained Motivation：(a). The motivation to combine both MMS and MXLS is not well explained in the paper. While the paper mentions that there is little attention given to the M^{3}S task, it is not strong, it is better to provide a more convincing motivation. (b). The paper's rationale for using the explicit target-oriented contrastive objective instead of implicit learning via the MMS objective lacks sufficient explanation and depth. While the paper mentions that implicit learning may limit the potential of visual features, there is no comprehensive analysis or concrete evidence to support this claim. It is better to give a clear and detailed explanation of the limitations of implicit learning and how the target-oriented contrastive objective addresses those limitations.\n\n**Response to Comment 1**: **(a)** Essentially, the MXLS model needs to simultaneously conduct machine translation and summarization [1] while the MMS model only conducts summarization. Obviously, it is harder to train an MXLS model than to learn an MMS model and that is why researchers [2,3] take the MMS model as the teacher to help the MXLS student model. However, when the MXLS model achieves a level of multilingual and cross-lingual ability, the MXLS model can better transfer and share task knowledge among different languages. Therefore, the MXLS model, in turn, can guide the MMS model to conduct summarization in diverse languages (English->English, Indonesian->Indonesian, and Urdu->Urdu), especially for low-resource ones. That is why we propose DKD to mutually enhance their performance as we explained in Lines 239-259. We will follow your constructive suggestions and add more detailed explanations in the Introduction part in the next version.\n\n**(b)** Actually, we follow this claim \"implicit learning via the MMS objective may limit the potential of visual features\" from [4]. For example, though the object “中国制造” in the fifth image of Fig. 1 is associated with the article content (\"made in China\"), the object contributes little to the summary. That is, the MAS model should focus on summary-oriented visual features. However, the visual features are generally implicitly learned via the MAS objective, which cannot help the model learn to explicitly discard such needless visual information. \nTherefore, we propose a target-oriented vision modeling objective to explicitly filter summary-unrelated visual features. The results show that the model with enhanced visual features achieves better results than vanilla visual features (Row 5 vs. Row 0 in Table 2), demonstrating the effectiveness of the proposed target-oriented vision modeling with faster training speed.\n\n[1]A survey on cross-lingual summarization. TACL 2022.\n\n[2] Improving neural cross-lingual abstractive summarization via employing optimal transport distance for knowledge distillation. AAAI 2021\n\n[3] Assist non-native viewers: Multimodal cross-lingual summarization for how2 videos. EMNLP 2022.\n\n[4] Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023. (appeared to Arxiv on Dec 15, 2022)\n\n**Comment 2**: Insufficient Comparison with Existing Work: The abstract mentions that little research has been done on the M^{3}S task, but it's essential for the paper to provide a more comprehensive comparison with existing related works in the field such as “Liu et al. (2022)” mentioned in line 054 in the paper. It is better to adequately situate itself in the existing literature and have a thorough comparison with state-of-the-art methods.\n\n**Response to Comment 2**: Actually, we have compared with the previously state-of-the-art model in [4]. The results are shown in Row 4 of Table 2, which shows that our model can significantly surpass it with faster training speed. Besides, we followed this constructive suggestion and have re-implemented the triple-stage training method of [5] as a stronger baseline. For example, the averaged results in terms of ROUGE-1 / ROUGE-2 / ROUGE-L / BERTSCORE scores are listed as follows:\n\n Models | English->*|Indonesia->*|Russian->*|and Urdu->*\n-|-|-|-|-\nLiang et al. (2022)[4] | 30.19 / 10.57 / 24.03 / 68.45 |31.42 / 11.49 / 25.33 / 68.88 |30.75 / 10.59 / 24.41 / 68.64 |29.96 / 10.61 / 24.32 / 68.31\nLiu et al. (2022) [5] |29.88 / 10.36 / 23.78 / 68.33 | 31.09 / 11.60 / 24.83 / 68.83 |29.58 / 10.87 / 23.99 / 68.45 |29.63 / 10.82 / 23.53 / 68.12\nOurs |30.77 / 10.72 / 24.53 / 68.83 |32.08 / 11.71 / 25.63 / 69.09 |31.27 / 11.24 / 25.13 / 68.96 |30.70 / 10.91 / 24.71 / 68.75\n\nThese results further show the effectiveness of our proposed approach. We will include these results in the revised version.\n\n[4]: Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.\n\n[5]: Assist non-native viewers: Multimodal cross-lingual summarization for how2 videos. EMNLP 2022.\n\n**Comment 3**: Inadequate Evaluation on the M^{3}Sum Dataset: While the paper contributes a new dataset called M^{3}Sum, however the evaluation on this dataset is only taken on 4 randomly selected languages, thus is insufficient and lacks diversity, it might raise doubts about the reliability of the results and the generalizability of the proposed approach. Moreover, the intuitional analysis of whether their Target-oriented Contrastive Objective discarding the irrelevant visual information is missing.\n\n**Response to Comment 3**: (1) In existing work [1], it selects 4 languages to conduct experiments. Inspired by this work, we randomly chose 4 languages covering 16 language directions. (2) Another realistic reason is our limited GPU resources and space, we have to utilize 4 languages. We think the results in these directions have already verified our M3Sum dataset and the effectiveness of our model on the benchmark. Despite all this, we will follow your insightful suggestions and would like to include six languages in the appendix.\n\n(2) Row 5 vs. Row 4 in Table 2 shows that our target-oriented contrastive objective (TCO) can achieve a competitive performance to CAT[2] with faster training speed. The CAT [2] proposes two explicit auxiliary tasks including the vision to summary task and the masked image modeling task to discard irrelevant visual information, showing that they can filter target-unrelated visual features and obtain SOTA performance. Therefore, we believe that our method also can discard such irrelevant visual information and thus achieve a competitive performance to CAT. \n\n(3) Besides, we use the extracted visual features to reconstruct the original images (we aim to present which part of the image our model focuses on.). Taking Figure 1 as an example, the results show that the people, stock in the first image, and outline of money in the third image can be fuzzily obtained while the fifth image is nearly blurred. This shows that the model indeed can focus on the target-related part and ignore the irrelevant information to some extent. We will add more analysis about this in the revised version.\n\n[1] Models and Datasets for Cross-Lingual Summarisation. EMNLP2021.\n\n[2] Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.\n\n**Comment 4**: What is the motivation for the proposed task and modules in your paper? Could you explain one of your important claims: “implicit learning via the MMS objective may limit the potential of visual features” in detail?\n\n**Response to Comment 4**: (a) Under the background of globalization and the rapid increase of multimedia data, there is still a lack of many-to-many multimodal summarization research, which may prevent diverse conversations and thus hinder the globalization process. This is why we propose the many-to-many multimodal summarization task. As for the modules of our architecture, we designed them from the perspective of the new task: (1) the dual knowledge distillation for many-to-many summarization (MMS and MXLS consist of many-to-many setting); (2) the target-oriented vision modeling to enhance the fusion of multimodality.\n\n(b) Actually, we follow this claim \"implicit learning via the MMS objective may limit the potential of visual features\" from [4]. For example, though the object “中国制造” in the fifth image of Fig. 1 is associated with the article content (\"made in China\"), the object contributes little to the summary. Thus, the MAS model should focus on summary-oriented visual features. However, the visual features are generally implicitly learned via the MAS objective, which cannot help the model learn to explicitly discard such needless visual information. Therefore, we propose an explicit summary-oriented training objective to filter target-unrelated visual features.\n\n[4]: Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.\n\n**Comment 5**: B. How does your model perform compared with prior baselines?\n\n**Response to Comment 5**: Actually, besides the general baselines, e.g., MMS, MXLS, MMS+MXLS (multi-task learning), and vanilla-KD under the mT5 and mBART50 backbones, we have compared with the previously state-of-the-art model in [4]. The results are shown in Row 4 of Table 2, which shows that our model can significantly surpass it with faster training speed. We also have re-implemented the triple-stage training method of [5] as a stronger baseline. The averaged results in English->*, Indonesia->*, Russian->*, and Urdu->*  are 29.88 / 10.36 / 23.78 / 68.33, 31.09 / 11.60 / 24.83 / 68.83, 29.58 / 10.87 / 23.99 / 68.45, and 29.63 / 10.82 / 23.53 / 68.12, respectively. All these results of prior baselines further show the effectiveness of our proposed approach. \n\n[4]: Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.\n\n[5]: Assist non-native viewers: Multimodal cross-lingual summarization for how2 videos. EMNLP 2022.\n\n**Comment 6**: How does your model perform on settings containing more language, and is there any evidence to prove that your target-oriented contrastive objective can discard irrelevant visual information?\n\n**Response to Comment 6**: (1) Row 5 vs. Row 4 in Table 2 shows that our target-oriented contrastive objective (TCO) can achieve a competitive performance to CAT [4] with faster training speed. The CAT [4] proposes two explicit auxiliary tasks including the vision to summary task and the masked image modeling task to discard irrelevant visual information, showing that they can filter target-unrelated visual features and obtain SOTA performance. Therefore, we believe that our method also can discard such irrelevant visual information and thus achieve a competitive performance to CAT. \n\n(2) Besides, we use the extracted visual features to reconstruct the original images (we aim to present which part of the image our model focuses on.). Taking Figure 1 as an example, the results show that the people, stock in the first image, and outline of money in the third image can be fuzzily obtained while the fifth image is nearly blurred. This shows that the model indeed can focus on the target-related part and ignore the irrelevant information to some extent.\n\n[4] Summary-oriented vision modeling for multimodal abstractive summarization. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Significance: Given the increasing availability of multimodal data on the media. The M^{3}S task, which combines both multimodal monolingual summarization and multimodal cross-lingual summarization, offers a comprehensive and practical solution for generating summaries across multiple languages and modalities.\n\n2. Novelty: The paper addresses a relatively under-explored problem in the field of natural language processing, namely the Many-to-Many Multimodal Summarization (M^{3}S) task. By introducing this task and proposing a new framework to tackle it, the paper brings fresh insights and contributions to the research community.\n\n3. Experimental Results: The paper presents extensive experiments in a many-to-many setting, showcasing the certain effectiveness of the proposed approach. The inclusion of the newly contributed M^{3}Sum dataset with 44 languages further provides a resource for the community.\n", "reasons_to_reject": "1. **Insufficiently Explained Motivation**：(a). The motivation to combine both MMS and MXLS is not well explained in the paper. While the paper mentions that there is little attention given to the M^{3}S task, it is not strong, it is better to provide a more convincing motivation. (b). The paper's rationale for using the explicit target-oriented contrastive objective instead of implicit learning via the MMS objective lacks sufficient explanation and depth. While the paper mentions that implicit learning may limit the potential of visual features, there is no comprehensive analysis or concrete evidence to support this claim. It is better to give a clear and detailed explanation of the limitations of implicit learning and how the target-oriented contrastive objective addresses those limitations.\n\n2. **Insufficient Comparison with Existing Work**: The abstract mentions that little research has been done on the M^{3}S task, but it's essential for the paper to provide a more comprehensive comparison with existing related works in the field such as “Liu et al. (2022)” mentioned in line 054 in the paper. It is better to adequately situate itself in the existing literature and have a thorough comparison with state-of-the-art methods.\n\n3. **Inadequate Evaluation on the M^{3}Sum Dataset**: While the paper contributes a new dataset called M^{3}Sum, however the evaluation on this dataset is only taken on 4 randomly selected languages, thus is insufficient and lacks diversity, it might raise doubts about the reliability of the results and the generalizability of the proposed approach. Moreover, the intuitional analysis of whether their Target-oriented Contrastive Objective discarding the irrelevant visual information is missing.\n", "questions_to_authors": "A. What is the motivation for the proposed task and modules in your paper? Could you explain one of your important claims: “implicit learning via the MMS objective may limit the potential of visual features” in detail?\n\nB. How does your model perform compared with prior baselines?\n\nC. How does your model perform on settings containing more language, and is there any evidence to prove that your target-oriented contrastive objective can discard irrelevant visual information?\n", "paper_topic_and_main_contributions": "The article's main theme is \"Many-to-Many Multimodal Summarization (M^{3}S)\" task, which aims to generate summaries in any language using document inputs in any language and the corresponding image sequence, combining both multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. The contribution of the paper can be summarized as follows:\nIntroduction of M^{3}S Task: The paper introduces the M^{3}S task as a general and practical problem, highlighting the lack of attention given to it in existing research.\n\nDual Knowledge Distillation and Target-Oriented Vision Modeling Framework: The framework utilizes a dual knowledge distillation method, leading to mutual improvement of both tasks. Additionally, a target-oriented contrastive objective is designed to provide target-oriented visual features, discarding irrelevant visual information.\n\nContribution of M^{3}Sum Dataset: The paper contributes a new dataset called \"M^{3}Sum\" for the M^{3}S task, which consists of 44 languages. This dataset is expected to facilitate future research in the field of many-to-many multimodal summarization.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1cdOVQSBAi", "reviewer": "Reviewer_KUUr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for the careful reviews. These comments are all valuable and very helpful for revising and improving our work, as well as significant to guide our research. The comments are laid out below and specific concerns have been numbered. Our responses are given point by point. We have studied the comments carefully and made corresponding corrections. We hope the newly provided content could address your concerns or clarify some misunderstandings. \n\nWith the strong support of our proposed M3Sum task and dataset by **Reviewer #2&#3&#4** and the strong backing of our novel approach by **Reviewer  #1&#2&#3**, we believe that this work has the potential to lay the foundation and facilitate future research in multimodal many-to-many abstractive summarization fields. We will release our dataset and codes with step-by-step instructions for reproducing our experiments.\n\n**Comment 1**: The motivation behind the Dual Knowledge Distillation module is unclear, and it would be preferable to provide vivid explanations with relevant examples in the Introduction section.\n\n**Response to Comment 1**: Essentially, the MXLS model needs to simultaneously conduct machine translation and summarization [1] while the MMS model only conducts summarization. Obviously, it is harder to train an MXLS model than to learn an MMS model and that is why researchers [2,3] take the MMS model as the teacher to help the MXLS student model. However, when the MXLS model achieves a level of multilingual and cross-lingual ability, the MXLS model can better transfer and share task knowledge among different languages. Therefore, the MXLS model, in turn, can guide the MMS model to conduct summarization in diverse languages (English->English, Indonesian->Indonesian, and Urdu->Urdu), especially for low-resource ones. That is why we propose DKD to mutually enhance their performance as we explained in Lines 239-259. We will follow your suggestions and add more detailed explanations in the Introduction part in the next version.\n\n[1] A survey on cross-lingual summarization. TACL 2022.\n\n[2] Improving neural cross-lingual abstractive summarization via employing optimal transport distance for knowledge distillation. AAAI 2021\n\n[3] Assist non-native viewers: Multimodal cross-lingual summarization for how2 videos. EMNLP 2022.\n\n**Comment 2**: Figure 2 is not sufficiently clear, and the DKD and TCO modules are not presented in the MMS model. It is recommended to revise and modify them accordingly.\n\n**Response to Comment 2**: Thanks for your insightful suggestions. We will revise our model architecture to clearly present the relationship between different modules.\n\n**Comment 3**: Recent relevant work is not be mentioned, such as \"CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarization\".\n\n**Response to Comment 3**: Because the submission deadline of EMNLP2023 is before the appearance of this work. Therefore, I cannot mention this work. I will discuss such close work in the new version. \nThe work \"CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarization\" first appeared to Arxiv on July 06 2023 and the date of ACL is also July. However, the submission deadline for EMNLP is June 24, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work introduces a new task called M$^3$S and presents a corresponding dataset.\n2. The paper is well-written and easy to understand, making it accessible to a broad audience.", "reasons_to_reject": "1. The motivation behind the Dual Knowledge Distillation module is unclear, and it would be preferable to provide vivid explanations with relevant examples in the Introduction section.\n2. Figure 2 is not sufficiently clear, and the DKD and TCO modules are not presented in the MMS model. It is recommended to revise and modify them accordingly.\n3. Recent relevant work is not be mentioned, such as \"CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarization\".", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is the first to introduce the comprehensive many-to-many multimodal summarization (M$^3$S) task and contribute a corresponding benchmark dataset. To achieve this, a dual knowledge distillation and target-oriented vision modeling framework (D2TV) is proposed.\nExperiments on the M3Sum benchmark with four languages and 16 language directions show that this work builds new state-of-the-art performance, demonstrating the effectiveness of the D2TV approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper brings merits to Many-to-Many Multimodal Summarization, which is interesting. The model part is also reasonable. The major weakness is marginal results make it hard to justify how much vision can contribute to this task.
Decision: Accept-Findings