So-TVAE: Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting
Authors: Fengyi Fu
Shancheng Fang
Weidong Chen
Yan Song
Zhendong Mao
Yongdong Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automatic live video commenting
batch attention
cross-modal fusion

Abstract: Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the sentiment consideration of the generated comments is missing from the current methods. Thus, in this paper, we introduce and investigate a task, namely sentiment-guided automatic live video commenting, which aims to generate live video comments based on sentiment guidance. To address this problem, we propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network, which consists of a sentiment-oriented diversity encoder module and a batch-attention module. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments. Furthermore, a batch attention module is also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data imbalance, which is common in live videos as the popularity of video varies. Extensive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods in terms of the quality and diversity of generated comments. Related codes will be released.

Under review as a conference paper at ICLR 2023
So-TVAE: SENTIMENT-ORIENTED TRANSFORMER- BASED VARIATIONAL AUTOENCODER NETWORK FOR LIVE VIDEO COMMENTING
Anonymous authors Paper under double-bling review
ABSTRACT
Automatic live video commenting is with increasing attention due to its signifi- cance narration generation, topic explanation, etc. However, the sentiment con- sideration of the generated comments is missing from the current methods. Senti- mental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, introduce and investigate task, namely sentiment-guided automatic live video commenting, which aims to generate live video comments based on sentiment guidance. To address this problem, propose Sentiment- oriented Transformer- based Variational Autoencoder (So-TVAE) network, which consists of sentiment-oriented diversity encoder module and batch-attention module. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism achieve semantic diversity under sentiment guidance, which is then fused with cross- rros-modaa features to generate live video comments. Furthermore, a batch attention module also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data im- balance, which is common in live videos the popularity of video varies. Exten- sive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods terms of the quality and di- versity of generated comments. Related codes will be released.
INTRODUCTION
Live video commenting, commonly known "danmaku" or "bullet screen", is a new interactive mode on online video websites which allows viewers to write real-time comments to interact with others. Recently, the automatic live video commenting (ALVC) task, which aims to generate real- time video comments for viewers, is increasingly used narration generation, topic explanation, and video science popularization However, previous works aim to generate factual and subjective comments without considering the sentimental factor. In real-world applications, it is difficult for comments with single sentiment to resonate with everyone (Wang & Zong, 2021 Yan et al., (02T). On the other hand, ciciimenntgguudee comments would help video labeling and distribution, and further encourage the human-interacted comment generation. Thus, in this paper, we introduce and investigate a task, namely sentiment- guided automatic live video commenting, which aims to generate live video comments based on sentiment guidance.
assist in attracting the attention and discussion of viewers.
Two major difficulties make this task extremely challenging. Firstly, sentiment, comment, and video are heterogeneous modalities, and they are both important in live video comment generation with sentiment. Previous works take their effort on sentiment-guice
text generation (Li at. 2021 Kim
et al. 2022 Sabour al., 2021) or video comment generation (Ma et al.) 2019 Wang et al. 2020), but cannot apply to the complex situation that needs considering all three modalities simultancously. Secondly, the imbalance of video data (Wu et al. 202T) hinders the generation of comments with the desired sentiment. The imbalance of video data lies in two aspects. On the one hand, as the popularity of the videos varies, the number of comments is a huge difference between videos, and on the other hand, a certain video, comments usually show some sentimental trend, but lack comments on the other sentiments.

---

Under review as a conference paper at ICLR 2023
Title: 22 BAEE) 4F
"A miniature origami umbrella can shrink und fold. It's the same as a real umbrella, but it's more fun and lavelly
00009 E1
01:13 53 A8R
"Look Noe white color sidko hoartifu
diticshi"
difficali
tine stpp has been dithouit me." Wiet + QUUT
COL14 OF-3219---
Leantt
"Retam afler sacezndial learning 00.07
"Its kind like folding lily""
hands have own Bitkking
0039
00:24
Figure 1: A live video commenting example in Livebot with selected video frames and live com- Positive comments. Black: neutral comments. Blue: Negative comments.
ments. Green:
To this end, in this paper, we propose Sentiment-oriented Transformer-basee Variational Autoen- coder (So-TVAF) network, which consists of a sentiment-oriented diversity encoder module and batch attention module, to deal with above two challenges. The proposed sentiment-oriend diver- sity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance. Specifically, we firstly leverage a Gaussian mixture distribution mapping guided by sentimental information to learn diverse comments with different sentiments. In addition, to effectively avoid VAE ignoring the input information and directly learns the mapping of the latent space, propose novel sentiment-oriented random mask encoding mechanism, balancing the learning ability of the model to various modalities, which further improves the performance.
Moreover, batch-attention module is proposed in this paper alleviate the data imbalance prob- lem existing in live video. We leverage batch-level attention along with multi-modal attention in parallel way to simultancously deal with the multi-modal features in batch dimension and spatial dimension. In this way, the proposed batch-level attention module can mitigate the gap between samples, explore the sample relationships in a mini-batch, and further improve the quality and di- versity of generated comments.
The main contributions of this work are as follows:
Firstly, we introduce and investigate a task in this paper, namely sentiment-guiee automatic live video commenting, which aims to generate live video commenting by sentiment guidance. To this end, we propose Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) net- work, which consists a sentiment-oriented diversity encoder module and a batch attention module.
Secondly, we propose sentiment-orienten diversity encoder module, which elegantly combines VAE and random mask mechanism to achieve semantic diversity and further align sentiment features with language and video modalities under sentiment guidance. Mcanwhile, propose batch- attention module for sample relationship learning, to alleviate the problem of missing sentimental samples caused by the data imbalance.
Thirdly, we perform extensive experiments on two public datasets. The experimental results based on all evaluation metries prove that our model outperforms the state-of-the-an models in terms of quality and diversity.
2 APPROACH
2.1 PROBLEM FORMULATION AND MULTI-MODAL ENCODER
Given the video clip V, with & frames I, (II...,, ..... Ix) and surrounding comments S, {#1, $2,...,... } which are firstly concatenated into a word sequence x; = {=1, 42, , Jp T2...... where p represents the total number of words of S,, automatic live video commenting aims to generate human-like comments at timestamp t. The overview of the framework is illustrated Figure 2
Specifically, the input frames and surrounding comments are first represented as the initial features with pre-trained convolution neural network (ResNet) He et al. (016)) and Long short-term mem- ory network (LSTM) Hochreiter et al. (1997), respectively. Then employ co-attention module further enhance feature encoding, and generate the attended visual representations V, and the
2

---

Under review as a conference paper at ICLR 2023
Output.
Softmax
FFN
(B)
Output
N
Transformer
Comment
Comment
2
Sentiment Prediction
Encoder
Decoder
Decoder
Mark
Zalobha
Seiteant
Multi-modal Attention
Multi-modal
Batch
Sr
2N:77
"So bARELE
Figure Overview of our proposed Sentiment-oriented Transformer-based Variational Auto- Encoder (So-TVAE) Network. (A) represents the proposed sentiment-oriented diversity encoder module; whereas, (B) represents the proposed batch attention module.
attended textual representations V.:
Vi Re), F. = LSTM((((()) Eq-1),
(1)
Ve, VI Co-Attention Vi),
(2)
where e(-) is the word embedding processing. Co-attention acts on the initial frame features V1 (Vi, V2.-.., Vi Bxkxd and text feature V. (F1, E2,.... Epp Bxpxd for modeling multi-modal interaction.
2.2 SENTIMENT-ORIENTE DIVERSITY ENCODER
The previous methods of the ALVC task can only achieve one-to-one generation as they encode the source input to fixed feature vector to guide the reconstruction of comments. Therefore, based on the principle of Variational Autoencoder, we propose a diversity encoder that can explicitly model one-to-many mapping between the latent space and the target comments. By sampling multiple latent vectors z from the trained latent space, our model can generate diverse comments.
2.2. SENTIMENTAL DIVERSITY
To make up for the lack sentiment annotation in the original datasets, pre-trained sentiment anal- ysis model SKEP (Tian al. 2020 is introduced to evaluate the sentiment of target comment y, pro- ducing sentiment label y, (ye {0, 10.1.... N 1}, N is the number of sentiment categories). We use one-hot encoding the sentiment label to obtain sentiment weight vector s, followed by embed- ding layer to generate the direct sentiment feature representations V, Embedding, (one-hot(y..))
2.2.2 SEMANTIC DIVERSITY
Furthermore, we model one-to-many mapping to learn the diverse semantic information of differ- ent sentiments. In the training stage, the semantic diversity module samples sentiment-oriented latent vector z from the latent space with the guidance of target comment y. By making z obey a prior probability distribution, our model can learn the diversity mapping relationship from the latent space to the reconstructed comments. Thus, in the inference stage, our model can generate diverse comments by sampling different sentiment-oriented latent vectors from the prior distribution.

---

Under review as a conference paper at ICLR 2023
To effectively integrate the sentimental information and the semantic information, model explic- itly structures the latent space with the guidance of sentimental information. As the prior distribution of the latent vector determines how the learned latent space is structured, which is crucial to the performance of the model, we construct as sentiment-based Gaussian mixture model:
plz 8) = Σ sf
where I the identity matrix, the sentiment weight mentioned before, Pj and oj represent the mean vector and standard deviation of the j-th Gaussian distribution.
Concretely, we leverage Transformer encoder layer map the target comment y and sentiment weight s to sampling region in the latent space for training. The last hidden state hr of the
Transformer layer is transformed into N mean vectors Pú and N log variances log oj. using a linear network for each:
(4)
Fg log of = LNj(hr), for = I,..,, N Nii oill,o"" :5).
(5)
7osst
(6)
where ppsst is the posterior latent vector, obtained by following the Gaussian mixture sampling mode of prior distribution in Equation B
2.2.3 SENTIMENT-ORIENTE RANDOM MASK
Contrary to the posterior collapse problem of VAE existing in other tasks, model confronts new mode imbalance problem where the model tends to rely heavily on the latent vector rather than the source input. Different from other tasks, the sentiment-guided ALVC is more difficult to reconstruct the target comments from the source input. As the latent space is infinite, the model tends ignore the input information and directly leams the mapping between the encoding region of latent space and the reconstructed comments, which is useful in training, but meaningless at inference. Therefore, we propose masked encoding mechanism balance the learning ability of the model to various features by sentimental masking the posterior latent vector 4post on the random mask region m. With linear network encode the latent vector, the multi-semantic sentiment feature representation is calculated as:
Vz Embedding, m1) 3 Zpost + m C Zprior):
where m. a binary mask region, which regional proportion is determined by the mask ratio A, O element-wise multiplication, and Zprior is the latent vector sampling from the prior distribution of the corresponding sentiment.
2.3 BATCH ATTENTION MODULE
To address the data imbalance problem, inspired by the batch interaction thought of Hou et al. (2022), we devise a batch attention module to explore the sample relationship during training. The proposed batch attention module leverages the batch multi-modal attention parallelly with the origi- nal multi-modal attention, which in turn leads to dealing with the multi-modal features in both batch dimension and spatial dimension. this manner, the proposed module can mitigate the gap between samples and model their relations.
Given a batch sample = X2,,X...... XB) with size B, the proposed batch attention module implicitly augments B virtual samples for each sample X; by modeling the relationship among samples in the mini-batch, resulting an improvement on data scarcity. Specifically, we adopt co-attention module acting the input features obtain the batch-attended textual and visual features:
e V (Co-AAtentiong (VB. vB))
B
(8)
where VB E Bxd and VB E R Bxd are reshaped from the original frame feature and text feature, to enable the co-attention module working on the batch dimension.
4

---

Under review as a conference paper at ICLR 2023
In the inference stage, considering the single-sample input, the batch attention module cannot be used directly. Therefore, we introduce a new auxiliary decoder, which conducts comment decoding based on the batch-attended visual and textual features V Ve and the sentiment feature V, and vz- By simply sharing weights between the auxiliary decoder and the final decoder, the model can still benefit from the sample relationship learned by the batch attention module, without utilizing the batch attention module in inference.
2.4 DECODER
2.4.1
SENTIMENT- DEETIMEET-PREDICTION PREDICTION
To predict the sentiment most expected by the viewers for a video clip, we design a sentiment- prediction module predict the most likely comment sentimental types based on the visual and textual context. The predicted sentiment label fs is calculated as:
pre LayerNormi W Vi + W.Ve). V p(ù. e,e) Sigmoid W pre Vpre);
(9)
(10)
where Wr, W. Rdxderr and W pre X learnable weight matrixes. At inference, the sentiment label ds predicted based on context will be used for generation, replacing the sentiment label ys from target comments.
2.4.2 COMMENT DECODER
The goal of the comment decoder is to generate comments for the target timestamp based context information and diverse sentimental information. More specifically, we decode the word ý, at every timestep & with Transformer decoder layer followed by softmax layer:
W. Transformer Y<o V1, Ve, Vs, Vz),
(11)
Plu yo. 3/-1, ye z) = Softmax (w)).
(12)
where <e denotes the masked comment (vo, ( 1h Inside the comment decoder, there five multi-head attention blocks, using Ye as the query to attend to y <t. Vr. V.. V, and V,, respec- tively. The auxiliary comment decoder has the same structure, but using yt the query attend to
<D VI. Ve V, and Vz, respectively.
2.5
LOSS FUNCTION
The traditional encoder-decoder model is realized by maximizing the log-likelihood function of generated comment ý. As generation model is controlled by latent vector z, the objective is changed to maximize:
p(f) - p(8 z)p(z)dz.
(13)
As it is impossible traverse all latent vectors for integration, by referring to the mathematical derivation in the Variational Autoencoder (Kingmm & Welling 2014), we use the Evidence Lower Bound (ELBO) approximate the log-likelihood function of generated comment ý:
log (yg)()) 2 [otesy)[oggd z)] - DKrlg(z p()),
(14)
where qz 8) corresponds to the probability distribution obtained by the diversity encoder and used to approximate the posterior probability distribution p(z y) of the decoder.
Since the model also affected by the video frames I, surrounding comments e, and generated or selected sentiment type ye. the objective function is changed to:
logp(; I. (.y.y) y+) 2 Eg(r)s [log p( I, y=)]
Dærlg(z I, p(z I, y.)]]
(15)
The first item encourages the model to generate higher quality comments, which is the loss of model reconstruction. The second item encourages the latent vector distribution obtained by training to be
5

---

Under review as a conference paper at ICLR 2023
close as possible to the prior distribution p(z I,e, Us), ie., p(z ys). which is the KL distance between them:
los8; = Dxila(z 8, e, M.), vs)]
22 0 + Mi Mj 1|
Σ
log
1
(16)
20
2
==1
Moreover, considering the
module, adopt cross-entropy loss optimize:
losspre log po( ys),
(17)
where 0 corresponds to model parameters, Us and ys are the predicted sentiment and ground truth sentiment respectively.
In short, the total loss function of the model is defined as:
C= lossre + 3 loss, + losSpre
(18)
where 3 and are hyper-parametten
3 EXPERIMENT
Experiment Setting. We conduct experiments on two live video commenting datasets: Livebot (Ma et al. 2019) and VideoIC (Wang et al. 2020). In all experiments, layers with 6 blocks, 8 attention heads, and set the hidden size of the multi-head attention layers and the feed-forward layers to 512 and 2, 048. Our model is optimized by Adam optimizer (Kingmal & Ba 2014) with the base learning rate set to - 4 and 3e 5 on datasets Livebot and VideoIC respectively, and decayed by 1/4 every 2 epochs after 4 epochs. The batch size is 128 and dropout rate is 0.1. The number of selected video frames and surrounding comments k and m are both set 5. For the joint loss function, the corresponding parameters are set as: 2 and y 0.3. We adopt as the number of layers of the co-attention module. In practice, the standard deviation of each prior distribution is fixed at 0.2. Unless otherwise stated, the sentiment category N is set to 3, as N = 3 exactly corresponds to the three sentiments of negative, neutral, and positive.
consider the Transformer
Evaluation Metrics. Since a time stamp contains multiple various comments, which makes the reference-basee metrics not applicable, we use the retrieval-based evaluation metrics proposed by Ma et al. (0199) for quality evaluation. As evaluation is formulated as a ranking problem, the model is asked to rank set of candidate comments based on the log-likelihood score. We provide a candidate comment set that consists of diverse ground-truth comments and three types of improper comments follows: 1) Correct: the ground-truie comments of the corresponding video and corresponding moment provided by human. 2) Plausible: 50 comments most similar to the video title which measured based on the cosine similarity of their TF-IDF values. 3) Popular: 20 most popular comments from the dataset which carry less information, such as "Great", "hahaha". 4) Random: after selecting the correct, plausible, and popular comments, randomly select some comments to make sure the number of the candidate set contains 100 comments.
The model ranks the ground-truth comments higher with higher generation probability, reflect- ing better model performance. We measure the ranking results with the following metrics: Re- call@k (the proportion of ground-truth comments in the top-k sorted comments), Mean Rank (the mean rank of the ground-truth truth comments), Mean Reciprocal Rank (the mean reciprocal rank of the ground-truth comments).
3.1 COMPARISON WITH STATE-OF-THE-ART
In order to verify the effectiveness of the proposed model, conduct experiments to compare So-TVAE with the state-of-the the-art methods, including Fusional RNN (Ma et al. 2019) Unified Transformer (Ma et al.] 2019). MML-CG (Wang et al. 2020s, DCA (Zhang al.] 2020b) and Matching Transformer (Duan et al. 2020). Besides, to further explore the impact of sentiment, we conduct the experiments with different sentiment categories (3, and 10).
the comparison results. Our proposed So- 6V-TVAE exhibits better performances on both two datasets. It is worth noting that oentinent-orientee model does simply improve as the

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparisons with the state-of-the-ar methods, where R@k, MR, MRR, are short for Recall@k, Mean Rank and Mean Reciprocal Rank. indicates that the performance is better with a lower score, 3,5,10 annotates the results of setting = 3,5, 10.
Livebot
VideoIC
Model
R@1
R@5
R@
10
MR
MRR
R@
R@5
R@ 10
MR MRR
F-RNN
17.25
3796
56.10
16.14 0.271
25.38
51.02
64.36
11.73 0.400
U-Trans
18.01
38.12
55.78
16.01 0.275
26.34
54.66
64.37
12,66 0.390
MML-CG
18.57
43.80
54.09
15.37 0.312
27.50
56.12
65.68
12.21 0.402
DCA
25.80
44.20
58.40
15.10
0.353
M-Trans
22.71
46.71
62.87
11.19
0.352
So-TVAEs
25.88
50.64
65.68
11.10
0.384
29.58
55.35
69.67
9.538
0.413
So-TVAE;
27.07
52.97
67.44
10.54
0.397
27.50
54.73
69.93
9.632 0.407
TV-Eo
24.04
51.46
66.83
10.73
0.374
29.59
55.09
70.34
9,431
0.415
3500
Human evaluation result on the Livebot test split.
Table 2:
3094
3000
2500
2457
Quamity
2000
2094
Model
Flu.
Rel.
Cor.
Div.
1500
1000
F-RNN
2.44
2.52
2.40
1.32
U-Trans
3.44
3.28
3.52
1.48
MML-CG
3.82
3.44
3.84
1.42
So-TVAE
4.36
3.72
4.32
3.88
Number generated
Human
4.64
4.12
4.52
4.24
Figure 3: Quantity distribution curve of mean- ingful generated comments.
increase of sentiment categories. The model achieves the best performances on Livebot dataset with N 5 and VideoIC dataset with N 10, meaning that the best effects will be obtained when the fine granularity of sentiment is matching the sentimental distribution of the dataset.
Human Evaluation. We test the model ability of generating human- comments. The generated comments are evaluated from four aspects: Fluency (fluency of comments), Relevance (relevant comments video content), Correctness (the confidence that the comments are made by humans), and Diversity (diversity of multiple generated comments). We randomly select 200 video clips, which include 1, 0000 comments from three baseline models, our So-TVAE, and ground-truth. Three human annotators with the higher the better. Finally, we take the average score of all annotators as the human evalu- ation results. As can be seen from Table 2 proposed So-TVAE achieves better performance in all aspects, especially in fluency, correctness and diversity. The great improvement in correctness indicates that our model can generate more human-style comments.
asked score the selected instances four metrics in the range of [1, 5],
Qualitative Results. Figure eesureeexhitee exhibits several live video commenting results of the baseline models and our So-TVAE, coupled with the video frames and contextual comments. Generally, compared with the baseline model, our So-TVAE produces more accurate and detailed comments. Taking the image Figure R (b) an example, our model generates more detailed and accurate phrase "tired", while the baseline models only predict high-frequenc, phrase "the first day". On the one hand, as we can see from Figure +. by choosing different sentimental types, the current model can successfully generate comments of different sentiments for single video frame, which proves the effectiveness of the sentimental diversity module. On the other hand, after selecting the same senti- mental type, our model can generate comments of different semantics with the same sentiment, such as "Really good" and "The harmony is so great" part (d) Figure verifying the effectiveness of the semantic diversity module. In general, the sentiment-oriented multi-semantic comment gen- eration method is more in line with human thinking habits, resulting in more humanized comments with the language characteristies of multiple perspectives and multiple sentiments.

---

Under review as a conference paper at ICLR 2023
(a). This 61/ ih AERUMM, SEA H Frleshura
(b). Tille 19:90ink22. HUMBUR WEEN-
Sharghai scheod!"
secerddgyi
leprose my bady dupp)
broken)
1o
(AL
(The day!
256 (Crytng, hiboon
R secund day, doch)
1itle 6.22 Hilve Mt HIIEEE At0/111
"The Clinew misuution tivieciiote in those yoars No.""
"Lieder the Bolng deeds, bes Insmurrdd ar stegles, in loscbee"
R AE (Bolhoftlen pood)
HH.. 11
(Sovadoo
(itamnoyy smunca prea))
PE POAA EL (Ther stag high worewer wuthy
(Thb
4/9 (Grer)
1-Trams
U-Tram
2 [Tble prake
MIE3RAAA (Skng wey well
So TYAE
Se:TVAK
priis 1hle1111
BULLE TI bumnem lesegovat
94309-933---12 watched whan was kid)
XLON (Reals poall
A A: I (Th tnalle le gros))
(M: Youth, youth, youth)
SEREI (Just sese)
Figure 4: Examples of generated comments by the baseline and So-TVAE for live video comment- ing. Green; positive comments. Red: neutral comments. Blue: negative comments. Context: human posted comments in context. The underline marks the generated comments of predicted sentiment.
Table 3: The results of ablation studies on Livebot and VideoIC.
Livebot
VideoIC
Module
Variant
R@1
R@5
R
MR-
MRR
R@1 1
R@5
R@
MRR
w/o
24.34
47.21
62.47
12.36
0.363
28.44
52.10
66.39
10.73
0.399
Diversity
SEND
24.39
47.24
62.83
12.61
0.362
28.73
53.48
67.70
10.30
0.412
SMD
25.22
50.50
64.76
11.65
0.382
29.20
54.47
68.41
10.10
0.418
A=
24.50
49.18
63.96
11.97
0.370
28.48
54.44
67.87
9,964
0.403
Mask
1=15%
25.21
50.42
64.76
11.48
0.379
29.22
55.27
68.89
9.798
0.421
A=45%
24.84
50.61
65.60
11.28
0.378
29.05
55.08
69.40
9.614
0.419
w/o
21.39
45.12
59.66
12.83
0.336
21.77
45.57
59.98
12.76
0.342
Batch-Atteation
Origin
22.93
47.64
61.94
12.33
0.354
29.00
53.42
67.43
10.55
0.412
Co-attention
w/o
8.678
20.04
34.34
22.12
0.171
16.98
38.08
53.26
15.69
0.284
Full model
So-TVAE
25.88
50.64
65.68
11.10
0.384
29.58
$5.35
69.67
9.538
0.413
3.2 ABLATION STUDY
To fully examine the contribution of each design in our model, we conduct ablation studies to com- pare different variants of So-TVAE. The corresponding variants of each module are shown below:
Diversity Encoding: !) the base model without any diversity components; 2) SMD: uses the stan- dard normal distribution the prior distribution without sentimental information; 3) SEND: simply encodes the sentimental information as the input to the decoder.
Mask: the models with different mask radio X, where
means without masked encoding.
Batch Attention | 1) the base model without any batch attention module; 2) the model which uses the original BatchFormer module proposed in Hou al. (2022) to model the sample relationship.
Co-attention the model without co-attention module.
According to the results shown in Table we have the following observations: (1) For diversity encoding, both the semantic diversity module and the sentimental diversity module bring an im- provement with respect to the base model. (2) When the mask ratio À is too large too small, the
See Appendix B for more details.
"See Appendix A for more details.
8

---

Under review as a conference paper at ICLR 2023
model will not learn enough source input information diversity information. By setting the appro- priate mask ratio to capture a balance between diversity and average quality, the model achieves the best overall performance with À 30% (full model). (3) The model with proposed batch attention module substantially outperforms the model without batch attention or with the original module, proving the effectiveness of our reconstruction of sample relational learning.
In addition, to further demonstrate the effectiveness of the proposed batch-attentinn module on data imbalance problems, we carry out extensive experiments. For the sentiment-guided live video com- menting, an unfavourable situation of data imbalance is that with the absence of comment samples with corresponding sentiment, the model difficult to learn this type of comment information and tends to generate an empty sentence. Thus, we use the model with and without batch attention module to generate N' (3, 5, and 10) comments for all 3, 768 samples the Livebot test split, ex- ploring the distribution of meaningful comment numbers. Figure 3 shows the distribution results. Obviously, we can observe that the model with batch-attention module can generate more meaning- ful comments than without utilizing batch-attention module. Taking the 10 sentiments situation as an example, 3, 099 samples can generate comments for each sentiment using the full model, while only 2, 084 samples can generate all sentimental comments without batch-attention. These results further prove that our proposed batch-attention module can effectively alleviate the problem of data imbalance in sentiment-guided ALVC.
RELATED WORK
Similar to but different from video captioning (Pan et al. 2020 Tan al. 2020 image captioning (Zhang et al. 2020a| /Zhao et al., 2020) and diverse text generation (Dathathri et al. 2020 Nie et al, 2019), automatic live video commenting aims at generating interactive comments, requiring a fully understanding of surrounding multimodal context. Ma et all (2019) first proposed the ALVC task and proposed two basic neural networks to jointly encode the visual and textual information into video comments. Duan et al. (2020); Zhang et al. (20206) explored the interaction between multi- modal information. Wang et al. (2020) constructed the VideoIC dataset on the consideration of higher comment density and diversity, and proposed a multimodal and multitask generation model to achieve effective commenting and temporal relation prediction. Zeng et al. (202T); Wu et al. 2021) further explored highlight detection and temporal interaction for ALVC. However, these works ignore the importance multi-sentiment in the application of live video commenting.
To achieve the sentiment-guie guided live video commenting, we consider some one-to-many generation methods for paired data, such as the mixture of experts (Shen et al. 2019), discrete domain encod- Variational Autoencoder (Kingma & Welling 2014 Child 2021 Zhu et al.] 2020, Shao et al., 2020). Among these, Variational Autoencoders were proposed by Kingma & Welling (2014)) 40: Rezende et al. (2014), and were initially applied in the
ing (Lachaux et al.) 2020; Gan et al. 2017)
computer vision task. [owmmnn et al. (2015) introduced VAES into natural language processing for text generation for the first time, then VAEs have been extended by many following works in various language generation tasks (Wen et al. 2017 Zhao et al. 2018 Hu et al. (017). To date, Conditional Variable Autoencoders (CVAEs) have been used in image generation (Child 2021), text summa- rization WWang et al. (2019), image question generation Jain et al. (2017), image caption (Wang et al., 2017), dialog model Zhao et al. 2017) and other tasks, our work the first to apply them to the ALVC task. By addressing the problem of multi-modal learning and sentiment information fusion, our model achieves the generation of diverse comments under sentiment guidance.
5 CONCLUSION
In this paper, we propose Sentiment-oriented Transformer-based Variational Auto-Encoder (So- TVAE) for the sentiment-guice -uided ALVC task which can achieve sentiment-guided live video com- menting. Specifically, we propose a sentiment-oriented diversity encoder to model the rich semantic diversity information and the sentimental diversity information in the target comments. To alleviate the mode imbalance problem, we propose a novel sentiment-orrieted random mask mechanism to ensure the quality and diversity of the generation model. In addition, we also propose batch atten- tion module to model the sample relationship. which is useful to address the video data imbalance problem. Extensive experiments verify the effectiveness of our proposed method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
S. R. Bowman, L. Vilnis, O. Vinyals. A. M. Dai, and S. Bengio. Generating sentences from continuous space. Computer Science, 2015.
R. Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference Learning Representations (ICIR), 2021.
Dathathri, A. Madotto, Lan, Hung, Frank, P. Molino, Yosinski, and R. Liu, Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations (ICLR), 2020.
Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, and Tiejun Zhao. Multimodal matching transformer for live commenting. ECAL, arXiv preprint arXiv:2002.02644 2020.
C. Gan, G. Zhe, X. He, Gao, and D Li. Stylenet: Generating attractive visual captions with styles. In IEEE Conference on Computer Vision and Pattem Recognition, 2017.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Hochreiter, Sepp, Schmidhuber, and Jurgen. Long short-term memory. Neural Computation, 1997.
Z. Hou, Yu, and D. Tao. Batchformer: Learning to explore sample relationships for robust representation learning. In CVPR, 2022.
Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing. Toward controlled generation of text. In International Conference on Machine Learning (ICML), 2017.
U. Jain, Z. Zhang, and A. Schwing. Creativity: Generating diverse questions using variational
autoencoders. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
W. Kim, Y. Ahn, D. Kim, and K. H. Lee. Emp-rft: Empathetic response generation via recognizing feature transitions between utterances. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI. 2022, 2022.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. Computer Science, 2014.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. In In International Conference Learning Representations (ICLR). 2014.
M. A. Lachaux, A. Joulin, and G. Lample. Target conditioning for one-to-many generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Li, H. Yan, and X. Qiu. Contrast and generation make bart a good dialogue emotion recognizer. In Conference Artificial Intelligence, AAAI, 2021.
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, and Xu Sun. Livebot: Generating live video comments based on visual and textual contexts. In AAAI, volume 33, pp. 6810-6817, 2019.
W. Nie, N. Narodytska, and A. Patel. Relgan: Relational generative adversarial networks for text generation. In International Conference on Learning Representations (ICLR), 2019.
B. Pan, H. Cai, D. A. Huang, K. H. Lee, A. Gaidon, E. Adeli, and J. C. Niebles. Spatio-temporal graph for video captioning with knowledge distillation. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
S. Sabour, C. Zheng, and M. Huang. Cem: Commonsense-awarc empathetic response generation. In Conference Artificial Intelligence, AAAI, 2021.
10

---

Under review as a conference paper at ICLR 2023
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang. Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In International Con- ference Machine Learning, pp. 8655- PMLR, 2020.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning. pp- PMLR, 2019.
G. Tan, D. Liu, M. Wang, and Z. J. Zha. Learning to discretely compose reasoning module net- works for video captioning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI, 2020.
H. Tian, C. Gao, X. Xiao, H. Liu, B. He, H. Wu, H. Wang, and F, Wu. Skep: Sentiment knowledge enhanced pre-training for sentiment analysis. Association for Computational Linguistics, ACL, 2020.
Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. Advances in Neural Information Processing Systems (NIPS), 2017.
W. Wang, Z. Gan, H. Xu, R. Zhang, G. Wang, D. Shen, C. Chen, and L. Carin. Topic-guided variational autoencoders for text generation. In NAACL, 2019.
Weiying Wang. Jieting Chen, and Qin Jin. Videoic: A video interactive comments dataset and mul- timodal multitask learning for comments generation. In The 28th ACM International Conference Multimedia, 2020.
X. Wang and C. Zong. Distributed representations of emotion categories in emotion space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
Tsung Hsien Wen, Yishu Miao, Phil Blunsom, and Steve Young. Latent intention dialogue models. In International Conference on Machine Learning (ICML), 2017.
Hao Wu, Gareth James F Jones, and Franois Pitié. Knowing where and what to write in automated live video comments: A unified multi-task approach. In International Conference on Multimodal Interaction (ICMI), 2021.
H. Yan, L. Gui, G. Pergola, and Y. He. Position bias mitigation: A knowledge-aware graph model for emotioncause extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing VVllue I: Long Papers), 2021.
Z. Zeng, N. Gao, C. Xue, and C. Tu. PLVCG: Pretraining Based Model for Live Video Comment Generation. Advances in Knowledge Discovery and Data Mining, 2021.
Z. Zhang, K. Chen, R. Wang. M. Utiyama, and H. Zhao Neural machine translation with universal visual representation. In ICLR 2020: Eighth International Conference Learning Representa- tions, 2020a.
Z. Zhang, Z. Yin, S. Ren, X. Li, and Li. DCA: Diversified Co-atiention Towards Informative Live Video Commenting. NLPCC(2), 2020b.
Zhao, Z. Ran, and M. Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Association for Computational Linguistics,ACIA 2017.
T. Zhao, K. Lee, and M. Eskenazi. Unsupervised discrete sentence representation learning for inter- pretable neural dialog generation. In Association for Computational Linguistics, ACL(1), 2018.
W. Zhao, X. Wu, and X. Zhang. Memeap: Memorizing style knowledge for image captioning. Proceedings of the AAAI Conference Artificial Intelligence, 34(7):12984-133 2020.
Q. Zhu, J Su. W. Bi, X Liu, X Ma. X Li, and D Wu. batch normalized inference network keeps the kl vanishing away. In ACI., 2020.
11


Reviews:
[{"review_id": "KWZ01NbzZo", "reviewer": "Reviewer_bz3b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents methods for the task of Automatic Live Video commenting (ALVC). The paper proposed a sentiment oriented robust diversity encoder and a batch level decoder network that can handle missing sentiment samples in the dataset. The authors conduct extensive experiments on two datasets and demonstrate that this method outperforms other known methods for automatic live video commenting. ", "summary_of_the_review": "Overall, the paper is pretty well-written for the task of ALVC. The proposed sentiment-oriented diversity encoder combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance. The paper also uses a batch-attention module to alleviate the data imbalance problem. The work leverages batch-level attention along with multi-modal attention in a parallel way (using auxiliary tied together decoder model) to simultaneously deal with the multi-modal features in batch dimension and spatial dimension. Results and ablations show that the method is promising for the problem. ", "strength_and_weaknesses": "Weaknesses:\n- The paper introduces this task as a feature of interactive mode online video websites. The authors mention that that this feature is increasingly being used on websites for narration generation etc. Some references to the use cases will be helpful to underscore the relevance of this task. \n- Authors introduce 3 contributions in the paper: the contributions on introducing a new task as well as extensive experiments are not really novel contributions (given that the task is ALVC and running experiments and evaluation isn't really a novel contribution of the work) \n- Authors do not discuss the inference times for running the various VAE based models as well as the sentiment prediction models. Can this method be used efficiently at inference time? \n\nStrengths:\n- There isn't novelty in the individual algorithms proposed but the overall pipeline that includes blocks on sentiment diversity encoder, the batch level auxiliary decoder and the combination of the encoder and decoder using VAEs looks very elegant. \n- Experiments are conducted on 2 datasets with extensive evaluation (automatic and human) and top of line results. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HGian_Clk7K", "reviewer": "Reviewer_5558", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Sentiment-oriented Transformer-based Variational Autoencoder (SO-TVAE) model for the automatic live video commenting task. The SO-TVAE takes sentiment clues of the surrounding comments to generate new automatic live video comments. The model also adopts the VAE to achieve commenting sentences with diverse sentiments and semantics.", "summary_of_the_review": "The paper studies a novel and interesting task ALVC. A new model is proposed to solve the sentiment and semantic diversity problem in ALVC, which achieves significant improvement compared with previous works.", "strength_and_weaknesses": "Strengths:\n1. Novel and interesting task.\n2. Well designed model, and achieve state-of-the-art performance on relative datasets.\n\nWeaknesses:\n1. It’s a bit unclear when introducing the motivation. According to Section 2.1, this task aims to generate the new automatic comments based on the context of some historical comments. Naturally, the historical comments may have multiple sentiments. Then, what kind sentiments should be used to guide generation of new comments, and why?\n\n2. Why does Equation 4 has three inputs. According to Figure 2, the Transformer Encoder takes $s$ and $y$ as inputs.\n\n3. In Equation 18, the denote $L_rc$ should be pre-defined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5x2bvpTaG_a", "reviewer": "Reviewer_b3zH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper a Sentiment- oriented Transformer-based Variational Autoencoder (So-TVAE) , which consists of a sentiment-oriented diversity encoder module and a batch-attention module.\nThe encoder part combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments,\nwhile the batch-attention module is to alleviate the problem of missing sentimental samples.\nAuthors perform experiments on Livebot and VideoIC datasets and compare it with the state-of-the-art methods in terms of the quality and diversity of generated comments. ", "summary_of_the_review": "The authors propose and evaluate a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network in this paper to achieve sentiment-guided live video commenting.\nHowever, the novelty and theoretical basis of the proposed model and its evaluation are insufficient.", "strength_and_weaknesses": "Strength\\\n*Automatic live video commenting is interesting task.\n\nWeaknesses\\\n*Low technical novelty of model\\\nThe model is a straightforward collection of known frameworks (Transformers and VAE), \nits motivation is unclear, and the framework itself is a bit dated.\nVAE and random mask mechanism may solve the semantic diversity under sentiment guidance, \nbut how does this mechanism or batch-level attention along with multi-modal attention solve the modality gap between text and video?\\\nIs there any reason to use Transformer and LSTM, as the initial features layer, together instead of just Transformer?\netc.,\n\n*Technical terms and descriptions are incomplete\\\nIf $s_{i}$ is the weight of $I$-th sentiment?, what makes $N()$ in eq(3) different than $N()$ in eq (6) ?\\\nWhere is  $W_{w_{t}}$  and $L_{rc}$ defined?\nIs there any reason why the random weight is more efficient than learnable weights in eq (7)?\n\n*Lack of existing research on Transformer-based models  and the resulting distrust of ideas for proposed architecture.\\\nWhy is there no research and comparison on the application of pre-trained models like GPT-2 as the decoder?\\\nThis lack leads to the weakness of the comparative experiments,\nand can not support the theoretical basis of the approach.\n\n*The purpose and results of the experiment do not adequately support the claim.\\\nTable 2 show that an assessment of the consistency of the comments as man-made comments, but does not appear to be an assessment of whether they reflect emotion.\\\nWhat are the video categories and their ratios?\\\nHow much is the ratio of positive-negatives in the data set?\\\nAs the proposed model is a generative model but does not use METEOR, ROUGE, or BLEU as the valuation metrics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2w0v9Y2WCz", "reviewer": "Reviewer_TAXK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors identify the importance of the sentiment information in automatic live video comments. Toward this end, the authors work on sentiment-guided automatic live video commenting, which aims to generate live video comments based on sentiment guidance.  More specifically, they propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network, which consists of a sentiment-oriented diversity encoder module and a batch-attention module.", "summary_of_the_review": "The authors work on sentiment guided live video commenting. The problem is interesting and challenging. However, the motivation regarding sentiment is not well supported by the empirical results. More details can be referred to Strength And Weaknesses section. ", "strength_and_weaknesses": "Strength:\nThe authors work on an important problem and the proposed method is able to introduce sentiment information into complicated live video comment generation.  Additionally, the authors propose a new batch attention mechanism to further improve the performance. \n\nWeakness:\n1. The authors hypothesize the important of the sentiment information. However, such a motivation is not clearly supported by the empirical studies. In the table 3,  SMD (uses the standard normal distribution as the prior distribution without sentimental information) is able to provide a very close performance to the proposed method. Such a result cannot support the motivation of the proposed method, which is sentiment-guided automatic live video commenting. \n2. The improvement gap brought by the proposed method are not very large. The multiple runs are needed to validate the significance. \n3. The technical novelty is limited. The proposed method is build upon combination of the existing techniques without clear novel designs. \n4. In the ablation studies, some of the results from the ablation baselines are better than the proposed method, like the last column in VideoIC, several values  in term of MRR are higher than the bolded value 0.413.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject