Multimodal Backchannel Prediction with Video
Authors: Jun-Yeong Moon, Min-Jae Kim, Gyeong-Moon Park


Reviewer paper summaries: This work introduces a method for predicting multimodal backchannel signals by incorporating video, audio, and text. It proposes a multi-level, multimodal alignment and cross-attention alignment with contrastive learning to enhance representation learning. The study demonstrates that this alignment method outperforms the naive concatenation approach in the ETRI backchannel corpus task, showing the model's improved understanding of multimodal interactions.
Summary of strengths: (1) Meaningful task and comprehensive metric: The study focuses on backchannel prediction, a crucial yet often overlooked aspect of conversation and QA research. It offers various types of F1 scores to comprehensively evaluate performance and focus on data imbalancement.

(2) Sufficient technical details: The study provides ample details regarding the dataset and implementation. It also clearly explains the unimodal models and baselines used in the research.
Summary of weaknesses: (1) Lack of validation split: Based on Table 5, is there a specific reason for selecting layers 9-11? Does this work include a development set to help with model selection? It seems like there might be some overfitting on the test set due to strict layer selection to achieve the highest performance. Additionally, the F1 results for all experiments appear quite similar. Are these differences statistically significant? Is there a p-value that we can refer to?

(2) More baselines:  Related to backchannel prediction task, it should be included in the results, in addition to BPM-MT.

(3) Limited model types: The methods are based on KoBERT, HuBERT, and VideoMAE. It seems like all models are required to have the same number of layers to make it work. Does this method work for models of different sizes with varying numbers of layers?

(4) Limited dataset: The study is validated only on the ETRI backchannel corpus dataset. This might not be sufficient to verify the effectiveness of the proposed method. At least one more dataset should be included to prove that the method is generally useful for all backchannel task.

(5) Lack of detailed MMA loss analysis: Beyond the current analysis, it would be beneficial to include more insights into which part of the MMA loss is the most critical and which is not

(6) Case study section: Including a section with case studies would help readers understand both successful and failed cases with concrete examples.
Comments, suggestions and typos: There is still space for more case study and analysis to fit 8 page limit. More analysis and ablation study can be potentially useful to prove the effectiveness of the method.
Overall assessment scores: 2.5