FakingRecipe: Detecting Fake News on Short Video Platforms from the Perspective of Creative Process

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-04 15:48:13

Abstract: As short-form video-sharing platforms become a significant channel for news consumption, fake news in short videos has emerged as a serious threat in the online information ecosystem, making developing detection methods for this new scenario an urgent need. Compared with that in text and image formats, fake news on short video platforms contains rich but heterogeneous information in various modalities, posing a challenge to effective feature utilization. Unlike existing works mostly focusing on analyzing what is presented, we introduce a novel perspective that considers how it might be created. Through the lens of the creative process behind news video production, our empirical analysis uncovers the unique characteristics of fake news videos in material selection and editing. Based on the obtained insights, we design FakingRecipe, a creative process-aware model for detecting fake news short videos. It captures the fake news preferences in material selection from sentimental and semantic aspects and considers the traits of material editing from spatial and temporal aspects. To improve evaluation comprehensiveness, we first construct FakeTT, an English dataset for this task, and conduct experiments on both FakeTT and the existing Chinese FakeSV dataset. The results show FakingRecipe's superiority in detecting fake news  on short video platforms.

Main paper: 
 empirical analysis: 
 We exhibit the manifestation differences between real and fake news videos in different phases of news video creation by conducting empirical analysis on real-world datasets, including the publicly available Chinese dataset FakeSV [35] and the newly constructed English dataset FakeTT. We identify the discrepancies between real and fake news production processes and provide plausible explanations for these phenomena, highlighting the nuances in the creative process to evaluate the news video veracity. Considering that consistent results were observed across both datasets, we only present findings from FakeSV here due to space limitations and attach results on FakeTT in the appendix.Observation 1. When selecting audio materials, fake news tends to opt for more emotionally charged audio.Considering background music (BGM) is a predominant option for short video news creators and the nature of BGM it serves primarily to evoke emotional responses, our analysis of audio selection behaviors focuses on the emotional aspect. We leverage the pre-trained wav2vec model [39] that has been fine-tuned for audio emotion classification. Depicted in Figure 2 , we can see that fake news videos exhibit an inclination towards using emotionally charged audio. Given that prior work [9] has indicated emotionality significantly boosts content sharing, we attribute this bias in audio selection to creators' intentions to maximize viewer engagement.Observation 2. When selecting visual materials, fake news often employs clips that exhibit a relatively lower semantic consistency with the accompanying text.We analyze creators' visual selection behaviors from the perspective of consistency between selected video materials and accompanying text. Specifically, we leverage the pre-trained text-image representation model CLIP [38] to extract textual and visual features. By normalizing these features and calculating the Jensen-Shannon (JS) Divergence between the attached text and each frame's visual content, we derive an average JS Divergence score across multiple frames as an indicator of text-visual consistency for the entire video.A lower indicator value signifies a higher semantic consistency between the video's textual and visual content. Figure 3 illustrates the distinct distributions of JS Divergence between textual and visual materials for fake and real news. The discrepancies have been statistically confirmed through the Kolmogorov-Smirnov (KS) test, with a p-value of less than 0.05. We find that fake news tends to utilize visual clips with noticeably lower semantic consistency with the accompanying text. We attribute the observed biases in video material selection to the nature that fabricated news inherently lacks access to a rich array of related video materials.We investigate two fundamental editing operations in video creation: spatial editing behaviors and temporal editing behaviors.Observation 3. When spatially imposing text, fake news tends to display relatively plain textual visuals.Spatial editing behaviors involve overlaying additional layers on top of the original visual materials. This can include adding animated stickers, text, and other elements. Among them, text imposition is a widely used operation in short news videos (85% in the FakeSV dataset), with variations reflected in decisions regarding the text's placement, color, typeface, and font. Here we quantified the color characteristics of the text visual areas in real and fake news videos respectively to explore the differences in color choice behaviors during text imposition. Figure 4 illustrates that real news videos tend to use a richer color palette for text presentation. We attribute this preference to that real news creators often follow conventional editorial norms and invest more effort to improve the presentation quality. Conversely, fake news creators often employ a monochromatic color scheme when imposing text, likely due to a lack of expertise in news production, leaving them unaware of the potential impact these details can have on viewers. Observation 4. When temporally splicing materials, fake news tends to adopt a relatively simple arrangement.Temporal editing behaviors, on the other hand, refer to the reorganization and splicing of multiple material segments. The duration and positioning of different segments can subtly influence viewers' perceptions of the news video. Here we examine the temporal editing behaviors related to text exposure, analyzing differences in the temporal arrangement of text segments between real and fake news. Specifically, we developed an indicator, 𝐼 𝐷 , to measure the dynamism of text presentation. By calculating the mean (𝜇) and standard deviation (𝜎) of exposure durations (𝑑 1 , 𝑑 2 , ...) for different text phrases within a video, 𝐼 𝐷 is defined as 𝜎 (1 -𝜇), based on the principle that shorter exposure times and greater variance among text exposure durations indicate stronger text temporal editing dynamism. Figure 5 shows the fitted sample density distribution of the on-screen text dynamic scores in the FakeSV dataset, revealing significant differences between the temporal editing behaviors of real and fake news, with real news exhibiting more dynamic text presentations. We ascribe this tendency to two factors: First, the disparity in video creation capabilities, wherein most creators of authentic news, endowed with professional media training, possess a nuanced understanding of effectively integrating text with visual elements. Second, the constraints posed by the availability of materials. Fabricated news, inherently characterized by its scant and biased content, often lacks the robust information necessary for dynamic presentations. This deficiency compels creators to resort to the static placement of limited information in specific areas of the screen. 
method 3.1 overview: 
 Drawing on the insights from our empirical analysis, we present FakingRecipe (Figure 6 ), a creative process-aware fake news video detection model. The model observes the given news video from the two pivotal phases of the creative process to unearth veracity indicating clues. Treating the feature from two phases as independent viewpoints, FakingRecipe is structured with dual branches operating separately and employs a late fusion strategy to integrate predictions from these independent perspectives.Based on prior analysis, we examine the creators' material selection behavior from two aspects (i.e., sentiment and semantic). The dominant role of different modalities varies in conveying information: Audio primarily expresses emotion, text renders emotional tones while conveying semantic information, and visuals generally complement the text to communicate semantic content. Therefore, we strategically select combinations of modalities for multifaceted feature extraction, subsequently fusing multimodal features from multiple viewpoints.Specifically, for the sentimental aspect, we consider audio and textual content as the primary sources. We utilize fine-tuned versions of HuBERT [17] and XLM-RoBERTA [8] as encoders to extract audio sentimental features H 𝑆𝐸𝑁 -𝐴 and textual sentimental features H 𝑆𝐸𝑁 -𝑇 , respectively. These sentimental features from different modalities are then concatenated and fed into a standard Transformer layer [46] . By leveraging self-attention, the transformer layer fuses multimodal sentimental features to produce a unified sentimental feature representation H 𝑆𝐸𝑁 .In the semantic aspect, visual and textual contents take precedence, while the audio mainly serves as background music, playing a minimal role. Keyframes are extracted from videos, serving as the basis for visual analysis. Utilizing CLIP [38] , we encode text and keyframes to token/frame-level text semantic features H 𝑆𝐸𝑀 -𝑇 and visual semantic features H 𝑆𝐸𝑀 -𝑉 . Interaction between text and visual content is facilitated through a co-attention transformer [27] , resulting in visually enhanced textual features H' 𝑆𝐸𝑀 -𝑇 and textually enhanced visual features H' 𝑆𝐸𝑀 -𝑉 . These features are then averaged, concatenated, and input into a transformer layer mirroring the structure used in the sentimental analysis. This process integrates semantic features from various modalities into a singular semantic feature representation H 𝑆𝐸𝑀 .The sentimental feature H 𝑆𝐸𝑁 and semantic feature H 𝑆𝐸𝑁 are then concatenated and fed into a two-layer MLP to derive the fake news predicted score Ŷ𝑆 from the material selection analysis perspective:EQUATIONIn mining detecting clues from the perspective of creator editing behaviors, we focus on spatial and temporal aspects, identified as critical in our empirical analysis. Spatially, we examine the prevalent practice of imposing text. Given a video 𝑉 , we select a representative text-rich frame 𝐹 𝑇 , identified based on the size of the text presence area, as our starting point. We first employ an OCR spotting model, CRAFT [1] , to delineate text regions 𝑇 𝑅 = {box 1 , box 2 , ...} within 𝐹 𝑇 . These regions are subsequently transformed into prompt embeddings H 𝑃𝑀𝑇 employing a methodology inspired by the prompt encoder in Segment Anything Model (SAM) [23] . In parallel, 𝐹 𝑇 undergoes processing by a pre-trained Vision Transformer (ViT) [10] to produce initial encodings H 𝐼 𝑀𝐺 . Both H 𝐼 𝑀𝐺 and H 𝑃𝑀𝑇 are then fed into a Two-Way Attention block, mirroring the structure used by the SAM's mask decoder. The block leverages both prompt self-attention and cross-attention mechanisms, functioning in two directions (promptto-image and image-to-prompt). This dual attention strategy is employed to refine the initial visual encoding H 𝐼 𝑀𝐺 , ensuring it focuses more accurately on text regions within the frame. Following attention processing, the updated H' 𝐼 𝑀𝐺 undergoes downsampling via two convolutional layers and then flattened to derive the spatial pattern feature H 𝑆𝑃𝐴 . Figure 7 provides a detailed depiction of the Two-Way Attention block and the downsampling network.Temporally, we examine the splicing practice of text segment and video segment. The audio track is omitted due to the observation that most audio tracks consist of continuous background music. , FrameIdx 𝑒𝑛𝑑 𝑖 ] marks the time interval of the 𝑖-th segment's appearance. The input also incorporates frame rate (fps) and the total frame count (vframes) of the video to contextualize duration. Each modality's temporal structure is initially modeled separately, followed by an interaction phase to derive overall temporal editing features. Specifically, we design a Hierarchical Temporal Structure Extractor (HTSE) for extracting temporal structure features applicable to both modalities. HTSE first performs intra-segment fusion for content occurring in the same time span to derive segment content features Seg 𝑖 . For text, Seg 𝑇 𝑖 is obtained by concatenating multiple segments and encoding them collectively, while for visuals, it applies the self-attention (𝑆𝐴) mechanism for integration:EQUATIONwhere k is the frame count within segment 𝑖, and MEAN(•) denotes the mean pooling. To model the subtle influences of the duration and temporal position of different segments, we introduce each segment's temporal position and exposure duration information. Positional encoding (PE) is generated using sine and cosine functions to reflect each segment's temporal position, akin to that leveraged by Transformer [46] :EQUATIONwhere 𝑤 𝑘 = 1/(10000 2𝑘/dim 𝑃 𝐸 ) represents the frequency of the sinusoid for each dimension and PE 𝑖 is the 𝑖-th segment's positional embedding. For duration encoding (DE), we employ an equifrequency binning approach, determining duration groups through empirical analysis and assigning a learnable embedding to each group. Considering the audience's perception of exposure time in reality, both absolute and relative durations are evaluated:Dura 𝑎𝑏𝑠 𝑖 = (FrameIdx 𝑏𝑒𝑔𝑖𝑛 𝑖 -FrameIdx 𝑒𝑛𝑑 𝑖 )/fps, Dura 𝑟𝑒𝑙 𝑖 = (FrameIdx 𝑏𝑒𝑔𝑖𝑛 𝑖 -FrameIdx 𝑒𝑛𝑑 𝑖 )/vframes.Here Dura 𝑎𝑏𝑠 𝑖 and Dura 𝑟𝑒𝑙 𝑖 denote the absolute (in seconds) and relative (the proportion of the total duration) durations of segment 𝑖, respectively. The 𝑖-th segment's duration embedding is:EQUATION) where Group(•) maps a duration to its designated group and Emb(•) retrieves the corresponding embedding for that group.Integrating positional and duration encodings, the segment features are updated to SEG 𝑖 , serving as the input for inter-segment fusion, which captures the relationships between different segments using a similar self-attention mechanism, generating temporal pattern features for each modality:EQUATIONEQUATIONUtilizing EQUATIONwhere F (•, •) is the fusion function. Inspired by previous works [6, 49] , we adopt the tanh(•) function, which introduces non-linearity to enhance the fusion strategy's representational capacity.Following previous works [7, 35, 41] , we utilize cross-entropy loss to optimize our model:EQUATIONwhere 𝑌 is the ground-truth label for each short video news.To further supervise the material selection and material editing modeling, the final loss L incorporates the loss for Ŷ𝑆 and Ŷ𝐸 :EQUATIONwhere 𝛼 and 𝛽 are hyperparameters that balance the impacts on the back-propagation of the three branches. L 𝑆 and L 𝐸 denote the Cross-Entropy( Ŷ𝑆 , 𝑌 ) and Cross-Entropy( Ŷ𝐸 , 𝑌 ), respectively. 
experiments: 
 In this section, we conduct extensive experiments on two real-world datasets to verify the effectiveness of FakingRecipe by comparing it with seven representative baselines and the FakingRecipe variants.To validate the generalizability of the proposed FakingRecipe, we conduct experiments on two datasets of different languages:FakeSV2 : The largest publicly available Chinese dataset for fake news detection on short video platforms, featuring samples from Douyin and Kuaishou, two popular Chinese short video platforms. Each sample in FakeSV contains the video itself, its title, comments, FakeTT: Our newly constructed English dataset for a comprehensive evaluation in English-speaking contexts. 3 Curated from TikTok, this dataset follows a similar collection process to [35] , focusing on videos related to events reported by the fact-checking website Snopes 4 . Each video was rigorously annotated for authenticity by at least two independent annotators, resulting in a collection of 1,172 fake news videos and 819 real news ones from May 2019 to March 2024, with video, audio, and text description (title) available. See more details in the appendix.Table 1 shows the statistics of the two datasets. To simulate real-world scenarios, we adopt a temporal split strategy for our experiments, dividing the data chronologically into training, validation, and testing sets with ratios of 70%, 15%, and 15%, respectively. Such a data split reflects the potential of applying compared methods in reality. Handcraft Feature-based Baselines: (1) HCFC-Hou [16] utilizes linguistic features from speech, acoustic emotion features, and user engagement statistics with a linear kernel SVM for classification. (2) HCFC-Medina [40] extracts TF-IDF vectors from video titles and the first hundred comments, applying SVM for detection.Neural Network-based Baselines: (1) FANVM [7] harnesses visual features from keyframes and textual features from titles and comments, using an adversarial network to extract topic-agnostic multimodal features for classification. (2) TikTec [41] employs speech text-guided visual object features and MFCC-guided speech textual features, using a co-attention mechanism for fusion and classification. (3) SVFEND [35] leverages cross-modal transformers to boost interaction between modalities and integrates content with social context features via a self-attention mechanism.(M)LLM Baselines: (1) GPT-4 [32] is one of the most powerful LLMs currently and is used to make predictions based on video news titles and extracted on-screen text. We use a zero-shot prompt template inspired by Hu et al. [18] . (2) GPT-4V [52] is the variant of GPT-4 that supports visual inputs. We include the video's thumbnail in the inputs to explore the capabilities of (M)LLMs in this task.Given that we focus on content-only detection at the early news spreading stage, all baselines are adapted to rely solely on content.For data preprocessing, we select the frame with the largest text region for spatial editing feature learning and segment video frames using TransNetv2 [43] for temporal behavior modeling. All the MLPs in our experiments consist of three layers with a ReLU activation and a dropout rate of 0.1. The coattention mechanism features four heads, and convolutional layers in the downsampling network are configured with a stride of 2 and padding of 1. Training parameters include setting hyperparameters 𝛼 and 𝛽 at 0.1 and 2.0, respectively, learning rates of 5e-5 for FakeSV and 1e-3 for FakeTT, and a batch size of 128. The model undergoes training for 30 epochs, incorporating early stopping to mitigate overfitting, and employs the Adam [22] for optimization. We report the average results of multiple runs.We report Accuracy and macro F1 as primary evaluation metrics, which are widely used in existing works [35, 37] . To account for imbalanced label distributions, we additionally report the F1-score, Precision, and Recall for each label (i.e., Fake or Real). First, the zero-shot (M)LLM-based methods, namely GPT-4(V), underperform the methods specifically tailored for fake news video detection, which indicates the complexity of the task and the necessity of specialized models for this task currently. Notably, GPT-4(V) exhibits biases in authenticity judgments, tending to classify videos as real in the FakeSV dataset and as fake in the FakeTT dataset, possibly due to different knowledge accumulation in the tested large models.Second, neural network-based baselines generally outperform handcraft feature-based baselines, demonstrating the superiority of automated neural network models in handling complex fake news detection tasks. However, in some instances, the handcraft featurebased baselines surpass certain neural network models, particularly TikTec. This suggests that integrating human-guided knowledge into the models may bring additional advantages in specific cases.Finally, FakingRecipe outperforms all competing methods in Accuracy and macro F1 on both datasets, validating its effectiveness in detecting fake news videos. Notably, the improvements are more pronounced on the FakeSV dataset (5.53% increase in accuracy and 5.33% in macro F1) compared to that on FakeTT (2.61% in accuracy and 2.79% in macro F1), possibly reflecting the moderate pattern differences of the creative process in different cultural background.To rigorously evaluate the individual contributions of each component within FakingRecipe, we conduct extensive ablation studies, the results of which are detailed in Table 3 . We first focus on the performances of the two core modules: Material Selection-Aware Modeling (MSAM) and Material Editing-Aware Modeling (MEAM). It is observed that MSAM generally outperforms MEAM, with a more notable performance gap observed on the FakeTT dataset compared to FakeSV. While MEAM showed relatively lower performance on its own, it provides crucial complementary insights that significantly enhance the overall effectiveness of the combined model beyond what is achieved by MSAM alone.Further exploration into each specific aspect within these modules: sentimental and semantic for MSAM, and spatial and temporal for MEAM. By systematically removing each aspect and comparing the altered model's performance to the original, the results confirm that each component plays a vital role in the model's overall effectiveness. Among these aspects, while the spatial component shows the smallest improvement in performance, the sentimental aspect is most impactful for FakeSV, and the semantic aspect is particularly effective for FakeTT. This detailed analysis not only demonstrates the essential contribution of each aspect but also underscores the synergy that their integration brings to the effectiveness of Fak-ingRecipe in detecting fake news videos.The performance improvements in FakingRecipe are attributed to the enhancements by the MSAM and MEAM modules. Specifically, MSAM facilitates multimodal content understanding and MEAM introduces a novel perspective on mining content utilization. In this section, we conduct a deeper investigation into these two modules and present two additional findings:The synergy of creative process-aware encoding and fusion strategy deepens understanding of video materials, leading to better detection performance. We implement a simplified version of MSAM, termed MSAM (vanilla), which directly concatenates features from multiple encoders for classification. As depicted in Figure 8 , MSAM (vanilla) performs better than SVFEND which employs universal encoders for multimodal content understanding, confirming the efficacy of our material selection-aware multimodal content encoding strategy. However, its performance still falls behind the full MSAM configuration, emphasizing the crucial role of the advanced fusion structure in performance improvement. This exploration underscores the individual effectiveness of both the encoding and fusion strategy and their synergy within MSAM. 5Creative process-aware modeling introduces new effective clues that can even bring improvements to other existing models. We assess the generalizability of MEAM, which introduces a novel perspective in modeling multimodal content utilization. We directly integrate MEAM into TikTec and SVFEND using the same late fusion strategy as FakingRecipe. The results on two datasets are shown in Table 4 . We see that incorporating MEAM resulted in performance gains on both baselines, with TikTec showing significant improvements, affirming MEAM's capacity to elevate performance under effective fusion.We further demonstrate the complementary capabilities of MSAM and MEAM in detecting fake news videos through two real examples from the FakeSV dataset in Figure 9 . In the left example, a video with high-quality production and visually rich materials is evaluated. Influenced by the video's polished appearance, MEAM classifies it as real. However, MSAM assesses the situation from a different angle, detecting emotionally charged language in the video's title, such as "what a heinous act," which identifies as a potential indicator of misinformation. This nuanced analysis by MSAM accurately flags the video as fake, showcasing its ability to probe deeper than superficial qualities. Conversely, the right example presents a video with a neutral expression, which initially leads MSAM to classify it as authentic. Here, MEAM provides critical complementary information. It scrutinizes the video's sparse visual content and simplistic textual presentation, cues that suggest a lack of authenticity. This focused evaluation by MEAM correctly identifies the video as fake, highlighting its essential role in the overall analysis. These case studies underscore the complementary nature of MSAM and MEAM in FakingRecipe, enabling a layered and comprehensive assessment of news videos. We provide the failure case analysis in the appendix. 
related work: 
 Fake News Video Detection. The early work closely related to fake news video detection traces its roots to multimedia forensics research. Forensics-based works follow a basic idea about veracity that misinformation videos are often produced using forgery techniques [4, 12] . However, with the prevalence of user-friendly editing tools, manipulating visual content has become a common practice across social media platforms, significantly limiting the applicability of this detection approach. Thus, recent investigations have shifted their methodology towards mining detection clues from multimodal content. Handcraft features tailored for fake news video detection [16, 33, 34, 40] like linguistic patterns, acoustic emotion, and user engagement statistics are designed. Further studies incorporate visual expression and leverage deep neural networks [7, 19, 24, 26, 35, 41] for falsehood identification. Building on the foundation of multimodal content clues within individual samples, some researchers propose to incorporate the neighborhood relationship in an event for fake news video detection, exemplified by the NEED framework [37] . Though effective, its dependency on existing data accumulations limits its applicability in real-world scenarios. Instead, our method is suitable for detection at the early stage as it only requires the video content as the input.Narrative-aware Fake News Detection. News reporting has long been seen as involving a form of storytelling [2, 45, 47] . From this perspective, applying narrative theory, a discipline focusing on how stories are depicted persuasively [3] , to characterize fake news emerges as an intuitive idea. Narrative theory emphasizes analyzing the "what" (the content of the story) and the "how" (the strategy of storytelling) as its two pivotal aspects [11] , echoing the perspective of the creative process. The potential of applying narrative theory for detecting fake news has been demonstrated by studies on news articles [14, 20, 48] . However, within the realm of multimodal news, related research remains limited. Current studies in multimodal fake news detection [5, 21, 36, 50, 51, 53] typically concentrate solely on the analysis of presented content, neglecting the broader narrative structures. Tseng et al. [45] make the first foray into understanding narratives of disinformation in TV news videos. A multimodal discourse analysis scheme is proposed to uncover narrative strategies [2] . However, their focus is to assist manual statistical analysis using web-based tools [25] and thus is inapplicable to automatic detection. Our study takes the first step to detect fake news on short video platforms from the perspective of the creative process, which can be seen as a practical solution of the narrative theory for this task. 
conclusion: 
 We proposed to detect fake news on short video platforms from the perspective of the creative process and designed the creative process-aware detector, FakingRecipe. It observes the given video from material selection and editing perspectives to capture the unique production characteristics of fake news videos. We conducted experiments on the English FakeTT dataset newly constructed by us and the popular Chinese FakeSV dataset and validated the effectiveness of FakingRecipe.Limitations and Future Work. Though bringing a new perspective and experimentally shown effective, our model design mainly relies on empirical analysis, and thus may not fully correspond to the existing theoretical knowledge in the analysis of fake news creation. Since spreading and combating fake news is constantly adversarial, the model may require periodic updates in real applications. In the future, we plan to draw inspiration from journalism and communication literature to make the creative process modeling more intrinsic. Also, it is still worthwhile exploring how to equip (M)LLMs with our method, possibly via advanced techniques [28] . 
: 
 of fake news videos, posing new serious threats to the online information ecosystem [4, 44] . Consequently, customizing methods for detecting short video fake news is of urgent need.Unlike fake news in text or image formats, fake news on short video platforms shows unique characteristics and is increasingly indistinguishable from real news, posing new challenges to developing effective detectors [4] . First, the easy-to-use video editing tools largely democratize news creation and enable almost everyone to edit a news video on par with professional journalists [31] , making the edit traces widely exist in both real and fake news videos. Second, due to the public nature of short video platforms, even a real news video is likely to be repurposed or re-edited for news faking. However, existing methods for fake news video detection mostly follow ideas from the research line of text-image-based detection and focus on modeling what is presented via analyzing the authenticity of multimodal content (e.g., detecting deepfakes [12] ) and modeling cross-modal correlation [7, 35, 41] , which are more likely to be misled by edited and repurposed contents. Faced with the more vague boundary between the real and the fake, it is necessary to find new perspectives and capture more effective clues for fake news video detection.In this paper, we propose to switch the perspective from analyzing what is presented in a fake news video to considering how it might be created. Our idea is based on a straightforward assumption: Fake news creators on short video platforms often lack first-hand, genuine news materials and professional editing skills while aiming to produce fake news for specific purposes intentionally [42] . This might leave unique characteristics of the resulting video. Figure 1 provides an intuitive example of the creative process of a fake news video about hanging national flags during the COVID-19 pandemic. The process typically unfolds in two main phases: material selection and material editing. For selecting the material, the creator first confirmed the news topic (i.e., residents hanging national flags during the pandemic) and the positive sentiment keynote and crafted an attractive narrative that diverges from the truth. Due to the lack of real visual materials (unlike real news), the creator had to repurpose historical materials collected from the Internet to make the fake video more convincing. Finally, an emotionally charged song is selected to impress audiences and achieve its underlying purpose. For the editing phase, the creator might consider arranging materials from the temporal and spatial views with the help of simple editing techniques. Due to the constraint of material sufficiency and editing skills, the collected visual materials were arranged with simple splicing temporally, and the text material was then spatially overlaid on the visual content for a straightforward textual-visual expression. Through this example, we intuitively find that the production of fake news videos may leave the nuances different from that of real ones in terms of material selection and editing. Therefore, modeling from the creative process perspective may help us capture more valuable instrumental clues for fake news video detection.Inspired by the observation, in this paper, we first quantitatively examine how effective the clues from the creative process perspective are in distinguishing fake and real news videos via an empirical analysis (Section 2). The results validate that statistical discrepancies exist between real and fake news videos in material selection and editing. For instance, we find that compared with real ones, fake news videos exhibit a propensity for selecting more emotionally charged music, using a limited palette of colors, and adopting a less dynamic on-screen text presentation. Based on the empirical analysis, we design FakingRecipe, a creative process-aware model for detecting fake news short videos. 1 FakingRecipe is a dual-branch network that models the characteristics of material selection and editing. In the two branches, the Material Selection-Aware Modeling (MSAM) module extracts multimodal features via attention to capture the sentiment resonance between audio and text and the semantic relevance between text and visual frames. The Material Editing-Aware Modeling (MEAM) module models typical spatial and temporal editing behaviors, via 1) analyzing the visual area and on-screen texts for the spatial editing; and 2) building hierarchical temporal structure that considers both intra-and inter-segment fusion for temporal editing. Ultimately, predictions from both branches are integrated through a late fusion function for the final prediction. Experiments on two real-world datasets demonstrate the superiority of the proposed FakingRecipe over seven baseline methods. Our main contributions are as follows:• Idea: We for the first time consider the creative process as a pivotal aspect for detecting fake news on short video platforms and demonstrate the feasibility of this perspective through empirical analysis.• Method: We propose FakingRecipe, a novel dual-branch model for fake news video detection that captures useful clues from 1 The creative process of faking a news video is metaphorically similar to cooking a dish following a recipe, so we use FakingRecipe to highlight the model's uniqueness.  
 
 Appendix: https://github.com/ICTMCG/FakeSVShang et al. [41] did collect an English TikTok dataset but did not release it. We did not receive any reply to our email for the dataset inquiry.https://www.snopes.com/The analysis of different fusion strategies is in the appendix. 


Decision: The reviewers are unanimous in their opinion that the paper should be accepted to the program of ACM Multimedia 2024. They are particularly positive about the idea of studying the properties of fake news production process. In addition, they appreciated the introduced FakeTT dataset, hoping that it will be made public. However, some reviewers were questioning novelty of the proposed approach, which suggests that the authors should pay special attention to that aspect when revising the paper.

Taking into account the initial reviews and rebuttal process, my opinion is that the paper should be accepted.

Reviews: review: paper_summary: paper_summary: The paper introduces an innovative approach for detecting fake news on short video platforms by focusing on the creative process involved in producing these videos. The authors propose the "FakingRecipe," a detection model that evaluates videos from the perspectives of material selection and editing, aiming to identify the unique production characteristics associated with fake news.
strength_weakness: strength_weakness: 1. The paper identifies discrepancies between real and fake news production processes and provides plausible explanations for these phenomena, highlighting the nuances in the creative process to evaluate the veracity of news videos.
2. The paper constructs FakeTT, an English short video dataset for fake news detection.
3. The paper proposes FakingRecipe, a novel dual-branch model for fake news video detection that captures useful clues from the perspective of the creative process, i.e., material selection and editing phases.
4. The proposed network achieves state-of-the-art performance on two public real-world datasets.
limitations: limitations: 1. Section 3.3 on the spatial aspects lacks some formal mathematical expressions.
2. The implementation details of the Two-Way Attention Block are not provided, specifically the internal details of Image to Prompt Attention and Prompt to Image Attention.
3. Parameter analysis is missing.
4. Implementation details for Table 4 are not provided.
paper_summary: paper_summary: The paper studies fake news detection on short video platforms from an interesting perspective that considers how it might be created and studies the unique characteristics of fake news videos in material selection and editing empirically. Based on the analysis, the authors propose a FakingRecipe that models material selection from sentimental and semantic aspects and considers the traits of material editing from spatial and temporal aspects. Extensive experiments show the proposed method performs better in detecting fake news on short video platforms.
strength_weakness: strength_weakness: 1. Studying how fake news might be created is an interesting perspective. The empirical analysis provides novel insights into the fake news data.
2. The paper is well-written and organized. The proposed method is clearly formatted and easy to follow.
3. The authors deliver an English dataset for fake news detection in the short video platform, contributing to the community.
limitations: limitations: 1. The proposed method lacks novelty and hardly corresponds to modeling material selection and editing.
2. The loss functions of the proposed method are confusing. Both the outputs of the two branches and the final outputs are aligned to the ground-truth label Y.
3. The case study is not insightful. The visualized cases fail to reflect the working mechanism and the effects of the two branches of the proposed model.
paper_summary: paper_summary: The authors propose a novel model that examines the creative process during the production stage of news videos. They introduce FakingRecipe, a model that detects fake news by analyzing the choices in material selection and editing. Besides, the authors construct a new English fake news video detection dataset.
strength_weakness: strength_weakness: 1. The authors construct an English fake news video dataset, which is meaningful for the community.
2. The proposed FakingRecipe provides a new respective from the video creator and achieves excellent performance in two datasets.
3. The experiment results are convincing and solid.
limitations: limitations: 1. The proposed dataset FakeTT is not open source. The demo or download link should be available.
2. A more detailed collection process should be explained. Besides more statics should be displayed about the dataset.
3. The absence of parameter sensitivity analysis in the experiment.
paper_summary: paper_summary: The authors propose a novel and interesting perspective and design a creative process-aware model for multimodal fake news detection on short video platforms. Meanwhile, they construct an English dataset Fake TT to evaluate the effectiveness of their methods.
strength_weakness: strength_weakness: 1. The objective of this work is clearly stated and paper is well structured.
2. The idea of considering how fake video news might be created is interesting. This novel perspective has the potential to significantly contribute to the advancement of research in this field.
3. Extensive experiments demonstrate the effectiveness of their method.
4. It is commendable to publicly release the new dataset to facilitate further research.
limitations: limitations: 1. Due to the limited informational value density of short video content, it seems more reasonable to conduct detection from the perspective of 'how it might be created' than 'what is presented'. However, in real-world settings, the creative patterns of fake news can exhibit considerable variability. Can a creative process-aware model effectively address the complexities of such environments? While this poses a challenging problem, I am curious to know if the author has taken this aspect into consideration.
2. It appears that there are some typos, such as 'ei' in Eq.3.