Coarse-to-Fine Proposal Refinement Framework For Audio Temporal Forgery Detection and Localization

Authors: junyan  wu
wei  lu
xiangyang  luo
rui  yang
qian  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 14:31:54

Abstract: Recently, a novel form of audio partial forgery has posed challenges to its forensics, requiring advanced countermeasures to detect subtle forgery manipulations within long-duration audio. However, existing countermeasures still serve a classification purpose and fail to perform meaningful analysis of the start and end timestamps of partial forgery segments. To address this challenge, we introduce a novel coarse-to-fine proposal refinement framework (CFPRF) that incorporates a frame-level detection network (FDN) and a proposal refinement network (PRN) for audio temporal forgery detection and localization. Specifically, the FDN aims to mine informative inconsistency cues between real and fake frames to obtain discriminative features that are beneficial for roughly indicating forgery regions. The PRN is responsible for predicting confidence scores and regression offsets to refine the coarse-grained proposals derived from the FDN. To learn robust discriminative features, we devise a difference-aware feature learning (DAFL) module guided by contrastive representation learning to enlarge the sensitive differences between different frames induced by minor manipulations. We further design a boundary-aware feature enhancement (BAFE) module to capture the contextual information of multiple transition boundaries and guide the interaction between boundary information and temporal features via a cross-attention mechanism. Extensive experiments show that our CFPRF achieves state-of-the-art performance on various datasets, including LAV-DF, ASVS2019PS, and HAD.

Main paper: 
 introduction: 
 Driven by massive amounts of data and powerful deep learning networks [9-12, 17, 24] , lifelike productions of audio artificial intelligence generated content (AIGC) [2, 14-16, 22, 23] have become more accessible to the public. While AIGC-related industries have greatly improved the quality of people's lives, they also bring convenience to malicious users. Currently, a novel form of audio partial forgery using AIGC and a large language model has raised public concerns about the security and authenticity of network information. Figure 1 illustrates the process of fabricating disinformation that AIGC productions can be integrated seamlessly into the original audio by minor modifications. Unfortunately, current countermeasures mainly address the entire synthesized utterances [1, 13, 26, 28, 40] , leaving the door open for malicious users to conceal forgery traces at a low cost. To promote the safe and proper use of AIGC, it is significant to study a more accurate and reliable countermeasure against partial forgery audio.Recent research efforts have been devoted to creating audio partial forgery datasets [4, 7, 34, 35, 37] and proposed a challenging task known as audio partial forgery detection (PFD). As shown in the 'countermeasure' part of Figure 1 , the PFD task aims to detect partial forgery segments, which may be only a small portion of the real audio, at different detection resolutions. Two main solutions have been proposed for the audio PFD, including forgery content detection [18, 31, 33, 37, 41] and forgery boundary detection [6, 8] . The former primarily focus on determining the content authenticity, with the goal of distinguishing which segments within audio are real or forgeries. The latter focus on detecting the transition boundary between real and forgery segments. Among them, self-supervised learning (SSL) models play a significant role in the detection performance improvement. The basic idea is to leverage powerful audio representations from the pre-trained SSL model to facilitate downstream forgery detection tasks. Despite the remarkable advancement, there still exist some issues.(1) Classification limitation: Current audio PFD solutions still aim at classification, i.e., ranging from predicting utterance-level results to frame-level results. However, providing temporal forgery regions within the modified audio can help users better understand the analysis results of the audio forgery content, which aligns more closely with the practical needs and applications of audio forensics. To fill the gap in audio forensics, we introduce a temporal forgery localization (TFL) task, which not only identifies each audio segment as real or forgery but also determines the precise timestamps at which these forgery segments start and end.(2) Small forgery segment challenge: If malicious users have phonological knowledge, they can alter the original semantics by manipulating vowels and even consonants smaller than the word level. TFL network aims to predict specific forgery regions, and it may be challenging to locate small forgery segments consisting of a single frame or several consecutive frames in long-duration audio. Thus, it is desirable to improve the localization performance of small forgery segments by embedding prior information of frame-level detection scores from the PFD network into the TFL network.(3) Multiple forgery segment challenge: Malicious users may not simply manipulate a single segment within audio but deliberately employ forgery in multiple segments. This challenge causes localization performance to gradually weaken with the increasing number of forgery segments. The transition boundary can provide valuable cues to enhance the detection of multiple forgery segments, as it indicates artifacts like inconsistency in speech and inconsistency in ambient noise [30] . Therefore, it is vital to introduce the contextual information at forgery transition boundaries.In this work, we are motivated to propose a novel two-stage framework called the coarse-to-fine proposal refinement framework (CFPRF) 1 for audio temporal forgery detection and localization (TFDL). Unlike existing audio PFD methods, we leverage a frame-level detection network (FDN) in the first stage to learn 1 Code and pre-trained models are available at: https://github.com/ItzJuny/CFPRF. robust representations for better indicating rough forgery regions and employ a proposal refinement network (PRN) in the second stage to produce fine-grained proposals. To address the audio TFDL challenges, our core idea is to mine temporal inconsistency cues by forcing the model to perceive the subtle differences between different frames and to capture the contextual information of multiple transition boundaries. To enlarge the sensitive differences caused by small forgery segments, we employ contrastive representation learning (CRL) [33] in the difference-aware feature learning (DAFL) module to guide distinct frame pairs to exhibit dissimilarity in the embedding space while ensuring identical frame pairs demonstrate similarity. In addition, we design a boundary-aware feature enhancement (BAFE) to guide the interaction between boundary context information and temporal difference-aware features to enhance the detection of multiple forgery segments. Benefiting from these modules, the robust frame-level features can be learned to provide pivotal coarse-grained proposals, which are conducive to locating small and multiple forgery segments and further being refined in the PRN. The results demonstrate the effectiveness of our CFPRF and the practicality of the designed PRN. Overall, the main contributions of this paper can be concluded as follows:• We innovatively propose a two-stage framework called CF-PRF, including an FDN for frame-level forgery detection and a PRN for temporal forgery localization. • We introduce a DAFL module involving dual-attention layers and guided by CRL to enlarge the sensitive differences between different frames caused by minor manipulations. • We devise a BAFE module incorporating the cross-attention mechanism to enhance the temporal features with context information of multiple transition boundaries. • We present a plug-and-play PRN to predict confidence scores and regression offsets to refine the coarse-grained proposals derived from the FDN. Moreover, PRN can be integrated with existing audio PFD models for localization. 
related work 2.1 audio forgery detection: 
 The audio forgery detection task aims to determine whether an audio contains forgery content. Early audio forgery detection methods [1, 13, 26, 28, 40] focus on introducing a well-designed model to capture forgery traces from a global view to detect entirely synthesized utterances. For example, AASIST [13] introduced a heterogeneous graph fusion network to model the audio temporal-spectral subgraphs, which are then downsampled to utterance-level scores. In addition, ASDG [32] utilized aggregation and separation domain generalization to learn an optimal feature space that can aggregate real audio and fake audio, resulting in better generalizability in detecting unseen target domains. Although cheerful performance has been achieved in various utterance-level detection scenarios, an initial investigation into audio partial forgery detection [38] revealed limitations. PFD task requires finer detection resolutions to identify subtle manipulations that affect only a small portion of the original audio. In this regard, researchers have explored PFD models under various time resolutions [6, 8, 18, 31, 33, 37, 41] , ranging from longer segment-level resolution (e.g., 320 milliseconds) down to frame-level one (20 milliseconds). For instance, QASAM [31] leveraged the question-answering strategy and self-attention mechanism into a designed fake span detection module, directing the model to focus away from less generalization shortcuts into fake spans. LS-HMI [41] used a hybrid multi-instance learning to mitigate the gradient conflict problem and applied local self-attention to compute local correlations between segments from a temporal perspective, thus enhancing feature representation. In addition, PSDL [37] and MGBF [18] implemented an SSL front-end model and combined multiple detection resolutions to improve the flexibility and accuracy of detection. TDL [33] was drew on the SSL front-end and designed an embedding similarity module to separate real and fake frames in the feature space. Moreover, WBD [8] and IFBDN [6] also introduced an SSL model and applied transform blocks to detect the probability of being a boundary for each frame. However, existing audio PFD methods have classification limitations that can not specify the start and end timestamps of partial forgery segments, thus limiting the practicality of audio forensics in realistic scenarios. To overcome this limitation, the temporal forgery localization task was introduced.Recently, temporal forgery localization has received considerable attention in video forensics. It aims to acquire informative visualaudio representations by exploring the interaction between multimodal features and to integrate an established localization decoder [19, 25, 36] to realize the localization of partial forgery segments. LAV-DF [7] was the first to introduce a content-driven multi-modal TFL dataset and proposed a benchmark network BA-TFD guided via contrastive learning, frame classification, and boundary matching network (BMN) [19] for the TFL task. In their subsequent work, BA-TFD+ [5] enhanced the BA-TFD backbone with a multi-scale transformer and adopted a BSN++ [25] -based boundary module to improve the localization performance significantly. In addition, AV-TFD [21] also drew on the decoder structure of BMN and used a cross-modal attention mechanism and an embedding-level fusion mechanism for learning visual-audio representations. To improve the localization accuracy of the model, UMMAFormer [39] devised a temporal feature abnormal attention module and a parallel crossattention feature pyramid network for enhancing subtle features and incorporated an ActionFormer decoder for localization [36] .However, we observe that existing vision-audio TFL methods cannot be directly applied to the audio TFL task. The forgery video usually manipulates successive video frames to effect semantic changes, while the forgery audio alters meaning with just a vowel change (single frame). In this context, these end-to-end TFL methods may not address small-and multi-forgery segment challenges in audio scenarios. Moreover, current audio forgery boundary detection models are unable to determine the start and end timestamps of partial forgery segments. In light of these issues, we propose a solution called CFPRF for audio TFDL.Given a partial forgery audio, we aim to detect partial forgeries at the frame level and propose the locations of these forgery segments.Frame-level Forgery Detection. Take the input audio 𝑥 as an example, which contains 𝑇 frames. First, frame-level features are learned by a designed encoder, denoted as 𝐹 𝑥 ∈ R 𝑇 ×𝐷 . Then, 𝐹 𝑥 is decoded into forgery probability scores Ŷ𝑥 = { ŷ1 , ŷ2 , ..., ŷ𝑇 }. This task can be considered as a frame-level classification challenge, utilizing labels 𝑌 𝑥 = {𝑦 1 , 𝑦 2 , ..., 𝑦 𝑇 } where 0 denotes a fake frame and 1 denotes a real frame, respectively.𝐺 𝑥 = {(𝑠 𝑚 , 𝑑 𝑚 )} 𝑀 𝑚=1, where 𝑀 is the number of forgery segments within the audio. For the 𝑚-th forgery segment, 𝑠 𝑚 and 𝑑 𝑚 are the start timestamp and duration time, respectively. This task needs to predict a set of confidence scores 𝐶 𝑥 = {𝑐 1 , 𝑐 2 , ..., 𝑐 𝐻 } for each coarse-grained proposal𝑃 † 𝑥 = {(𝑠 † ℎ , 𝑑 † ℎ )} 𝐻 ℎ=1 derived from FDN outputs,where 𝐻 is the number of coarse-grained proposals. Then, positive proposals are refined using predicted regression offsets R𝑥 = {( r𝑠 ℎ , r𝑑 ℎ )}𝐻 𝑝𝑜𝑠 ℎ=1, where 𝐻 𝑝𝑜𝑠 is the number of positive proposals, each with a confidence score 𝑐 ℎ ≥ 𝜃 𝑝 . For the ℎ-th proposal, labels for the duration offset 𝑟 𝑑 ℎ (𝑟 𝑑 ℎ = 𝑙𝑜𝑔(𝑑 ℎ /𝑑 † ℎ )) and the shift offset 𝑟 𝑠 ℎ (𝑟 𝑠 ℎ = (𝑠 ℎ -𝑠 † ℎ )/𝑑 ℎ ) are calculated based on matched ground truth. 
proposed method 3.1 frame-level detection network: 
 To learn robust discriminative features, we first design a frame-level detection network to mine inconsistency cues to better distinguish between fake and real audio frames. Figure 2 illustrates the detail of our FDN, which contains difference-aware feature learning and boundary-aware feature enhancement modules that are adept at capturing minute forgery traces. As the SSL model (namely, Wav2Vec2-XLSR300M) has been shown to produce powerful audio representations for downstream tasks, we first extract front-end features from the last hidden states using an embedding size of 1024. To reduce feature dimension, we apply a linear layer to get the output feature map 𝐹 𝑠𝑠𝑙 ∈ R 𝑇 ×128 , where 𝑇 is the number of audio frames. These features are then fine-tuned to be tailored for the PFD task. Considering features 𝐹 𝑠𝑠𝑙 are not enough for modeling spectral and channel-wise information for each frame, we utilize six CNN-based residual blocks to learn a higher-level feature map 𝐹 𝑠𝑐 ∈ R 𝐶×𝑇 ×𝑆 . Here, 𝐶 and 𝑆 are the number of channels and spectral bins for each frame, respectively. EQUATIONwhere 𝐸 𝑎𝑙 (•) denotes the encoder of the attention layer, followed by a softmax function to normalize these weights. The integration of spectral-channel information has better potential to leverage complementary information to model temporal forgery representations, which can be denoted asEQUATIONwhere 𝐹 𝑡 ∈ R 𝑇 ×𝐷 (𝐷 = 𝐶 + 𝑆) is the temporal representation, ⊙ denotes the element-wise multiplication, and ⊕ indicates the element-wise addition.To enlarge the differences between real and fake frames in the feature space, we further enable a pair of real and fake frames from different positions to exhibit dissimilarity. In contrast, a pair of frames that are both either real or fake should show similarity. Using cosine similarity to measure the similarity between feature vectors is a popular approach. Specifically, the cosine similarity between two frames 𝑓 𝑎 and 𝑓 𝑏 can be denoted asEQUATIONThen, the contrastive learning loss 𝐿 𝑟 is utilized to guide differenceaware features 𝐹 𝑑𝑎 to cluster similar frames together while separating dissimilar frames, denoted as𝐿 𝑟 = 1 𝐽 𝐽 𝑗=1 𝐼 𝑗 (1 -𝑆𝐼𝑀 (𝑓 𝑗 , 𝑓 + 𝑗 )) 2 + (1 -𝐼 𝑗 )𝑚𝑎𝑥 (0, 𝑆𝐼𝑀 (𝑓 𝑗 , 𝑓 - 𝑗 ) -𝛼) 2(4) where 𝐽 is the number of frame pairs. For the 𝑗-th pair, 𝑓 𝑗 is the feature vector of the reference frame, while 𝑓 + 𝑗 and 𝑓 - 𝑗 are the feature vectors of the similar and dissimilar frames, respectively. 𝐼 𝑗 set to 1 is used to indicate similar pairs and 0 otherwise, while 𝛼 is a margin parameter for the dissimilar pairs. Boundary-aware Feature Enhancement Module. The transition boundaries of partial forgery segments are often presented unnaturally, such as discontinuities in speech and inconsistencies in ambient noise [30] , leaving the door open for detection. Inspired by this, we design a boundary-aware feature enhancement module, as shown in Figure 2 (c), with the goal of enhancing features by modeling context information of multiple transition boundaries.To model long-term information and capture boundary inconsistencies, a boundary encoder based on MLPs with gating and a tiny self-attention (aMLP) [20] is utilized to get the boundary feature map 𝐹 𝑏 . Further, a full connected layer is applied to downsample 𝐹 𝑏 to boundary probability scores Ŷ𝑏 = { ŷ1 𝑏 , ŷ2 𝑏 , ..., ŷ𝑇 𝑏 }. Considering the imbalance problem in boundary labels, we employ a hyperparameter-free loss function, namely P2sGrad-based mean squared error (MSE) [29] , which can be defined as follows:EQUATIONwhere 𝑀𝑆𝐸 𝑝𝑔 (•) is the P2sGrad-based MSE loss, 𝑇 is the number of audio frames. Each frame is assigned a boundary label, where 𝑦 𝑡 𝑏 is 1 denoting the 𝑡-th frame as a transition point, and 0 otherwise. For enhancing the temporal difference-aware features, we employ a cross-attention transformer block to facilitate the interaction between boundary information and temporal features. Given the permutation-invariant problem, the positional encoding 𝑃𝐸 (•) is added to the inputs to provide the positional information. Within this block, the boundary-aware correlation map 𝑀 𝑏𝑎 between temporal features 𝐹 𝑑𝑎 and boundary features 𝐹 𝑏 are first derived and normalized as follows:EQUATIONwhere learning matrices {𝑊 𝑞 ,𝑊 𝑘 } ∈ R 𝐷×𝐷 ′ is utilized to transform features {𝐹 𝑑𝑎 , 𝐹 𝑏 }, and 1dentoes the scaling factor. To get the output features enhanced by the boundary-aware attention, a dotproduction is performed on the temporal features projected by𝑊 𝑣 ∈ R 𝐷×𝐷 ′ and the cross-attention map 𝑀 𝑏𝑎 , which can be denoted asEQUATIONAdditionally, the enhanced features 𝐹 𝑐𝑎 are added to the original features 𝐹 𝑑𝑎 and fed to a feed-forward network with layer normalization to generate boundary-aware representations 𝐹 𝑏𝑎 . Finally, an aMLP-based decoder is utilized to downsample 𝐹 𝑏𝑎 to frame-level forgery probability scores Ŷ𝑓 = { ŷ1 𝑓 , ŷ2 𝑓 , ..., ŷ𝑇 𝑓 }. Also, we employ the P2sGrad-based MSE function to compute the forgery detection loss 𝐿 𝑓 asEQUATIONwhere 𝑇 is the number of audio frames, and label 𝑦 𝑡 𝑓 is 0 denoting the 𝑡-th frame as a forgery frame, and 1 otherwise.To realize the localization of partial forgery segments, we devise a proposal refinement network in the second stage, whose structure is shown at the bottom of Figure 2(a) . We utilize the frame-level features and detection scores derived from the PFD network to better locate small and multiple forgery segments in long-duration audio. The frame-level detection scores are used to provide coarse regions, and region features are utilized to predict confidence scores and regression offsets to produce fine-grained proposals. Following a two-step strategy, we fix the network parameters of the FDN to train the PRN.Coarse-grained Proposal. To be specific, PRN takes framelevel forgery probability scores Ŷ𝑓 = { ŷ𝑡 𝑓 } 𝑇 𝑡 =1 and the last hidden states 𝐹 𝑏𝑎 = {𝑓 𝑡 𝑏𝑎 } 𝑇 𝑡 =1 derived from the FDN as input, where ŷ𝑡 𝑓 and 𝑓 ℎ 𝑏𝑎 denote the score and the feature vector of the ℎ-th frame, respectively. Given predicted forgery scores, a temporal region (𝑠, 𝑑) that likely to contain forgeries is formed by consecutive forgery frames {( ŷ𝑡 𝑓 )| ŷ𝑡 𝑓 > 𝜃 𝑓 } 𝑠+𝑑 𝑡 =𝑠 , each forgery frame with a score higher than a threshold 𝜃 𝑓 . In this context, 𝑠 and 𝑑 are the start timestamp and the duration length of the region, respectively. Furthermore, coarse-grained proposals 𝑃 † = {(𝑠 ℎ , 𝑑 ℎ )} 𝐻 ℎ=1 can be defined as a set of pairs containing the start timestamp and the duration length, where 𝐻 is the number of coarse-grained proposals. For the ℎth proposal, the corresponding region features can be extracted as𝑅(𝑃 † ℎ ) = {𝑓 ℎ 𝑏𝑎 } 𝑠 ℎ +𝑑 ℎ ℎ=𝑠 ℎ. To enhance the incorporation of globalcontext information for subsequent validation and regression, we design a pooling encoder consisting of two CNN-based residual blocks and a max-pooling layer, where each residual block contains two convolutional layers followed by BN and SELU functions. Consequently, the pooling encoder processes coarse-grained region features and outputs TRoI features 𝐹 𝑟 ∈ R 𝐻 ×𝐷 ′ with a fixed size. Fine-grained Proposal. PRN is responsible for producing the fine-grained proposals and the confidence scores. To this end, two MLP-based headers are employed that take TRoI features 𝐹 𝑟 as input to decide whether a proposal is positive and refine these positive proposals using predicted regression offsets. We define three types of proposals based on the temporal intersection-over-union (TIoU) with their matched groundtruths: mostly-complete proposals with 𝑇 𝐼𝑜𝑈 > 0.7, incomplete proposals with 0.4 <= 𝑇 𝐼𝑜𝑈 < 0.7, and otherwise negative proposals. Then, a binary cross-entropy loss is adopted to derive the verification loss 𝐿 𝑣 , which can be denoted as follows:EQUATIONwhere 𝑦 ℎ 𝑣 is the true label for the ℎ-th proposal, with 𝑦 ℎ 𝑣 = 1 indicating a positive proposal and 0 otherwise.In addition, a regression header is crucial for precisely approaching positive proposals to their groundtruths. Consequently, 𝐹 𝑟 within 𝐻 𝑝𝑜𝑠 positive proposals are fed into the regression header to obtain regression offsets R = {( r𝑠 ℎ , r𝑑 ℎ )} 𝐻 𝑝𝑜𝑠 ℎ=1 , including predicted start offsets and duration offsets. The regression loss is calculated only for positive proposals and can be expressed asEQUATION)where 𝐻 𝑝𝑜𝑠 is the number of positive samples, 𝑠 ℎ and 𝑑 ℎ are the true labels of the start offset and the duration offset for the ℎ-th proposal, respectively. In this context, Soomth-L1 distance is adopted, where 𝑦 ℎ 𝑣 = 1 indicates the positive proposal and 0 otherwise. Finally, fine-grained proposals 𝑃 = {(𝑠 ℎ + ŝ ℎ , 𝑑 𝑖 + dℎ )} 𝐻 𝑝𝑜𝑠 ℎ=1 are generated. Ideally, our proposed PRN can be integrated with other PFD networks to achieve temporal forgery localization using similar processes upon frame-level features and forgery scores.Training. We utilize a two-step strategy to train the CFPRF: the FDN is first trained on the PFD task, and then its network parameters are fixed to conduct the subsequent the PRN training. The training of the FDN is guided by the CRL loss 𝐿 𝑐 , the forgery detection loss 𝐿 𝑓 , and the boundary detection loss 𝐿 𝑏 . And the total loss 𝐿 𝐹 𝐷𝑁 for training FDN is calculated asEQUATIONwhere 𝜆 𝑐 = 0.15 and 𝜆 𝑏 = 0.1 are hyper-parameters to balance the total loss. In addition, the total loss 𝐿 𝑃𝑅𝑁 for training the PRN is calculated asEQUATIONwhere 𝐿 𝑣 and 𝐿 𝑟 are the verification loss and the regression loss, respectively. And the hyper-parameter 𝜆 𝑟 is set to 0.15. Given that coarse-grained proposals derived from forgery probability scores are mostly composed of positive proposals, causing an imbalance problem in training the verification header. To ensure stable training, we randomly sample negative proposals to balance the number of negative and positive proposals.Inference. For inference, we utilize the FDN to generate forgery probability scores Ŷ𝑓 , and then apply the PRN to derive coarsegrained proposals 𝑃 † = {(𝑠 † ℎ , 𝑑 † ℎ )} 𝐻 ℎ=1 and corresponding TRoI features 𝐹 𝑟 . Next, confidence scores are predicted for each proposal and regression offsets r = {( ŝℎ , dℎ )} 𝐻 𝑝𝑜𝑠 ℎ=1 are generated to refine positive proposals, denoted as𝑃 = {(𝑠 † ℎ + ŝℎ , 𝑑 † ℎ + dℎ )} 𝐻 𝑝𝑜𝑠ℎ=1 . Finally, Soft-NMS [3] is used to post-process these proposals. 
experiments 4.1 experimental setting: 
 Datasets. To validate the proposed method, we conduct extensive experiments on three different datasets: LAV-DF [7] , ASVS2019PS [37] (referred to as PS), and HAD [34] . Among these, the ASVS2019PS dataset is more challenging as it contains multiple and small partial forgery segments within audio.Compared methods. We compare our CFPRF with existing state-of-the-art PFD methods, including PSDL [37] and IFBDN [6] . Our proposed plug-and-play PRN is integrated with these methods to realize localization results. We also adopt multi-modal TFL methods, including BA-TFD [7] , BA-TFD+ [5] and UMMAF [39] , and adapt them to the audio TFL task by using audio-only features.Experiment Metrics. To evaluate the PFD task, we report equal error rate (EER), area under the curve (AUC), false negative rate (FNR), false positive rate (FPR), and 𝐹 1 -Score. In addition, we adopt average precision (AP) at various TIoU thresholds {0.5, 0.75, 0.9, 0.95}, average recall (AR) with different average number of proposals Implementation Details. We implement the proposed CFPRF using the PyTorch toolkit and conduct all experiments on the same platform with a single NVIDIA GeForce RTX 3090 GPU and the same random seed. We do not choose the experimental hyperparameters for training CFPRF detailed. The Adam optimizer is adopted to optimize the proposed CFPRF. The epoch, learning rate and weight decay are set to 30 (or 50), 10 -7 (or 10 -3 ) and 10 -4 (or 10 -3 ) respectively for training FDN (or PRN). The batch size is set to 2.4.2.1 Partial Forgery Detection. We show the PFD results in Table 1 . Compared to other state-of-the-art PFD methods, our CFPRF achieves the best results on all three datasets. We observe that the length and number of forgery segments within an audio can affect the detection performance of different models. For the challenging ASVS2019PS dataset, we achieve 7.41% EER (↓) and 93.89% 𝐹 1 -Score, which show a relative improvement of 23.45% and 2.07% over the second-best method. Although our DAFL and BAFE modules are tailored for fine-grained forgery scenarios, they also provide incremental improvement for other datasets. For example, we achieve 0.08% EER on the HAD dataset and slightly outperform IFBDN (0.35% EER) and PSDL (0.18% EER). These results demonstrate the effectiveness of our proposed CFPRF for the audio PFD task.We compare CFPRF with state-of-the-art TFL methods and PFD methods incorporated with the proposed PRN. The results are shown in Table 2 , which indicates that our method outperforms the comparison methods on all three datasets in terms of mAP. As the difficulty of datasets increases, a variable decline in localization performance is observed. Specifically, our CFPRF achieves mAP scores of 99.23%, 93.01%, and 55.22% and AR@20 scores of 99.38%, 93.51%, and 66.53% on the HAD, LAV-DF, and ASVS2019PS datasets, respectively. In addition, the results achieved by TFL methods indicate the importance of feature encoders and localization headers used for the audio-only evaluation. For example, the BA-TFD and BA-TFD+ can not achieve satisfactory results due to a lack of specialized design for audio features and an unsuitable localization header BMN. While the UM-MAF performs well on the datasets (HAD, LAV-DF), where audio Table 2 : Temporal forgery localization result. Performance comparison with state-of-the-art TFL methods and PFD models integrated with PRN on evaluated datasets using different evaluation metrics, where PRN † indicates coarse-grained proposals.Dataset Method AP@0.5 AP@0.75 AP@0.9 AP@0.95 mAP AR@1 AR@2 AR@5 AR@10 AR@20 contains a single or few forgery segments with a long span. Its performance degraded when facing small-and multi-forgery segment challenges, as evidenced by the ASVS2019PS dataset. Moreover, we observe that the localization performance improves when using features derived from PFD methods that are specifically designed for the audio modality. This further demonstrates the effectiveness of our proposed PRN, as evidenced by the 39.7%↑ (21.5%↑) relative improvement in mAP scores for the IFPRN (or PSDL) method on the ASVS2019PS dataset when incorporating PRN. In this context, we achieve relative improvements of 2.17%, 1.43%, and 13.18% respectively, on the HAD, LAV-DF and PS datasets over the second-best PFD method in terms of the mAP metrics. These results underscore the robustness of the discriminative features learned from our designed FDN and the effectiveness of PRN for two-stage localization.4.3.1 Impact of FDN architecture. We first investigate the indispensability of the various components of the proposed FDN. Specifically, we remove residual blocks (RB), dual attention layers (DAL), the DAFL module, and the BAFE module to assess their extra effectiveness for the entire network, respectively. The results are shown in Table 3 , where the baseline refers to the entire FDN. We observe that DAFL has the greatest impact on performance as it enables the model to discern subtle forgery differences between real and fake frames, reducing the EER values by 0.15%, 1.82%, and 0.20% for the entire network. In addition, benefiting from the extraction of global boundary information, the BAFE module reduces the EER values by straightforward to observe that replacing PRN with BMN greatly weakens the localization performance, indicating the two-stage is more suitable for the audio TFL task. In addition, the TIoU features extracted from the pooling encoder that deploys RB-PE before the pooling layer can provide global context information to facilitate the identification of the verification header, thus giving better performance. Moreover, the dual prediction headers play a crucial role in the refinement of coarse-grained proposals, where VegH evaluates the completeness of these proposals, giving a 16.45% absolute improvement in the mAP score on the ASVS2019PS dataset. As shown in Figure 3 , true negative proposals can be removed by the predicted confidence scores from VegH, and the RegH refines positive proposals by regression to approach groundtruths, giving a 3.46% absolute improvement in the mAP score on the ASVS2019PS dataset. In short, these components are essential to the PRN.To explore the contribution of each loss function in guiding frame-level feature learning, we train three models with different combinations and use t-SNE [27] to visualize the last hidden states on the different dataset samples. As shown in Figure 3 , the results reveal the distinct feature distributions under different loss integrations. Specifically, the p2sgard-based MSE loss can alleviate the bias of the detection model towards classifying fake frames as real. The introduction of the CRL loss causes the feature distributions of the real and fake frames to diverge, with only a few dots intermixed at the boundary. Finally, with the inclusion of the boundary loss, we find a markedly clearer separation between the real and fake frames, as well as the fake frames within different fake regions are also clearly distinguished. This suggests that boundary information can provide global cues to enhance the detection of multiple forgery segments within an audio.Impact on model efficiency. Finally, to compute the model efficiency, we start the experiment with 50 warm-up rounds to stabilize GPU performance, followed by 300 iterations for processing a 10-second audio. We repeat this five times and report the average results in terms of frames per second (FPS) and model parameters (#Params) in Table 5 . Specifically, the FDN is supported by 320M parameters and achieves an FPS of 12270 for a 10-second audio (500 frames), suggesting a moderate speed and a high complexity that can potentially capture more discriminative features. In addition, the PRN utilizes 129K parameters and shows a high FPS of 701445, demonstrating its efficiency for plug-and-play localization header. 
conclusion: 
 In this paper, we introduced a novel two-step framework called CFPRF, which incorporates an FDN and a PRN to address the emerging audio TFDL challenge. Following a two-step strategy, we first devised DAFL and BAFE modules in the FDN to mine temporal inconsistency cues caused by small and multiple forgery segments to learn robustness representations that distinguish between fake and real frames. Then, in the PRN, we derived coarse-grained proposals and region features from the FDN outputs and subsequently employed dual prediction headers to generate fine-grained proposals.Extensive experiments demonstrate that existing PFD models can achieve satisfactory localization performance with our designed PRN. Moreover, on the challenging ASVS2019PS dataset, our CFPRF could achieve a reduction of 23.45% EER for the PFD task and an improvement of 13.18% mAP for the TFL task over the second-best PFD comparative method, respectively. Finally, it is crucial to locate partial forgery segments modified by various types to provide explainability, which is the focus of our next research efforts. 
 
 Appendix:  


Decision: The initial ratings are 1 WA and 2 BA, and they are 3 Accept after rebuttal. The reviewers all appreciated excellent writing, clear logic, substantial content, detailed exposition, and are satisfied with the authors' responses. 
The response to reviewers should be added in the final version.
I agree with the reviewers and recommend an acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel framework (CFPRF) to detect and localize partial forgeries in audio data. The framework addresses challenges in current countermeasures that fail to analyze the start and end timestamps of forgery segments. CFPRF includes two main components: a Frame-level Detection Network (FDN) and a Proposal Refinement Network (PRN).
strength_weakness: strength_weakness: • The paper effectively tackles several key challenges in audio forgery detection, such as the detection of small forgery segments, and the localization of multiple forgery segments.
 • The proposed method allows for precise localization of forgery segments. This detailed approach addresses the limitation of existing methods that mainly focus on classification without precise timestamp analysis.
 • The PRN component is designed to be plug-and-play. This flexibility enhances the framework’s applicability and ease of adoption.
limitations: limitations: • The framework requires setting certain thresholds, and the module is the combination of some previous methods.
 • Some parts and labels in the figure need to be clarified, which is hard to understand.
 • The proposed framework requires a two-step training process. This might be a constraint for rapid development and deployment, particularly in dynamic scenarios.
paper_summary: paper_summary: The paper introduces a Coarse-to-Fine Proposal Refinement Framework (CFPRF) for detecting and localizing temporal audio forgeries. It addresses the challenge of identifying subtle manipulation within long audio files, where existing methods mainly classify authenticity but fail to figure out forgery timestamps precisely. Experiments show that the proposed method achieves optimal performance in both detecting and precisely localizing audio forgeries.
strength_weakness: strength_weakness: 1. This paper is characterized by excellent writing, clear logic, substantial content, and detailed exposition.
2. The paper proposes an effective Coarse-to-Fine Proposal Refinement Framework (CFPRF) for the challenge of identifying subtle manipulation within long audio files, in the task of detecting and localizing temporal audio forgeries.
3. Extensive experiments validate the effectiveness of the proposed CFPRF.
limitations: limitations: 1. Figure 2 comprises too many elements, resulting in overlapping and obscuring, which impairs both the aesthetics and readability of the structural diagram.
2. It is better to add the up or down arrows to the all "EER" and "mAP" symbol in Table 3 and Table 4, not only when they first appear.
3. Considering the forgery detection and localization is a task needs practical application, I am curious about the model complexity and inference speed comparison of the proposed method.
paper_summary: paper_summary: The paper introduces a novel Coarse-to-Fine Proposal Refinement Framework (CFPRF) designed for audio temporal forgery detection and localization. It aims to improve upon existing methods by addressing the challenges of detecting subtle forgery manipulations within audio clips, particularly focusing on the accurate identification of start and end timestamps of these forgeries. The framework utilizes a Frame-level Detection Network (FDN) and a Proposal Refinement Network (PRN), incorporating techniques like difference-aware feature learning (DAFL) and boundary-aware feature enhancement (BAFE). Extensive experimental results demonstrate that the CFPRF achieves state-of-the-art performance on various datasets, including LAV-DF, ASVS2019PS, and HAD.
strength_weakness: strength_weakness: 1. The paper introduces a novel two-stage framework for audio Temporal Forgery Localization (TFL). The two-stage model demonstrates significant performance improvements over traditional single-stage models, providing a new direction for future research in TFL tasks.
2. The FDN effectively learns robust discriminative features to mine inconsistency cues, making good use of contrastive learning techniques. 
3. Extensive experiments on multiple datasets (LAV-DF, ASVS2019PS, and HAD) demonstrate the effectiveness of the proposed framework.
4. The paper is well-structured and clearly written, with each component of the proposed model thoroughly explained.
5. Figures clearly represent the algorithm's pipeline and the challenges it addresses, enhancing reader comprehension.
limitations: limitations: 1. The proposed PRN network is relatively simple and lacks innovative design. In the experimental section, it may be beneficial to replace the PRN with other existing single-stage TFL or Action Localization, Sound Event Localization models. This would allow for testing the performance of the Frame-level Detection Network (FDN) and demonstrate the innovative value of the FDN.
2. The related work mentions UMMAFormer, which has already conducted experiments on audio partial forgery localization. However, the summary of this related work has some issues. Additionally, there is no performance comparison with UMMAFormer, nor have experiments been conducted on the Psynd dataset used by UMMAFormer.
3. The paper describes three types of forgery scenarios: replacement, deletion, and insertion. However, there is no evaluation or testing of the performance in these specific scenarios. For instance, it is important to investigate whether the deletion scenario might cause the FDN to fail due to the absence of significant anomalies in the feature expression, as deletion results in zero values.
4. The paper does not explain why the wav2vec2 model was chosen, nor does it provide the rationale and basis for this choice compared to other common feature extractors, including spectrogram-based or self-supervised pretrained models. It is necessary to explain the reasons and basis for choosing the wav2vec2 model and discuss its performance advantages over other common feature extractors.