dataesr/scientific-paragraphs-categorization huggingface dataset:
Number of Alarms In our dataset, we have a total of 15,990 different full alarm identifiers. Such identifiers are not evenly distributed across sites, we can see in Figure  9  that the number of full alarm identifiers that appear on each site varies from site to site. It is also clear that faulty sites feature a higher number of active alarms each day. This is an indication of the fact that alarms indeed carry useful information to discriminate normal sites from faulty sites. As described above, our models work with the summarized version of the full alarm identifiers. We have 737 unique summarized alarm identifiers. Figure  10  shows the distribution of the number of single summarized alarms across normal and faulty sites. For each site, we count the average daily number of single triplets < ResourceInstance, ResourceType, AlarmType > appearing in its log.
Finally, we tested the trained model on a test set that is disjoint from the two sets used during training. For the survival probability model, we first needed to train the random survival forest model to predict the risk scores for each site for each day. After having performed this on a training set containing normal and faulty sites, we used a validation set to determine the best hyper-parameters of both the random survival forest and the two added hyperparameters: window (of the average rolling window) and risk threshold. We split the entire dataset into five subsets and we performed a nested cross-validation. Let us consider the model based on the RNN: the same logic applies to the other models for faulty site-detection. At each round, we picked a subset as a test set and we set it apart.
This section presents the results for fault prediction at the component level. Following the layered approach of Section 4, faulty component detection was performed just on sites that have been detected as faulty by one of the site-level algorithms. However, in order to evaluate the performance of component-level algorithms independently from the quality of site-level algorithms, we first present the (conditional) results assuming all faulty sites are correctly identified, i. e. , we execute faulty component prediction algorithm just on actually faulty sites, assuming we have an ideal site-level fault prediction with recall = precision = 100%. The accuracy of faulty component identification computed on all faulty sites of the labeled dataset is depicted in Figure  13. It shows the precision-recall trade-off with respect to different score thresholds (in the range [0.
Additionally, the faulty module identification performs better in terms of precision for the higher precision site-level predictions. For example, when applied on top of the RNN with site-level precision = 100% and recall = 26%; for high score thresholds at the component level, we obtained an end-to-end precision equal to the one of the site-level, i. e. , 100%, while for the site level predictions with 92% precision, the best end-to-end precision is getting lower-70%. This means that there are faults that are easier to predict than others at both layers. Our analysis shows that faults on some components are better predicted. The results per component are sensitive and cannot be presented in detail in this paper. However, we can mention that by selecting the top 15 out of the 18 components present in the dataset, the precision and recall are increased by 20%.
Automatic speaker verification is susceptible to various manipulations and spoofing, such as text-to-speech synthesis, voice conversion, replay, tampering, adversarial attacks, and so on. We consider a new spoofing scenario called "Partial Spoof" (PS) in which synthesized or transformed speech segments are embedded into a bona fide utterance. While existing countermeasures (CMs) can detect fully spoofed utterances, there is a need for their adaptation or extension to the PS scenario. We propose various improvements to construct a significantly more accurate CM that can detect and locate short-generated spoofed speech segments at finer temporal resolutions. First, we introduce newly developed self-supervised pre-trained models as enhanced feature extractors. Second, we extend our PartialSpoof database by adding segment labels for various temporal resolutions.
Since the short spoofed speech segments to be embedded by attackers are of variable length, six different temporal resolutions are considered, ranging from as short as 20 ms to as large as 640 ms. Third, we propose a new CM that enables the simultaneous use of the segment-level labels at different temporal resolutions as well as utterance-level labels to execute utterance-and segment-level detection at the same time. We also show that the proposed CM is capable of detecting spoofing at the utterance level with low error rates in the PS scenario as well as in a related logical access (LA) scenario. The equal error rates of utterance-level detection on the PartialSpoof database and ASVspoof 2019 LA database were 0. 77 and 0. 90%, respectively.
This is similar to the standard spoof vs. bona fide classification in the LA and DF scenarios. The second function is to detect which segments in an utterance are spoofed. This is to show which specific parts of an audio sample may have been generated using TTS or VC and to improve the explainability of the CMs. We refer to the former as "utterance-level detection" and the latter as "segment-level detection. " While earlier we described examples of spoofing at the phone and word levels, in practice, there is no restriction on the speech unit to be used by attackers. Segment-level detection should be carried out without prior knowledge of the length of the unit used by the attacker and needs to support variable-length segments. In our previous study  [5] , we constructed a speech database called "PartialSpoof" designed for the PS scenario and reported initial results for utterance-and segment-level detection.
In this preliminary work, segment-level detection was applied in a straightforward manner with a fixed temporal resolution. Whether the detection is per-utterance or per-segment, the features needed for CMs are likely to be similar. Therefore, in our subsequent work  [6] , we constructed a CM that simultaneously executes detection at the utterance level and (fixed) segment level using multi-task learning. However, these two studies showed that there is much room for improvement. We thus extend our previous studies to improve the utteranceand segment-level detection of CMs in the PS scenario. We use self-supervised pre-trained models based on wav2vec 2. 0  [7]  and WavLM  [8]  as enhanced feature extractors. We also extend the PartialSpoof database. Specifically, we add segment-level labels for various temporal resolutions, instead of only for a fixed temporal resolution.
The spoofed audio segments to be embedded by attackers are of variable length; thus, by using these labels, CM models can be trained to execute detection at various temporal resolutions. In the extended PartialSpoof database, there are six different temporal resolutions for these segment-level labels, ranging from as fine as 20 ms to as coarse as 640 ms. We also propose a CM that enables the simultaneous use of the segment-level labels at different temporal resolutions as well as utterance-level labels to execute utterance-and segmentlevel detection at the same time. The remainder of this paper is structured as follows. In Section II, we explain the PS scenario in more detail and review relevant topics. In Section III, we explain our PartialSpoof database with new segment labels at various temporal resolutions.
C. Databases for PS Scenario During the same time as when we built the initial PartialSpoof database in 2021  [5] , another database was also proposed for the PS scenario  [16]  for which a single multi-speaker TTS system was used to replace a single word within an utterance. This later became a part of the Audio Deep synthesis Detection (ADD) challenge database  [17]. In addition to the limited number of TTS systems, the database has predetermined the target for replacement as a word, which is a form of prior knowledge even on the detection side. In contrast, we consider a spoofing scenario based on variable-length spoofed segments generated using several multi-speaker TTS and VC systems.
III. NEW PARTIALSPOOF DATABASE With the goal of stimulating research on CM models suitable for the PS scenario, an early version of the PartialSpoof 2 database  [5]  containing the spoofed speech needed for model training was built and made available to the research community in 2021. In the PS scenario, attackers may create partiallyspoofed audio in various ways. Attackers may insert segments of spoofed speech generated by TTS or VC into a natural utterance to add new content or substitute parts of the original utterance with spoofed speech to replace content. Alternately, instead of starting with a natural utterance and inserting or replacing segments with spoofed speech, they may start with an utterance generated by TTS or VC and insert or replace segments with natural speech, for example, in the case in which the attackers only have a small amount of bona fide data.
Thus, it is necessary for the database to contain spoofed speech with different proportions of generated audio segments within a single utterance. We call the ratio of the duration of generated audio segments in an utterance to the total duration of the utterance "intra-speech generated segment ratio. " An efficient way to achieve this is to prepare pairs of speech utterances consisting of one that is entirely generated using TTS or VC and one original bona fide speech utterance then randomly substitute short segments within the pair. This enables us to create a large number of spoofed audio files with different intra-speech generated segment ratios. This is a key concept of the PartialSpoof database and we describe its construction policies and procedures as well as newly generated segment-level labels for a new version of the PartialSpoof database.
A. Construction Policies Since the methods used by attackers are unknown in practice, these will likely be different from the training data used to train a real system. Therefore, we should assume that TTS/VC methods for generating spoofed audio segments in the training set are mostly different from those in the evaluation set. To achieve this, each subset of the ASVspoof 2019 LA database  [18]  was used to construct each corresponding subset of the PartialSpoof database. It was also assumed that the attacker could use generated audio segments at a variety of acoustic units, not limited to linguistic units such as words. Hence, we used variable-length speech segments found by voice activity detection (VAD) regardless of the content of the speech. Finally, we assumed that the attacker could carry out not advanced but basic digital signal processing (DSP) to reduce artifacts of the spoofed audio for the purpose of a reliable assessment of CMs.
B. Construction Procedure The following is the procedure for constructing the Partial-Spoof database automatically. As shown in Fig. 1 , the entire processing pipeline involves five stages: Step 1. Normalization and VAD: When substituting segments into another speech utterance, the volume of the segments and utterance may be different. If there are significant volume mismatches, it becomes straightforward to detect the substituted segments, which would interfere with the CM's ability to learn fine-grained mismatches. Therefore, we first normalized the waveform amplitudes of the original and spoofed utterances contained within the ASVspoof 2019 LA database to -26 dBov according to the ITU-T SV56 standard  [19]. Next, to select the variable-length candidate segments, three types of VAD  [20] ,  [21] ,  [22]  were used, and a majority vote was taken.
Step 5. Post-processing: To balance the subsets, intra-speech generated segment ratios of the concatenated audio were quantized into ten levels, and then random sampling was done for each level. As a result, audio files with a small intra-speech generated segment ratio and those with a large intra-speech generated segment ratio were equally included in the database and in each subset. 3 The number of files that we randomly sampled was also the same as the spoof class of the ASVspoof 2019 LA database.
C. Database Statistics The total number of samples for each temporal resolution is shown in Table  II. The sample size increases as resolution increases from utterance (utt. ) to 20 ms. The percentage of samples belonging to the spoof class in each temporal resolution is shown in Table  III. If any part of the generated waveform is contained in the target segment to be annotated, it is labeled as spoof. We can see that the finer the temporal resolution of the segment, the less likely it is to contain generated audio signals; thus, the number of samples in the bona fide class increases at finer-grained resolutions. As can be seen from the table, the data have a bias toward the spoof class on a per utterance basis, but the extreme bias is eliminated on finer segment bases. In multi-task learning using labels at multiple temporal resolutions, such differences in bias may lead to more robust model learning.
D. Limitations In the PartialSpoof database, the variable-length speech segments found by the VADs are replaced without considering the meaning of sentences and words as well as the phonemes before and after the segments. Therefore, the speech in the database is not partially-spoofed speech that is intended to deceive listeners and deliver wrong linguistic messages to them but an approximation of it. On the other hand, since partially-spoofed segments of the PartialSpoof database are of variable length, CMs built on this database can execute segment-level detection at various temporal resolutions, and their accuracy can be evaluated.
Finally, s u is computed from e u using linear transformation. The back-end is typically trained by minimizing the standard cross-entropy (CE) or other advanced criteria given an utterance-level score s u and groundtruth label y u for a sufficiently large dataset. 2) Adapting Utterance-Level CM for PS Scenario: The conventional utterance-level CM has to be modified so that it can produce segment-level scores for the PS scenario. Two studies proposed to revise the LCNN-based back-end  [5] ,  [16]. As shown in Fig. 2 (c),  [5]  and  [16]  remove pooling layer(s) from the utterance-level CM to derive s 1:M from h 1:M. While there are differences in the implementations, the common idea is to add a sub-network to transform hidden features into segmentlevel scores h 1:M Linear ---→ s 1:M. The revised CM may have a bipartite structure as shown in Fig.
D. Training Strategies Supposing that ground-truth labels are available for all temporal resolutions, the proposed CM can be trained using two types of strategies: 1) Multiple Temporal Resolutions: For the k-th temporal resolution, a loss L k is computed given the score s k and label y k. Similar to the CMs described in Section IV, the loss function can be CE or other advanced metrics. The gross loss function can thus be written as L = K+1 k=0 L k 8  , where we use L K+1 to denote the utterance-level loss. 2) Single Temporal Resolution: As described earlier, CM training using multiple temporal resolutions requires groundtruth labels for all temporal resolutions. While such labels are provided with the PartialSpoof database, they might not be available for other databases. In such a case, the CM can still be learned without application of the scoring modules and loss functions at temporal resolutions that have no segment-level labels.
VI. EXPERIMENTS We conducted experiments to test the proposed CM on the PartialSpoof database. We first conducted a pilot study to select the most suitable SSL-based front-end and back-end scoring module. We then conducted a comparative study to compare the two CM training strategies. After that, we analyzed differences in the tendencies of the proposed CM's performance on the development and evaluation sets. Furthermore, we conducted a cross-scenario experiment using the ASVspoof 2019 LA database  [18] , the goal of which was to show that the proposed CM optimized for the PS scenario can also be applied to the LA scenario. After explaining the experimental setup and model configurations, we describe each experiment in detail.
B. Comparing Front-and Back-Ends for Proposed CM As explained in Section V, we used a pre-trained SSL model for the front-end and designed a powerful scoring module for the back-end. In this experiment, we compared several candidates for both. Performance was measured using the utterance-level detection EER on the development set of the PartialSpoof database. The CM was trained following the same conventional utterance-level training as in the LA scenario, that is, using the single temporal resolution training strategy with utterance-level labels only. For the front-end, we tested the four pre-trained SSL models listed in Table V; w2v2-base is based on the Wav2vec 2. 0 Base model, while w2v2-large and w2v2-xlsr are based on the Wav2vec 2. 0 Large model. The difference is that the   [29]  blocks while Wav2vec 2.
A gMLP block is similar to basic Multilayer Perceptrons (MLPs) with a gating unit. It was found to be simple and powerful compared with other alternatives. We compared each of the above components in terms of the EER on the development set of the PartialSpoof database. To reduce the time cost, we used w2v2-base as the front-end when comparing the back-end scoring modules. The EERs listed in Table VI show that using five gMLP blocks gives the best performance. We then compared the SSL models while using the architecture of five gMLP blocks in the back-end scoring module. The results indicate that the three Large SSL models outperformed w2v2-base, and that w2v2-large performed best. Given these results, all following experiments were conducted using w2v2-large and five gMLP layers.
C. Comparing Training Strategies on the Proposed CM In this section, we compared the two CM training strategies, that is, training at a single temporal resolution or multiple resolutions. Performance was measured on the development set and evaluation set of the PartialSpoof database. We first explored the two training strategies on segment-level detection. The results are shown in Fig. 3(a). For the proposed CM trained at a single temporal resolution (black circle), since the segment-level ground-truth labels are available at the six segment-level temporal resolutions, we trained six versions of the proposed CM, one for each segment-level temporal resolution. After training, each CM version produces scores at the corresponding temporal resolution. We also trained the CM at multiple temporal resolutions (red triangle) that can derive scores for all resolutions at once.
D. EER Gap Between Development and Evaluation Sets We have two hypotheses to explain the significant differences between the EERs on the development and evaluation sets, and we examined both using the CM trained at multiple temporal resolutions. 1) Hypothesis 1: More difficult spoofing systems exist in the evaluation set. For the LA scenario and ASVspoof 2019 LA database, existing studies have measured the EER over fully spoofed trials from each spoofing system and found that a voice-conversion-based spoofing system called A17 significantly increased the CMs' EERs on the evaluation set  [3] ,  [41]. Since the PartialSpoof database used spoofed trials from the ASVspoof 2019 LA database, we hypothesize that the strong attacks in the evaluation set may have led to the higher EER. Because the segments in a partially-spoofed audio sample can be from different spoofing methods, we did not use the same investigation method as previous studies on the LA scenario and instead used a leave-one-out evaluation approach.
For each spoofing method, we excluded the test trials that contain at least one segment produced by that spoofing method. We then measured the EER on the remaining trials and compared it with the original EER computed on all the test trials. We conducted the analysis on both the development and evaluation sets. The results are shown in Fig. 4. Each column in the figure corresponds to the results on segment-level detection at one   [5] ,  [6] ,  [16]  USED ONLY ONE OR TWO TEMPORAL RESOLUTIONS temporal resolution, while each row to the results of analysis on a specific spoofing method. The IDs of the spoofing systems such as A07 were inherited from those in the ASVspoof 2019 LA database. It was observed that, if the spoofed trials that contain spoofed segments from A15 10 were removed, the EER on the rest of the evaluation set decreased significantly.
F. Cross-Scenario Study Since a CM for the PS scenario can execute utterance-level detection, it can also be applied to the LA scenario. To determine whether the improvements brought by multi-resolution training for the PS scenario also translate to the LA scenario, we investigated the performance of the proposed CM on utterance-level detection for both LA and PS scenarios. The data for these scenarios were from the ASVspoof 2019 LA and PartialSpoof databases, respectively. These experiments involved two training settings, one using the training set of the ASVspoof 2019 LA database and the other using that of the PartialSpoof database. For the ASVspoof 2019 LA database, since the training data only contains utterancelevel labels, the proposed CM was trained using the single temporal resolution training strategy for utterance-level detection.
For CMs trained using the PartialSpoof data, we used the two CMs trained using either the single temporal resolution strategy or the multiple resolution one as described in Section V-D. The first used only utterance-level labels and was trained using the same strategy as the CM trained on the ASVspoof 2019 LA database. The latter used the full set of utterance and segment labels from the PartialSpoof database and trained using the multiple temporal resolution strategy. We used each CM to produce scores for the ASVspoof 2019 LA and PartialSpoof development and evaluation sets for the utterance-level detection. The results are shown in Table  VIII. Focusing first on single resolution and utterance-level training, the performance for the ASVspoof 2019 LA database was shown to be competitive. However, the EERs increased significantly beyond 10% in the PartialSpoof dataset.
This was expected because the CMs trained using the LA data are not exposed to partial spoofs, thus perform poorly. When trained using the PartialSpoof dataset; however, performance in the case of PS data improved significantly, while performance when tested using the LA data remained competitive. Further improvements were achieved when the proposed CM was trained at multiple temporal resolutions. In summary, when trained on the PartialSpoof dataset, the proposed CM demonstrated better utterance-level detection performance for both LA and PS scenarios. Using PartialSpoof training data is beneficial even when the CM is tested on LA data. 11.
VII. CONCLUSION We reported a new spoofing scenario, PS, in which only a fraction of speech segments are spoofed, with the remaining segments containing bona fide speech. Successful approaches to spoofing detection in this scenario can be applied at either the utterance level or segment level. The latter requires the assessment of spoofing classifiers using a database of bona fide and spoofed speech, with the latter labeled at the segment level. We described the new PartialSpoof database which is labeled at multiple temporal resolutions from 20 to 640 ms. After formulating CM tasks required to tackle the PS scenario, we introduced SSL models as an enhanced front-end and proposed new neural architectures and training strategies that exploit segment-level labels for simultaneous, multi-resolution training.