Reviewerly/exHarmony huggingface dataset:
Algorithm selection using edge ML and case-based reasoning

Abstract In practical data mining, a wide range of classification algorithms is employed for prediction tasks. However, selecting the best algorithm poses a challenging task for machine learning practitioners and experts, primarily due to the inherent variability in the characteristics of classification problems, referred to as datasets, and the unpredictable performance of these algorithms. Dataset characteristics are quantified in terms of meta-features, while classifier performance is evaluated using various performance metrics. The assessment of classifiers through empirical methods across multiple classification datasets, while considering multiple performance metrics, presents a computationally expensive and time-consuming obstacle in the pursuit of selecting the optimal algorithm. Furthermore, the scarcity of sufficient training data, denoted by dimensions representing the number of datasets and the feature space described by meta-feature perspectives, adds further complexity to the process of algorithm selection using classical machine learning methods. This research paper presents an integrated framework called eML-CBR that combines edge edge-ML and case-based reasoning methodologies to accurately address the algorithm selection problem. It adapts a multi-level, multi-view case-based reasoning methodology, considering data from diverse feature dimensions and the algorithms from multiple performance aspects, that distributes computations to both cloud edges and centralized nodes. On the edge, the first-level reasoning employs machine learning methods to recommend a family of classification algorithms, while at the second level, it recommends a list of the top-k algorithms within that family. This list is further refined by an algorithm conflict resolver module. The eML-CBR framework offers a suite of contributions, including integrated algorithm selection, multi-view meta-feature extraction, innovative performance criteria, improved algorithm recommendation, data scarcity mitigation through incremental learning, and an open-source CBR module, reshaping research paradigms. The CBR module, trained on 100 datasets and tested with 52 datasets using 9 decision tree algorithms, achieved an accuracy of 94% for correct classifier recommendations within the top k=3 algorithms, making it highly suitable for practical classification applications.
Boosting Rule-Based Grapheme-to-Phoneme Conversion with Morphological Segmentation and Syllabification in Bengali

This paper presents a novel approach to enhance rule-based Bengali grapheme-to-phoneme (G2P) conversion by leveraging morphological segmentation and syllabification techniques. In this approach, input words are first morphologically segmented into valid morphological chunks, each having a different stem and semantic. Applying the G2P rules on each of these chunks, their pronunciations are generated. An intermediate pronunciation for the whole input word is attained by merging these pronunciations. Using syllabification, this intermediate pronunciation is further divided into valid syllabic sequences that offer accurate morphological boundaries. Finally, the final pronunciation is achieved using syllable-specific orthographic rules on these syllabic sequences. The performance of the proposed G2P approach is assessed using measures representing (i) direct accuracy and (ii) enhancements in different speech-related applications. According to the performances noted for the direct accuracy-based measures, the proposed approach predicted the appropriate pronunciations for about 90% cases, especially for compound and inflected words. This performance is around 22% and 10% better than the performance of a rule-based system and a previous state-of-the-art system, respectively. On the other hand, application-based measures guarantee that the generated phone sequences (i) sound natural and (ii) improve the quality of speech synthesis and recognition systems. These quantitative and qualitative assessment plans answered research questions pertinent to speech and linguistics.
Exploring the Impact of Different Approaches for Spoken Dialect Identification of Konkani Language

This work aims to identify dialects for Konkani language. In this work, various state-of-the-art methods in language identification are explored for the identification of dialects of the Konkani language. The initial base model is constructed using fully connected neural network which is trained on frame-level Mel-frequency cepstral coefficient (MFCC) features. This base model trained on frame-level features is then used for comparison with state-of-the-art models from language identification task that are built for dialect identification (DID) that use utterance-level embeddings, namely x-vector and u-vector. The x-vector and u-vector based models are trained on segment-level features. This work explores segment-level features namely phone-state bottleneck features (BNFs) and wav2vec features extracted from pretrained feature extractors. The x-vector based model uses time delay neural network (TDNN) for the extraction of an utterance-level embedding from sequence of speech segments. A u-vector based model uses bidirectional LSTM (BLSTM) to extract utterance-level embeddings from sequence of speech segments. This work also proposes a novel transformer-based model to extract utterance-level embedding from sequence of speech segments. Results show the effectiveness of the proposed methods for DID of Konkani. It is observed that proposed transformer-based model outperform the other explored models. The results also show the superiority of wav2vec features over the phone-state BNFs for DID task.
Pre- and Post-Textual Contexts in Assessment of a Message as Offensive or Defensive Aggression Verbalization

Contrary to the public stereotype that aggression is evil, scientific paradigm attributes aggression the fundamental importance for humans as biological and social beings. Despite the fact that most definitions of the aggression concept are based on the desire of an individual to harm another person, many aggressive actions do not have the goal of causing harm and damage. Aggression can serve the purpose of increasing self-esteem, self-affirmation, social and psychological adaptation, self-preservation. It is conditionally possible to distinguish offensive, defensive and instrumental types of aggression that are detected by people in verbal communication. Defensive type of aggression is aimed at maintaining an optimally stable psychological state of the individual under negative influence from the outside world. The paper describes an experiment on classification of written communication assessed as aggression verbalization. We explore messages in Russian language extracted out of the VKontakte social network site in interpretations given by a group of Russian natives. We confirm the assumption that the presence of pre- and post- textual contexts changes the result of classification of a written message: the message that previously (without any contexts) was assessed as offensive aggression verbalization, within textual contexts (messages provoking aggressive verbal reaction in response) is being assessed as defensive aggression verbalization.
Everyday Conversations: A Comparative Study of Expert Transcriptions and ASR Outputs at a Lexical Level

The study examines the outcomes of automatic speech recognition (ASR) applied to field recordings of daily Russian speech. Everyday conversations, captured in real-life communicative scenarios, pose quite a complex subject for ASR. This is due to several factors: they can contain speech from a multitude of speakers, the loudness of the conversation partners’ speech signals fluctuates, there’s a substantial volume of overlapping speech from two or more speakers, and significant noise interferences can occur periodically. The presented research compares transcripts of these recordings produced by two recognition systems: the NTR Acoustic Model and OpenAI’s Whisper. These transcripts are then contrasted with expert transcription of the same recordings. The comparison of three frequency lists (the expert transcription, the acoustic model, and Whisper) reveals that each model has its unique characteristics at the lexical level. At the same time, both models perform worse in recognizing the following groups of words typical for spontaneous unprepared dialogues: discursive words, pragmatic markers, backchannel responses, interjections, conversational reduced word forms, and hesitations. These findings aim to foster improvements in ASR systems designed to transcribe conversational speech, such as work meetings and daily life dialogues.
Enhancing Language Identification in Indian Context Through Exploiting Learned Features with Wav2Vec2.0

This work proposes the utilization of a self-supervised pre-trained network for developing a Language Identification (LID) system catering to low-resource Indian languages. The framework employed is Wav2vec2.0-XLSR-53, pre-trained on 53k hours of unlabeled speech data. The unsupervised training of the model enables it to learn the acoustic patterns specific to a language. Given that languages share phonetic space, multi-lingual pre-training is instrumental in learning cross-lingual information and building systems that cater to low-resource languages. Further fine-tuning with a limited amount of labeled data significantly boosts the model’s accuracy. The results showcase a relative improvement of 33.2% over the DNN-A (DNN with attention) model and 19.04% over Dense Resnets for the Language Identification task on the IIITH-ILSC database using the proposed features (Shivang Gupta and Kowshik Siva Sai Motepalli share first authorship).
Design and Development of Voice OTP Authentication System

Voice OTP Authentication (VOA) provides authorization for a speaker by validating the spoken One-Time Password(OTP) and the speaker’s identity. Even though Speaker Recognition and Digit Recognition techniques are fairly mature, the exploration in the direction of the development of VOA systems is limited. This work proposes a speaker and speech representation-based framework to develop the VOA system. Our design uses ECAPA-TDNN based speaker representation and wav2vec conformer-based digit representation to perform VOA. The achieved performance of the speaker identification, OTP identification, and the combined VOA system in the DigitUtter-IITDH dataset in terms of identification accuracy are $$96.75\%$$ , $$83.25\%$$ and $$78.92\%$$ , respectively. Further, to deploy the VOA system on an edge device, we conduct a comprehensive performance analysis by deploying the proposed VOA system from a high-end server class machine to an embedded edge device. Our experimental results indicate that the average inference time for an OTP Authentication using an edge device is 3.14 seconds, while it takes 0.05 seconds on the server class system.
End-to-End Native Language Identification Using a Modified Vision Transformer(ViT) from L2 English Speech

Native langauge identification involves identifying the mother tongue of a person from an audio recording of their speech in second language. Improving native language identification holds potential in advancing the development of more sophisticated human-computer interfaces that rely on audio inputs. Automatic speech recognition systems show a downgrade in performance when used on non-native speech, this can be mitigated by using L1 identification. Presently, the majority of research efforts in L1 identification have concentrated on employing Convolutional Neural Networks (CNNs) on audio spectrograms to predict the native language. With the emergence of Vision Transformers, which have demonstrated exceptional performance in object identification, we have adopted a modified version of the Vision Transformer model to analyze audio spectrograms for L1 identification. This approach has yielded promising outcomes on the NISP dataset which contains audio recordings of English speech of 5 regional lannguages(Hindi, Tamil, Telugu, Kannada, Malayalam) of 345 speakers. The proposed model was able to achieve an overall accuracy of 97.87% on the test dataset.
Significance of Audio Quality in Speech-to-Text Translation Systems

Research on Speech-to-Text Translation systems has shown that their performance is influenced by the size and quality of the training corpus. However, most efforts on corpus creation referred to the size and audio-transcription-translation alignment and text translation as measure of quality of a corpus while not emphasizing much on the audio quality as they are scrapped directly from web-based sources and not recorded in a studio environment. Hence, the absence of higher quality audio may present challenges for these systems in learning the mappings between audio and its corresponding text effectively by the Transformer based encoder-decoder architecture. Even with larger corpora, Direct Speech-to-Text Translation systems have struggled to achieve comparable performance as by their cascaded counterparts, unless they utilize additional external data resources or techniques. A comparative study was conducted on the performance of direct and cascaded speech translation systems without using any external data sources directly or indirectly. This study involved comparing the performance by using both the original and generated high-quality audios generated by a Text-to-Speech Synthesis system from two distinct corpora: Prabhupadavani and MuST-C. The findings revealed that direct Speech-to-Text systems can perform similar to their cascaded counterparts, given that they are trained with a larger corpus that contains high-quality audio. The findings suggest that, when building efficient and robust direct speech-to-text translation systems, it is crucial to consider not only the size and translation quality of the corpus but also the audio quality. By incorporating high-quality audio data into the training process, researchers can enhance the performance of direct Speech-to-Text translation systems forming a viable alternative to cascaded systems.
A feature mapping technique for complex data object generation with likelihood and deep generative approaches

When a sufficient amount of training data is available, Machine Learning (ML) models show great promise for solving problems involving complex and dynamic patterns. Social and behavioral domains are rich with such challenging problems, with complex object data extracted from documents, surveys, etc., and represented in forms such as graphs and trees. However, many social and behavioral data sets are inherently sparse and incomplete. The same data field may be unavailable in different records of a data set due to different causes, e.g., because it was not measured, not known, or simply not applicable to that particular record. Furthermore, collection challenges, cost, lack of participation, small affected populations, etc., result in very small sets of data. Resulting unconventional datasets cannot be directly used with potent approaches such as machine learning. A technique to model and synthesize large sets of such complex data objects while maintaining the same statistical and topological characteristics of original data helps overcome these challenges. To the best of our knowledge, our work is the first attempt to synthesize unconventional datasets. We propose a novel feature mapping technique to eliminate data inconsistencies and model data objects from unconventional datasets. The feature-mapped data objects are used to synthesize data using two likelihood approaches, i.e., multi-variate Gaussian and regular vine copulas, and one generative adversarial approach using an adversarial autoencoder (AAE). We demonstrate the robustness of the proposed technique with three real-world datasets representing disparate domains and validate the performance of likelihood and deep-generative approaches with these object synthesis strategies.
Video Quality Assessment System Using Deep Optical Flow and Fourier Property

Ensuring superior video quality is essential in various fields such as VFX film production, digital signage, media facades, product advertising, and interactive media, as it directly elevates the viewer’s engagement and experience. The ability to accurately quantify a video’s visual quality not only influences its valuation but is pivotal in maintaining high standards. Among the attributes influencing video quality, subjective quality stands out, however, several other elements also contribute significantly. Although automated video evaluations offer efficiency, there are situations necessitating expert editorial insight to measure nuanced subjective attributes. Our research primarily focuses on two prevalent issues undermining video quality: erratic camera motions and suboptimal focus. We employed a deep learning-driven optical flow technique to quantify inconsistent camera movements and adopted a Fast Fourier Transform (FFT)-based algorithm for blur detection. Moreover, our proposed adaptive threshold, grounded in statistical analysis, effectively delineates scenes as either desirable or substandard. Testing this framework on a diverse set of videos, we found it proficiently assessed video quality within a practical threshold range.
On the Effectiveness of Activation Noise in Both Training and Inference for Generative Classifiers

Neurons of the brain never produce the same output twice even when the same stimuli are presented; this is because of their internal noisy biological processes. Likewise, in this work, we study activation noise jointly during training, and inference, in a generative energy-based modeling setting, and observe that the joint activation noise yields noteworthy outcomes: while it is known that applying activation noise during training serves the purpose of regularization and data augmentation, activation noise is not quite well-understood during inference, and most importantly, the relationship between the former and the latter, when we adopt both at the same time, is yet unknown. In this work, therefore, we analyze the role of activation noise at inference time and demonstrate it to be utilizing sampling. Then, we prove that the best performance is achieved when the activation noise follows the same distribution both during training and inference. Leveraging the proof of the optimal joint activation noise during training and inference, we achieve significant improvement in performance (classification accuracy). To help understand this phenomenon, we provide theoretical results that illuminate the roles of activation noise during training, inference, and their mutual influence on performance. To further confirm our theoretical results, we conduct extensive experiments on autoencoder architecture for five datasets and seven different distributions of activation noise.
A Blockchain Reputation Management System for the Internet of Vehicles (IoVT) With Cryptocurrency-Based Recovery System

Internet of Vehicular Things (IoVT), as a subset of the Internet of Things (IoT), enhances safety, traffic management, and driver experience by connecting vehicles and facilitating data exchange. Indeed, the implementation of IoVT comes with its fair share of challenges, which include transparent and secure service management, security, privacy preservation, and prevention of malware attacks. Researchers have proposed reputation-based systems, including blockchain-supported ones, as effective solutions to address the challenges in IoVT, offering benefits such as integrity, authenticity, transparency, and privacy preservation support. On the other hand, cryptocurrency offers several advantages, such as decentralization, enhanced security, and lower transaction costs, making it an effective form of payment that can bypass traditional intermediaries and facilitate fast and borderless transactions with greater financial privacy and control for users. In this paper, a blockchain-based reputation management system is proposed that can collect information about the surroundings from intelligent vehicles, convert that data into appropriate information, and make necessary decisions regarding reported incidents. A cryptocurrency-based recovery system allows defaulters to recover their missing points through the use of digital tokens or assets, providing them with a transparent and decentralized mechanism for restoring their lost value or reputation. The proposed method for the cryptocurrency-based recovery system is implemented and tested using virtual machines, specifically utilizing the Ethereum blockchain and smart contract programming as proof of concept to showcase the functionality and feasibility of the system. Additionally, the packet structure and the throughput are also demonstrated to prove the efficiency of the proposed method.
Deep learning-based detection of mask rule check violations in curvilinear mask

In recent years, Curvilinear Mask technology has emerged as a next-generation resolution enhancement method for photomasks, providing optimal margins by maximizing the degree of freedom in pattern design. However, this technology presents challenges in defining layout design rule limit based solely on geometric information rules based solely on geometric information such as width, space, and corner-to-corner. With the introduction of Multi Beam Mask Writers for Curvilinear pattern production, brand-new violations of Mask Rule Check(MRC) have occurred, which cannot be explained by geometry terms alone. In this study we propose a deep learning-based method for detecting MRC violations using the state-of-the-art Transformer architecture, drawing an analogy between the vertices of curvilinear patterns and words in natural language. The proposed MRC binary classifier demonstrates improved performance compared to traditional rule-based MRC and CNN-based MRC methods. Importantly, our method exhibits robustness in recall, ensuring that defective patterns are not misclassified as normal, preventing missed defects. Moreover, by employing attention maps to visualize deep learning results, we gain explainability and reveal that MRC violations may arise from issues related to the fabrication of specific designs, rather than being solely attributable to geometric features. This insight contributes to a better understanding of the challenges associated with Curvilinear Mask technology and offers an effective solution for detecting MRC violations.
Response Time of Cloud-Based Facial Recognition System Utilizing Homomorphic Encryption

We are developing a system that augments user accessibility by integrating various facial recognition engines across multiple enterprises, thereby facilitating the widespread adoption of facial recognition technology within societal structures. In this system, homomorphic encryption is employed to mitigate the risk of data leakage during facial recognition. However, the use of homomorphic encryption in facial recognition significantly increases latency, making it challenging to meet practical response time requirements. We experimentally evaluated the impact of adopting homomorphic encryption on the response time. The evaluation revealed that the number of registrants per facial recognition engine should be "< 120". Additionally, we evaluated a clustering strategy for reducing the response time to the level of practical application.
Interpretability of Deep Network Structures for SAR Image Target Recognition Based on Generative Adversarial Mask

Deep neural networks (DNN) have been widely used in SAR image Target recognition and have achieved great success. However, the black box attribute of DNN makes the mechanism of the network inexplicable and confusing. In this paper, a network layer analysis model based on generative adversarial mask (GAM) is proposed to enhance the structure interpretability of DNNs. Inspired by network adaptive search, we introduce a mask layer and construct a layer analysis model. After sparse training and generative adversarial training, the contribution of the network layer can be calculated based on the mask layer parameters. We conducted sufficient experiments on several typical DNNs to verify the effectiveness of GAM.