Reviewerly/exHarmony huggingface dataset:
Real-Time Embedded Deep Learning on Elk Audio OS

Recent years have witnessed significant advancements in deep learning architectures for music, along with the availability of more powerful embedded computing platforms specific to low-latency audio processing tasks. These recent developments have opened promising avenues for new Smart Musical Instruments and audio devices that rely on the execution of deep learning models on small embedded computers. Despite these new opportunities, there is a lack of instructions on how to deploy neural networks to many promising embedded audio platforms, including the embedded real-time Elk Audio OS. In this paper, we introduce a procedure for deploying audio deep learning models on embedded systems utilizing the Elk Audio OS. The procedure covers the entire process, from creating a compatible code project to executing and diagnosing it on a Raspberry Pi. Moreover, we discuss different approaches for the real-time execution of deep learning inference on embedded devices and provide alternatives for handling larger neural network models. To facilitate implementation and support future updates, we provide an online repository with a detailed guide, code templates, functional examples, and precompiled library binaries for the TensorFlow Lite and ONNX Runtime inference engines. This work aims to bridge the gap between deep learning model development and real-world deployment on embedded systems, fostering the development of self-contained digital musical instruments and other audio devices equipped with real-time deep learning capabilities. By promoting the deployment of neural networks to embedded devices, we contribute to the development of Smart Musical Instruments that are capable of providing musicians and audiences with unprecedented services.
Feature Enhancement Algorithm for Recognition of Small Targets in Low Resolution Images

Small object detection is a challenging task in computer vision due to limited pixels, low resolution, and sparse feature information. Existing infrared-based methods for small object detection suffer from low accuracy and susceptibility to environmental disturbances. This paper proposes a novel approach using an enhanced lightweight YOLOv5 network for high-precision infrared small object detection. The method involves capturing depth data using a ToF sensor, converting it to a four-channel RGB image through pseudo-color encoding, and applying HSV conversion. By preprocessing the depth values and incorporating them into the lightweight YOLOv5 model, the proposed approach extracts infrared small object detection features and improves the network's efficiency. Experimental results demonstrate that the proposed method achieves a significant improvement in detection performance, with an average precision of 97.4% on an enhanced dataset compared to pseudocolor encoded images.
EFL Text Readability Assessment Based on Feature Engineering

Chinese students learning English in a Chinese environment are EFL (English as a foreign language) learners . Most of the current English readability studies focus on first language learning and English as a second language (ESL) learners in an English-speaking environment, while Chinese students learning English as a foreign language (EFL) have fewer readability studies, mainly using the formula method, but this method only focuses on surface features, involves fewer features, and has low accuracy. Therefore we proposes a readability assessment method designed for Chinese learners of EFL. Using 253 features in five aspects, the final accuracy is above 85%. Meanwhile, this paper studies the predictive ability of different aspects of linguistic features on the readability of middle school texts, and the results show that shallow traditional features have the highest accuracy.
Self-Attention for Visual Reinforcement Learning

Reinforcement learning has been extensively applied to playing video games that involve visual observations, using convolutional neural networks (CNNs) to process the image input. Recent research has demonstrated that attention mechanisms in neural networks can be highly effective in various tasks, although their potential for visual reinforcement learning has not been fully explored.This study explores three approaches to incorporating attention into reinforcement learning. The first approach is to add self-attention to the later dense layers of the CNN. The second approach involves using a convolutional self-attention module within the convolutional layers of the network. The third approach incorporates self-attention with a large kernel convolution within the convolutional layers.To assess the efficacy of the proposed approaches, we conducted a comparative analysis of their performance against a baseline CNN feature extractor for a Proximal Policy Optimization (PPO) agent across multiple image-based Atari environments. Our findings suggest that attention mechanisms can be advantageous in scenarios where the PPO agent needs to adapt to minor environmental fluctuations. However, in games with fixed setups, baseline PPO tends to outperform the variations with the attention mechanism.
Adaptively Weighted Copy-Decoupling Resampling Strategy for Long-Tailed Multi-label Classification

Typical natural images are associated with multiple classes, and different classes typically exhibit a long tail distribution. The Binary Relevance strategy is commonly employed to deal with multi-label image classification. However, this strategy faces two major challenges: the resampling strategy’s failure due to label co-occurrence and the classifier’s bias towards negative samples with imbalanced properties of positive and negative samples. To overcome these limitations, this paper proposes two novel approaches: a simple but efficient sampling strategy, i.e., Copy-Decoupling Resampling (CD-RS), and an adaptive weighted loss function, i.e., Adaptive Focal Loss (AFL). Specifically, CD-RS converts a multi-label image into multiple single-label images with special labels, eliminating the effect of label co-occurrence on the resampling strategy. Meanwhile, AFL adaptively adjusts the focal factor in focal loss according to model performance on samples, which gives more appropriate weights to samples and alleviates the bias caused by sample imbalance during classifier training. Finally, this paper proposed method achieved impressive results on popular datasets VOC-MLT and COCO-MLT, with mAP of 78.96 and 55.35, respectively.
Using Image Enhancement for Target Tracking

In UAV aerial photography scenes, lighting changes, image blur, low resolution and other factors have a great impact on tracking performance, and previous target tracking algorithms mainly studied robust tracking under sufficient light and high resolution. This article proposes an adaptive image enhancement algorithm for unmanned aerial vehicle target tracking, which achieves robust tracking of unmanned aerial vehicles under dark changes in lighting and insufficient lighting. Firstly, an adaptive image enhancement module is constructed to identify the dark light scene and compensate for the corresponding image brightness and image contrast. Secondly, a dynamic constraint strategy is added to constrain the difference in tracking response, enabling the tracker to achieve time adaptation. Finally, experiments on two UAV benchmarks have proven the superiority of our method compared to other nine state-of-the-art trackers.
Co-FT: Effective Fine-tuning for Pretrained Language Model via Adding Collaborator

Dialogue understanding is a common application in many online webs, which recognizes the intents of customers via utterance classification. The state-of-the-art methods finetune a pretrained language model (PTLM) on a task-specific dataset, which is sub-optima because it makes the PTLM forget the knowledge learned in pretraining. To address it, we propose an effective finetuning method (Co-FT) via adding collaborators for PTLM. Instead of finetuning the whole PTLM, we add an extra collaborator utterance encoder (CoUE). The CoUE consists of independent tokenizer and network weights, which can be implemented via rich flexible network architectures. We aggregate the features extracted by PTLM and the CoUE, and then feed them to the intent predictor (a feed-forward classifier) for dialogue intent recognition. Given a domain-specific dataset, we only update the weights of CoUE via a 2-stage training method. Our experiments show that Co-FT outperforms the PTLM finetuning method and achieves 78.43% f1-score. Besides, Co-FT also shows a good capacity in the few-shot learning.
FPGA-based Low-Latency Audio Coprocessor for Networked Music Performance

Networked Music Performance (NMP) applications are acknowledged to be a particularly challenging field due to their stringent latency requirements and their demand for high audio quality. Most solutions developed in the last decades tried to overcome these obstacles by leveraging software approaches, that can introduce excessive time delays as a consequence of the general-purpose nature of the architectures on which they are implemented. Alternatively, a dedicated audio processor can be employed to minimize the mouth-to-ear latency.This paper presents the ongoing development of an hardware system that exploits an Application-Specific Instruction set Processor (ASIP) implemented on a Field-Programmable Gate Array (FPGA) to accelerate audio sample management. Specifically, a Transport Triggered Architecture (TTA) is being investigated as a processor design that aligns well with the required application domains. Preliminary empirical results indicate that the proposed solution has the potential to achieve extremely low latency, compatible with NMP requirements. Further optimizations and enhancements are actively being pursued to address the yet open challenges posed by NMP applications.
Research on Bidirectional LSTM Recurrent Neural Network in Speech Recognition

Speech recognition is a very useful artificial intelligence technology, which has been widely used in various fields and has great potential for development. With the continuous progress of technology, speech recognition will become an indispensable part of people’s daily life, but in the process of further development of speech recognition, there are still many problems that have not been solved. For example, speech signals are susceptible to a variety of interference factors, such as noise, speaker accent, which may lead to the degradation of speech recognition performance. In speech recognition, it is often necessary to process long time sequences of speech signals, such as continuous conversations of speakers, long sentences in speech recognition tasks. Traditional recurrent neural networks may suffer from gradient disappearance or gradient explosion, which makes it difficult for the model to capture long-term dependencies. In this paper, our primary focus is designing a bidirectional LSTM recurrent network structure [10]. We utilize the CTC algorithm to calculate the loss and employ the cluster search decoding algorithm for decoding the network’s output. Compared with the bidirectional LSTM recurrent neural network with 5 hidden layers, the pinyin text recognition can reach 85% correct rate, 5% less training times, and 6% less wrong pinyin words in individual sentences at 15%. Some of the recognition rates presented in this paper may appear suboptimal due to noise in individual speech signals and other factors. How to denoise has been the focus of speech recognition in neural networks.
Réaltacht: Creating Immersive and Accessible Experiences of Irish Traditional Music in Virtual Reality

In this article we report work in progress on an interactive immersive experience that brings several recordings of live music performances in the Irish tradition to virtual reality (VR) headsets and online 360° streaming. Using 360° green screen technology we virtually locate the musicians in three well-known cultural institutions of Ireland including a traditional pub, a former church repurposed as an arts centre and Siamsa Tíre - The National Folk Theatre of Ireland. Immersive audio is achieved using close-miking techniques, virtual ambisonics and impulse response captures of the reverberant destination spaces. We discuss the wider motivations of the project in the context of inclusive and accessible concert attendance as well as cultural heritage. The production pipeline is described, from inception and design to studio and venue capture followed by VR post-production, implementation and next steps.
Applying the Eye-Tracking Method for the Classification of Neurological Disorders, Mental Diseases, and Speech Impairments Based on Machine Learning: an Overview

Eye tracking is a non-invasive technology that facilitates the real-time monitoring of eye movements. It offers insights into visual behavior and cognitive processes. The analysis of eye movement data involves two key components: fixations, (periods of stable gaze), and saccades (rapid shifts in focus between points of interest). This brief overview evaluates significant recent studies conducted between 2018 and 2023 that employ eye tracking for task related to the medical classification. Main algorithms that process eye movement data are used to extract features like pupil positions, fixation duration, and saccade characteristics. Our analysis showed that investigators use machine learning algorithms such as support vector machines, k-nearest neighbors, random forest, and convolutional neural networks for distinguishing normal and pathological states, typically in binary classification tasks, measured by accuracy and AUC. Generally, the size of the datasets is limited. However, authors achieved reliable classification results, ranging from 52% to 95%. As technology continues to evolve, the integration of eye tracking and machine learning offers a promising path toward enhancing our understanding of cognitive processes and medical diagnostic capabilities.
Task-Agnostic Quantize-Aware Training Using Knowledge Feature Distillation

The more intelligent and complex model has brought huge computing demand and high Memory bandwidth. It is deployed on the embedded neural network accelerator and meets the need to reduce the model bit width in real time to reduce the computing capacity requirements of the model on embedded devices. With the expansion of intelligent task scenarios, the model must through quantitative perception training, but it will also introduce expensive training costs, making the Toolchain bloated and complex, which is not conducive to the rapid expansion of new scenarios. To address this problem, a task-agnostic knowledge distillation low-bit-width quantization-aware training method is proposed. Specifically, by quantization-aware training the model to low bitwidth, the memory and bandwidth requirements of the model are greatly reduced. Combining with the knowledge distillation strategy, which includes Attention Feature Fusion (AFF) and Multi-level Contextual Knowledge (MCK), the limitations of the actual task scenarios can be overcome. By extracting the feature outputs of the teacher model, knowledge distillation is performed directly between different model levels and different feature scales to achieve knowledge penetration, transfer and fusion. Finally, knowledge refinement and distillation are realized, which can accelerate the rapid iterative development of model deployment. The experimental results show the effectiveness of combining the task-agnostic knowledge distillation strategy with the low-bit-width quantization-aware training strategy proposed in this paper. The accuracy of the 1-bit MobileNetV2 model on the MNIST dataset is increased by 71.1%. The accuracy of the 1-bit YoloV3-tiny model on the VOC2007 dataset is increased by 27.4%.
Matchmaking Engine for Energy Data Marketplace Using Word Embedding Techniques

This work presents an approach for a matchmaking engine in the energy data marketplace. The engine uses a hybrid recommender system that combines knowledge-based and content-based filtering to suggest the best data assets to users. The matchmaking service utilizes annotated data to identify the most suitable match between data consumers' needs and available data in the energy marketplace. The recommended datasets are ordered by their relevance to the user's needs. To accomplish this, the matchmaking engine employs two popular word embedding techniques: TF-IDF and word2vec, for content-based filtering. Additionally, the knowledge-based recommender leverages the semantic annotation of data obtained from mapping data to a data model.
Verification &amp; Validation Methods for Complex AI-enabled Cyber-Physical Learning-Based Systems: A Systematic Literature Review

This systematic literature review (SLR) investigates how systems- and software engineers should approach verification and validation (V&V) of complex Artificial Intelligence (AI)-enabled cyber-physical learning-based systems. General trust in learning-based, complex, self-adaptive cyber-physical systems (CPSs) is low. For the technology to be accepted and trusted for more industrial and commercial use, proper V&V techniques, methods, frameworks, and theories are required to ensure desired outcomes. Principal findings show that traditional V&V methods could be classified as either inspection, demonstration, testing, or analysis. These conventional methods are highly nuanced for complex systems. They may even work for large CPSs where determinism and linear internal properties are assumed. However, the self-adaptive learning nature of AI-systems calls for adapted or new V&V methods. Primary complications encountered in V&V activities for AI-enabled systems relate to input data quality, model and requirements management, non-determinism, and AI black-box effect related to poor explainability, scale, and complexity issues. This review highlights the importance of a designated data officer with authority and capability to manage the relevant data for being relevant, available, bias-free, and adversarial sample free. Also, an issue list or complications may also provide caution for implementing AI-systems.
JumpApp: an online didactic game for music training and education

Musical learning is beneficial for individual of all ages. In particular, the use of interactive software tools is considered nowadays essential for increased engagement, experience personalisation, and effectiveness. This is particularly true for the task of music education, which has to face the problem of complex rules and notation. For these reasons, we developed JumpApp, a musical didactic game. The devised software is a completely web-based application built on JavaScript and accessible through Chrome browser. The objectives of JumpApp are twofold. First, the user is trained to improve the relative pitch with an engaging listening experience. Secondly, such a learning is decoupled from the standard western music notation, through a user-friendly interface that maps notes in an octave in an innovative way, thus avoiding the learning of complex theoretical rules and musical notation of intervals and scales. The quality of experience, versatility and learning effectiveness of JumpApp, have been evaluated by musicians with different music background through a questionnaire. The paper is structured in a top-down fashion. We start from a description of the features of JumpApp. We then discuss how these features have been implemented in the software and the signal processing blocks that attain the goal. Finally, results of the evaluation process are reported along with final considerations and future works.
Multi-layer Long Short-Term Memory Recursive Perceptual Network for Temporal Knowledge Graph Representation Learning

Temporal Knowledge Graphs (TKG) has been widely applied in many fields of life, and the reasoning of TKG has naturally become a research hotspot for many researchers. To set up extrapolation for TKG, we need to observe historical information, reflect entity preferences through historical information, and ultimately achieve the goal of predicting future events. However, for current inference models, how to make the model obtain more stable and accurate prediction results is still a challenge. Therefore, we propose a model called MLRPN (GCN-based Multi-layer Long Short-Term Memory Recursive Perceptual Network) that combines Graph Convolution Networks (GCN), Long Short-Term Memory (LSTM) and Multi-Layer Perceptrons (MLP) for regression modeling of temporal knowledge graph sequences. This model obtains evolutionary representations of entities and relations, and ultimately achieves prediction tasks for entities and relations. It consists of two modules: the Evolutionary Modeling Unit and the Score Calculation Function Unit. The Evolutionary Modeling Unit encodes and regresses the historical KG sequence to obtain evolutionary representations of entities and relations with longer temporal dependencies. The Score Calculation Function Unit contains the score functions for the two sub-tasks, which calculate the optimal solutions through score functions to obtain the best prediction output results. MLRPN was evaluated on four common TKG datasets, the experimental results showed that MLRPN outperforms other models on multiple datasets.
Vehicles Detection and Counting Method from Overhead View based on Embedded Vision

In modern society, vehicles are indispensable carriers in transportation and daily life. The rapid increase in the number of vehicles has contributed to the development of intelligent traffic monitoring. Vehicle counting allows for a quick understanding of the traffic flow on the road and the parking situation in the parking lot. This enables the management of traffic. In recent years, vehicle counting techniques which is based on image processing have been innovated. However, there are still challenges to effectively utilize image processing methods for vehicle counting. In this paper, an efficient vehicle counting algorithm using ORB algorithm and Sliding Window Method is proposed. To be able to improve the efficiency of vehicle detection, the Frame Difference Method is used to detect whether there are moving objects. Based on the results of vehicle counting on the parking lot video. The method is able to efficiently and accurately identify the number of vehicles in a parking lot. And it can be deployed to small embedded devices.
Adversarial Robust Scene Classification based on Proactive-Reactive Deep Ensemble Defenses

As a safety-related application, visual systems based on convolutional neural networks (CNNs) in modern unmanned aerial vehicles (UAVs) show adversarial vulnerability when performing the real-time inference. Aimed at scene classification task of remote sensing images (RSIs), this paper proposes to use ensemble methods for both proactive and reactive defenses in solving the problem. Because of serious adversarial transferability for malicious RSIs crafted against CNNs, we train a robust ensemble by promoting the orthogonality between sub-models’ loss fields and inhibiting the loss fields strength of sub-models for the proactive defense. The adversarial transferability in the ensemble then can be greatly reduced. In terms of reactive defense, we fuse scoring functions of several classical detection algorithms with the hidden features and average output confidences from sub-models as a second fusion. Extensive experiments on two benchmark datasets with multiple scenes demonstrate that both deep ensemble defenses achieve the state-of-the-art performances in most cases and generalize well for non-malicious data, which enhances the adversarial robustness of CNN models for scene classification on UAVs.