Reviewerly/exHarmony huggingface dataset:
Learning Adapters for Code-Switching Speech Recognition

Multilingual code-switching speech recognition has been an emerging research direction in real-world applications since most of speakers are bilingual or multilingual. A code-switching sentence is the mixing of two or more languages especially within the same sentence. It is crucial to work out a multilingual speech recognition via a code-switching scheme based on a parameter-efficient learning by utilizing a pretrained encoder. Using this scheme, it is essential to identify the languages within a single spoken utterance for code-switching speech recognition. However, collecting speech data from monolingual language is easier than collecting code-switching speech in multiple languages. This study develops a new code-switching Mandarin-English speech recognition by utilizing a large-scale pre-trained backbone model containing 53 single-code languages. The fine-tuning of backbone model is implemented by introducing the controllable language or task adapters and incorporating small number of Mandarin-English code-switching speech where the backbone model is frozen and only individual adapters for Mandarin and English are estimated. A limited amount of controllable parameters can be sufficiently calculated. Experiments on code-switching speech recognition for Taiwanese Mandarin and English shows the merit of the proposed method.
Incorporating Pinyin into Pipeline Named Entity Recognition from Chinese Speech

Named Entity Recognition (NER) from speech is usually implemented through a two-step pipeline that consists of (1) processing the audio using an Automatic Speech Recognition (ASR) system and (2) applying an NER tagger to the ASR output. In this paper, we incorporate pinyin <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> — spelled sounds of Chinese characters — into the pipeline NER from Chinese speech, aiming to improve the NER performance through two steps. First, we take the pretrained model ChineseBERT to embed pinyin, as well as glyph and char, as the input of NER tagger. Second, we introduce homophone noises into training data for NER tagger, as homophone errors most likely exist in ASR output for Chinese speech. Using the two-step pipeline method with pinyin incorporated into the NER tagger, the F1 score is improved by nearly 1% absolute points in the experiment on the AISHELL-NER dataset, which is a significant improvement in the field of NER. And the F1 score outperforms the current state-of-the-art (SOTA) result on the AISHELL-NER dataset by 0.4% absolute points, despite the slightly worse Character Error Rate (CER) of our ASR.
A Comparative Study on Multichannel Speaker-Attributed Automatic Speech Recognition in Multi-party Meetings

Speaker-attributed automatic speech recognition (SA-ASR) in multi-party meeting scenarios is one of the most valuable and challenging ASR tasks. It was shown that SingleChannel Frame-level Diarization with Serialized Output Training (SC-FD-SOT), Single-Channel Word-level Diarization with SOT (SC-WD-SOT), and joint training of Single-Channel Target-Speaker separation and ASR (SC-TS-ASR) can be exploited to partially solve this problem, where the latter achieves the best performance. In this paper, we propose three corresponding MultiChannel (MC) SA-ASR approaches, namely MC-FD-SOT, MC-WD-SOT, and MC-TS-ASR. For different tasks/models, different multichannel data fusion strategies are considered, including channel-level cross-channel attention for MC-FD-SOT, frame-level cross-channel attention for MC-WD-SOT and neural beamforming for MC-TS-ASR. Experimental results on the Al-iMeeting corpus reveal that our proposed multichannel SA-ASR models can consistently outperform the corresponding single-channel counterparts in terms of the speaker-dependent character error rate (SD-CER).
HDR Image Watermarking based on Saliency Detection and Quantization Index Modulation

High-dynamic range (HDR) images are circulated rapidly over the internet with risks of being exploited for unauthorized usage. To protect these images, some HDR image-based watermarking methods were put forward. However, they inherited the same problem faced by the conventional IW methods for standard dynamic range images, where only trade-offs among conflicting requirements are managed instead of simultaneous improvement. In this paper, a novel saliency (eye-catching object) detection based trade-off independent HDR-IW is proposed, to simultaneously improve robustness, imperceptibility and payload. First, the host image goes through our proposed salient object detection model to produce a saliency map, which is, in turn, exploited to segment the foreground and background of the host image. Therefore, a convolutional deep learning model is trained with various datasets containing multiple salient objects. Next, the binary watermark is partitioned into the foregrounds and backgrounds using the same mask and scrambled using a random permutation algorithm. Finally, the watermark segments are embedded into the selected bit-plane of the corresponding host segments using quantized indexed modulation. Experimental results suggest that the proposed work outperforms state-of-the-art methods in terms of concurrently improving the aforementioned conflicting requirements.
Detecting Distracted Drivers Using Convolutional Neural Networks

Driver distraction has become a major concern for road safety as distracted driving is a leading cause of car accidents, resulting in a large number of injuries and fatalities every year. This project aims to address the issue of driver distraction as a leading cause of traffic accidents worldwide. Utilizing various image datasets, the project explores the use of Convolutional Neural Networks (CNNs), specifically the DarkNet53 model, to classify whether drivers are focused on the road or distracted. A collection of publicly available datasets and a new dataset consisting of 4024 images produced by the authors were used in training the model, with image augmentation being used to significantly enhance the model performance. The training and testing processes were conducted using MATLAB and different techniques such as transfer learning and layer freezing were utilized to improve the training process. The real-time performance of the DarkNet53 model was also evaluated against other models and it was shown that the DarkNet53 had adequate performance while maintaining a high level of accuracy. Furthermore, the model misclassifications were evaluated using confusion matrices and different statistics such as the F1 score were reported. The study concludes with a recommendation for further development of datasets to train more accurate models without sacrificing real-time performance. The datasets should be generalized, including different lighting conditions and camera angles.
1M parameters are enough? A lightweight CNN-based model for medical image segmentation

Convolutional neural networks (CNNs) and Transformer-based models are being widely applied in medical image segmentation thanks to their ability to extract high-level features and capture important aspects of the image. However, there is often a trade-off between the need for high accuracy and the desire for low computational cost. A model with higher parameters can theoretically achieve better performance but also result in more computational complexity and higher memory usage, and thus is not practical to implement. In this paper, we look for a lightweight U-Net-based model which can remain the same or even achieve better performance, namely U-Lite. We designed U-Lite based on the principle of Depthwise Separable Convolution so that the model can both leverage the strength of CNNs and reduce a remarkable number of computing parameters. Specifically, we propose Axial Depthwise Convolutions with kernels 7× 7 in both the encoder and decoder to enlarge the model’s receptive field. To further improve the performance, we use several Axial Dilated Depthwise Convolutions with filters 3× 3for the bottleneck as one of our branches. Overall, U-Lite contains only 878K parameters, 35 times less than the traditional U-Net, and much more times less than other modern Transformer-based models. The proposed model cuts down a large amount of computational complexity while attaining an impressive performance on medical segmentation tasks compared to other state-of-the-art architectures. The code will be available at: https://github.com/duong-db/U-Lite.
Speech-Emotion Control for Text-to-Speech in Spoken Dialogue Systems Using Voice Conversion and x-vector Embedding

In this paper, we propose an algorithm to control both speaker individuality and emotional expressions in synthesized speech, where the most important feature is the controllability of intensity in emotional expressions. An aim of the proposed algorithm is to generate various responses including emotions in text-to-speech (TTS) for spoken dialogue systems (SDS), which results in making the system more human-like. An idea is to control emotion and its intensity in line with the user’s utterances. For example, when a user happily talks to SDS, the agent of the SDS responses with happy voice. Generally, voice quality of a user and the agent are different. Therefore, the proposed algorithm consists of two steps: (1) voice conversion to change speaker individuality including emotional expressions and (2) TTS with x-vector acting as an embedding vector to mainly control speech quality related to the intensity of emotions. Evaluation experiments are carried out using a scenario of a spoken dialogue system, where a teacher system of TTS encourages or cheers up students according to students’ utterances. The experiment results showed that TTS can successfully reproduce the emotion and its intensity that are extracted from students’ utterances, while maintaining the teacher’s speaker individuality.
Indoor Pedestrian-Following System by a Drone with Edge Computing and Neural Networks: Part 2 - Development of Tracking System and Monocular Depth Estimation

This paper is the second installment in a series on indoor drone pedestrian tracking utilizing edge computing and neural networks. Building upon the SLAM and EKF technologies introduced in Part 1, this paper introduces Monocular Depth Estimation to reduce camera costs and overall weight. The system leverages AI-driven depth information for indoor positioning and real-time human tracking. Experiments demonstrate the drone's ability to autonomously track a specific individual indoors using vision and IMU sensors. Key contributions encompass an AI-based tracking system employing YOLO v3 and a novel depth estimation approach that supersedes traditional depth cameras.
On reading and interpreting black box deep neural networks

Abstract The deep neural networks used in computer vision and in recent large language models are widely recognized as black boxes, a term that describes their complicated architectures and opaque decision-making mechanisms. This essay outlines several different strategies through which humanist researchers and critics of machine learning might better understand and interpret the class of deep learning methods known as Transformers. These strategies expose different aspects of what might be “learned” as Transformers are trained and used in the analysis of language and can help critics at least partially open the black box of machine learning. They are also especially useful for digital humanists using these models as part of a research program informed by tool criticism in which the use of computational tools is conceived of as a metainterpretive act.
Emotion Recognition in Conversations: A Survey Focusing on Context, Speaker Dependencies, and Fusion Methods

As a branch of sentiment analysis tasks, emotion recognition in conversation (ERC) aims to explore the hidden emotions of a speaker by analyzing the sentiments in utterance. In addition, emotion recognition in multimodal data from conversation includes the text of the utterance and its corresponding acoustic and visual data. By integrating features from various modalities, the emotion of utterance can be more accurately predicted. ERC research faces challenges in context construction, speaker dependency design, and multimodal heterogeneous feature fusion. Therefore, this review starts by defining the ERC task, developing the research work, and introducing the utilized datasets in detail. Simultaneously, we analyzed context modeling in conversations, speaker dependency, and methods for fusing multimodal information concerning existing research work for evaluation purposes. Finally, this review also explores the research, application challenges, and opportunities of ERC.
Central Bank Digital Currencies: a proper reaction to private digital money?

The emergence of private digital currencies poses a threat to payment systems and monetary policy because they challenge all functions of money as we know them. In this paper, we focus mainly on the banking and monetary policy issues raised by stablecoins and Central Bank Digital Currencies (CBDCs). We begin by describing the current working of bank-centered payment systems. We next touch upon cryptoassets and focus on the domestic and international impact of stablecoins. We identify two problems with stablecoins: their role in the breakup of uniform currency, and as a source of financial instability due to the lack of a monetary backstop in the event of a run to withdraw funds in adverse situations. We then deal with the pros and cons of CBDCs, whether they are an adequate response to the challenges raised by stablecoins, their possible impact on monetary and banking policy, and some open economy issues. While we do not see major advantages in private digital currencies as a payment system or as an investment, we also do not find any robust motivation for the introduction of CBDCs beyond geopolitical reasons and the oversight of a technological area.
A Federated Learning Approach to Support the Decision-Making Process for ICU Patients in a European Telemedicine Network

A result of the pandemic is an urgent need for data collaborations that empower the clinical and scientific communities in responding to rapidly evolving global challenges. The ICU4Covid project joined research institutions, medical centers, and hospitals all around Europe in a telemedicine network for sharing capabilities, knowledge, and expertise distributed within the network. However, healthcare data sharing has ethical, regulatory, and legal complexities that pose several restrictions on their access and use. To mitigate this issue, the ICU4Covid project integrates a federated learning architecture, allowing distributed machine learning within a cross-institutional healthcare system without the data being transported or exposed outside their original location. This paper presents the federated learning approach to support the decision-making process for ICU patients in a European telemedicine network. The proposed approach was applied to the early identification of high-risk hypertensive patients. Experimental results show how the knowledge of every single node is spread within the federation, improving the ability of each node to make an early prediction of high-risk hypertensive patients. Moreover, a performance evaluation shows an accuracy and precision of over 90%, confirming a good performance of the FL approach as a prediction test. The FL approach can significantly support the decision-making process for ICU patients in distributed networks of federated healthcare organizations.
Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel Benchmarks

Abstract Transformer-based Pre-Trained Language Models currently dominate the field of Natural Language Inference (NLI). We first survey existing NLI datasets, and we systematize them according to the different kinds of logical inferences that are being distinguished. This shows two gaps in the current dataset landscape, which we propose to address with one dataset that has been developed in argumentative writing research as well as a new one building on syllogistic logic. Throughout, we also explore the promises of ChatGPT. Our results show that our new datasets do pose a challenge to existing methods and models, including ChatGPT, and that tackling this challenge via fine-tuning yields only partly satisfactory results.
Augmented and virtual reality as a teaching resource to attend to the diversity of students with special educational needs: a systematic review

The irruption of Information and Communication Technologies in society in general, and immersive technology, in particular, are playing an important role in the educational field, among other reasons, due to the way they favour the inclusion of students with Special Educational Needs. The main objective was to carry out a systematic review of the scientific production of the last decade (2012–2022) in relation to the use of Augmented and Virtual Reality in students with Special Education Needs. For this, a search was carried out in the Web of Science and Scopus databases following the PRISMA statement. The main results place these technologies as an appropriate intervention resource by expanding educational possibilities and opening learning opportunities in new contexts that significantly support the development of the teaching-learning process of these students. However, this study has demonstrated, at the same time, a significant lack of studies that can cover all Special Education Needs groups in the different educational stages.
Towards participatory activities with augmented reality for cultural heritage: A literature review

The preservation of cultural heritage is a basilar to support the feeling of identity and community. In this context, cultural tourism emerges as an effective strategy that, when combined with participatory activities, may increment tourist perceptions, contributing simultaneously to the acquisition of knowledge associated with cultural heritage. This literature review aimed to explore how mobile augmented reality applications using games, narrative, or craft design and convey knowledge associated with the cultural heritage of a particular place. In addition, it was intended to understand how their evaluation protocol is administered, as well as how can this genre of applications influence the tourist experience. The literature search was operationalized in the Scopus, Web of Science and ACM Digital Library electronic databases and used studies from 2018 until November 16, 2022. In total, 16 studies were analyzed in this literature review. The results demonstrate that the integration of game, narrative, and craft in augmented reality mobile applications for cultural heritage is diverse, with narrative being the most prevalent. Regarding how the evaluation phase is conducted, despite using each study using different protocols, the data collection instruments can be patterned, highlighting the use of independent questionnaires and scales. Nevertheless, the findings of this literature review suggest that augmented reality mobile applications with participatory activities can positively influence the touristic experience, constituting a powerful element of engagement and immersion, guaranteeing that some creative and technical aspects are ensured.
Augmentation and translation crowdsourcing

Abstract This paper critically discusses how crowdsourcing relate to the emergence of the so-called translation augmentation paradigm. As a broad generalization, these two technology-driven phenomena emerged as possible solutions to achieve better, more efficient or speedier solutions to complex problems than those that single individuals can solve on their own. They both can be contextualized within 4EA approaches to cognition, reframing problem-solving towards a distributed and extended perspective that goes beyond the limitations of the individual mind. Nevertheless, augmented translation refers to the coupling of human and machines in a technology-centric approach that attempts to increase the cognitive capabilities of translators, who remain in the loop and in charge ( Lommel 2020 ; Angelone 2023 ). It relates to existing technological integrated systems beyond just the automation brought by TM and MT that learn and adapt from humans. The paper (1) interrelates distributed and extended cognitive approaches and crowdsourcing ( Risku, Pein-Weber &amp; Rogl 2016 ; Jiménez-Crespo 2017 ; Risku &amp; Windhager 2020) in relation to AI and intelligence augmentation (IA), (2) analyzes the existing fuzziness in the theoretical and conceptual constructs to translation augmentation , (3) explores issues related to the locus of control and the center of the cognitive system in augmented crowdsourcing, as well as (4) looks ahead by introducing unexplored areas such as social augmentation . In doing so, the paper will attempt to answer whether human translators in crowdsourcing initiatives are, in fact, “in charge” and – or augmented .
Bringing order into the realm of Transformer-based language models for artificial intelligence and law

Abstract Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development.
The Effect of Avatar Self-Integration on Consumers’ Behavioral Intention in the Metaverse

This study explores the relationship between users and avatars, focusing on the degree of facial resemblance and the realism of fashion items as the main factors of avatar self-integration that affect consumers' behavioral intentions. Through a contactless survey, this study empirically identified how avatar–user resemblance and fashion item realism influence consumers' future spending intention on virtual fashion items and the continuous usage intention of the metaverse by mediating psychological identification. We also demonstrate the moderating role of users' self-monitoring of the effect of avatar–user resemblance on psychological identification. We found a positive mediation role of psychological identification between avatar–user resemblance, fashion item realism, and users' consumption intentions. Additionally, a higher self-monitoring tendency strengthens the positive effect of avatar–user resemblance on psychological identification. The study establishes an academic basis for metaverse retailing by uncovering connections between predictors contributing to users' consumption intentions in the metaverse.