Reviewerly/exHarmony huggingface dataset:
Lip-Reading Advancements: A 3D Convolutional Neural Network/Long Short-Term Memory Fusion for Precise Word Recognition

Lip reading, the art of deciphering spoken words from the visual cues of lip movements, has garnered significant interest for its potential applications in diverse fields, including assistive technologies, human–computer interaction, and security systems. With the rapid advancements in technology and the increasing emphasis on non-verbal communication methods, the significance of lip reading has expanded beyond its traditional boundaries. These technological advancements have led to the generation of large-scale and complex datasets, necessitating the use of cutting-edge deep learning tools that are adept at handling such intricacies. In this study, we propose an innovative approach combining 3D Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to tackle the challenging task of word recognition from lip movements. Our research leverages a meticulously curated dataset, named MobLip, encompassing various speech patterns, speakers, and environmental conditions. The synergy between the spatial information extracted by 3D CNNs and the temporal dynamics captured by LSTMs yields impressive results, achieving an accuracy rate of up to 87.5%, showcasing robustness to lighting variations and speaker diversity. Comparative experiments demonstrate our model’s superiority over existing lip-reading approaches, underlining its potential for real-world deployment. Furthermore, we discuss ethical considerations and propose avenues for future research, such as multimodal integration with audio data and expanded language support. In conclusion, our 3D CNN-LSTM architecture presents a promising solution to the complex problem of word recognition from lip movements, contributing to the advancement of communication technology and opening doors to innovative applications in an increasingly visual world.
Automated Classification of User Needs for Beginner User Experience Designers: A Kano Model and Text Analysis Approach Using Deep Learning

This study introduces a novel tool for classifying user needs in user experience (UX) design, specifically tailored for beginners, with potential applications in education. The tool employs the Kano model, text analysis, and deep learning to classify user needs efficiently into four categories. The data for the study were collected through interviews and web crawling, yielding 19 user needs from Generation Z users (born between 1995 and 2009) of LEGO toys (Billund, Denmark). These needs were then categorized into must-be, one-dimensional, attractive, and indifferent needs through a Kano-based questionnaire survey. A dataset of over 3000 online comments was created through preprocessing and annotating, which was used to train and evaluate seven deep learning models. The most effective model, the Recurrent Convolutional Neural Network (RCNN), was employed to develop a graphical text classification tool that accurately outputs the corresponding category and probability of user input text according to the Kano model. A usability test compared the tool’s performance to the traditional affinity diagram method. The tool outperformed the affinity diagram method in six dimensions and outperformed three qualities of the User Experience Questionnaire (UEQ), indicating a superior UX. The tool also demonstrated a lower perceived workload, as measured using the NASA Task Load Index (NASA-TLX), and received a positive Net Promoter Score (NPS) of 23 from the participants. These findings underscore the potential of this tool as a valuable educational resource in UX design courses. It offers students a more efficient and engaging and less burdensome learning experience while seamlessly integrating artificial intelligence into UX design education. This study provides UX design beginners with a practical and intuitive tool, facilitating a deeper understanding of user needs and innovative design strategies.
Needs Analysis and Requirements Delivery: Guiding the Development Approach with an Intuitive Framework

Abstract The needs assessment and requirements engineering domains involve tasks that help us fully understand a project's motivation. The functions of this domain include exploring the problem or opportunity and recommending a possible solution and development approach to the problem. From a project perspective, this is a critical stage as we can understand the value a viable solution brings to the customer and identify an appropriate development approach. A business investing resources to develop a solution must be pragmatic about the product development approach. Our research paper presents a framework to aid decision-making within this context. It involves considering attributes such as delivery cadence, risk, degree of innovation, and requirements certainty. Knowledge of the impact of these attributes on project outcomes will help educate engineers to become successful entrepreneurs and product managers.
Exploring Virtual Reality as a Design Observation Training Tool for Engineering Students

Abstract Observation is an essential tool in human-centered engineering design. Observations allow engineering designers to collect rich data on design problem environments and stakeholder behaviors in ways that are not always possible with other methods, as well as allow for the identification of discrepancies in other data (e.g., verbal reports from stakeholders) (Blomberg & Burrel, 2009). Developing design observation skills presents challenges, however, as observation often involves uncontrolled environments and the management of significant levels of uncertainty, which engineering students have been shown to struggle with (Jordan, 2015). Practical observation experience may also require extra effort for students to reach observation locations outside the classroom. In addition, students may be confronted with high-pressure or sensitive observation environments like medical facilities in co-curricular and extracurricular design projects and internships, which are likely to be especially difficult to prepare for in a classroom. Virtual reality (VR) has been promoted as a promising design training tool for its ability to provide controlled, yet immersive access to scenarios that are otherwise difficult to replicate in a classroom (Özgen et al., 2021). There is mixed empirical evidence on the benefits and drawbacks of VR-based training simulations in higher education (Merchant et al., 2013), however, and no studies have characterized the use of VR for engineering design observation training. To explore the effectiveness of VR for supporting engineering design observation training, this study explored students' perceptions of a prototype VR-based design observation training tool. Four undergraduate engineering students were each interviewed twice: once after the VR training experience and again after in-person design observation practice within a clinical setting during a co-curricular design program. Approximately six hours of data were collected and analyzed inductively to characterize student perceptions of advantages, disadvantages, and effectiveness of VR as a training tool compared to conventional classroom training (i.e., lecture and in-classroom practice). We found that student participants reported VR-based design observation training to be a more engaging format than other classroom-based training formats, such as lectures or two-dimensional videos. Participants also reported that VR-based training did not fully prepare them for responding to the social queues and interactions present in in-person design observation experiences, however. Preliminary recommendations for VR-based observation training experiences and related future research are also discussed.
What makes a solar engineer?

The international response to climate change has driven growing demand for engineers trained to design solar energy systems and integrate them into electricity infrastructure.There is a need for some guidance as to what training is necessary for future engineers to meet this demand.This study gathers data from literature, industry perspectives and current educational practice to help establish suitable learning objectives for training undergraduate engineers to be prepared for solar project development.In general, results suggest that students be trained to understand the complete workflow of concepts related to design of photovoltaic systems including solar positioning, the solar resource and irradiance data sources, design of a photovoltaic system from both a solar resource and an electrical perspective and performing calculations to model or support validation of photovoltaic systems.Professionals also highlighted the importance of discussing various types of typical financing structures for solar energy systems, though these were less common in existing courses.The paper summarizes these outcomes with a proposed set of learning objectives that encompasses these highlight areas and that could form the backbone of an undergraduate course on solar energy engineering.
Exploring Magic Interactions for Collaboration in Virtual Reality Learning Factory

Abstract A hands-on curriculum that blends theory and practical skills is essential to teach manufacturing. An integral part of such a curriculum is a learning factory, which allows engineering students to experience the entire manufacturing cycle of a product in a realistic factory environment. In addition to learning the required technical skills, students can practice their collaborative skills and communication via teamwork in a learning factory. With virtual reality (VR), environments can be made using game engines that simulate their real-world equivalents, providing realistic experiences. Compared to traditional remote learning, VR-based learning together with online remote learning is experiential, allows for natural interaction, and is only limited by the capabilities of the hardware running the virtual environments. The cost of VR devices has dramatically reduced with standalone VR devices such as Meta Quest 2, making these devices a compelling option for specialized educational simulations. A VR Learning Factory should support synchronous collaboration of multiple learners in the same environment. This is a critical advantage of using VR, since collaboration is an essential skill for engineers. To maximize this benefit, it is imperative to develop an appropriate VR interaction mode, because it can greatly influence the effectiveness of collaborations. In this research, we explore multi-user interaction within the context of the VR Learning Factory and compare two modes of virtual user interaction that we call natural and magic. Magic interactions include three additional tools: object container, holographic representation, and multi-object selection. We conduct an analysis of the two modes of VR interaction in a craft production task and show increased performance of using magic interactions.
Speech Melody Research as the Interdisciplinary Foundation of the Petrograd Institute of the Living Word

The assumption of similarity between artistic speech melody and music was deeply rooted in Russian Symbolism and based on the culturally established analogy between poetry/lyrical prosody and music. This connection was the basis for a wide range of performative practices focused on performed word such as the experiments of director Vsevolod Meyerhold and composer Mikhail Gnesin in Petrograd theater studios in 1900–1910s, and the collective declamation of Vasilii Serezhnikov and Vsevolod Vsevolodskii-Gerngross. However, after the October revolution, this analogy not only inspired new artistic paths, but also new approaches in humanities. This article explores the correlation between a practice-based strategy and advanced theory that characterized the structure and curricula of the Petrograd Institute of the Living Word (Institut zhivogo slova; 1918–1924). Its specific institutional features affected the development of disciplines in the fields of linguistics, poetics, and literary studies. The earlier period of its work (1918–1921) was defined by the search for common ground, which could unite representatives of different disciplines. The study of the melody of speech, which this article is focused on, became one of the key joint research projects of the Institute’s team. It is the perspective of the Institute of the Living Word’s research projects and performance-related art practices that is used for analysis of the Russian Formalist approaches in the 1910s–20s, specifically articles and books of philologist Boris Eikhenbaum on the melody and composition of verse intonation.
Brain-driven facial image reconstruction via StyleGAN inversion with improved identity consistency

The reconstruction of visual stimuli from fMRI data represents a major technological and scientific challenge at the forefront of contemporary neuroscience research. Deep learning techniques have played a critical role in advancing decoding models for visual stimulus reconstruction from fMRI data. Particularly, the use of advanced GANs has resulted in significant improvements in the quality of image generation, providing a powerful tool for addressing the challenges of this complex task. However, none of these studies have taken into account the inherent characteristics of the stimulus contents themselves; This, in turn, leads to unsatisfactory outcomes, as demonstrated by the inconsistent identity between reconstructed faces and ground truth in the decoding of facial images. In order to tackle this challenge, we introduce a new framework aimed at enhancing the accuracy of reconstructing facial images from fMRI data. Our key innovation involves extracting and disentangling multi-level visual information from brain signals in the latent space and optimizing high-level features for facial identity control using identity loss. Specifically, our framework uses StyleGAN inversion to extract hierarchical latent codes from images, which are then bridged to fMRI data through transformation blocks. Additionally, we introduce a multi-stage refinement method to enhance the accuracy of reconstructed faces, which involves progressively updating fMRI latent codes with custom loss functions designed for both feature- and image-wise optimization. Our experimental results demonstrate that our proposed framework effectively achieves two critical objectives: (1) accurate facial image reconstruction from fMRI data and (2) preservation of identity characteristics with a high level of consistency.
An Abnormal External Link Detection Algorithm Based on Multi-Modal Fusion

Website link detection is an important means to ensure the security of the external chain. In the past, it was mainly realized through blacklisting and feature engineering-based machine learning, which has the problems of slow detection speed and weak model generalization ability. The development of neural networks has brought a new solution to the security detection of the external chain of the website. To address the performance bottleneck caused by the variable content length of web pages, this article introduces an innovative approach: a website external link security detection algorithm based on multi-modal fusion. It extracts text, dynamic script, and image features separately, and constructs a deep fusion model that combines these multi-modal features. Compared with the previous research results, the proposed method is superior to the traditional single-mode method, and can quickly and accurately identify malicious web pages. The accuracy and F1 value are improved by 2.7% and 0.026.
Building bonds: an examination of relational bonding in continuous content contribution behaviors on metaverse-based non-fungible token platforms

Purpose The proliferation of non-fungible token (NFT)-based crypto-art platforms has transformed how creators manage, own and earn money through the creation, assets and identity of their digital works. Despite this, no studies have examined the drivers of continuous content contribution behavior (CCCB) toward NFTs. Hence, this study draws on the theory of relational bonds to examine how various relational bonds affect feelings of psychological ownership, which, in turn, affects CCCB on metaverse platforms. Design/methodology/approach Using structural equation modeling and importance-performance matrix analysis, an online survey of 434 content creators from prominent NFT platforms empirically validated the research hypotheses. Findings Financial, structural, and social bonds positively affect psychological ownership, which in turn encourages CCCBs. The results of the importance-performance matrix analysis reveal that male content creators prioritized virtual reputation and social enhancement, whereas female content creators prioritized personalization and monetary gains. Originality/value We examine Web 3.0 and the NFT creators’ network that characterizes the governance practices of the metaverse. Consequently, the findings facilitate a better understanding of creator economy and meta-verse commerce.
Quantum steering and coherence evolution of two atoms under noisy environments

By considering two kinds of non-Markovian (NM) environments, the dynamical behavior of quantum steering, Bell nonlocality and coherence of a system of two atoms initially immersed in a vacuum has been studied. In this context, the cases of a cavity little off-resonance with the atomic transition frequency and non-perfect photonic band-gap (NPPBG) has been considered. The control and preservation of the quantumness measures is discussed in terms of the physical parameters. Furthermore, the situations for which the atoms state is steerable with the presence of Bell nonlocality and coherence has been illustrated. Finally, the monotonic dependence among the quantumness measures is displayed with respect to the model parameters.
On a discrete symmetric optimal associated kernel for estimating count data distributions

The nonparametric estimator using discrete kernels is one competing alternative to the frequency estimator. We investigate a discrete symmetric “optimal” kernel. Its properties are established and studied by simulations. An application on real count data is given.
Y-GAN: Learning dual data representations for anomaly detection in images

We propose a novel reconstruction-based model for anomaly detection in image data, called’Y-GAN’. The model consists of a Y-shaped auto-encoder and represents images in two separate latent spaces. The first captures meaningful image semantics, which are key for representing (normal) training data, whereas the second encodes low-level residual image characteristics. To ensure the dual representations encode mutually exclusive information, a disentanglement procedure is designed around a latent (proxy) classifier. Additionally, a novel representation-consistency mechanism is proposed to prevent information leakage between the latent spaces. The model is trained in a one-class learning setting using only normal training data. Due to the separation of semantically-relevant and residual information, Y-GAN is able to derive informative data representations that allow for efficacious anomaly detection across a diverse set of anomaly detection tasks. The model is evaluated in comprehensive experiments with several recent anomaly detection models using four popular image datasets, i.e., MNIST, FMNIST, CIFAR10, and PlantVillage. Experimental results show that Y-GAN outperforms all tested models by a considerable margin and yields state-of-the-art results. The source code for the model is made publicly available at https://github.com/MIvanovska/Y-GAN.
Self-Distillation and Pinyin Character Prediction for Chinese Spelling Correction Based on Multimodality

Chinese spelling correction (CSC) constitutes a pivotal and enduring goal in natural language processing, serving as a foundational element for various language-related tasks by detecting and rectifying spelling errors in textual content. Numerous methods for Chinese spelling correction leverage multimodal information, including character, character sound, and character shape, to establish connections between incorrect and correct characters. Research indicates that a majority of spelling errors stem from pinyin similarity, with character similarity accounting for half of the errors. Consequently, effectively modeling character pinyin and character relationships emerges as a key challenge in the CSC task. In this study, we propose enhancing the CSC task by introducing the pinyin character prediction task. We employ an adaptive weighting method in the pinyin character prediction task to address predictions in a more granular manner, achieving a balance between the two prediction tasks. The proposed model, SPMSpell, utilizes ChineseBERT as an encoder to capture multimodal feature information simultaneously. It incorporates three parallel decoders for character prediction, pinyin prediction, and self-distillation modules. To mitigate potential overfitting concerning pinyin, a self-distillation method is introduced to prioritize character information in predictions. Extensive experiments conducted on three SIGHAN benchmark tests showcase that the model introduced in this paper attains a superior level of performance. This substantiates the correctness and superiority of the adaptive weighted pinyin character prediction task and underscores the effectiveness of the self-distillation module.
GPONet: A Two-Stream Gated Progressive Optimization Network for Salient Object Detection

The salient object detection task is to locate and detect salient regions in images, which is widely applied in various fields. In this paper, we propose a gated progressive optimization network (GPONet) for salient object detection. Firstly, to extract salient regions more accurately, we design a multi-level feature fusion module with a gating mechanism named gate fusion network (GFN). GFN focuses on the semantic information of high-level features as well as the detailed information of low-level features, enabling purposeful delivery of high-level features to low-level features. The gate fusion unit (GFU) is also able to maintain valid information and suppress redundant information in the fusion process. Secondly, while some existing methods have shown that the additional edge supervision can facilitate salient object detection, edge pixels are often much less common than non-edge pixels, leading to the challenge of class imbalance. To overcome this issue, we introduce detail labels that provide additional internal details as a supplementary supervisory signal. Combining these labels with proposed Detail Perception Loss enables our network to learn edge information of salient objects more effectively. To complement each other and guide information exchange between the two branches, we propose a cross guide module (CGM) to control the information flow transfer between them. Finally, we develop a simple and efficient attention fusion strategy to merge the prediction maps of the two branches to generate the final salient prediction map. Extensive experimental results validate that our method reaches optimal or comparable performance on several mainstream datasets. The code of GPONet is available from https://github.com/antonie-z/GPONet