Reviewerly/exHarmony huggingface dataset:
Crowd Counting in Harsh Weather using Image Denoising with Pix2Pix GANs

Visual crowd counting estimates the density of the crowd using deep learning models such as convolution neural networks (CNNs). The performance of the model heavily relies on the quality of the training data that constitutes crowd images. In harsh weather such as fog, dust, and low light conditions, the inference performance may severely degrade on the noisy and blur images. In this paper, we propose the use of Pix2Pix generative adversarial network (GAN) to first denoise the crowd images prior to passing them to the counting model. A Pix2Pix network is trained using synthetic noisy images generated from original crowd images and then the pretrained generator is then used in the inference engine to estimate the crowd density in unseen, noisy crowd images. The performance is tested on JHU-Crowd dataset to validate the significance of the proposed method particularly when high reliability and accuracy are required.
Sequential Image Storytelling Model Based on Transformer Attention Pooling

The Visual Storytelling Task (VST) extends beyond describing a single image, such as image captioning, to sequential image descriptions in the form of a coherent story. However, such descriptions present challenges in handling varying language styles, relational role-modeling, consistency, and events not evident in individual images. A common limitation in existing approaches is their inability to fully describe relations and visual changes, leading to stories devoid of linguistic cohesion between multiple sentences. To address this challenge, we introduce a novel framework, Sequential Image Storytelling Model (SISM), based on the Transformer architecture. Our model contextualizes input images by dividing them into multiple 16 × 16 patches and associates them with language content using the encoder-decoder technique. It incorporates cross-attention and attention pooling to identify the most relevant relationships between input images. Our proposed model achieves state-of-the-art performance on the recently published SSID dataset and performs competitively on VIST dataset, achieving the top score on BLEU-1 metric.
Unsupervised End-to-End Transformer based approach for Video Anomaly Detection

Anomaly detection in videos is a challenging task due to multiple constraints including the imbalanced nature of anomalies, limited annotated data, and limitation of existing supervised or semi-supervised algorithms to learn in such imbalanced situations. To overcome these issues, Transformer-based unsupervised learning may be a promising solution. In this paper, we proposed a generative transfer learning (GTL) algorithm for video anomaly detection using an unsupervised learning approach. This framework consists of three major parts: (1) a feature extractor; (2) a generator transformer; and (3) a discriminator transformer. The feature extractor generates spatio-temporal features from unlabeled input segments, and passes them to the generator transformer which tries to reconstruct these features. Using cooperative learning between the generator transformer and the discriminator transformer, we train our network so that anomalies have high reconstruction error and non-anomalies do not. We test our proposed model on two well-known anomaly detection datasets (UCF-Crime and ShangaiTech) and report state-of-the-art.
Auditory Temporal Hints in AR Piano Tutoring

Learning to play a musical instrument can be challenging for beginners, requiring ample practice and coordination. Previous studies have investigated the use of screens, projectors, and Augmented Reality (AR) headsets to provide visual feedback to piano students and several authors showed that such tools can improve learning. However, little work has been done investigating auditory aids and we are not aware of any analysis of their effect in combination with AR piano tutoring tools. In this research we use a virtual metronome to convey rhythm and we investigate its effect when combined with an AR piano roll. We performed a user study measuring users’ piano performance, cognitive load, and preferences when using an AR piano tutoring tool with and without a virtual metronome. Our results found no significant differences for the entire cohort and users with no piano experience. However, we found evidence that the virtual metronome helps to decrease rhythm errors and cognitive load for users with at least some piano experience. We suggest that the virtual metronome is most useful when used in a staged approach after introducing visual aids.
Neural Network Feature Explanation Using Neuron Activation Rate Based Bipartite Graph

Deep Neural Networks (DNNs) are popular machine learning models that have gained popularity due to its good predictive accuracy and ability to automatically learn features from raw data. Convolutional Neural Networks (CNNs) are one such models that have gained popularity in the field of Computer Vision (CV). Despite the popularity, these models are notoriously black-box models. The decisions made by these models are not explainable. In this paper we propose a method to create a Neuron Activation Rate based Bipartite Graph (NARBG) , that can explain the decisions made by the model, based on the contributions of class specific features. In the proposed method, the features are extracted from the raw data using a CNN based architecture. From the extracted features, neuron activation rate is calculated. Based on these neuron activation rates, influential features for the target class prediction are identified. Then a bipartite graph named NARBG is trained using these influential features. The predictions of NARBG can be explained based on the features and the path in the graph that got activated for a given target class prediction. The proposed method performs on par with the other state-of-the-art methods in terms of accuracy.
Shape from Shading under changing Illumination using a CNN

This paper presents an alternative method to the classical yet foundational problem of recovering the geometry of a 3D object from a single 2D brightness image, i.e., shape-from-shading. Drawing inspiration from traditional approaches and recent work in single-image 3D reconstruction, we employ a convolutional neural network trained on synthetic images of characteristic shapes as well as high-frequency surfaces to address the shape-from-shading problem. Although the resulting network is less accurate than non-deep learning (DL) approaches, it exhibits greater resilience to common pitfalls, such as variations in light source direction and the presence of noise in the input image.
Fusing exocentric and egocentric real-time reconstructions for embodied immersive experiences

Experiencing one’s own body in virtual and mixed reality can enhance applications such as 3D teleconferencing, physical and psychological rehabilitation, and natural 3D user interfaces. Embodied experiences require a dynamic virtual body to represent the user. Typical virtual bodies consist of rigged mesh models which are animated using expensive and cumbersome motion capture systems, or heavy reliance on models of human movement. Models of human appearance and movement are notoriously susceptible to undesirable and "uncanny" appearances, and are often unconvincing as a result. An alternative is to reconstruct the user in real time without relying on a motion capture system or on visual or movement models. With this approach the appearance of the virtual body and its motion are inherently natural by virtue of being directly captured, and embodied experiences are possible even at low levels of detail. However, the necessarily sparse arrangement of reconstruction cameras often produces incomplete virtual bodies. This is partly due to the disparity between the egocentric (first-person) view of the virtual body and the typically exocentric (third-person) perspectives of the reconstruction cameras. In this paper we present a method for reconstructing a more complete view of the user with a minimal number of cameras by combining head-worn egocentric with exocentric depth-sensing cameras, with a focus on the first-person view of the virtual body. We describe our approach to producing the virtual body, including camera registration methods and key technical performance metrics. We also provide insights from a user study with 26 participants indicating that our approach has the potential to increase the sense of embodiment and the perception of the completeness of the virtual body.
NRspttemVQA: Real-Time Video Quality Assessment Based on the User’s Visual Perception

There is a strong need for non-reference video quality metrics for user-generated video content to prevent loss of video quality caused by distortion during recording, compression, and signal transmission. Here we contribute to advancing the issue of streaming quality by creating a large-scale dataset with video compression and transmission artefacts. Our final dataset consists of 4.1 million video quality perceptual thresholds by users. We also created a new first non-reference video quality metric that includes the psychophysical features of the user’s video experience, which provides stability in predicting the user’s subjective rating of a video. Our experimental results show that the proposed video quality metric achieves the most stable performance on three independent video datasets. We believe our study will expand further research into deep learning-based video quality metrics modelling.
Identifying Athletics Tracks using Keypoint Detection

This paper proposes the first known approaches to locating an athletics track from GPS data of a running workout. This is done by mapping the GPS points onto an image and applying Computer Vision methods, including: performing keypoint detection using a Keypoint Region-Based Convolutional Neural Network (R-CNN) model with a Residual Neural Network backbone; and applying Convolutional Kernels over the image. The Keypoint R-CNN model achieved a Root Mean Square Error (RMSE) of 1.29 on synthetic data, which is lower than the error of other keypoint detection models of 2.23, but performed poorly on real data. The Convolutional Method achieved an RMSE of 4.20. The methods and results of this paper can be used in future applications for error correction and feature extraction of GPS data for running workout analysis.
ACENet: Attention-Driven Contextual Features-Enhanced Lightweight EfficientNet for 2D Hand Pose Estimation

Computer Vision (CV) has seen remarkable advancements recently, but traditional input devices remain outdated, especially considering the critical role of hand movements in human interactions. This has prompted intensified research in Hand Pose Estimation (HPE) for applications ranging from gesture-controlled car systems to virtual reality. Despite deep learning’s success in HPE, challenges like hand flexibility, occlusion, and size variability persist. As the emphasis shifts towards RGB-based 3D pose estimation over pricier multi-view camera systems, we proposed a 2D HPE approach called ACENet: Attention-Driven Contextual Features-Enhanced Lightweight EfficientNet. The framework employs the EfficientNet backbone integrated with a Squeeze-and-Excitation (SE) block for feature extraction and is enhanced by a Global Context (GC) block. It effectively predicts complex hand poses from single RGB images, emphasizing crucial features for enhanced representation. Tests on the Carnegie Mellon University (CMU) Panoptic dataset showed our model’s average accuracy increases by 2.11% with respect to OCPM, underscoring its significant potential across various sectors.
Thinking through the tool: collaborative archaeological bodywork in immersive virtual reality

Thanks to currently available very high-resolution three-dimensional (3D) models via photogrammetric techniques as a primary method of archaeological documentation, constructing immersive, high-fidelity simulacra is imminently possible. This paper considers how the scale at which the human body interacts with immersive digital models is especially important for understanding the affordances and ergonomics of past things and places. The implications of this isometry between archaeological objects of analysis and emerging capabilities to interact with them through digital surrogates in the present are manifold. By enabling interaction with objects and contexts in immersive virtual space, such observational experiences create in silico engagements that are repeatable, distributable, and collaborative. In particular, it is the collaborative capacity of this technology that this paper explores using online immersive virtual reality (iVR). Collaborative online iVR is used in this research as a key instrument for enhancing understanding and reinterpreting the digital records of two archaeological sites under excavation in Peru. The case studies analyzed show a variety of cultural, geographic, and temporal contexts in the Andean region, which illustrates the broad potential of iVR for archaeological hermeneutics. Through iVR frameworks, the authors engage with embodied reconsiderations of Catholic ritual spaces within a planned colonial town in the southern Peruvian highlands and the pre-Columbian site of Huaca Colorada on the north coast. Synchronous scalar experiences that privilege the affordances of architectural space within digital models create opportunities for embodied experience and collaborative dialogue. A fundamental argument is the capacity to digitally inhabit these places and manipulate materials holds subtle as well as profound epistemological and hermeneutic implications for archaeological knowledge construction.
Thai-English Neural Machine Translation Method with Local and Global Syllable Feature

Neural Machine Translation (NMT) has superseded Statistical Machine Translation (SMT) owing to the advent of deep learning in natural language processing. However, enhancing the quality of low-resource NMT involves more nuanced strategies beyond data-dependent methods, especially for languages like Thai that lack substantial parallel corpus data for Thai-English translation. Addressing this problem, we introduce a Thai-English NMT methodology that amalgamates local and global syllabic features. Our approach employs a CNN-based module for extracting local syllable characteristics and capturing interdependencies among Thai syllable characters. Concurrently, a global syllable feature extraction mechanism is used to preserve sentence-level syllable traits. The integration of these features significantly augments Thai syllable context representation. In experiments conducted on the Thai-English corpus from the AL T dataset, our method improved the BLEU score by 2.11 points above the baseline, achieving a score of 12.99. This research holds potential implications for the translation of other syllable-based, low-resource languages, as well.
Towards the Creation of the Filipino Wordnet: A Two-Way Approach

As databases of lexical information on words and their lexical relationships, WordNets are important for various downstream natural language processing applications. However, the construction of WordNets can be challenging, especially for low-resource languages such as Filipino. The existing Filipino WordNet has not been maintained, and lacks contextual information for identifying the evolution of word senses. In this study, we built a corpus of 5,370,667 unique tokens and used it to construct a Filipino WordNet via a two-way approach that combines natural language processing and network science. For the natural language processing approach, we utilized only two linguistic sources: our corpus and a RoBERTa-based language model that generates sentence embeddings. For the network science approach, we created a temporal-multiplex network that represents the co-occurrence of words, their semantic relationships, and their usage in different sources across time. We show that our proposed method can induce existing senses (30% of our validation data, as evaluated by matching with the senses from Princeton WordNet) and generate 9,549 semantic sets.
A Blockchain-Enhanced Self-Sovereign Identity Platform for Corporate Resource Security

In an era dominated by concerns of data breaches, and identity theft, security of corporate resources and assets has become paramount. Centralized identity management systems traditionally present vulnerabilities that can fundamentally threaten corporate security. This paper introduces a novel platform to identity management in organizations, leveraging the principles of Self-Sovereign Identity (SSI) and the technological robustness of block- chain. By giving individuals unwavering control over their digital identities and reducing dependence on centralized intermediaries, SSI provides a transformative advancement in security and privacy. When combined with blockchain's immutable, decentralized, and transparent nature, this model ensures a verifiable, tamper-proof, and holistic iden- tity management system. Beyond individual identity man- agement, this paradigm provides corporations with a ro- bust mechanism to protect their assets, both digital and physical. We explore the architectural design and benefits of implementing the proposed system, BlockSSI-CRS, emphasizing its transformative potential for corporate resource protection. Through rigorous analysis, this paper highlights the feasibility of a blockchain-enhanced SSI platform in the context of corporate security needs.
Enhancing Kinship Verification through Multiscale Retinex and Combined Deep-Shallow features

The challenge of kinship verification from facial images represents a cutting-edge and formidable frontier in the realms of pattern recognition and computer vision. This area of study holds a myriad of potential applications, spanning from image annotation and forensic analysis to social media research. Our research stands out by integrating a preprocessing method named Multiscale Retinex (MSR), which elevates image quality and amplifies contrast, ultimately bolstering the end results. Strategically, our methodology capitalizes on the harmonious blend of deep and shallow texture descriptors, merging them proficiently at the score level through the Logistic Regression (LR) method. To elucidate, we employ the Local Phase Quantization (LPQ) descriptor to extract shallow texture characteristics. For deep feature extraction, we turn to the prowess of the VGG16 model, which is pre-trained on a convolutional neural network (CNN). The robustness and efficacy of our method have been put to the test through meticulous experiments on three rigorous kinship datasets, namely: Cornell Kin Face, UB Kin Face, and TS Kin Face.
A Study of Project Description Inference Using Method Name Elements for Software Upcycling

In software development, there is often a situation where products developed in the past could be managed as more valuable assets. Conversely, such products may contain the reusable value. We are currently exploring utilizing existing project assets for new development, a concept we define as "software upcycling". To facilitate upcycling, the challenge lies in comprehending the software overview without documentation and making it easily referenceable. In a previous study, we proposed a methodology for inferring the system overview using a project corpus. We collected constituent words from class names and conducted a validation study to assess their utility in inferring the system’s overview. In this research, we shift our focus to method names in order to further enhance accuracy. We attempt to understand the software’s architecture efficiently by assigning weighted importance to words in the project corpus and visualizing them through Tag cloud images. We apply our proposed methodology to 136 Java projects managed within our laboratory and evaluate its effectiveness from two perspectives: the functions provided by the software and the technologies that enable the software to operate.
LLM-based Attack Scenarios Generator with IT Asset Management and Vulnerability Information

As businesses become more dependent on IT due to digital transformation, a variety of attackers are targeting companies, government agencies, and individuals to steal information and disrupt services. To reduce the risks these cyber threats pose, penetration testing and red teaming are important. On the other hand, these initiatives require skills and knowledge, and there is a shortage of human resources. This research aims to demonstrate the effectiveness of a system that inputs asset management data and vulnerability information into ChatGPT and searches for attack routes with a high threat level. Specifically, ChatGPT uses information used for IT asset management (OS type, version, device usage, account), vulnerability information published by CISA, and network information as input values to verify whether it is possible to output attack routes that are useful for penetration testing and red teaming. The results of the experiment confirmed that attack vectors for penetration testing and red teaming could be used to effectively uncover cybersecurity threats within an organization and perform risk assessments.
Human Action Recognition using Attention EfficientNet

The integration of Human Activity Recognition (HAR) within automated residences and surveillance systems has significantly elevated the significance of this field for advanced investigation. HAR entails the deployment of artificial intelligence (AI)-driven algorithms to autonomously discern human actions. Given the remarkable potency of deep learning (DL) methodologies in deciphering intricate tasks, their supremacy over conventional machine learning (ML) techniques has rendered DL-driven HAR increasingly prevalent. This study centers on the application of the EfficientNet transfer learning architecture with attention mechanism to the realm of HAR image recognition, specifically utilizing skeleton-based representations. The objective was to discern and evaluate the architecture’s performance attributes in this context. Notably, our results unveiled an impressive classification accuracy of 99% for EfficientNet. These findings underscore the efficacy of the EfficientNet model in enhancing HAR capabilities, showcasing its potential for pivotal roles in automated settings.
IMU-based Human Activity Recognition using Machine Learning and Deep Learning models

Research into Human Activity Recognition (HAR) with wearable sensors is attracting a lot of attention due to its wide range of applications. In this paper, we use a microcontroller with an integrated Inertial Measurement Unit (IMU) to design various Machine Learning (ML) and Deep Learning (DL) models. Five different activities were captured: walking, walking up a staircase, walking down a staircase, jumping, and falling. Both ML and DL models were trained on a variety of IMU data. A comparison was performed among the models based on their accuracy scores and confusion matrices to determine the most effective one. The proposed LSTM model achieved an accuracy score of 99% when trained with accelerometer and gyroscope data from the IMU.