Reviewerly/exHarmony huggingface dataset:
Quantifying the Impact of XR Visual Guidance on User Performance Using a Large-Scale Virtual Assembly Experiment

The combination of Visual Guidance and Extended Reality (XR) technology holds the potential to greatly improve the performance of human workforces in numerous areas, particularly industrial environments. Focusing on virtual assembly tasks and making use of different forms of supportive visualisations, this study investigates the potential of XR Visual Guidance. Set in a web-based immersive environment, our results draw from a heterogeneous pool of 199 participants. This research is designed to significantly differ from previous exploratory studies, which yielded conflicting results on user performance and associated human factors. Our results clearly show the advantages of XR Visual Guidance based on an over 50% reduction in task completion times and mistakes made; this may further be enhanced and refined using specific frameworks and other forms of visualisations/Visual Guidance. Discussing the role of other factors, such as cognitive load, motivation, and usability, this paper also seeks to provide concrete avenues for future research and practical takeaways for practitioners.
Persona-Coded Poly-Encoder: Persona-Guided Multi-Stream Conversational Sentence Scoring

Recent advances in machine learning and deep learning have led to the widespread use of Conversational AI in many practical applications. However, it is still very challenging to leverage auxiliary information that can provide conversational context or personalized tuning to improve the quality of conversations. For example, there has only been limited research on using an individual’s persona information to improve conversation quality, and even state-of-the-art conversational AI techniques are unable to effectively leverage signals from heterogeneous sources of auxiliary data, such as multi-modal interaction data, demographics, SDOH data, etc. In this paper, we present a novel Persona-Coded Poly-Encoder method that leverages persona information in a multi-stream encoding scheme to improve the quality of response generation for conversations. To show the efficacy of the proposed method, we evaluate our method on two different persona-based conversational datasets, and compared against two state-of-the-art methods. Our experimental results and analysis demonstrate that our method can improve conversation quality over the baseline method Poly-Encoder by $ 3.32\%$ and $ 2.94\%$ in terms of BLEU score and HR@1, respectively. More significantly, our method offers a path to better utilization of multi-modal data in conversational tasks. Lastly, our study outlines several challenges and future research directions for advancing personalized conversational AI technology.
Scene Retrieval in Traffic Videos with Contrastive Multimodal Learning

Retrieval of scenes from traffic videos is an important task in intelligent transportation systems (ITS) for efficient traffic management in AI smart cities. This work proposes natural language-based vehicle retrieval from traffic monitoring videos, emphasizing the significance of temporal information and context. We present contrastive learning as a technique to optimize joint representations of vision and language modalities within a shared latent representation space. The approach involves training contrastive losses to keep similar encodings closer in joint feature representation space by minimizing the distance between positive visual-text pairs and maximizing the distance between negative visual-text pairs. Our study employs state-of-the-art vision models for visual encoding and transformer-based language models for text encoding. We analyze the impact of feature selection from visuals and text on retrieval performance. We evaluate the efficacy of our proposed method on the AI City Challenge 2022 dataset for natural language-based vehicle retrieval, achieving performance accuracy of 49.84% Mean Reciprocal Rank (MRR) on the test dataset, securing second position on the leader board. Our approach highlights the effectiveness of feature selection and contrastive learning for enhancing multimodal retrieval tasks.
Multi-Modal Sarcasm Detection Based on Cross-Modal Composition of Inscribed Entity Relations

Sarcasm, a linguistic technique employed to express emotions opposite to their literal meaning, has garnered significant attention from researchers due to the rise of social media. Detecting sarcasm in a multi-modal context has become a focal point in recent studies. However, existing research primarily relies on identifying inconsistencies between text semantics and image semantics, often lacking a deep understanding of images. Consequently, capturing inconsistencies between images and texts poses a challenge in many cases. In this paper, we propose the Entity-Relational Graph Convolutional Network (ERGCN) as a solution to detect sarcasm by examining the relationship between entities within images. Our approach involves extracting entities and text descriptions from each image, which provides valuable entity information. Subsequently, we employ external knowledge to construct a cross-modal graph for each text and image pair, emphasizing the presence of internal contradictory information. Finally, we utilize the graph convolutional network to identify inconsistent information across modalities and successfully detect sarcasm. Experimental results demonstrate that our model achieves state-of-the-art performance on a widely used multimodal Twitter dataset.
UDA-HOID: Unsupervised Domain Adaptation for Human-Object Interaction Detection

Human-object interaction (HOI) detection has witnessed remarkable advancements in recent years, primarily driven by deep learning networks and the availability of large-scale HOI datasets. However, there are still scenarios where HOI detection lacks a suitable training database. The process of expanding existing datasets by manually annotating more images can be challenging because it is time-consuming and labor-intensive. In response to this challenge, domain adaptation has emerged as a promising approach to address the scarcity of annotated data. Drawing inspiration from this, we propose an innovative unsupervised adaptive approach for HOI detection called UDA-HOID. Our method aims to adapt HOI detection from a label-rich source domain to a label-poor target domain, thereby reducing annotation costs and enhancing detection performance. Specifically, UDA-HOID leverages attention-based full alignment for low-level features, which tackles domain shifts caused by image style, illumination, and other factors. Additionally, semantic-based weak alignment is applied to high-level features, as domain shifts in these features contain more semantic information. Adversarial learning techniques are employed to facilitate this alignment process. To evaluate the effectiveness of our proposed method, we conduct experiments on the test data of the low-light HOI image set (LLHOI). The results demonstrate that our approach achieves a relative improvement of 9.8 percent in the mean average precision of roles compared to existing methods.
Can We Trust the Default Vulnerabilities Severity?

As software systems become increasingly complex and interconnected, the risk of security debt has risen significantly, increasing cyber-attacks and data breaches. Vulnerability prioritization is a critical activity in software engineering as it helps identify and address security vulnerabilities in software systems promptly and effectively. With the increasing complexity of software systems and the growing number of potential threats, it is essential to have a systematic approach to vulnerability prioritization to ensure that the most critical vulnerabilities are addressed first. The present study aims to investigate the agreement between the default and the National Vulnerability Database (NVD) severity levels. We analyzed 1626 vulnerabilities encompassing 12 unique types of vulnerabilities associated with 125 Common Platform Enumeration identifiers belonging to 105 Apache projects. Our results show a scarce correlation between the default and NVD severity levels. Thus, the default severity of vulnerabilities is not trustworthy. Moreover, we discovered that, surprisingly, the same type of vulnerability has several NVD severity; therefore, no default prioritization can be accurate based only on the type of vulnerability. Future studies are needed to accurately estimate the priority of vulnerabilities by considering several aspects of vulnerabilities rather than only the type.
Calibrating Deep Learning-based Code Smell Detection using Human Feedback

Code smells are inherently subjective in nature. Software developers may have different opinions and perspectives on smelly code. While many attempts have been made to use deep learning-based models for code smell detection, they fail to consider each developer’s subjective perspective while detecting smells. Ignoring this aspect defies the purpose of using deep learning-based smell detection methods because the models are not customized to the developer’s context. This paper proposes a method that considers human feedback to account for such subjectivity. Towards this, we created a plugin for IntelliJ IDEA and developed a container-based web-server to offer services of our baseline deep learning model. The setup allowed developers to see code smells within the IDE and provide feedback. Using this setup, we conducted a controlled experiment with 14 participants divided into experimental and control groups. In the first round of our experiment, we show code smells predicted using the baseline deep learning model and collect feedback from the participants. In the second round, we fine-tune the model based on the experimental group’s feedback and reevaluate its performance before and after adjustment. Our results show that using such calibration improves the performance of the smell detection model by 15.49% in F1 score on average across the participants of the experimental group. Our work carries implications for both researchers and practitioners. Practitioners can apply our approach to enhance the quality of their code in day-to-day development activities, aligning it with their own code smell definitions. Furthermore, software engineering researchers can leverage this study to adopt analogous approaches for addressing similar issues, including code review.
GeneticFlow: Exploring Scholar Impact with Interactive Visualization

Visualizing a scholar’s scientific impact is important for many challenging tasks in academia such as tenure evaluation and award selection. Existing visualization and profiling approaches do not focus on the analysis of individual scholar’s impact, or they are too abstract to provide detailed interpretation of high-impact scholars. This work builds over a new scholar-centric impact-oriented profiling method called GeneticFlow. We propose a visualization design of scholar’s self-citation graphs using a time-dependent, hierarchical representation method. The graph visualization is augmented with color-coded topic information trained with cutting-edge deep learning techniques, and also temporal trend chart to illustrate the dynamics of topic/impact evolution. The visualization method is validated on a benchmark dataset established for the visualization field. Visualization results reveal key patterns of high-impact scholars and also demonstrate its capability to serve ordinary researchers for their impact visualization task.
Image super resolution via multi-regularization combining hybrid Tikhonov-TV prior and deep denoiser prior

In a real scenario, the image is often corrupted by complex degradation, and a lot of useful information is lost, which makes super-resolution (SR) reconstruction seriously ill-posed. To effectively solve such a problem, it is crucial to correctly exploit image prior knowledge. Although existing deep learning-based methods can obtain excellent results, they cannot deal with the complex degradation effectively, which would lead to the loss of texture details and the destruction of edge details. In this paper, an efficient multi-regularization method for SR is proposed, which can simultaneously exploit both internal and external image priors within a unified framework. The hybrid Tikhonov-TV prior and deep denoiser prior are introduced to constrain the reconstruction process. That is, the proposed model combines the superiority of the piecewise-smooth prior and deep prior. Moreover, an adaptive weight parameter is employed to make the hybrid component more detail-preserving. Experimental demonstrate that the proposed method achieves better performance in image detail protection than advanced methods.
Aligning Speakers: Evaluating and Visualizing Text-based Speaker Diarization Using Efficient Multiple Sequence Alignment

This paper presents a novel evaluation approach to text-based speaker diarization (SD), tackling the limitations of traditional metrics that do not account for any contextual information in text. Two new metrics are proposed, Text-based Diarization Error Rate and Diarization F1, which perform utterance- and word-level evaluations by aligning tokens in reference and hypothesis transcripts. Our metrics encompass more types of errors compared to existing ones, allowing us to make a more comprehensive analysis in SD. To align tokens, a multiple sequence alignment algorithm is introduced that supports multiple sequences in the reference while handling high-dimensional alignment to the hypothesis using dynamic programming. Our work is packaged into two tools, align4d, providing an API for our alignment algorithm and TranscribeView for visualizing and evaluating SD errors, which can greatly aid in the creation of high-quality data, fostering the advancement of dialogue systems.
Explaining Transformer-based Code Models: What Do They Learn? When They Do Not Work?

In recent years, there has been a wide interest in designing deep neural network-based models that automate downstream software engineering tasks on source code, such as code document generation, code search, and program repair. Although the main objective of these studies is to improve the effectiveness of the downstream task, many studies only attempt to employ the next best neural network model, without a proper in-depth analysis of why a particular solution works or does not, on particular tasks or scenarios. In this paper, using an example eXplainable AI (XAI) method (attention mechanism), we study two recent large language models (LLMs) for code (CodeBERT and GraphCodeBERT) on a set of software engineering downstream tasks: code document generation (CDG), code refinement (CR), and code translation (CT). Through quantitative and qualitative studies, we identify what CodeBERT and GraphCodeBERT learn (put the highest attention on, in terms of source code token types), on these tasks. We also show some of the common patterns when the model does not work as expected (performs poorly even on easy problems) and suggest recommendations that may alleviate the observed challenges.
Effects of data distribution and granularity on color semantics for colormap data visualizations

To create effective data visualizations, it helps to represent data using visual features in intuitive ways. When visualization designs match observer expectations, visualizations are easier to interpret. Prior work suggests that several factors influence such expectations. For example, the dark-is-more bias leads observers to infer that darker colors map to larger quantities, and the opaque-is-more bias leads them to infer that regions appearing more opaque (given the background color) map to larger quantities. Previous work suggested that the background color only plays a role if visualizations appear to vary in opacity. The present study challenges this claim. We hypothesized that the background color would modulate inferred mappings for colormaps that should not appear to vary in opacity (by previous measures) if the visualization appeared to have a “hole” that revealed the background behind the map (hole hypothesis). We found that spatial aspects of the map contributed to inferred mappings, though the effects were inconsistent with the hole hypothesis. Our work raises new questions about how spatial distributions of data influence color semantics in colormap data visualizations.
LRAGAD: Local Information Recognition for Attribute Graph Anomaly Detection

Anomaly detection is a crucial technique for comprehending the intricate structures of data, and it has garnered significant attention in various real-life applications, including finance, transportation, and network security. Presently, existing anomaly detection methods primarily concentrate on shallow techniques like residual analysis and community discovery, as well as deep learning methods that employ self-encoders as the underlying framework. However, these current methods overlook the local information during training, resulting in subpar and unexplained results. To recognize the local information, in this paper, we propose a novel Local Information Recognition method for Attribute Graph Anomaly Detection (LRAGAD). Specifically, to use the contextual structural information, LRAGAD first constructs a contrastive learning representation by generating different substructures from the target nodes, while reconstructing the whole graph using self-encoders that can utilize the neighborhood information of target nodes. Moreover, to better understand the complex graph structure, LRAGAD uses anomaly score estimation for outlier prediction. Experimental results on five real-world datasets demonstrate that the proposed LRAGAD method obtains better performance on AUC scores.
Generation of Rule-Based Expert Systems with Certainty Factors from Datasets

We present an approach for extracting rules from datasets using Certainty Factor values to represent uncertainty and generating Expert Systems. This paper focuses on the different methods we use for extracting rules from datasets with nominal values, the way we calculate the Certainty Factors and also our approach for combining conclusions from different sources. Furthermore, we briefly present a tool we have developed that uses these methods to semi-automatically generate Expert Systems coded in the CLIPS language. Finally, we present experimental results that show the approach to have comparable performance in the classification task with other popular methods like decision trees.
Which Words Pillar the Semantic Expression of a Sentence?

In the realm of machine learning, a profound understanding of sentence semantics holds paramount importance for various applications, notably text classification. Traditionally, this comprehension has been entrusted to deep learning models, despite their computationally intensive nature, particularly when dealing with lengthy sequences. The nuanced impact of individual words within a sentence on semantic expression necessitates a strategic removal of less pertinent words to alleviate the computational burden of the model. Presently, prevailing approaches for word removal predominantly employ methods such as truncation, stop-word elimination and attention mechanisms. Regrettably, these techniques often lack a robust theoretical foundation concerning semantics and interpretability. To bridge this conceptual gap, our study introduces the concept of ‘Semantic Pillar Words’ (SPW) within a sentence, anchored in a Semantic Euclidean space. Here, the semantics of a word are represented as a constellation of semantic points, with a text sequence encapsulating the convex hull of these semantic points of words. We propose a novel method for Semantic Pillar Word extraction, known as ‘SPW-Conv’, which dynamically and interpretably prunes text segments, striving to preserve the semantic pillars inherent in the original text. Our extensive experimentation encompasses three diverse text classification datasets, revealing that SPW-Conv outperforms existing methods. Remarkably, it becomes evident that retaining less than 80% of the words within a sentence suffices to capture its semantics adequately, all while achieving classification accuracy levels comparable to those obtained using the entire original text.
Vis-SPLIT: Interactive Hierarchical Modeling for mRNA Expression Classification

We propose an interactive visual analytics tool, Vis-SPLIT, for partitioning a population of individuals into groups with similar gene signatures. Vis-SPLIT allows users to interactively explore a dataset and exploit visual separations to build a classification model for specific cancers. The visualization components reveal gene expression and correlation to assist specific partitioning decisions, while also providing overviews for the decision model and clustered genetic signatures. We demonstrate the effectiveness of our framework through a case study and evaluate its usability with domain experts. Our results show that Vis-SPLIT can classify patients based on their genetic signatures to effectively gain insights into RNA sequencing data, as compared to an existing classification system.
DBERT-ELVA: Discourse-Aware Extractive Text Summarization with Autoencoder

In recent advancements in text summarization, BERT has gained popularity for encoding documents. However, sentence-based extractive models often lead to redundant or uninformative phrases in the generated summaries. Additionally, BERT, which is pretrained on sentence pairs rather than full documents, struggles to capture long-range dependencies present within a document. To overcome these challenges, we introduce DBERT-ELVA, a discourse-aware neural summarization model. DBERT-ELVA extracts sub-sentential discourse units, offering a more refined granularity for extractive selection, in contrast to traditional sentence-based approaches. To learn a compressed representation of these discourse units while capturing the inter-dependencies among them, an Autoencoder is designed, utilizing Extreme Learning for improved generalization performance. Experimental evaluations on popular summarization benchmarks demonstrate that the proposed model significantly outperforms state-of-the-art methods, including other BERT-based models, by a substantial margin.
The Feasibility of Introducing Mobile Technology in Kenyan Schools that Utilize the Competency Based Curriculum

Competency Based Learning is an emerging curriculum in Kenya that provides students with a well-rounded education focused on learning both foundational and practical skills. The Competency Based Curriculum (CBC) was recently introduced but is not integrating well into the current system because there is a lack of instructor training and knowledge in the new educational program. The CBC incorporates digital literacy; however, it does not provide schools with the necessary technology. Through two experiments performed in Kisumu, Kenya, the time saved using a tablet as an assessment tool and time to learn an unfamiliar educational app were quantified. Instructor comfort level was also recorded. It was found that instructors were able to comfortably use multiple tablets to assess multiple students at the same time. Additionally, a tutorial video was sufficient for them learn to assess a student and perform basic tasks.