OpenRaiser/Intern-Atlas huggingface dataset:
Do Subjectivity and Objectivity Always Agree? A Case Study with Stack Overflow Questions
Venue: arXiv cs.SE 2023
Year: 2023

In Stack Overflow (SO), the quality of posts (i.e., questions and answers) is subjectively evaluated by users through a voting mechanism. The net votes (upvotes - downvotes) obtained by a post are often considered an approximation of its quality. However, about half of the questions that received working solutions got more downvotes than upvotes. Furthermore, about 18% of the accepted answers (i.e., verified solutions) also do not score the maximum votes. All these counter-intuitive findings cast doubts on the reliability of the evaluation mechanism employed at SO. Moreover, many users raise concerns against the evaluation, especially downvotes to their posts. Therefore, rigorous verification of the subjective evaluation is highly warranted to ensure a non-biased and reliable quality assessment mechanism. In this paper, we compare the subjective assessment of questions with their objective assessment using 2.5 million questions and ten text analysis metrics. According to our investigation, four objective metrics agree with the subjective evaluation, two do not agree, one either agrees or disagrees, and the remaining three neither agree nor disagree with the subjective evaluation. We then develop machine learning models to classify the promoted and discouraged questions. Our models outperform the state-of-the-art models with a maximum of about 76% - 87% accuracy.
Towards Automated 3D Search Planning for Emergency Response Missions
Venue: arXiv cs.RO 2023
Year: 2023

The ability to efficiently plan and execute automated and precise search missions using unmanned aerial vehicles (UAVs) during emergency response situations is imperative. Precise navigation between obstacles and time-efficient searching of 3D structures and buildings are essential for locating survivors and people in need in emergency response missions. In this work we address this challenging problem by proposing a unified search planning framework that automates the process of UAV-based search planning in 3D environments. Specifically, we propose a novel search planning framework which enables automated planning and execution of collision-free search trajectories in 3D by taking into account low-level mission constrains (e.g., the UAV dynamical and sensing model), mission objectives (e.g., the mission execution time and the UAV energy efficiency) and user-defined mission specifications (e.g., the 3D structures to be searched and minimum detection probability constraints). The capabilities and performance of the proposed approach are demonstrated through extensive simulated 3D search scenarios.
Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks
Venue: arXiv cs.LG 2023
Year: 2023

Previous work has established that RNNs with an unbounded activation function have the capacity to count exactly. However, it has also been shown that RNNs are challenging to train effectively and generally do not learn exact counting behaviour. In this paper, we focus on this problem by studying the simplest possible RNN, a linear single-cell network. We conduct a theoretical analysis of linear RNNs and identify conditions for the models to exhibit exact counting behaviour. We provide a formal proof that these conditions are necessary and sufficient. We also conduct an empirical analysis using tasks involving a Dyck-1-like Balanced Bracket language under two different settings. We observe that linear RNNs generally do not meet the necessary and sufficient conditions for counting behaviour when trained with the standard approach. We investigate how varying the length of training sequences and utilising different target classes impacts model behaviour during training and the ability of linear RNN models to effectively approximate the indicator conditions.
Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains
Venue: arXiv cs.NE 2023
Year: 2023

Quality-Diversity (QD) algorithms are designed to generate collections of high-performing solutions while maximizing their diversity in a given descriptor space. However, in the presence of unpredictable noise, the fitness and descriptor of the same solution can differ significantly from one evaluation to another, leading to uncertainty in the estimation of such values. Given the elitist nature of QD algorithms, they commonly end up with many degenerate solutions in such noisy settings. In this work, we introduce Archive Reproducibility Improvement Algorithm (ARIA); a plug-and-play approach that improves the reproducibility of the solutions present in an archive. We propose it as a separate optimization module, relying on natural evolution strategies, that can be executed on top of any QD algorithm. Our module mutates solutions to (1) optimize their probability of belonging to their niche, and (2) maximize their fitness. The performance of our method is evaluated on various tasks, including a classical optimization problem and two high-dimensional control tasks in simulated robotic environments. We show that our algorithm enhances the quality and descriptor space coverage of any given archive by at least 50%.
Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data
Venue: arXiv cs.LG 2023
Year: 2023

Generating synthetic data through generative models is gaining interest in the ML community and beyond. In the past, synthetic data was often regarded as a means to private data release, but a surge of recent papers explore how its potential reaches much further than this -- from creating more fair data to data augmentation, and from simulation to text generated by ChatGPT. In this perspective we explore whether, and how, synthetic data may become a dominant force in the machine learning world, promising a future where datasets can be tailored to individual needs. Just as importantly, we discuss which fundamental challenges the community needs to overcome for wider relevance and application of synthetic data -- the most important of which is quantifying how much we can trust any finding or prediction drawn from synthetic data.
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Venue: arXiv cs.CY 2023
Year: 2023

As the capabilities of generative language models continue to advance, the implications of biases ingrained within these models have garnered increasing attention from researchers, practitioners, and the broader public. This article investigates the challenges and risks associated with biases in large-scale language models like ChatGPT. We discuss the origins of biases, stemming from, among others, the nature of training data, model specifications, algorithmic constraints, product design, and policy decisions. We explore the ethical concerns arising from the unintended consequences of biased model outputs. We further analyze the potential opportunities to mitigate biases, the inevitability of some biases, and the implications of deploying these models in various applications, such as virtual assistants, content generation, and chatbots. Finally, we review the current approaches to identify, quantify, and mitigate biases in language models, emphasizing the need for a multi-disciplinary, collaborative effort to develop more equitable, transparent, and responsible AI systems. This article aims to stimulate a thoughtful dialogue within the artificial intelligence community, encouraging researchers and developers to reflect on the role of biases in generative language models and the ongoing pursuit of ethical AI.
SparseFormer: Sparse Visual Recognition via Limited Latent Tokens
Venue: arXiv cs.CV 2023
Year: 2023

Human visual recognition is a sparse process, where only a few salient visual cues are attended to rather than traversing every detail uniformly. However, most current vision networks follow a dense paradigm, processing every single visual unit (e.g,, pixel or patch) in a uniform manner. In this paper, we challenge this dense paradigm and present a new method, coined SparseFormer, to imitate human's sparse visual recognition in an end-to-end manner. SparseFormer learns to represent images using a highly limited number of tokens (down to 49) in the latent space with sparse feature sampling procedure instead of processing dense units in the original pixel space. Therefore, SparseFormer circumvents most of dense operations on the image space and has much lower computational costs. Experiments on the ImageNet classification benchmark dataset show that SparseFormer achieves performance on par with canonical or well-established models while offering better accuracy-throughput tradeoff. Moreover, the design of our network can be easily extended to the video classification with promising performance at lower computational costs. We hope that our work can provide an alternative way for visual modeling and inspire further research on sparse neural architectures. The code will be publicly available at https://github.com/showlab/sparseformer
Compact and High-Performance TCAM Based on Scaled Double-Gate FeFETs
Venue: arXiv cs.ET 2023
Year: 2023

Ternary content addressable memory (TCAM), widely used in network routers and high-associativity caches, is gaining popularity in machine learning and data-analytic applications. Ferroelectric FETs (FeFETs) are a promising candidate for implementing TCAM owing to their high ON/OFF ratio, non-volatility, and CMOS compatibility. However, conventional single-gate FeFETs (SG-FeFETs) suffer from relatively high write voltage, low endurance, potential read disturbance, and face scaling challenges. Recently, a double-gate FeFET (DG-FeFET) has been proposed and outperforms SG-FeFETs in many aspects. This paper investigates TCAM design challenges specific to DG-FeFETs and introduces a novel 1.5T1Fe TCAM design based on DG-FeFETs. A 2-step search with early termination is employed to reduce the cell area and improve energy efficiency. A shared driver design is proposed to reduce the peripherals area. Detailed analysis and SPICE simulation show that the 1.5T1Fe DG-TCAM leads to superior search speed and energy efficiency. The 1.5T1Fe TCAM design can also be built with SG-FeFETs, which achieve search latency and energy improvement compared with 2FeFET TCAM.
LSGDDN-LCD: An Appearance-based Loop Closure Detection using Local Superpixel Grid Descriptors and Incremental Dynamic Nodes
Venue: arXiv cs.CV 2023
Year: 2023

Loop Closure Detection (LCD) is an essential component of visual simultaneous localization and mapping (SLAM) systems. It enables the recognition of previously visited scenes to eliminate pose and map estimate drifts arising from long-term exploration. However, current appearance-based LCD methods face significant challenges, including high computational costs, viewpoint variance, and dynamic objects in scenes. This paper introduced an online appearance based LCD using local superpixel grids descriptor and dynamic node, i.e, LSGDDN-LCD, to find similarities between scenes via hand-crafted features extracted from LSGD. Unlike traditional Bag-of-Words (BoW) based LCD, which requires pre-training, we proposed an adaptive mechanism to group similar images called $\textbf{\textit{dynamic}}$ $\textbf{\textit{node}}$, which incrementally adjusted the database in an online manner, allowing for efficient and online retrieval of previously viewed images without need of the pre-training. Experimental results confirmed that the LSGDDN-LCD significantly improved LCD precision-recall and efficiency, and outperformed several state-of-the-art (SOTA) approaches on multiple typical datasets, indicating its great potential as a generic LCD framework.
GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation
Venue: arXiv cs.IR 2023
Year: 2023

Recent advancements in Natural Language Processing (NLP) have led to the development of NLP-based recommender systems that have shown superior performance. However, current models commonly treat items as mere IDs and adopt discriminative modeling, resulting in limitations of (1) fully leveraging the content information of items and the language modeling capabilities of NLP models; (2) interpreting user interests to improve relevance and diversity; and (3) adapting practical circumstances such as growing item inventories. To address these limitations, we present GPT4Rec, a novel and flexible generative framework inspired by search engines. It first generates hypothetical "search queries" given item titles in a user's history, and then retrieves items for recommendation by searching these queries. The framework overcomes previous limitations by learning both user and item embeddings in the language space. To well-capture user interests with different aspects and granularity for improving relevance and diversity, we propose a multi-query generation technique with beam search. The generated queries naturally serve as interpretable representations of user interests and can be searched to recommend cold-start items. With GPT-2 language model and BM25 search engine, our framework outperforms state-of-the-art methods by $75.7\%$ and $22.2\%$ in Recall@K on two public datasets. Experiments further revealed that multi-query generation with beam search improves both the diversity of retrieved items and the coverage of a user's multi-interests. The adaptiveness and interpretability of generated queries are discussed with qualitative case studies.
Rayleigh quotients of Dillon's functions
Venue: arXiv cs.DM 2023
Year: 2023

The Walsh--Hadamard spectrum of a bent function uniquely determines a dual function. The dual of a bent function is also bent. A bent function that is equal to its dual is called a self-dual function. The Hamming distance between a bent function and its dual is related to its Rayleigh quotient. Carlet, Danielsen, Parker, and Sole studied Rayleigh quotients of bent functions in ${\mathcal PS}_{ap}$, and obtained an expression in terms of a character sum. We use another approach comprising of Desarguesian spreads to obtain the complete spectrum of Rayleigh quotients of bent functions in $\mathcal{PS}_{ap}$.
The Short Text Matching Model Enhanced with Knowledge via Contrastive Learning
Venue: arXiv cs.CL 2023
Year: 2023

In recent years, short Text Matching tasks have been widely applied in the fields ofadvertising search and recommendation. The difficulty lies in the lack of semantic information and word ambiguity caused by the short length of the text. Previous works have introduced complement sentences or knowledge bases to provide additional feature information. However, these methods have not fully interacted between the original sentence and the complement sentence, and have not considered the noise issue that may arise from the introduction of external knowledge bases. Therefore, this paper proposes a short Text Matching model that combines contrastive learning and external knowledge. The model uses a generative model to generate corresponding complement sentences and uses the contrastive learning method to guide the model to obtain more semantically meaningful encoding of the original sentence. In addition, to avoid noise, we use keywords as the main semantics of the original sentence to retrieve corresponding knowledge words in the knowledge base, and construct a knowledge graph. The graph encoding model is used to integrate the knowledge base information into the model. Our designed model achieves state-of-the-art performance on two publicly available Chinese Text Matching datasets, demonstrating the effectiveness of our model.
Knowledge Relation Rank Enhanced Heterogeneous Learning Interaction Modeling for Neural Graph Forgetting Knowledge Tracing
Venue: arXiv cs.LG 2023
Year: 2023

Recently, knowledge tracing models have been applied in educational data mining such as the Self-attention knowledge tracing model(SAKT), which models the relationship between exercises and Knowledge concepts(Kcs). However, relation modeling in traditional Knowledge tracing models only considers the static question-knowledge relationship and knowledge-knowledge relationship and treats these relationships with equal importance. This kind of relation modeling is difficult to avoid the influence of subjective labeling and considers the relationship between exercises and KCs, or KCs and KCs separately. In this work, a novel knowledge tracing model, named Knowledge Relation Rank Enhanced Heterogeneous Learning Interaction Modeling for Neural Graph Forgetting Knowledge Tracing(NGFKT), is proposed to reduce the impact of the subjective labeling by calibrating the skill relation matrix and the Q-matrix and apply the Graph Convolutional Network(GCN) to model the heterogeneous interactions between students, exercises, and skills. Specifically, the skill relation matrix and Q-matrix are generated by the Knowledge Relation Importance Rank Calibration method(KRIRC). Then the calibrated skill relation matrix, Q-matrix, and the heterogeneous interactions are treated as the input of the GCN to generate the exercise embedding and skill embedding. Next, the exercise embedding, skill embedding, item difficulty, and contingency table are incorporated to generate an exercise relation matrix as the inputs of the Position-Relation-Forgetting attention mechanism. Finally, the Position-Relation-Forgetting attention mechanism is applied to make the predictions. Experiments are conducted on the two public educational datasets and results indicate that the NGFKT model outperforms all baseline models in terms of AUC, ACC, and Performance Stability(PS).
Pump It Up: Predict Water Pump Status using Attentive Tabular Learning
Venue: arXiv cs.LG 2023
Year: 2023

Water crisis is a crucial concern around the globe. Appropriate and timely maintenance of water pumps in drought-hit countries is vital for communities relying on the well. In this paper, we analyze and apply a sequential attentive deep neural architecture, TabNet, for predicting water pump repair status in Tanzania. The model combines the valuable benefits of tree-based algorithms and neural networks, enabling end-to-end training, model interpretability, sparse feature selection, and efficient learning on tabular data. Finally, we compare the performance of TabNet with popular gradient tree-boosting algorithms like XGBoost, LightGBM,CatBoost, and demonstrate how we can further uplift the performance by choosing focal loss as the objective function while training on imbalanced data.
BRAMAC: Compute-in-BRAM Architectures for Multiply-Accumulate on FPGAs
Venue: arXiv cs.AR 2023
Year: 2023

Deep neural network (DNN) inference using reduced integer precision has been shown to achieve significant improvements in memory utilization and compute throughput with little or no accuracy loss compared to full-precision floating-point. Modern FPGA-based DNN inference relies heavily on the on-chip block RAM (BRAM) for model storage and the digital signal processing (DSP) unit for implementing the multiply-accumulate (MAC) operation, a fundamental DNN primitive. In this paper, we enhance the existing BRAM to also compute MAC by proposing BRAMAC (Compute-in-$\underline{\text{BR}}$AM $\underline{\text{A}}$rchitectures for $\underline{\text{M}}$ultiply-$\underline{\text{Ac}}$cumulate). BRAMAC supports 2's complement 2- to 8-bit MAC in a small dummy BRAM array using a hybrid bit-serial & bit-parallel data flow. Unlike previous compute-in-BRAM architectures, BRAMAC allows read/write access to the main BRAM array while computing in the dummy BRAM array, enabling both persistent and tiling-based DNN inference. We explore two BRAMAC variants: BRAMAC-2SA (with 2 synchronous dummy arrays) and BRAMAC-1DA (with 1 double-pumped dummy array). BRAMAC-2SA/BRAMAC-1DA can boost the peak MAC throughput of a large Arria-10 FPGA by 2.6$\times$/2.1$\times$, 2.3$\times$/2.0$\times$, and 1.9$\times$/1.7$\times$ for 2-bit, 4-bit, and 8-bit precisions, respectively at the cost of 6.8%/3.4% increase in the FPGA core area. By adding BRAMAC-2SA/BRAMAC-1DA to a state-of-the-art tiling-based DNN accelerator, an average speedup of 2.05$\times$/1.7$\times$ and 1.33$\times$/1.52$\times$ can be achieved for AlexNet and ResNet-34, respectively across different model precisions.
Analysis of Sampling Strategies for Implicit 3D Reconstruction
Venue: arXiv cs.CV 2023
Year: 2023

In the training process of the implicit 3D reconstruction network, the choice of spatial query points' sampling strategy affects the final performance of the model. Different works have differences in the selection of sampling strategies, not only in the spatial distribution of query points but also in the order of magnitude difference in the density of query points. For how to select the sampling strategy of query points, current works are more akin to an enumerating operation to find the optimal solution, which seriously affects work efficiency. In this work, we explored the relationship between sampling strategy and network final performance through classification analysis and experimental comparison from three aspects: the relationship between network type and sampling strategy, the relationship between implicit function and sampling strategy, and the impact of sampling density on model performance. In addition, we also proposed two methods, linear sampling and distance mask, to improve the sampling strategy of query points, making it more general and robust.
A new transformation for embedded convolutional neural network approach toward real-time servo motor overload fault-detection
Venue: arXiv cs.LG 2023
Year: 2023

Overloading in DC servo motors is a major concern in industries, as many companies face the problem of finding expert operators, and also human monitoring may not be an effective solution. Therefore, this paper proposed an embedded Artificial intelligence (AI) approach using a Convolutional Neural Network (CNN) using a new transformation to extract faults from real-time input signals without human interference. Our main purpose is to extract as many as possible features from the input signal to achieve a relaxed dataset that results in an effective but compact network to provide real-time fault detection even in a low-memory microcontroller. Besides, fault detection method a synchronous dual-motor system is also proposed to take action in faulty events. To fulfill this intention, a one-dimensional input signal from the output current of each DC servo motor is monitored and transformed into a 3d stack of data and then the CNN is implemented into the processor to detect any fault corresponding to overloading, finally experimental setup results in 99.9997% accuracy during testing for a model with nearly 8000 parameters. In addition, the proposed dual-motor system could achieve overload reduction and provide a fault-tolerant system and it is shown that this system also takes advantage of less energy consumption.
Bipol: A Novel Multi-Axes Bias Evaluation Metric with Explainability for NLP
Venue: arXiv cs.CL 2023
Year: 2023

We introduce bipol, a new metric with explainability, for estimating social bias in text data. Harmful bias is prevalent in many online sources of data that are used for training machine learning (ML) models. In a step to address this challenge we create a novel metric that involves a two-step process: corpus-level evaluation based on model classification and sentence-level evaluation based on (sensitive) term frequency (TF). After creating new models to detect bias along multiple axes using SotA architectures, we evaluate two popular NLP datasets (COPA and SQUAD). As additional contribution, we created a large dataset (with almost 2 million labelled samples) for training models in bias detection and make it publicly available. We also make public our codes.