OpenRaiser/Intern-Atlas huggingface dataset:
Constructing Enhanced Mutual Information for Online Class-Incremental Learning
Venue: arXiv cs.LG 2024
Year: 2024

Online Class-Incremental continual Learning (OCIL) addresses the challenge of continuously learning from a single-channel data stream, adapting to new tasks while mitigating catastrophic forgetting. Recently, Mutual Information (MI)-based methods have shown promising performance in OCIL. However, existing MI-based methods treat various knowledge components in isolation, ignoring the knowledge confusion across tasks. This narrow focus on simple MI knowledge alignment may lead to old tasks being easily forgotten with the introduction of new tasks, risking the loss of common parts between past and present knowledge.To address this, we analyze the MI relationships from the perspectives of diversity, representativeness, and separability, and propose an Enhanced Mutual Information (EMI) method based on knwoledge decoupling. EMI consists of Diversity Mutual Information (DMI), Representativeness Mutual Information (RMI) and Separability Mutual Information (SMI). DMI diversifies intra-class sample features by considering the similarity relationships among inter-class sample features to enable the network to learn more general knowledge. RMI summarizes representative features for each category and aligns sample features with these representative features, making the intra-class sample distribution more compact. SMI establishes MI relationships for inter-class representative features, enhancing the stability of representative features while increasing the distinction between inter-class representative features, thus creating clear boundaries between class. Extensive experimental results on widely used benchmark datasets demonstrate the superior performance of EMI over state-of-the-art baseline methods.
Utilising Explainable Techniques for Quality Prediction in a Complex Textiles Manufacturing Use Case
Venue: arXiv cs.LG 2024
Year: 2024

This paper develops an approach to classify instances of product failure in a complex textiles manufacturing dataset using explainable techniques. The dataset used in this study was obtained from a New Zealand manufacturer of woollen carpets and rugs. In investigating the trade-off between accuracy and explainability, three different tree-based classification algorithms were evaluated: a Decision Tree and two ensemble methods, Random Forest and XGBoost. Additionally, three feature selection methods were also evaluated: the SelectKBest method, using chi-squared as the scoring function, the Pearson Correlation Coefficient, and the Boruta algorithm. Not surprisingly, the ensemble methods typically produced better results than the Decision Tree model. The Random Forest model yielded the best results overall when combined with the Boruta feature selection technique. Finally, a tree ensemble explaining technique was used to extract rule lists to capture necessary and sufficient conditions for classification by a trained model that could be easily interpreted by a human. Notably, several features that were in the extracted rule lists were statistical features and calculated features that were added to the original dataset. This demonstrates the influence that bringing in additional information during the data preprocessing stages can have on the ultimate model performance.
Mechanism Design for Locating Facilities with Capacities with Insufficient Resources
Venue: arXiv cs.GT 2024
Year: 2024

This paper explores the Mechanism Design aspects of the $m$-Capacitated Facility Location Problem where the total facility capacity is less than the number of agents. Following the framework outlined by Aziz et al., the Social Welfare of the facility location is determined through a First-Come-First-Served (FCFS) game, in which agents compete once the facility positions are established. When the number of facilities is $m > 1$, the Nash Equilibrium (NE) of the FCFS game is not unique, making the utility of the agents and the concept of truthfulness unclear. To tackle these issues, we consider absolutely truthful mechanisms, i.e. mechanisms that prevent agents from misreporting regardless of the strategies used during the FCFS game. We combine this stricter truthfulness requirement with the notion of Equilibrium Stable (ES) mechanisms, which are mechanisms whose Social Welfare does not depend on the NE of the FCFS game. We demonstrate that the class of percentile mechanisms is absolutely truthful and identify the conditions under which they are ES. We also show that the approximation ratio of each ES percentile mechanism is bounded and determine its value. Notably, when all the facilities have the same capacity and the number of agents is sufficiently large, it is possible to achieve an approximation ratio smaller than $1+\frac{1}{2m-1}$. Finally, we extend our study to encompass higher-dimensional problems. Within this framework, we demonstrate that the class of ES percentile mechanisms is even more restricted and characterize the mechanisms that are both ES and absolutely truthful. We further support our findings by empirically evaluating the performance of the mechanisms when the agents are the samples of a distribution.
Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
Venue: arXiv cs.CL 2024
Year: 2024

The Mixture of Experts (MoE) model is a promising approach for handling code-switching speech recognition (CS-ASR) tasks. However, the existing CS-ASR work on MoE has yet to leverage the advantages of MoE's parameter scaling ability fully. This work proposes DLG-MoE, a Dynamic Language Group-based MoE, which can effectively handle the CS-ASR task and leverage the advantages of parameter scaling. DLG-MoE operates based on a hierarchical routing mechanism. First, the language router explicitly models the language attribute and dispatches the representations to the corresponding language expert groups. Subsequently, the unsupervised router within each language group implicitly models attributes beyond language and coordinates expert routing and collaboration. DLG-MoE outperforms the existing MoE methods on CS-ASR tasks while demonstrating great flexibility. It supports different top-$k$ inference and streaming capabilities and can also prune the model parameters flexibly to obtain a monolingual sub-model. The code has been released.
Signalling and Control in Nonlinear Stochastic Systems: An Information State Approach with Applications
Venue: arXiv cs.IT 2024
Year: 2024

We consider optimal signalling and control of discrete-time nonlinear partially observable stochastic systems in state space form. In the first part of the paper, we characterize the operational {\it control-coding capacity}, $C_{FB}$ in bits/second, by an information theoretic optimization problem of encoding signals or messages into randomized controller-encoder strategies, and reproducing the messages at the output of the system using a decoder or estimator with arbitrary small asymptotic error probability. Our analysis of $C_{FB}$ is based on realizations of randomized strategies (controller-encoders), in terms of information states of nonlinear filtering theory, and either uniform or arbitrary distributed random variables (RVs). In the second part of the paper, we analyze the linear-quadratic Gaussian partially observable stochastic system (LQG-POSS). We show that simultaneous signalling and control leads to randomized strategies described by finite-dimensional sufficient statistics, that involve two Kalman-filters, and consist of control, estimation and signalling strategies. We apply decentralized optimization techniques to prove a separation principle, and to derive the optimal control part of randomized strategies explicitly in terms of a control matrix difference Riccati equation (DRE).
DynamicTrack: Advancing Gigapixel Tracking in Crowded Scenes
Venue: arXiv cs.CV 2024
Year: 2024

Tracking in gigapixel scenarios holds numerous potential applications in video surveillance and pedestrian analysis. Existing algorithms attempt to perform tracking in crowded scenes by utilizing multiple cameras or group relationships. However, their performance significantly degrades when confronted with complex interaction and occlusion inherent in gigapixel images. In this paper, we introduce DynamicTrack, a dynamic tracking framework designed to address gigapixel tracking challenges in crowded scenes. In particular, we propose a dynamic detector that utilizes contrastive learning to jointly detect the head and body of pedestrians. Building upon this, we design a dynamic association algorithm that effectively utilizes head and body information for matching purposes. Extensive experiments show that our tracker achieves state-of-the-art performance on widely used tracking benchmarks specifically designed for gigapixel crowded scenes.
A dual ensemble classifier used to recognise contaminated multi-channel EMG and MMG signals in the control of upper limb bioprosthesis
Venue: arXiv cs.LG 2024
Year: 2024

Myopotential pattern recognition to decode the intent of the user is the most advanced approach to controlling a powered bioprosthesis. Unfortunately, many factors make this a difficult problem and achieving acceptable recognition quality in real-word conditions is a serious challenge. The aim of the paper is to develop a recognition system that will mitigate factors related to multimodality and multichannel recording of biosignals and their high susceptibility to contamination. The proposed method involves the use of two co-operating multiclassifier systems. The first system is composed of one-class classifiers related to individual electromyographic (EMG) and mechanomyographic (MMG) biosignal recording channels, and its task is to recognise contaminated channels. The role of the second system is to recognise the class of movement resulting from the patient's intention. The ensemble system consists of base classifiers using the representation (extracted features) of biosignals from different channels. The system uses a dynamic selection mechanism, eliminating those base classifiers that are associated with biosignal channels that are recognised by the one-class ensemble system as being contaminated. Experimental studies were conducted using signals from an able-bodied person with simulation of amputation. The results obtained allow us to reject the null hypothesis that the application of the dual ensemble foes not lead to improved classification quality.
VeriCHERI: Exhaustive Formal Security Verification of CHERI at the RTL
Venue: arXiv cs.CR 2024
Year: 2024

Protecting data in memory from attackers continues to be a concern in computing systems. CHERI is a promising approach to achieve such protection, by providing and enforcing fine-grained memory protection directly in the hardware. Creating trust for the entire system stack, however, requires a gap-free verification of CHERI's hardware-based protection mechanisms. Existing verification methods for CHERI target the abstract ISA model rather than the underlying hardware implementation. Fully ensuring the CHERI security guarantees for a concrete RTL implementation is a challenge in previous flows and demands high manual efforts. This paper presents VeriCHERI, a novel approach to security verification. It is conceptionally different from previous works in that it does not require any ISA specification. Instead of checking compliance with a golden ISA model, we check against well-established global security objectives of confidentiality and integrity. Fully covering these objectives, VeriCHERI uses as few as four unbounded properties to exhaustively prove or disprove any vulnerability. We demonstrate the effectiveness and scalability of VeriCHERI on a RISC-V based processor implementing a CHERI variant.
Rapid Object Annotation
Venue: arXiv cs.CV 2024
Year: 2024

In this report we consider the problem of rapidly annotating a video with bounding boxes for a novel object. We describe a UI and associated workflow designed to make this process fast for an arbitrary novel target.
RRO: A Regularized Routing Optimization Algorithm for Enhanced Throughput and Low Latency with Efficient Complexity
Venue: arXiv cs.NI 2024
Year: 2024

In the rapidly evolving landscape of wireless networks, achieving enhanced throughput with low latency for data transmission is crucial for future communication systems. While low complexity OSPF-type solutions have shown effectiveness in lightly-loaded networks, they often falter in the face of increasing congestion. Recent approaches have suggested utilizing backpressure and deep learning techniques for route optimization. However, these approaches face challenges due to their high implementation and computational complexity, surpassing the capabilities of networks with limited hardware devices. A key challenge is developing algorithms that improve throughput and reduce latency while keeping complexity levels compatible with OSPF. In this collaborative research between Ben-Gurion University and Ceragon Networks Ltd., we address this challenge by developing a novel approach, dubbed Regularized Routing Optimization (RRO). The RRO algorithm offers both distributed and centralized implementations with low complexity, making it suitable for integration into 5G and beyond technologies, where no significant changes to the existing protocols are needed. It increases throughput while ensuring latency remains sufficiently low through regularized optimization. We analyze the computational complexity of RRO and prove that it converges with a level of complexity comparable to OSPF. Extensive simulation results across diverse network topologies demonstrate that RRO significantly outperforms existing methods.
AutoRDF2GML: Facilitating RDF Integration in Graph Machine Learning
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we introduce AutoRDF2GML, a framework designed to convert RDF data into data representations tailored for graph machine learning tasks. AutoRDF2GML enables, for the first time, the creation of both content-based features -- i.e., features based on RDF datatype properties -- and topology-based features -- i.e., features based on RDF object properties. Characterized by automated feature extraction, AutoRDF2GML makes it possible even for users less familiar with RDF and SPARQL to generate data representations ready for graph machine learning tasks, such as link prediction, node classification, and graph classification. Furthermore, we present four new benchmark datasets for graph machine learning, created from large RDF knowledge graphs using our framework. These datasets serve as valuable resources for evaluating graph machine learning approaches, such as graph neural networks. Overall, our framework effectively bridges the gap between the Graph Machine Learning and Semantic Web communities, paving the way for RDF-based machine learning applications.
Towards Effective and Efficient Continual Pre-training of Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. To make the CPT approach more traceable, this paper presents a technical report for continually pre-training Llama-3 (8B), which significantly enhances the Chinese language ability and scientific reasoning ability of the backbone model. To enhance the new abilities while retaining the original abilities, we design specific data mixture and curriculum strategies by utilizing existing datasets and synthesizing high-quality datasets. Specifically, we synthesize multidisciplinary scientific question and answer (QA) pairs based on related web pages, and subsequently incorporate these synthetic data to improve the scientific reasoning ability of Llama-3. We refer to the model after CPT as Llama-3-SynE (Synthetic data Enhanced Llama-3). We also present the tuning experiments with a relatively small model -- TinyLlama, and employ the derived findings to train the backbone model. Extensive experiments on a number of evaluation benchmarks show that our approach can largely improve the performance of the backbone models, including both the general abilities (+8.81 on C-Eval and +6.31 on CMMLU) and the scientific reasoning abilities (+12.00 on MATH and +4.13 on SciEval), without hurting the original capacities. Our model, data, and codes are available at https://github.com/RUC-GSAI/Llama-3-SynE.
Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation
Venue: arXiv cs.CL 2024
Year: 2024

Applying differential privacy (DP) by means of the DP-SGD algorithm to protect individual data points during training is becoming increasingly popular in NLP. However, the choice of granularity at which DP is applied is often neglected. For example, neural machine translation (NMT) typically operates on the sentence-level granularity. From the perspective of DP, this setup assumes that each sentence belongs to a single person and any two sentences in the training dataset are independent. This assumption is however violated in many real-world NMT datasets, e.g., those including dialogues. For proper application of DP we thus must shift from sentences to entire documents. In this paper, we investigate NMT at both the sentence and document levels, analyzing the privacy/utility trade-off for both scenarios, and evaluating the risks of not using the appropriate privacy granularity in terms of leaking personally identifiable information (PII). Our findings indicate that the document-level NMT system is more resistant to membership inference attacks, emphasizing the significance of using the appropriate granularity when working with DP.
Exploiting Device Heterogeneity in Grant-Free Random Access: A Data-Driven Approach
Venue: arXiv cs.IT 2024
Year: 2024

Grant-free random access (GFRA) is now a popular protocol for large-scale wireless multiple access systems in order to reduce control signaling. Resource allocation in GFRA can be viewed as a form of frame slotted ALOHA, where a ubiquitous design assumption is device homogeneity. In particular, the probability that a device seeks to transmit data is common to all devices. Recently, there has been an interest in designing frame slotted ALOHA algorithms for networks with heterogeneous activity probabilities. These works have established that the throughput can be significantly improved over the standard uniform allocation. However, the algorithms for optimizing the probability a device accesses each slot require perfect knowledge of the active devices within each frame. In practice, this assumption is limiting as device identification algorithms in GFRA rarely provide activity estimates with zero errors. In this paper, we develop a new algorithm based on stochastic gradient descent for optimizing slot allocation probabilities in the presence of activity estimation errors. Our algorithm exploits importance weighted bias mitigation for stochastic gradient estimates, which is shown to provably converge to a stationary point of the throughput optimization problem. In moderate size systems, our simulations show that the performance of our algorithm depends on the type of error distribution. We study symmetric bit flipping, asymmetric bit flipping and errors resulting from a generalized approximate message passing (GAMP) algorithm. In these scenarios, we observe gains up to 40\%, 66\%, and 19\%, respectively.
Online Planning in POMDPs with State-Requests
Venue: arXiv cs.LG 2024
Year: 2024

In key real-world problems, full state information is sometimes available but only at a high cost, like activating precise yet energy-intensive sensors or consulting humans, thereby compelling the agent to operate under partial observability. For this scenario, we propose AEMS-SR (Anytime Error Minimization Search with State Requests), a principled online planning algorithm tailored for POMDPs with state requests. By representing the search space as a graph instead of a tree, AEMS-SR avoids the exponential growth of the search space originating from state requests. Theoretical analysis demonstrates AEMS-SR's $\varepsilon$-optimality, ensuring solution quality, while empirical evaluations illustrate its effectiveness compared with AEMS and POMCP, two SOTA online planning algorithms. AEMS-SR enables efficient planning in domains characterized by partial observability and costly state requests offering practical benefits across various applications.
Generative Adversarial Networks for Imputing Sparse Learning Performance
Venue: arXiv cs.LG 2024
Year: 2024

Learning performance data, such as correct or incorrect responses to questions in Intelligent Tutoring Systems (ITSs) is crucial for tracking and assessing the learners' progress and mastery of knowledge. However, the issue of data sparsity, characterized by unexplored questions and missing attempts, hampers accurate assessment and the provision of tailored, personalized instruction within ITSs. This paper proposes using the Generative Adversarial Imputation Networks (GAIN) framework to impute sparse learning performance data, reconstructed into a three-dimensional (3D) tensor representation across the dimensions of learners, questions and attempts. Our customized GAIN-based method computational process imputes sparse data in a 3D tensor space, significantly enhanced by convolutional neural networks for its input and output layers. This adaptation also includes the use of a least squares loss function for optimization and aligns the shapes of the input and output with the dimensions of the questions-attempts matrices along the learners' dimension. Through extensive experiments on six datasets from various ITSs, including AutoTutor, ASSISTments and MATHia, we demonstrate that the GAIN approach generally outperforms existing methods such as tensor factorization and other generative adversarial network (GAN) based approaches in terms of imputation accuracy. This finding enhances comprehensive learning data modeling and analytics in AI-based education.
A Flexible and Scalable Approach for Collecting Wildlife Advertisements on the Web
Venue: arXiv cs.IR 2024
Year: 2024

Wildlife traffickers are increasingly carrying out their activities in cyberspace. As they advertise and sell wildlife products in online marketplaces, they leave digital traces of their activity. This creates a new opportunity: by analyzing these traces, we can obtain insights into how trafficking networks work as well as how they can be disrupted. However, collecting such information is difficult. Online marketplaces sell a very large number of products and identifying ads that actually involve wildlife is a complex task that is hard to automate. Furthermore, given that the volume of data is staggering, we need scalable mechanisms to acquire, filter, and store the ads, as well as to make them available for analysis. In this paper, we present a new approach to collect wildlife trafficking data at scale. We propose a data collection pipeline that combines scoped crawlers for data discovery and acquisition with foundational models and machine learning classifiers to identify relevant ads. We describe a dataset we created using this pipeline which is, to the best of our knowledge, the largest of its kind: it contains almost a million ads obtained from 41 marketplaces, covering 235 species and 20 languages. The source code is publicly available at \url{https://github.com/VIDA-NYU/wildlife_pipeline}.