OpenRaiser/Intern-Atlas huggingface dataset:
Learning Interpretable Classifiers for PDDL Planning
Venue: arXiv cs.AI 2024
Year: 2024

We consider the problem of synthesizing interpretable models that recognize the behaviour of an agent compared to other agents, on a whole set of similar planning tasks expressed in PDDL. Our approach consists in learning logical formulas, from a small set of examples that show how an agent solved small planning instances. These formulas are expressed in a version of First-Order Temporal Logic (FTL) tailored to our planning formalism. Such formulas are human-readable, serve as (partial) descriptions of an agent's policy, and generalize to unseen instances. We show that learning such formulas is computationally intractable, as it is an NP-hard problem. As such, we propose to learn these behaviour classifiers through a topology-guided compilation to MaxSAT, which allows us to generate a wide range of different formulas. Experiments show that interesting and accurate formulas can be learned in reasonable time.
Safety-Aware Fine-Tuning of Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios.
Improving accuracy and convergence of federated learning edge computing methods for generalized DER forecasting applications in power grid
Venue: arXiv cs.LG 2024
Year: 2024

This proposal aims to develop more accurate federated learning (FL) methods with faster convergence properties and lower communication requirements, specifically for forecasting distributed energy resources (DER) such as renewables, energy storage, and loads in modern, low-carbon power grids. This will be achieved by (i) leveraging recently developed extensions of FL such as hierarchical and iterative clustering to improve performance with non-IID data, (ii) experimenting with different types of FL global models well-suited to time-series data, and (iii) incorporating domain-specific knowledge from power systems to build more general FL frameworks and architectures that can be applied to diverse types of DERs beyond just load forecasting, and with heterogeneous clients.
Programming of Cellular Automata in C and C++
Venue: arXiv cs.PL 2024
Year: 2024

This study explores running times of different ways to program cellular automata in C and C++, i.e. looping through arrays by different means, the effect of structures and objects, and the choice of data structure (array versus vector in C++) and compiler (GNU gcc versus Apple clang). Using arrays instead of vectors, using pointers instead of array indices, using C instead of C++, and using structures and objects instead of primitive data types has little to negligible effects on the running time. Also, the choice of compiler has only a minor effect, except for simple update functions, in which case clang seems to find better ways to optimise it, especially for index-based access. If one is interested in multi-state cellular automata, structures and objects can be used without loss of performance in C and C++, respectively. The study supports the recommendation from practice to use vector in C++ and adds to it the use of index-based access for cellular automata. Future studies might investigate Apple's Metal shader or compiler optimisation, especially with respect to the update function.
A Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics
Venue: arXiv cs.CL 2024
Year: 2024

The explosion of open-sourced models and Question-Answering (QA) datasets emphasizes the importance of automated QA evaluation. We studied the statistics of the existing evaluation metrics for a better understanding of their limitations. By measuring the correlation coefficients of each evaluation metric concerning human-like evaluation score, we observed the following: (1) existing metrics have a high correlation among them concerning the question type (e.g., single word, single phrase, etc.), (2) no single metric can adequately estimate the human-like evaluation. As a potential solution, we discuss how a Mixture Of Grader could potentially improve the auto QA evaluator quality.
Evaluating Effectiveness of Interactivity in Contour-based Geospatial Visualizations
Venue: arXiv cs.HC 2024
Year: 2024

Contour maps are an essential tool for exploring spatial features of the terrain, such as distance, directions, and surface gradient among the contour areas. User interactions in contour-based visualizations create approaches to visual analysis that are noticeably different from the perspective of human cognition. As such, various interactive approaches have been introduced to improve system usability and enhance human cognition for complex and large-scale spatial data exploration. However, what user interaction means for contour maps, its purpose, when to leverage, and design primitives have yet to be investigated in the context of analysis tasks. Therefore, further research is needed to better understand and quantify the potentials and benefits offered by user interactions in contour-based geospatial visualizations designed to support analytical tasks. In this paper, we present a contour-based interactive geospatial visualization designed for analytical tasks. We conducted a crowd-sourced user study (N=62) to examine the impact of interactive features on analysis using contour-based geospatial visualizations. Our results show that the interactive features aid in their data analysis and understanding in terms of spatial data extent, map layout, task complexity, and user expertise. Finally, we discuss our findings in-depth, which will serve as guidelines for future design and implementation of interactive features in support of case-specific analytical tasks on contour-based geospatial views.
StatioCL: Contrastive Learning for Time Series via Non-Stationary and Temporal Contrast
Venue: arXiv cs.LG 2024
Year: 2024

Contrastive learning (CL) has emerged as a promising approach for representation learning in time series data by embedding similar pairs closely while distancing dissimilar ones. However, existing CL methods often introduce false negative pairs (FNPs) by neglecting inherent characteristics and then randomly selecting distinct segments as dissimilar pairs, leading to erroneous representation learning, reduced model performance, and overall inefficiency. To address these issues, we systematically define and categorize FNPs in time series into semantic false negative pairs and temporal false negative pairs for the first time: the former arising from overlooking similarities in label categories, which correlates with similarities in non-stationarity and the latter from neglecting temporal proximity. Moreover, we introduce StatioCL, a novel CL framework that captures non-stationarity and temporal dependency to mitigate both FNPs and rectify the inaccuracies in learned representations. By interpreting and differentiating non-stationary states, which reflect the correlation between trends or temporal dynamics with underlying data patterns, StatioCL effectively captures the semantic characteristics and eliminates semantic FNPs. Simultaneously, StatioCL establishes fine-grained similarity levels based on temporal dependencies to capture varying temporal proximity between segments and to mitigate temporal FNPs. Evaluated on real-world benchmark time series classification datasets, StatioCL demonstrates a substantial improvement over state-of-the-art CL methods, achieving a 2.9% increase in Recall and a 19.2% reduction in FNPs. Most importantly, StatioCL also shows enhanced data efficiency and robustness against label scarcity.
Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning
Venue: arXiv cs.LG 2024
Year: 2024

In-Context Learning (ICL) emerges as a key feature for Large Language Models (LLMs), allowing them to adapt to new tasks by leveraging task-specific examples without updating model parameters. However, ICL faces challenges with increasing numbers of examples due to performance degradation and quadratic computational costs. In this paper, we propose Logit Arithmetic Reweighting Approach (LARA), a novel framework that enhances ICL by using logit-based ensembling of multiple demonstrations. Our approach divides long input demonstrations into parallelizable shorter inputs to significantly reduce memory requirements, and then effectively aggregate the information by reweighting logits of each group via a non-gradient optimization approach. We further introduce Binary LARA (B-LARA), a variant that constrains weights to binary values to simplify the search space and reduces memory usage by filtering out less informative demonstration groups. Experiments on BBH and MMLU demonstrate that LARA and B-LARA outperform all baseline methods in both accuracy and memory efficiency. We also conduct extensive analysis to show that LARA generalizes well to scenarios of varying numbers of examples from limited to many-shot demonstrations.
VNF Migration with Fast Defragmentation: A GAT-Based Deep Learning Method
Venue: arXiv cs.NI 2024
Year: 2024

Network function virtualization (NFV) enhances service flexibility by decoupling network functions from dedicated hardware. To handle time-varying traffic in NFV network, virtualized network function (VNF) migration has been involved to dynamically adjust resource allocation. However, as network functions diversify, different resource types may be underutilized due to bottlenecks, which can be described as multidimensional resource fragmentation. To address this issue, we firstly define a metric to quantify resource fragmentation in NFV networks. Then, we propose a multi-hop graph attention network (MHGAT) model to effectively extract resource features from tailored network layers, which captures the overall network state and produces high-quality strategies rapidly. Building on this, we develop an MHGAT method to implement fast defragmentation and optimize VNF migration. Simulations demonstrate that by fast defragmentation, the MHGAT method improves the acceptance ratio by an average of 12.8%, reduces the overload ratio by an average of 30.6%, and lowers migration loss by an average of 43.3% compared to the state-of-art benchmark.
Swift: High-Performance Sparse Tensor Contraction for Scientific Applications
Venue: arXiv cs.DS 2024
Year: 2024

In scientific fields such as quantum computing, physics, chemistry, and machine learning, high dimensional data are typically represented using sparse tensors. Tensor contraction is a popular operation on tensors to exploit meaning or alter the input tensors. Tensor contraction is, however, computationally expensive and grows quadratically with the number of elements. For this reason, specialized algorithms have been created to only operate on the nonzero elements. Current sparse tensor contraction algorithms utilize sub-optimal data structures that perform unnecessary computations which increase execution time and the overall time complexity. We propose Swift, a novel algorithm for sparse tensor contraction that replaces the costly sorting with more efficient grouping, utilizes better data structures to represent tensors, and employs more memory-friendly hash table implementation. Swift is evaluated against the state-of-the-art sparse tensor contraction algorithm, demonstrating up to 20x speedup in various test cases and being able to handle imbalanced input tensors significantly better.
Unified Representation of Genomic and Biomedical Concepts through Multi-Task, Multi-Source Contrastive Learning
Venue: arXiv cs.LG 2024
Year: 2024

We introduce GENomic Encoding REpresentation with Language Model (GENEREL), a framework designed to bridge genetic and biomedical knowledge bases. What sets GENEREL apart is its ability to fine-tune language models to infuse biological knowledge behind clinical concepts such as diseases and medications. This fine-tuning enables the model to capture complex biomedical relationships more effectively, enriching the understanding of how genomic data connects to clinical outcomes. By constructing a unified embedding space for biomedical concepts and a wide range of common SNPs from sources such as patient-level data, biomedical knowledge graphs, and GWAS summaries, GENEREL aligns the embeddings of SNPs and clinical concepts through multi-task contrastive learning. This allows the model to adapt to diverse natural language representations of biomedical concepts while bypassing the limitations of traditional code mapping systems across different data sources. Our experiments demonstrate GENEREL's ability to effectively capture the nuanced relationships between SNPs and clinical concepts. GENEREL also emerges to discern the degree of relatedness, potentially allowing for a more refined identification of concepts. This pioneering approach in constructing a unified embedding system for both SNPs and biomedical concepts enhances the potential for data integration and discovery in biomedical research.
Fast and Accurate Neural Rendering Using Semi-Gradients
Venue: arXiv cs.CV 2024
Year: 2024

We propose a simple yet effective neural network-based framework for global illumination rendering. Recently, rendering techniques that learn neural radiance caches by minimizing the difference (i.e., residual) between the left and right sides of the rendering equation have been suggested. Due to their ease of implementation and the advantage of excluding path integral calculations, these techniques have been applied to various fields, such as free-viewpoint rendering, differentiable rendering, and real-time rendering. However, issues of slow training and occasionally darkened renders have been noted. We identify the cause of these issues as the bias and high variance present in the gradient estimates of the existing residual-based objective function. To address this, we introduce a new objective function that maintains the same global optimum as before but allows for unbiased and low-variance gradient estimates, enabling faster and more accurate training of neural networks. In conclusion, this method is simply implemented by ignoring the partial derivatives of the right-hand side, and theoretical and experimental analyses demonstrate the effectiveness of the proposed loss.
Scalable Multi-Domain Adaptation of Language Models using Modular Experts
Venue: arXiv cs.CL 2024
Year: 2024

Domain-specific adaptation is critical to maximizing the performance of pre-trained language models (PLMs) on one or multiple targeted tasks, especially under resource-constrained use cases, such as edge devices. However, existing methods often struggle to balance domain-specific performance, retention of general knowledge, and efficiency for training and inference. To address these challenges, we propose Modular Domain Experts (MoDE). MoDE is a mixture-of-experts architecture that augments a general PLMs with modular, domain-specialized experts. These experts are trained independently and composed together via a lightweight training process. In contrast to standard low-rank adaptation methods, each MoDE expert consists of several transformer layers which scale better with more training examples and larger parameter counts. Our evaluation demonstrates that MoDE achieves comparable target performances to full parameter fine-tuning while achieving 1.65% better retention performance. Moreover, MoDE's architecture enables flexible sharding configurations and improves training speeds by up to 38% over state-of-the-art distributed training configurations.
Differentially Private Selection using Smooth Sensitivity
Venue: arXiv cs.DS 2024
Year: 2024

With the growing volume of data in society, the need for privacy protection in data analysis also rises. In particular, private selection tasks, wherein the most important information is retrieved under differential privacy are emphasized in a wide range of contexts, including machine learning and medical statistical analysis. However, existing mechanisms use global sensitivity, which may add larger amount of perturbation than is necessary. Therefore, this study proposes a novel mechanism for differentially private selection using the concept of smooth sensitivity and presents theoretical proofs of strict privacy guarantees. Simultaneously, given that the current state-of-the-art algorithm using smooth sensitivity is still of limited use, and that the theoretical analysis of the basic properties of the noise distributions are not yet rigorous, we present fundamental theorems to improve upon them. Furthermore, new theorems are proposed for efficient noise generation. Experiments demonstrate that the proposed mechanism can provide higher accuracy than the existing global sensitivity-based methods. Finally, we show key directions for further theoretical development. Overall, this study can be an important foundational work for expanding the potential of smooth sensitivity in privacy-preserving data analysis. The Python implementation of our experiments and supplemental results are available at https://github.com/ay0408/Smooth-Private-Selection.
Bounding $\varepsilon$-scatter dimension via metric sparsity
Venue: arXiv cs.DM 2024
Year: 2024

A recent work of Abbasi et al. [FOCS 2023] introduced the notion of $\varepsilon$-scatter dimension of a metric space and showed a general framework for efficient parameterized approximation schemes (so-called EPASes) for a wide range of clustering problems in classes of metric spaces that admit a bound on the $\varepsilon$-scatter dimension. Our main result is such a bound for metrics induced by graphs from any fixed proper minor-closed graph class. The bound is double-exponential in $\varepsilon^{-1}$ and the Hadwiger number of the graph class and is accompanied by a nearly tight lower bound that holds even in graph classes of bounded treewidth. On the way to the main result, we introduce metric analogs of well-known graph invariants from the theory of sparsity, including generalized coloring numbers and flatness (aka uniform quasi-wideness), and show bounds for these invariants in proper minor-closed graph classes. Finally, we show the power of newly introduced toolbox by showing a coreset for $k$-Center in any proper minor-closed graph class whose size is polynomial in $k$ (but the exponent of the polynomial depends on the graph class and $\varepsilon^{-1}$).
Yuan: Research on the Concept of Digital World Analogue Scientific Infrastructure and Science Popularization Communication Based on Suzhou Gardens Pattern
Venue: arXiv cs.CR 2024
Year: 2024

In the current digital era, high security relies significantly on advanced concepts such as native security. However, the design and implementation of these concepts face challenges in enterprises and organizations. Leveraging advancements in Large Language Models (LLMs), we draw inspiration from the design principles of Suzhou Gardens, a UNESCO World Heritage site. By examining its core features, which align closely with those of the AI world simulator Sora, we extract three concurrent concepts to enhance the security of future digital infrastructures.We propose three guiding principles to facilitate the preliminary construction of the "Space Spider," a hyper-large scientific infrastructure. These principles will steer the development of the "Yuan" digital garden, establishing a "Chinese Series" focused on the construction pathways of the Yuan AI world simulator. The initial pilot of Yuan is expected to generalize various hyper-large scientific infrastructure scenarios, ultimately expanding into numerous high-security digital applications.Through the design concept of Suzhou Gardens, we aim to promote science communication and talent training in the field of cybersecurity. With the support of Yuan, we intend to extend our efforts to various digital construction domains. This initiative is poised to contribute significantly to the future of digital world simulators, emphasizing the integration of hyper-large scientific infrastructure with science communication and research dissemination.
The formula for the completion time of project networks
Venue: arXiv cs.DM 2024
Year: 2024

This paper formulates the completion time $τ$ of a project network as $ τ=\|\mathbf{R} \mathbf{t} \|_\infty $ where the rows of $\mathbf{R}$ are simple paths of the network and $\mathbf{t}$ is a column vector representing the duration of the activities. Considering this product as a linear transformation leads to interesting findings on the topological relevance of both paths and activities using singular value decomposition. The notion of spectral networks is introduced to condense the fundamental structure of the project network. A definition of project stress is introduced to establish a comparison index between two alternatives in terms of slack. Additionally, the Moore-Penrose inverse of $\mathbf{R}$ is presented to find the configuration of the durations of the activities resulting in a given simple path duration vector. Then, the systematic mapping review process carried out to assess our claims' novelty is reported. Finally, we reflect on the notion of relevance for paths and activities and the relationship of the incidence matrix with the proposed approach.
Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis
Venue: arXiv cs.CV 2024
Year: 2024

The development of computational pathology lies in the consensus that pathological characteristics of tumors are significant guidance for cancer diagnostics. Most existing research focuses on the inner-contextual information within each WSI yet ignores the possible inter-correlations between slides. As the development of tumors is a continuous process involving a series of histological, morphological, and genetic changes that accumulate over time, the similarities and differences between WSIs across various stages, grades, locations and patients should potentially contribute to the representation of WSIs and deserve to be taken into account in WSI modeling. To verify the advancement of introducing the slide inter-correlations into the representation learning of WSIs, we proposed a generic WSI analysis pipeline SlideGCD that can be adapted to any existing Multiple Instance Learning (MIL) frameworks and improve their performance. With the new paradigm, the prior knowledge of cancer development can participate in the end-to-end workflow, which concurrently initializes and refines the slide representation, as a guide for message passing in the slide-based graph. Extensive comparisons and experiments are conducted to validate the effectiveness and robustness of the proposed pipeline across 4 different tasks, including cancer subtyping, cancer staging, survival prediction, and gene mutation prediction, with 7 representative SOTA WSI analysis frameworks as backbones.