OpenRaiser/Intern-Atlas huggingface dataset:
Explorations in Subexponential non-associative non-commutative Linear Logic (extended version)
Venue: arXiv cs.LO 2023
Year: 2023

In a previous work we introduced a non-associative non-commutative logic extended by multimodalities, called subexponentials, licensing local application of structural rules. Here, we further explore this system, considering a classical one-sided multi-succedent classical version of the system, following the exponential-free calculi of Buszkowski's and de Groote and Lamarche's works, where the intuitionistic calculus is shown to embed faithfully into the classical fragment.
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
Venue: arXiv cs.LG 2023
Year: 2023

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.
Beyond Intuition, a Framework for Applying GPs to Real-World Data
Venue: arXiv cs.LG 2023
Year: 2023

Gaussian Processes (GPs) offer an attractive method for regression over small, structured and correlated datasets. However, their deployment is hindered by computational costs and limited guidelines on how to apply GPs beyond simple low-dimensional datasets. We propose a framework to identify the suitability of GPs to a given problem and how to set up a robust and well-specified GP model. The guidelines formalise the decisions of experienced GP practitioners, with an emphasis on kernel design and options for computational scalability. The framework is then applied to a case study of glacier elevation change yielding more accurate results at test time.
JSONoid: Monoid-based Enrichment for Configurable and Scalable Data-Driven Schema Discovery
Venue: arXiv cs.DB 2023
Year: 2023

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is time-consuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.
BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
Venue: arXiv cs.CL 2023
Year: 2023

Automatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained model-based metrics that focus on measuring sentence semantics. However, these neural metrics, while achieving higher correlations with human evaluations, are often considered to be black boxes with potential biases that are difficult to detect. In this study, we systematically analyze and compare various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. Through Minimum Risk Training (MRT), we find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm. By incorporating token-level constraints, we enhance the robustness of evaluation metrics, which in turn leads to an improvement in the performance of machine translation systems. Codes are available at \url{https://github.com/powerpuffpomelo/fairseq_mrt}.
A Scalable Approach to Performing Multiplication and Matrix Dot-Products in Unary
Venue: arXiv cs.ET 2023
Year: 2023

Stochastic computing is a paradigm in which logical operations are performed on randomly generated bit streams. Complex arithmetic operations can be executed by simple logic circuits, resulting in a much smaller area footprint compared to conventional binary counterparts. However, the random or pseudorandom sources required for generating the bit streams are costly in terms of area and offset the advantages. Additionally, due to the inherent randomness, the computation lacks precision, limiting the applicability of this paradigm. Importantly, achieving reasonable accuracy in stochastic computing involves high latency. Recently, deterministic approaches to stochastic computing have been proposed, demonstrating that randomness is not a requirement. By structuring the computation deterministically, exact results can be obtained, and the latency greatly reduced. The bit stream generated adheres to a "unary" encoding, retaining the non-positional nature of the bits while discarding the random bit generation of traditional stochastic computing. This deterministic approach overcomes many drawbacks of stochastic computing, although the latency increases quadratically with each level of logic, becoming unmanageable beyond a few levels. In this paper, we present a method for approximating the results of the deterministic method while maintaining low latency at each level. This improvement comes at the cost of additional logic, but we demonstrate that the increase in area scales with the square root of n, where n represents the equivalent number of binary bits of precision. Our new approach is general, efficient, composable, and applicable to all arithmetic operations performed with stochastic logic. We show that this approach outperforms other stochastic designs for matrix multiplication (dot-product), which is an integral step in nearly all machine learning algorithms.
Pretty Good Strategies for Benaloh Challenge
Venue: arXiv cs.CR 2023
Year: 2023

Benaloh challenge allows the voter to audit the encryption of her vote, and in particular to check whether the vote has been represented correctly. An interesting analysis of the mechanism has been presented by Culnane and Teague. The authors propose a natural game-theoretic model of the interaction between the voter and a corrupt, malicious encryption device. Then, they claim that there is no "natural" rational strategy for the voter to play the game. In consequence, the authorities cannot provide the voter with a sensible auditing strategy, which undermines the whole idea. Here, we claim the contrary, i.e., that there exist simple rational strategies that justify the usefulness of Benaloh challenge.
To pretrain or not to pretrain? A case study of domain-specific pretraining for semantic segmentation in histopathology
Venue: arXiv cs.CV 2023
Year: 2023

Annotating medical imaging datasets is costly, so fine-tuning (or transfer learning) is the most effective method for digital pathology vision applications such as disease classification and semantic segmentation. However, due to texture bias in models trained on real-world images, transfer learning for histopathology applications might result in underperforming models, which necessitates the need for using unlabeled histopathology data and self-supervised methods to discover domain-specific characteristics. Here, we tested the premise that histopathology-specific pretrained models provide better initializations for pathology vision tasks, i.e., gland and cell segmentation. In this study, we compare the performance of gland and cell segmentation tasks with histopathology domain-specific and non-domain-specific (real-world images) pretrained weights. Moreover, we investigate the dataset size at which domain-specific pretraining produces significant gains in performance. In addition, we investigated whether domain-specific initialization improves the effectiveness of out-of-distribution testing on distinct datasets but the same task. The results indicate that performance gain using domain-specific pretrained weights depends on both the task and the size of the training dataset. In instances with limited dataset sizes, a significant improvement in gland segmentation performance was also observed, whereas models trained on cell segmentation datasets exhibit no improvement.
On Invariance, Equivariance, Correlation and Convolution of Spherical Harmonic Representations for Scalar and Vectorial Data
Venue: arXiv cs.LG 2023
Year: 2023

The mathematical representations of data in the Spherical Harmonic (SH) domain has recently regained increasing interest in the machine learning community. This technical report gives an in-depth introduction to the theoretical foundation and practical implementation of SH representations, summarizing works on rotation invariant and equivariant features, as well as convolutions and exact correlations of signals on spheres. In extension, these methods are then generalized from scalar SH representations to Vectorial Harmonics (VH), providing the same capabilities for 3d vector fields on spheres
Machine Learning to detect cyber-attacks and discriminating the types of power system disturbances
Venue: arXiv cs.LG 2023
Year: 2023

This research proposes a machine learning-based attack detection model for power systems, specifically targeting smart grids. By utilizing data and logs collected from Phasor Measuring Devices (PMUs), the model aims to learn system behaviors and effectively identify potential security boundaries. The proposed approach involves crucial stages including dataset pre-processing, feature selection, model creation, and evaluation. To validate our approach, we used a dataset used, consist of 15 separate datasets obtained from different PMUs, relay snort alarms and logs. Three machine learning models: Random Forest, Logistic Regression, and K-Nearest Neighbour were built and evaluated using various performance metrics. The findings indicate that the Random Forest model achieves the highest performance with an accuracy of 90.56% in detecting power system disturbances and has the potential in assisting operators in decision-making processes.
3D Environment Modeling for Falsification and Beyond with Scenic 3.0
Venue: arXiv cs.PL 2023
Year: 2023

We present a major new version of Scenic, a probabilistic programming language for writing formal models of the environments of cyber-physical systems. Scenic has been successfully used for the design and analysis of CPS in a variety of domains, but earlier versions are limited to environments which are essentially two-dimensional. In this paper, we extend Scenic with native support for 3D geometry, introducing new syntax which provides expressive ways to describe 3D configurations while preserving the simplicity and readability of the language. We replace Scenic's simplistic representation of objects as boxes with precise modeling of complex shapes, including a ray tracing-based visibility system that accounts for object occlusion. We also extend the language to support arbitrary temporal requirements expressed in LTL, and build an extensible Scenic parser generated from a formal grammar of the language. Finally, we illustrate the new application domains these features enable with case studies that would have been impossible to accurately model in Scenic 2.
ACDNet: Attention-guided Collaborative Decision Network for Effective Medication Recommendation
Venue: arXiv cs.LG 2023
Year: 2023

Medication recommendation using Electronic Health Records (EHR) is challenging due to complex medical data. Current approaches extract longitudinal information from patient EHR to personalize recommendations. However, existing models often lack sufficient patient representation and overlook the importance of considering the similarity between a patient's medication records and specific medicines. Therefore, an Attention-guided Collaborative Decision Network (ACDNet) for medication recommendation is proposed in this paper. Specifically, ACDNet utilizes attention mechanism and Transformer to effectively capture patient health conditions and medication records by modeling their historical visits at both global and local levels. ACDNet also employs a collaborative decision framework, utilizing the similarity between medication records and medicine representation to facilitate the recommendation process. The experimental results on two extensive medical datasets, MIMIC-III and MIMIC-IV, clearly demonstrate that ACDNet outperforms state-of-the-art models in terms of Jaccard, PR-AUC, and F1 score, reaffirming its superiority. Moreover, the ablation experiments provide solid evidence of the effectiveness of each module in ACDNet, validating their contribution to the overall performance. Furthermore, a detailed case study reinforces the effectiveness of ACDNet in medication recommendation based on EHR data, showcasing its practical value in real-world healthcare scenarios.
DIG: The Data Interface Grammar
Venue: arXiv cs.DB 2023
Year: 2023

Building interactive data interfaces is hard because the design of an interface depends on the data processing needs for the underlying analysis task, yet we do not have a good representation for analysis tasks. To fill this gap, this paper advocates for a Data Interface Grammar (DIG) as an intermediate representation of analysis tasks. We show that DIG is compatible with existing data engineering practices, compact to represent any analysis, simple to translate into an interface design, and amenable to offline analysis. We further illustrate the potential benefits of this abstraction, such as automatic interface generation, automatic interface backend optimization, tutorial generation, and workload generation.
Augmented Reality for Maintenance Tasks with ChatGPT for Automated Text-to-Action
Venue: arXiv cs.HC 2023
Year: 2023

Advancements in sensor technology, artificial intelligence (AI), and augmented reality (AR) have unlocked opportunities across various domains. AR and large language models like GPT have witnessed substantial progress and are increasingly being employed in diverse fields. One such promising application is in operations and maintenance (O&M). O&M tasks often involve complex procedures and sequences that can be challenging to memorize and execute correctly, particularly for novices or under high-stress situations. By marrying the advantages of superimposing virtual objects onto the physical world, and generating human-like text using GPT, we can revolutionize O&M operations. This study introduces a system that combines AR, Optical Character Recognition (OCR), and the GPT language model to optimize user performance while offering trustworthy interactions and alleviating workload in O&M tasks. This system provides an interactive virtual environment controlled by the Unity game engine, facilitating a seamless interaction between virtual and physical realities. A case study (N=15) is conducted to illustrate the findings and answer the research questions. The results indicate that users can complete similarly challenging tasks in less time using our proposed AR and AI system. Moreover, the collected data also suggests a reduction in cognitive load and an increase in trust when executing the same operations using the AR and AI system.
CSCLog: A Component Subsequence Correlation-Aware Log Anomaly Detection Method
Venue: arXiv cs.LG 2023
Year: 2023

Anomaly detection based on system logs plays an important role in intelligent operations, which is a challenging task due to the extremely complex log patterns. Existing methods detect anomalies by capturing the sequential dependencies in log sequences, which ignore the interactions of subsequences. To this end, we propose CSCLog, a Component Subsequence Correlation-Aware Log anomaly detection method, which not only captures the sequential dependencies in subsequences, but also models the implicit correlations of subsequences. Specifically, subsequences are extracted from log sequences based on components and the sequential dependencies in subsequences are captured by Long Short-Term Memory Networks (LSTMs). An implicit correlation encoder is introduced to model the implicit correlations of subsequences adaptively. In addition, Graph Convolution Networks (GCNs) are employed to accomplish the information interactions of subsequences. Finally, attention mechanisms are exploited to fuse the embeddings of all subsequences. Extensive experiments on four publicly available log datasets demonstrate the effectiveness of CSCLog, outperforming the best baseline by an average of 7.41% in Macro F1-Measure.
Weakly-supervised Contrastive Learning for Unsupervised Object Discovery
Venue: arXiv cs.CV 2023
Year: 2023

Unsupervised object discovery (UOD) refers to the task of discriminating the whole region of objects from the background within a scene without relying on labeled datasets, which benefits the task of bounding-box-level localization and pixel-level segmentation. This task is promising due to its ability to discover objects in a generic manner. We roughly categorise existing techniques into two main directions, namely the generative solutions based on image resynthesis, and the clustering methods based on self-supervised models. We have observed that the former heavily relies on the quality of image reconstruction, while the latter shows limitations in effectively modeling semantic correlations. To directly target at object discovery, we focus on the latter approach and propose a novel solution by incorporating weakly-supervised contrastive learning (WCL) to enhance semantic information exploration. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images, which is achieved by fine-tuning the feature encoder of a self-supervised model, namely DINO, via WCL. Subsequently, we introduce Principal Component Analysis (PCA) to localize object regions. The principal projection direction, corresponding to the maximal eigenvalue, serves as an indicator of the object region(s). Extensive experiments on benchmark unsupervised object discovery datasets demonstrate the effectiveness of our proposed solution. The source code and experimental results are publicly available via our project page at https://github.com/npucvr/WSCUOD.git.
Rationality and Behavior Feedback in a Model of Vehicle-to-Vehicle Communication
Venue: arXiv cs.GT 2023
Year: 2023

Vehicle-to-Vehicle (V2V) communication is intended to improve road safety through distributed information sharing; however, this type of system faces a design challenge: it is difficult to predict and optimize how human agents will respond to the introduction of this information. Bayesian games are a standard approach for modeling such scenarios; in a Bayesian game, agents probabilistically adopt various types on the basis of a fixed, known distribution. Agents in such models ostensibly perform Bayesian inference, which may not be a reasonable cognitive demand for most humans. To complicate matters, the information provided to agents is often implicitly dependent on agent behavior, meaning that the distribution of agent types is a function of the behavior of agents (i.e., the type distribution is endogenous). In this paper, we study an existing model of V2V communication, but relax it along two dimensions: first, we pose a behavior model which does not require human agents to perform Bayesian inference; second, we pose an equilibrium model which avoids the challenging endogenous recursion. Surprisingly, we show that the simplified non-Bayesian behavior model yields the exact same equilibrium behavior as the original Bayesian model, which may lend credibility to Bayesian models. However, we also show that the original endogenous equilibrium model is strictly necessary to obtain certain informational paradoxes; these paradoxes do not appear in the simpler exogenous model. This suggests that standard Bayesian game models with fixed type distributions are not sufficient to express certain important phenomena.
Line-Constrained $k$-Semi-Obnoxious Facility Location
Venue: arXiv cs.CG 2023
Year: 2023

Suppose we are given a set $\cal B$ of blue points and a set $\cal R$ of red points, all lying above a horizontal line $\ell$, in the plane. Let the weight of a given point $p_i\in {\cal B}\cup{\cal R}$ be $w_i>0$ if $p_i\in {\cal B}$ and $w_i<0$ if $p_i\in {\cal R}$, $|{\cal B}\cup{\cal R}|=n$, and $d^0$($=d\setminus\partial d$) be the interior of any geometric object $d$. We wish to pack $k$ non-overlapping congruent disks $d_1$, $d_2$, \ldots, $d_k$ of minimum radius, centered on $\ell$ such that $\sum\limits_{j=1}^k\sum\limits_{\{i:\exists p_i\in{\cal R}, p_i\in d_j^0\}}w_i+\sum\limits_{j=1}^k\sum\limits_{\{i:\exists p_i\in{\cal B}, p_i\in d_j\}}w_i$ is maximized, i.e., the sum of the weights of the points covered by $\bigcup\limits_{j=1}^kd_j$ is maximized. Here, the disks are the obnoxious or undesirable facilities generating nuisance or damage (with quantity equal to $w_i$) to every demand point (e.g., population center) $p_i\in {\cal R}$ lying in their interior. In contrast, they are the desirable facilities giving service (equal to $w_i$) to every demand point $p_i\in {\cal B}$ covered by them. The line $\ell$ represents a straight highway or railway line. These $k$ semi-obnoxious facilities need to be established on $\ell$ to receive the largest possible overall service for the nearby attractive demand points while causing minimum damage to the nearby repelling demand points. We show that the problem can be solved optimally in $O(n^4k^2)$ time. Subsequently, we improve the running time to $O(n^3k \cdot\max{(\log n, k)})$. The above-weighted variation of locating $k$ semi-obnoxious facilities may generalize the problem that Bereg et al. (2015) studied where $k=1$ i.e., the smallest radius maximum weight circle is to be centered on a line. Furthermore, we addressed two special cases of the problem where points do not have arbitrary weights.
Derivative Free Weight-space Ensembling
Venue: arXiv cs.CL 2023
Year: 2023

Recent work suggests that interpolating between the weights of two specialized language models can transfer knowledge between tasks in a way that multi-task learning cannot. However, very few have explored interpolation between more than two models, where each has a distinct knowledge base. In this paper, we introduce Derivative Free Weight-space Ensembling (DFWE), a new few-sample task transfer approach for open-domain dialogue. Our framework creates a set of diverse expert language models trained using a predefined set of source tasks. Next, we finetune each of the expert models on the target task, approaching the target task from several distinct knowledge bases. Finally, we linearly interpolate between the model weights using a gradient-free-optimization algorithm, to efficiently find a good interpolation weighting. We demonstrate the effectiveness of the method on FETA-Friends outperforming the standard pretrain-finetune approach.