OpenRaiser/Intern-Atlas huggingface dataset:
Schema First! Learn Versatile Knowledge Graph Embeddings by Capturing Semantics with MASCHInE
Venue: arXiv cs.AI 2023
Year: 2023

Knowledge graph embedding models (KGEMs) have gained considerable traction in recent years. These models learn a vector representation of knowledge graph entities and relations, a.k.a. knowledge graph embeddings (KGEs). Learning versatile KGEs is desirable as it makes them useful for a broad range of tasks. However, KGEMs are usually trained for a specific task, which makes their embeddings task-dependent. In parallel, the widespread assumption that KGEMs actually create a semantic representation of the underlying entities and relations (e.g., project similar entities closer than dissimilar ones) has been challenged. In this work, we design heuristics for generating protographs -- small, modified versions of a KG that leverage RDF/S information. The learnt protograph-based embeddings are meant to encapsulate the semantics of a KG, and can be leveraged in learning KGEs that, in turn, also better capture semantics. Extensive experiments on various evaluation benchmarks demonstrate the soundness of this approach, which we call Modular and Agnostic SCHema-based Integration of protograph Embeddings (MASCHInE). In particular, MASCHInE helps produce more versatile KGEs that yield substantially better performance for entity clustering and node classification tasks. For link prediction, using MASCHinE substantially increases the number of semantically valid predictions with equivalent rank-based performance.
Effective Intrusion Detection in Highly Imbalanced IoT Networks with Lightweight S2CGAN-IDS
Venue: arXiv cs.CR 2023
Year: 2023

Since the advent of the Internet of Things (IoT), exchanging vast amounts of information has increased the number of security threats in networks. As a result, intrusion detection based on deep learning (DL) has been developed to achieve high throughput and high precision. Unlike general deep learning-based scenarios, IoT networks contain benign traffic far more than abnormal traffic, with some rare attacks. However, most existing studies have been focused on sacrificing the detection rate of the majority class in order to improve the detection rate of the minority class in class-imbalanced IoT networks. Although this way can reduce the false negative rate of minority classes, it both wastes resources and reduces the credibility of the intrusion detection systems. To address this issue, we propose a lightweight framework named S2CGAN-IDS. The proposed framework leverages the distribution characteristics of network traffic to expand the number of minority categories in both data space and feature space, resulting in a substantial increase in the detection rate of minority categories while simultaneously ensuring the detection precision of majority categories. To reduce the impact of sparsity on the experiments, the CICIDS2017 numeric dataset is utilized to demonstrate the effectiveness of the proposed method. The experimental results indicate that our proposed approach outperforms the superior method in both Precision and Recall, particularly with a 10.2% improvement in the F1-score.
DashQL -- Complete Analysis Workflows with SQL
Venue: arXiv cs.HC 2023
Year: 2023

We present DashQL, a language that describes complete analysis workflows in self-contained scripts. DashQL combines SQL, the grammar of relational database systems, with a grammar of graphics in a grammar of analytics. It supports preparing and visualizing arbitrarily complex SQL statements in a single coherent language. The proximity to SQL facilitates holistic optimizations of analysis workflows covering data input, encoding, transformations, and visualizations. These optimizations use model and query metadata for visualization-driven aggregation, remote predicate pushdown, and adaptive materialization. We introduce the DashQL language as an extension of SQL and describe the efficient and interactive processing of text-based analysis workflows.
Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Tagging
Venue: arXiv cs.CL 2023
Year: 2023

The U.S. Securities and Exchange Commission (SEC) mandates all public companies to file periodic financial statements that should contain numerals annotated with a particular label from a taxonomy. In this paper, we formulate the task of automating the assignment of a label to a particular numeral span in a sentence from an extremely large label set. Towards this task, we release a dataset, Financial Numeric Extreme Labelling (FNXL), annotated with 2,794 labels. We benchmark the performance of the FNXL dataset by formulating the task as (a) a sequence labelling problem and (b) a pipeline with span extraction followed by Extreme Classification. Although the two approaches perform comparably, the pipeline solution provides a slight edge for the least frequent labels.
Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPU
Venue: arXiv cs.LG 2023
Year: 2023

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, it can result in much diminished predictive performance of the model. Fortunately, we found that this can be mitigated by introducing a penultimate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be uniform, in the sense that each output neuron will have the exact same number of incoming connections. This allows for efficient implementations of sparse matrix multiplication and connection redistribution on GPU hardware. Via a custom CUDA implementation, we show that the proposed approach can scale to datasets with 670,000 labels on a single commodity GPU with only 4GB memory.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Venue: arXiv cs.CL 2023
Year: 2023

While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often attributed to functional pressures. In the effort to identify these pressures, prior work has compared real and counterfactual word orders. Yet one functional pressure has been overlooked in such investigations: the uniform information density (UID) hypothesis, which holds that information should be spread evenly throughout an utterance. Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically. To this end, we use computational models to test whether real orders lead to greater information uniformity than counterfactual orders. In our empirical study of 10 typologically diverse languages, we find that: (i) among SVO languages, real word orders consistently have greater uniformity than reverse word orders, and (ii) only linguistically implausible counterfactual orders consistently exceed the uniformity of real orders. These findings are compatible with a pressure for information uniformity in the development and usage of natural languages.
Exploring the effects of robotic design on learning and neural control
Venue: arXiv cs.RO 2023
Year: 2023

The ongoing deep learning revolution has allowed computers to outclass humans in various games and perceive features imperceptible to humans during classification tasks. Current machine learning techniques have clearly distinguished themselves in specialized tasks. However, we have yet to see robots capable of performing multiple tasks at an expert level. Most work in this field is focused on the development of more sophisticated learning algorithms for a robot's controller given a largely static and presupposed robotic design. By focusing on the development of robotic bodies, rather than neural controllers, I have discovered that robots can be designed such that they overcome many of the current pitfalls encountered by neural controllers in multitask settings. Through this discovery, I also present novel metrics to explicitly measure the learning ability of a robotic design and its resistance to common problems such as catastrophic interference. Traditionally, the physical robot design requires human engineers to plan every aspect of the system, which is expensive and often relies on human intuition. In contrast, within the field of evolutionary robotics, evolutionary algorithms are used to automatically create optimized designs, however, such designs are often still limited in their ability to perform in a multitask setting. The metrics created and presented here give a novel path to automated design that allow evolved robots to synergize with their controller to improve the computational efficiency of their learning while overcoming catastrophic interference. Overall, this dissertation intimates the ability to automatically design robots that are more general purpose than current robots and that can perform various tasks while requiring less computation.
Development of On-Ground Hardware In Loop Simulation Facility for Space Robotics
Venue: arXiv cs.RO 2023
Year: 2023

Over a couple of decades, space junk has increased rapidly, which has caused significant threats to the LEO operation satellites. An Active Debris Removal $(ADR)$ concept continuously evolves for space junk removal. One of the ADR methods is Space Robotics, whose function is to chase, capture and de-orbit the space junk. This paper presents the development of an on-ground space robotics facility in the TCS Research for on-orbit servicing $(OOS)$ like refueling and debris capture experiments. A Hardware in Loop Simulation (HILS) system will be used for integrated system development, testing, and demonstration of on-orbit docking mechanisms. The HiLS test facility of TCS Research Lab will use two URs in which one UR is attached to the RG2 gripper, and the other is attached to a force-torque sensor and with a scaled mock-up model. The first UR5 will be mounted on a 7-axis linear rail and contain the docking probe. First, UR5 with a suitable gripper has to interface its control boxes. The grasping algorithm was run through the ROS interface line to demonstrate and validate the on-orbit operations. The manipulator will be mounted with LIDAR and a camera to visualize the mock-up model, find the target model's pose and rotational velocity estimation, and a gripper that will move relative to the target model. The other manipulator has the UR10 control, providing rotational and random motion to the mockup, enabling a dynamic simulator fed by force-torque data. The dynamic simulator is fed up with the orbit propagator, which will provide the orbiting environment to the target model. For the simulation of the docking and grasping of the target model, a linear rail of a 6m setup is still in the procurement process. Once reaching proximity, the grasping algorithm will be launched to capture the target model after reading the random motion of the mock-up model.
An Exponential Envy-Free Cake Cutting Protocol for $n$ Agents
Venue: arXiv cs.CC 2023
Year: 2023

We consider a classical envy-free cake cutting problem. The first limited protocol was proposed by Aziz and McKenzie in 2016 arXiv:1604.03655. The disadvantage of this protocol is its high complexity. The authors proved that the maximum number of queries required by the protocol is $n^{n^{n^{n^{n^n}}}}$. We made minor changes to the Aziz-Mackenzie protocol, improved estimation of the required number of queries and made an algorithm that uses at most $n^{8n^2(1 + o(1))}$ queries.
Pivotuner: automatic real-time pure intonation and microtonal modulation
Venue: arXiv cs.MM 2023
Year: 2023

Pivotuner is a VST3/AU MIDI effect plugin that automatically tunes note data in an adaptive pure intonation, in real time. Where previously pure intonation was out of reach for most musicians due to difficulty and impracticality, Pivotuner enables it to be achieved easily and straightforwardly by using novel yet simple algorithms. This may lead to more widespread exploration of pure intonation for a larger and more diverse crowd of musicians! This paper includes a review of prior systems for adaptive pure intonation systems, including Hermode Tuning/Kontakt Dynamic Pure Tuning and Just Intonation. The paper introduces the notion of an adaptive tuning center and how it serves as a flexible underlying concept for multiple tuning algorithms, as well as extensions to offer greater control for performers, including pitch and tuning center locking and resetting, and gradual interpolation between equal temperament and pure intonation. The paper then showcases some pieces which use Pivotuner effectively, then discusses areas for future exploration within Pivotuner's feature set, and plans for future development.
Causal interventions expose implicit situation models for commonsense language understanding
Venue: arXiv cs.CL 2023
Year: 2023

Accounts of human language processing have long appealed to implicit ``situation models'' that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent transformer models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a relatively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the candidate noun phrases the pronoun ultimately attends to. We then compare how this circuit behaves in a closely matched ``syntactic'' control where the situation model is not strictly necessary. These analyses suggest distinct pathways through which implicit situation models are constructed to guide pronoun resolution.
ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory
Venue: arXiv cs.AI 2023
Year: 2023

Large language models (LLMs) with memory are computationally universal. However, mainstream LLMs are not taking full advantage of memory, and the designs are heavily influenced by biological brains. Due to their approximate nature and proneness to the accumulation of errors, conventional neural memory mechanisms cannot support LLMs to simulate complex reasoning. In this paper, we seek inspiration from modern computer architectures to augment LLMs with symbolic memory for complex multi-hop reasoning. Such a symbolic memory framework is instantiated as an LLM and a set of SQL databases, where the LLM generates SQL instructions to manipulate the SQL databases. We validate the effectiveness of the proposed memory framework on a synthetic dataset requiring complex reasoning. The project website is available at https://chatdatabase.github.io/ .
Partial Inference in Structured Prediction
Venue: arXiv cs.LG 2023
Year: 2023

In this paper, we examine the problem of partial inference in the context of structured prediction. Using a generative model approach, we consider the task of maximizing a score function with unary and pairwise potentials in the space of labels on graphs. Employing a two-stage convex optimization algorithm for label recovery, we analyze the conditions under which a majority of the labels can be recovered. We introduce a novel perspective on the Karush-Kuhn-Tucker (KKT) conditions and primal and dual construction, and provide statistical and topological requirements for partial recovery with provable guarantees.
Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction
Venue: arXiv cs.CL 2023
Year: 2023

Task-oriented dialogues often require agents to enact complex, multi-step procedures in order to meet user requests. While large language models have found success automating these dialogues in constrained environments, their widespread deployment is limited by the substantial quantities of task-specific data required for training. The following paper presents a data-efficient solution to constructing dialogue systems, leveraging explicit instructions derived from agent guidelines, such as company policies or customer service manuals. Our proposed Knowledge-Augmented Dialogue System (KADS) combines a large language model with a knowledge retrieval module that pulls documents outlining relevant procedures from a predefined set of policies, given a user-agent interaction. To train this system, we introduce a semi-supervised pre-training scheme that employs dialogue-document matching and action-oriented masked language modeling with partial parameter freezing. We evaluate the effectiveness of our approach on prominent task-oriented dialogue datasets, Action-Based Conversations Dataset and Schema-Guided Dialogue, for two dialogue tasks: action state tracking and workflow discovery. Our results demonstrate that procedural knowledge augmentation improves accuracy predicting in- and out-of-distribution actions while preserving high performance in settings with low or sparse data.
Fifty Years of ISCA: A data-driven retrospective on key trends
Venue: arXiv cs.AR 2023
Year: 2023

Computer Architecture, broadly, involves optimizing hardware and software for current and future processing systems. Although there are several other top venues to publish Computer Architecture research, including ASPLOS, HPCA, and MICRO, ISCA (the International Symposium on Computer Architecture) is one of the oldest, longest running, and most prestigious venues for publishing Computer Architecture research. Since 1973, except for 1975, ISCA has been organized annually. Accordingly, this year will be the 50th year of ISCA. Thus, we set out to analyze the past 50 years of ISCA to understand who and what has been driving and innovating computing systems thus far. Our analysis identifies several interesting trends that reflect how ISCA, and Computer Architecture in general, has grown and evolved in the past 50 years, including minicomputers, general-purpose uniprocessor CPUs, multiprocessor and multi-core CPUs, general-purpose GPUs, and accelerators.
A Reference-less Slope Detection Technique in 65nm for Robust Sensing of 1T1R Arrays
Venue: arXiv cs.ET 2023
Year: 2023

Spin-Torque-Transfer RAM (STTRAM) is a promising technology however process variation poses serious challenge to sensing. To eliminate bit-to-bit process variation we propose a reference-less, destructive slope detection technique which exploits the MTJ switching from high to low state to detect memory state. A proof-of-concept fabricated test-chip using 96kb mimicked STTRAM bits in 65nm technology shows that slope sensing reduces failure rate by 120X in 2.5K-5K array@TMR=100% and 162X in 2.5K-5K@TMR=80% array compared to conventional voltage sensing.
Counterfactual Explanations and Predictive Models to Enhance Clinical Decision-Making in Schizophrenia using Digital Phenotyping
Venue: arXiv cs.AI 2023
Year: 2023

Clinical practice in psychiatry is burdened with the increased demand for healthcare services and the scarce resources available. New paradigms of health data powered with machine learning techniques could open the possibility to improve clinical workflow in critical stages of clinical assessment and treatment in psychiatry. In this work, we propose a machine learning system capable of predicting, detecting, and explaining individual changes in symptoms of patients with Schizophrenia by using behavioral digital phenotyping data. We forecast symptoms of patients with an error rate below 10%. The system detects decreases in symptoms using changepoint algorithms and uses counterfactual explanations as a recourse in a simulated continuous monitoring scenario in healthcare. Overall, this study offers valuable insights into the performance and potential of counterfactual explanations, predictive models, and change-point detection within a simulated clinical workflow. These findings lay the foundation for further research to explore additional facets of the workflow, aiming to enhance its effectiveness and applicability in real-world healthcare settings. By leveraging these components, the goal is to develop an actionable, interpretable, and trustworthy integrative decision support system that combines real-time clinical assessments with sensor-based inputs.
Evaluating the Potential of Disaggregated Memory Systems for HPC applications
Venue: arXiv cs.DC 2023
Year: 2023

Disaggregated memory is a promising approach that addresses the limitations of traditional memory architectures by enabling memory to be decoupled from compute nodes and shared across a data center. Cloud platforms have deployed such systems to improve overall system memory utilization, but performance can vary across workloads. High-performance computing (HPC) is crucial in scientific and engineering applications, where HPC machines also face the issue of underutilized memory. As a result, improving system memory utilization while understanding workload performance is essential for HPC operators. Therefore, learning the potential of a disaggregated memory system before deployment is a critical step. This paper proposes a methodology for exploring the design space of a disaggregated memory system. It incorporates key metrics that affect performance on disaggregated memory systems: memory capacity, local and remote memory access ratio, injection bandwidth, and bisection bandwidth, providing an intuitive approach to guide machine configurations based on technology trends and workload characteristics. We apply our methodology to analyze thirteen diverse workloads, including AI training, data analysis, genomics, protein, fusion, atomic nuclei, and traditional HPC bookends. Our methodology demonstrates the ability to comprehend the potential and pitfalls of a disaggregated memory system and provides motivation for machine configurations. Our results show that eleven of our thirteen applications can leverage injection bandwidth disaggregated memory without affecting performance, while one pays a rack bisection bandwidth penalty and two pay the system-wide bisection bandwidth penalty. In addition, we also show that intra-rack memory disaggregation would meet the application's memory requirement and provide enough remote memory bandwidth.
PyTrial: Machine Learning Software and Benchmark for Clinical Trial Applications
Venue: arXiv cs.AI 2023
Year: 2023

Clinical trials are conducted to test the effectiveness and safety of potential drugs in humans for regulatory approval. Machine learning (ML) has recently emerged as a new tool to assist in clinical trials. Despite this progress, there have been few efforts to document and benchmark ML4Trial algorithms available to the ML research community. Additionally, the accessibility to clinical trial-related datasets is limited, and there is a lack of well-defined clinical tasks to facilitate the development of new algorithms. To fill this gap, we have developed PyTrial that provides benchmarks and open-source implementations of a series of ML algorithms for clinical trial design and operations. In this paper, we thoroughly investigate 34 ML algorithms for clinical trials across 6 different tasks, including patient outcome prediction, trial site selection, trial outcome prediction, patient-trial matching, trial similarity search, and synthetic data generation. We have also collected and prepared 23 ML-ready datasets as well as their working examples in Jupyter Notebooks for quick implementation and testing. PyTrial defines each task through a simple four-step process: data loading, model specification, model training, and model evaluation, all achievable with just a few lines of code. Furthermore, our modular API architecture empowers practitioners to expand the framework to incorporate new algorithms and tasks effortlessly. The code is available at https://github.com/RyanWangZf/PyTrial.