OpenRaiser/Intern-Atlas huggingface dataset:
JuMP 1.0: Recent improvements to a modeling language for mathematical optimization
Venue: arXiv cs.PL 2023
Year: 2023

JuMP is an algebraic modeling language embedded in the Julia programming language. JuMP allows users to model optimization problems of a variety of kinds, including linear programming, integer programming, conic optimization, semidefinite programming, and nonlinear programming, and handles the low-level details of communicating with solvers. After nearly 10 years in development, JuMP 1.0 was released in March, 2022. In this short communication, we highlight the improvements to JuMP from recent releases up to and including 1.0.
DORA: Exploring Outlier Representations in Deep Neural Networks
Venue: arXiv cs.LG 2023
Year: 2023

Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (Data-agnOstic Representation Analysis), the first data-agnostic framework for analyzing the representational space of DNNs. Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns on data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models to illustrate that representations identified as outliers using the EA metric often correspond to undesired and spurious concepts.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Venue: arXiv cs.CL 2023
Year: 2023

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Unifying Multimodal Source and Propagation Graph for Rumour Detection on Social Media with Missing Features
Venue: arXiv cs.MM 2023
Year: 2023

With the rapid development of online social media platforms, the spread of rumours has become a critical societal concern. Current methods for rumour detection can be categorized into image-text pair classification and source-reply graph classification. In this paper, we propose a novel approach that combines multimodal source and propagation graph features for rumour classification. We introduce the Unified Multimodal Graph Transformer Network (UMGTN) which integrates Transformer encoders to fuse these features. Given that not every message in social media is associated with an image and community responses in propagation graphs do not immediately follow source messages, our aim is to build a network architecture that handles missing features such as images or replies. To enhance the model's robustness to data with missing features, we adopt a multitask learning framework that simultaneously learns representations between samples with complete and missing features. We evaluate our proposed method on four real-world datasets, augmenting them by recovering images and replies from Twitter and Weibo. Experimental results demonstrate that our UMGTN with multitask learning achieves state-of-the-art performance, improving F1-score by 1.0% to 4.0%, while maintaining detection robustness to missing features within 2% accuracy and F1-score compared to models trained without the multitask learning framework. We have made our models and datasets publicly available at: https://thcheung.github.io/umgtn/.
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Venue: arXiv cs.LG 2023
Year: 2023

Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training process is extremely memory-intensive and communication-intensive. These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency. To achieve this goal, some custom software frameworks such as Megatron-LM and DeepSpeed are developed. However, current 3D parallelism frameworks still meet two issues: i) they are not transparent to model developers, which need to manually modify the model to parallelize training. ii) their utilization of computation, GPU memory and network bandwidth are not sufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys with an automatic model partitioner, which uses a graph sharding algorithm on a proxy representation of the model. Merak also presents the non-intrusive API for scaling out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine in Merak. It uses several techniques to exploit available training resources, including shifted critical path pipeline schedule that brings a higher computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs show Merak can speedup the training performance over the state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42X, 1.39X, 1.43X, and 1.61X, respectively.
Neighborhoods of binary self-dual codes
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, we introduce and investigate the neighborhood of binary self-dual codes. We prove that there is no better Type I code than the best Type II code of the same length. Further, we give some new necessary conditions for the existence of a singly-even $(56,28,12)$-code and a doubly-even $(72,36,16)$-code.
Mathematical Theory of Bayesian Statistics for Unknown Information Source
Venue: arXiv cs.LG 2023
Year: 2023

In statistical inference, uncertainty is unknown and all models are wrong. That is to say, a person who makes a statistical model and a prior distribution is simultaneously aware that both are fictional candidates. To study such cases, statistical measures have been constructed, such as cross validation, information criteria, and marginal likelihood, however, their mathematical properties have not yet been completely clarified when statistical models are under- and over- parametrized. We introduce a place of mathematical theory of Bayesian statistics for unknown uncertainty, which clarifies general properties of cross validation, information criteria, and marginal likelihood, even if an unknown data-generating process is unrealizable by a model or even if the posterior distribution cannot be approximated by any normal distribution. Hence it gives a helpful standpoint for a person who cannot believe in any specific model and prior. This paper consists of three parts. The first is a new result, whereas the second and third are well-known previous results with new experiments. We show there exists a more precise estimator of the generalization loss than leave-one-out cross validation, there exists a more accurate approximation of marginal likelihood than BIC, and the optimal hyperparameters for generalization loss and marginal likelihood are different.
A Matrix Decomposition Model Based on Feature Factors in Movie Recommendation System
Venue: arXiv cs.IR 2023
Year: 2023

Currently, matrix decomposition is one of the most widely used collaborative filtering algorithms by using factor decomposition to effectively deal with large-scale rating matrix. It mainly uses the interaction records between users and items to predict ratings. Based on the characteristic attributes of items and users, this paper proposes a new UISVD++ model that fuses the type attributes of movies and the age attributes of users into SVD++ framework. By projecting the age attribute into the user's implicit space and the type attribute into the item's implicit space, the model enriches the side information of the users and items. At last, we conduct comparative experiments on two public data sets, Movielens-100K and Movielens-1M. Experiment results express that the prediction accuracy of this model is better than other baselines in the task of predicting scores. In addition, these results also show that UISVD++ can effectively alleviate the cold start situation.
RL-GA: A Reinforcement Learning-Based Genetic Algorithm for Electromagnetic Detection Satellite Scheduling Problem
Venue: arXiv cs.NE 2023
Year: 2023

The study of electromagnetic detection satellite scheduling problem (EDSSP) has attracted attention due to the detection requirements for a large number of targets. This paper proposes a mixed-integer programming model for the EDSSP problem and a genetic algorithm based on reinforcement learning (RL-GA). Numerous factors that affect electromagnetic detection are considered in the model, such as detection mode, bandwidth, and other factors. The RL-GA embeds a Q-learning method into an improved genetic algorithm, and the evolution of each individual depends on the decision of the agent. Q-learning is used to guide the population search process by choosing evolution operators. In this way, the search information can be effectively used by the reinforcement learning method. In the algorithm, we design a reward function to update the Q value. According to the problem characteristics, a new combination of <state, action> is proposed. The RL-GA also uses an elite individual retention strategy to improve search performance. After that, a task time window selection algorithm (TTWSA) is proposed to evaluate the performance of population evolution. Several experiments are used to examine the scheduling effect of the proposed algorithm. Through the experimental verification of multiple instances, it can be seen that the RL-GA can solve the EDSSP problem effectively. Compared with the state-of-the-art algorithms, the RL-GA performs better in several aspects.
Deadline-constrained Multi-resource Task Mapping and Allocation for Edge-Cloud Systems
Venue: arXiv cs.DC 2023
Year: 2023

In an edge-cloud system, mobile devices can offload their computation intensive tasks to an edge or cloud server to guarantee the quality of service or satisfy task deadline requirements. However, it is challenging to determine where tasks should be offloaded and processed, and how much network and computation resources should be allocated to them, such that a system with limited resources can obtain a maximum profit while meeting the deadlines. A key challenge in this problem is that the network and computation resources could be allocated on different servers, since the server to which a task is offloaded (e.g., a server with an access point) may be different from the server on which the task is eventually processed. To address this challenge, we first formulate the task mapping and resource allocation problem as a non-convex Mixed-Integer Nonlinear Programming (MINLP) problem, known as NP-hard. We then propose a zero-slack based greedy algorithm (ZSG) and a linear discretization method (LDM) to solve this MINLP problem. Experiment results with various synthetic tasksets show that ZSG has an average of $2.98\%$ worse performance than LDM with a minimum unit of 5 but has an average of $6.88\%$ better performance than LDM with a minimum unit of 15.
Neuromorphic Wireless Cognition: Event-Driven Semantic Communications for Remote Inference
Venue: arXiv cs.IT 2023
Year: 2023

Neuromorphic computing is an emerging computing paradigm that moves away from batched processing towards the online, event-driven, processing of streaming data. Neuromorphic chips, when coupled with spike-based sensors, can inherently adapt to the "semantics" of the data distribution by consuming energy only when relevant events are recorded in the timing of spikes and by proving a low-latency response to changing conditions in the environment. This paper proposes an end-to-end design for a neuromorphic wireless Internet-of-Things system that integrates spike-based sensing, processing, and communication. In the proposed NeuroComm system, each sensing device is equipped with a neuromorphic sensor, a spiking neural network (SNN), and an impulse radio transmitter with multiple antennas. Transmission takes place over a shared fading channel to a receiver equipped with a multi-antenna impulse radio receiver and with an SNN. In order to enable adaptation of the receiver to the fading channel conditions, we introduce a hypernetwork to control the weights of the decoding SNN using pilots. Pilots, encoding SNNs, decoding SNN, and hypernetwork are jointly trained across multiple channel realizations. The proposed system is shown to significantly improve over conventional frame-based digital solutions, as well as over alternative non-adaptive training methods, in terms of time-to-accuracy and energy consumption metrics.
Brain tumour segmentation with incomplete imaging data
Venue: arXiv cs.CV 2023
Year: 2023

The complex heterogeneity of brain tumours is increasingly recognized to demand data of magnitudes and richness only fully-inclusive, large-scale collections drawn from routine clinical care could plausibly offer. This is a task contemporary machine learning could facilitate, especially in neuroimaging, but its ability to deal with incomplete data common in real world clinical practice remains unknown. Here we apply state-of-the-art methods to large scale, multi-site MRI data to quantify the comparative fidelity of automated tumour segmentation models replicating the various levels of sequence availability observed in the clinical reality. We compare deep learning (nnU-Net-derived) segmentation models with all possible combinations of T1, contrast-enhanced T1, T2, and FLAIR sequences, trained and validated with five-fold cross-validation on the 2021 BraTS-RSNA glioma population of 1251 patients, with further testing on a real-world 50 patient sample diverse in not only MRI scanner and field strength, but a random selection of pre- and post-operative imaging also. Models trained on incomplete imaging data segmented lesions well, often equivalently to those trained on complete data, exhibiting Dice coefficients of 0.907 (single sequence) to 0.945 (full datasets) for whole tumours, and 0.701 (single sequence) to 0.891 (full datasets) for component tissue types. Incomplete data segmentation models could accurately detect enhancing tumour in the absence of contrast imaging, quantifying its volume with an R2 between 0.95-0.97, and were invariant to lesion morphometry. Deep learning segmentation models characterize tumours well when missing data and can even detect enhancing tissue without the use of contrast. This suggests translation to clinical practice, where incomplete data is common, may be easier than hitherto believed, and may be of value in reducing dependence on contrast use.
A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks
Venue: arXiv cs.NE 2023
Year: 2023

Spiking neural networks (SNNs) have demonstrated excellent capabilities in various intelligent scenarios. Most existing methods for training SNNs are based on the concept of synaptic plasticity; however, learning in the realistic brain also utilizes intrinsic non-synaptic mechanisms of neurons. The spike threshold of biological neurons is a critical intrinsic neuronal feature that exhibits rich dynamics on a millisecond timescale and has been proposed as an underlying mechanism that facilitates neural information processing. In this study, we develop a novel synergistic learning approach that involves simultaneously training synaptic weights and spike thresholds in SNNs. SNNs trained with synapse-threshold synergistic learning~(STL-SNNs) achieve significantly superior performance on various static and neuromorphic datasets than SNNs trained with two degenerated single-learning models. During training, the synergistic learning approach optimizes neural thresholds, providing the network with stable signal transmission via appropriate firing rates. Further analysis indicates that STL-SNNs are robust to noisy data and exhibit low energy consumption for deep network structures. Additionally, the performance of STL-SNN can be further improved by introducing a generalized joint decision framework. Overall, our findings indicate that biologically plausible synergies between synaptic and intrinsic non-synaptic mechanisms may provide a promising approach for developing highly efficient SNN learning methods.
Absolute Expressiveness of Subgraph-based Centrality Measures
Venue: arXiv cs.AI 2023
Year: 2023

In graph-based applications, a common task is to pinpoint the most important or ``central'' vertex in a (directed or undirected) graph, or rank the vertices of a graph according to their importance. To this end, a plethora of so-called centrality measures have been proposed in the literature. Such measures assess which vertices in a graph are the most important ones by analyzing the structure of the underlying graph. A family of centrality measures that are suited for graph databases has been recently proposed by relying on the following simple principle: the importance of a vertex in a graph is relative to the number of ``relevant'' connected subgraphs surrounding it; we refer to the members of this family as subgraph-based centrality measures. Although it has been shown that such measures enjoy several favourable properties, their absolute expressiveness remains largely unexplored. The goal of this work is to precisely characterize the absolute expressiveness of the family of subgraph-based centrality measures by considering both directed and undirected graphs. To this end, we characterize when an arbitrary centrality measure is a subgraph-based one, or a subgraph-based measure relative to the induced ranking. These characterizations provide us with technical tools that allow us to determine whether well-established centrality measures are subgraph-based. Such a classification, apart from being interesting in its own right, gives useful insights on the structural similarities and differences among existing centrality measures.
A Unified Approach for Dynamic Analysis of Tensegrity Structures with Arbitrary Rigid Bodies and Rigid Bars
Venue: arXiv cs.CE 2023
Year: 2023

This paper proposes a unified approach for dynamic modeling and simulations of general tensegrity structures with rigid bars and rigid bodies of arbitrary shapes. The natural coordinates are adopted as a non-minimal description in terms of different combinations of basic points and base vectors to resolve the heterogeneity between rigid bodies and rigid bars in three-dimensional space. This leads to a set of differential-algebraic equations with a constant mass matrix and free from trigonometric functions. Formulations for linearized dynamics are derived to enable modal analysis around static equilibrium. For numerical analysis of nonlinear dynamics, we derive a modified symplectic integration scheme which yields realistic results for long-time simulations, and accommodates non-conservative forces as well as boundary conditions. Numerical examples demonstrate the efficacy of the proposed approach for dynamic simulations of Class-1-to-$k$ general tensegrity structures under complex situations, including dynamic external loads, cable-based deployments, and moving boundaries. The novel tensegrity structures also exemplify new ways to create multi-functional structures.
Convergence for score-based generative modeling with polynomial complexity
Venue: arXiv cs.LG 2023
Year: 2023

Score-based generative modeling (SGM) is a highly successful approach for learning a probability distribution from data and generating further samples. We prove the first polynomial convergence guarantees for the core mechanic behind SGM: drawing samples from a probability density $p$ given a score estimate (an estimate of $\nabla \ln p$) that is accurate in $L^2(p)$. Compared to previous works, we do not incur error that grows exponentially in time or that suffers from a curse of dimensionality. Our guarantee works for any smooth distribution and depends polynomially on its log-Sobolev constant. Using our guarantee, we give a theoretical analysis of score-based generative modeling, which transforms white-noise input into samples from a learned data distribution given score estimates at different noise scales. Our analysis gives theoretical grounding to the observation that an annealed procedure is required in practice to generate good samples, as our proof depends essentially on using annealing to obtain a warm start at each step. Moreover, we show that a predictor-corrector algorithm gives better convergence than using either portion alone.
An adversarially robust data-market for spatial, crowd-sourced data
Venue: arXiv cs.DS 2023
Year: 2023

We describe an architecture for a decentralised data market for applications in which agents are incentivised to collaborate to crowd-source their data. The architecture is designed to reward data that furthers the market's collective goal, and distributes reward fairly to all those that contribute with their data. We show that the architecture is resilient to Sybil, wormhole, and data poisoning attacks. In order to evaluate the resilience of the architecture, we characterise its breakdown points for various adversarial threat models in an automotive use case.