OpenRaiser/Intern-Atlas huggingface dataset:
PolarFly: A Cost-Effective and Flexible Low-Diameter Topology
Venue: arXiv cs.NI 2023
Year: 2023

In this paper we present PolarFly, a diameter-2 network topology based on the Erdos-Renyi family of polarity graphs from finite geometry. This is a highly scalable low-diameter topology that asymptotically reaches the Moore bound on the number of nodes for a given network degree and diameter PolarFly achieves high Moore bound efficiency even for the moderate radixes commonly seen in current and near-future routers, reaching more than 96% of the theoretical peak. It also offers more feasible router degrees than the state-of-the-art solutions, greatly adding to the selection of scalable diameter-2 networks. PolarFly enjoys many other topological properties highly relevant in practice, such as a modular design and expandability that allow incremental growth in network size without rewiring the whole network. Our evaluation shows that PolarFly outperforms competitive networks in terms of scalability, cost and performance for various traffic patterns.
Learning Prior Feature and Attention Enhanced Image Inpainting
Venue: arXiv cs.CV 2023
Year: 2023

Many recent inpainting works have achieved impressive results by leveraging Deep Neural Networks (DNNs) to model various prior information for image restoration. Unfortunately, the performance of these methods is largely limited by the representation ability of vanilla Convolutional Neural Networks (CNNs) backbones.On the other hand, Vision Transformers (ViT) with self-supervised pre-training have shown great potential for many visual recognition and object detection tasks. A natural question is whether the inpainting task can be greatly benefited from the ViT backbone? However, it is nontrivial to directly replace the new backbones in inpainting networks, as the inpainting is an inverse problem fundamentally different from the recognition tasks. To this end, this paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model, which enjoys richer informative priors to enhance the inpainting process. Moreover, we propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions. Sufficient ablations have been discussed about the inpainting and the self-supervised pre-training models in this paper. Besides, experiments on both Places2 and FFHQ demonstrate the effectiveness of our proposed model. Codes and pre-trained models are released in https://github.com/ewrfcas/MAE-FAR.
EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables
Venue: arXiv cs.LG 2023
Year: 2023

While exogenous variables have a major impact on performance improvement in time series analysis, inter-series correlation and time dependence among them are rarely considered in the present continuous methods. The dynamical systems of multivariate time series could be modelled with complex unknown partial differential equations (PDEs) which play a prominent role in many disciplines of science and engineering. In this paper, we propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system in multivariate time series whose governing equations are parameterised by self-attention and gated recurrent neural networks. The proposed model, \underline{E}xogenous-\underline{g}uided \underline{P}artial \underline{D}ifferential \underline{E}quation Network (EgPDE-Net), takes account of the relationships among the exogenous variables and their effects on the target series. Importantly, the model can be reduced into a regularised ordinary differential equation (ODE) problem with special designed regularisation guidance, which makes the PDE problem tractable to obtain numerical solutions and feasible to predict multiple future values of the target series at arbitrary time points. Extensive experiments demonstrate that our proposed model could achieve competitive accuracy over strong baselines: on average, it outperforms the best baseline by reducing $9.85\%$ on RMSE and $13.98\%$ on MAE for arbitrary-step prediction.
Large scale analysis of gender bias and sexism in song lyrics
Venue: arXiv cs.CY 2023
Year: 2023

We employ Natural Language Processing techniques to analyse 377808 English song lyrics from the "Two Million Song Database" corpus, focusing on the expression of sexism across five decades (1960-2010) and the measurement of gender biases. Using a sexism classifier, we identify sexist lyrics at a larger scale than previous studies using small samples of manually annotated popular songs. Furthermore, we reveal gender biases by measuring associations in word embeddings learned on song lyrics. We find sexist content to increase across time, especially from male artists and for popular songs appearing in Billboard charts. Songs are also shown to contain different language biases depending on the gender of the performer, with male solo artist songs containing more and stronger biases. This is the first large scale analysis of this type, giving insights into language usage in such an influential part of popular culture.
Distributed On-Demand Routing for LEO Mega-Constellations: A Starlink Case Study
Venue: arXiv cs.NI 2023
Year: 2023

The design and launch of large-scale satellite networks create an imminent demand for efficient and delay-minimising routing methods. With the rising number of satellites in such constellations, pre-computing all shortest routes between all satellites and for all times becomes more and more infeasible due to space and time limitations. Even though distributed on-demand routing methods were developed for specific LEO satellite network configurations, they are not suited for increasingly popular mega-constellations based on Walker Delta formations. The contributions of this paper are twofold. First, we introduce a formal model that mathematically captures the time-evolving locations of satellites in a Walker Delta constellation and use it to establish a formula to compute the minimum number of ISL hops between two given satellites. In the second part, we present an on-demand hop-count-based routing algorithm that approximates the optimal path while achieving superior performance compared to classical shortest-path algorithms like Dijkstra.
Fuzzing Deep Learning Compilers with HirGen
Venue: arXiv cs.SE 2023
Year: 2023

Deep Learning (DL) compilers are widely adopted to optimize advanced DL models for efficient deployment on diverse hardware. Their quality has profound effect on the quality of compiled DL models. A recent bug study shows that the optimization of high-level intermediate representation (IR) is the most error-prone compilation stage. Bugs in this stage are accountable for 44.92% of the whole collected ones. However, existing testing techniques do not consider high-level optimization related features (e.g. high-level IR), and are therefore weak in exposing bugs at this stage. To bridge this gap, we propose HirGen, an automated testing technique that aims to effectively expose coding mistakes in the optimization of high-level IR. The design of HirGen includes 1) three coverage criteria to generate diverse and valid computational graphs; 2) full use of high-level IRs language features to generate diverse IRs; 3) three test oracles inspired from both differential testing and metamorphic testing. HirGen has successfully detected 21 bugs that occur at TVM, with 17 bugs confirmed and 12 fixed. Further, we construct four baselines using the state-of-the-art DL compiler fuzzers that can cover the high-level optimization stage. Our experiment results show that HirGen can detect 10 crashes and inconsistencies that cannot be detected by the baselines in 48 hours. We further validate the usefulness of our proposed coverage criteria and test oracles in evaluation.
Interpretable bilinear attention network with domain adaptation improves drug-target prediction
Venue: arXiv cs.LG 2023
Year: 2023

Predicting drug-target interaction is key for drug discovery. Recent deep learning-based methods show promising performance but two challenges remain: (i) how to explicitly model and learn local interactions between drugs and targets for better prediction and interpretation; (ii) how to generalize prediction performance on novel drug-target pairs from different distribution. In this work, we propose DrugBAN, a deep bilinear attention network (BAN) framework with domain adaptation to explicitly learn pair-wise local interactions between drugs and targets, and adapt on out-of-distribution data. DrugBAN works on drug molecular graphs and target protein sequences to perform prediction, with conditional domain adversarial learning to align learned interaction representations across different distributions for better generalization on novel drug-target pairs. Experiments on three benchmark datasets under both in-domain and cross-domain settings show that DrugBAN achieves the best overall performance against five state-of-the-art baselines. Moreover, visualizing the learned bilinear attention map provides interpretable insights from prediction results.
DropKey
Venue: arXiv cs.CV 2023
Year: 2023

In this paper, we focus on analyzing and improving the dropout technique for self-attention layers of Vision Transformer, which is important while surprisingly ignored by prior works. In particular, we conduct researches on three core questions: First, what to drop in self-attention layers? Different from dropping attention weights in literature, we propose to move dropout operations forward ahead of attention matrix calculation and set the Key as the dropout unit, yielding a novel dropout-before-softmax scheme. We theoretically verify that this scheme helps keep both regularization and probability features of attention weights, alleviating the overfittings problem to specific patterns and enhancing the model to globally capture vital information; Second, how to schedule the drop ratio in consecutive layers? In contrast to exploit a constant drop ratio for all layers, we present a new decreasing schedule that gradually decreases the drop ratio along the stack of self-attention layers. We experimentally validate the proposed schedule can avoid overfittings in low-level features and missing in high-level semantics, thus improving the robustness and stableness of model training; Third, whether need to perform structured dropout operation as CNN? We attempt patch-based block-version of dropout operation and find that this useful trick for CNN is not essential for ViT. Given exploration on the above three questions, we present the novel DropKey method that regards Key as the drop unit and exploits decreasing schedule for drop ratio, improving ViTs in a general way. Comprehensive experiments demonstrate the effectiveness of DropKey for various ViT architectures, e.g. T2T and VOLO, as well as for various vision tasks, e.g., image classification, object detection, human-object interaction detection and human body shape recovery.
Bayesian Optimization with Informative Covariance
Venue: arXiv cs.LG 2023
Year: 2023

Bayesian optimization is a methodology for global optimization of unknown and expensive objectives. It combines a surrogate Bayesian regression model with an acquisition function to decide where to evaluate the objective. Typical regression models are given by Gaussian processes with stationary covariance functions. However, these functions are unable to express prior input-dependent information, including possible locations of the optimum. The ubiquity of stationary models has led to the common practice of exploiting prior information via informative mean functions. In this paper, we highlight that these models can perform poorly, especially in high dimensions. We propose novel informative covariance functions for optimization, leveraging nonstationarity to encode preferences for certain regions of the search space and adaptively promote local exploration during optimization. We demonstrate that the proposed functions can increase the sample efficiency of Bayesian optimization in high dimensions, even under weak prior information.
Cryptoeconomic Security for Data Availability Committees
Venue: arXiv cs.CR 2023
Year: 2023

Layer 2 systems have received increasing attention due to their potential to scale the throughput of L1 blockchains. To avoid the cost of putting data on chain, these systems increasingly turn to off-chain data availability solutions such as data availability committees (DACs). However, placing trust on DACs conflicts with the goal of obtaining an L2 architecture whose security relies solely on the L1 chain. To eliminate such trust assumptions, we propose a DAC protocol that provides financial incentives to deter the DAC nodes from adversarial behavior such as withholding data upon request. We then analyze the interaction of rational DAC nodes and clients as a dynamic game, with a Byzantine adversary that can corrupt and bribe the participants. We also define a notion of optimality for the DAC protocols, inspired by fairness and economic feasibility. Our main result shows that our protocol is optimal and guarantees security with the highest possible probability under reasonable assumptions on the adversary.
Scale-friendly In-network Coordination
Venue: arXiv cs.NI 2023
Year: 2023

The programmability of modern network devices has led to innovative research in the area of in-network computing, i.e., offloading certain computations to the programmable data plane. Key-value stores, which offer coordination services for many large-scale data centres, benefited from this technological advancement. Previous research reduced the response latency of key-value requests by half through deploying the store in the programmable data plane. In this work, we identify previous design decisions that have led to increased traffic generation and latency for in-network coordination services. We have developed a new in-network key-value store platform that maintains strong consistency and fault-tolerance, while improving performance and scalability over the state-of-the-art. We have designed and implemented the platform in P4, and analysed the optimisations that unlock these performance improvements. Our evaluation shows a reduction of up to orders of magnitude in latency and significant improvements in throughput. We obtain up to nine times higher throughput for scenarios with multiple participating nodes, indicative of the superior scalability the platform can offer.
An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithms
Venue: arXiv cs.LG 2023
Year: 2023

In this work, we consider policy-based methods for solving the reinforcement learning problem, and establish the sample complexity guarantees. A policy-based algorithm typically consists of an actor and a critic. We consider using various policy update rules for the actor, including the celebrated natural policy gradient. In contrast to the gradient ascent approach taken in the literature, we view natural policy gradient as an approximate way of implementing policy iteration, and show that natural policy gradient (without any regularization) enjoys geometric convergence when using increasing stepsizes. As for the critic, we consider using TD-learning with linear function approximation and off-policy sampling. Since it is well-known that in this setting TD-learning can be unstable, we propose a stable generic algorithm (including two specific algorithms: the $λ$-averaged $Q$-trace and the two-sided $Q$-trace) that uses multi-step return and generalized importance sampling factors, and provide the finite-sample analysis. Combining the geometric convergence of the actor with the finite-sample analysis of the critic, we establish for the first time an overall $\mathcal{O}(ε^{-2})$ sample complexity for finding an optimal policy (up to a function approximation error) using policy-based methods under off-policy sampling and linear function approximation.
Proof-of-Learning is Currently More Broken Than You Think
Venue: arXiv cs.LG 2023
Year: 2023

Proof-of-Learning (PoL) proposes that a model owner logs training checkpoints to establish a proof of having expended the computation necessary for training. The authors of PoL forego cryptographic approaches and trade rigorous security guarantees for scalability to deep learning. They empirically argued the benefit of this approach by showing how spoofing--computing a proof for a stolen model--is as expensive as obtaining the proof honestly by training the model. However, recent work has provided a counter-example and thus has invalidated this observation. In this work we demonstrate, first, that while it is true that current PoL verification is not robust to adversaries, recent work has largely underestimated this lack of robustness. This is because existing spoofing strategies are either unreproducible or target weakened instantiations of PoL--meaning they are easily thwarted by changing hyperparameters of the verification. Instead, we introduce the first spoofing strategies that can be reproduced across different configurations of the PoL verification and can be done for a fraction of the cost of previous spoofing strategies. This is possible because we identify key vulnerabilities of PoL and systematically analyze the underlying assumptions needed for robust verification of a proof. On the theoretical side, we show how realizing these assumptions reduces to open problems in learning theory.We conclude that one cannot develop a provably robust PoL verification mechanism without further understanding of optimization in deep learning.
The Impact of Data Corruption on Named Entity Recognition for Low-resourced Languages
Venue: arXiv cs.CL 2023
Year: 2023

Data availability and quality are major challenges in natural language processing for low-resourced languages. In particular, there is significantly less data available than for higher-resourced languages. This data is also often of low quality, rife with errors, invalid text or incorrect annotations. Many prior works focus on dealing with these problems, either by generating synthetic data, or filtering out low-quality parts of datasets. We instead investigate these factors more deeply, by systematically measuring the effect of data quantity and quality on the performance of pre-trained language models in a low-resourced setting. Our results show that having fewer completely-labelled sentences is significantly better than having more sentences with missing labels; and that models can perform remarkably well with only 10% of the training data. Importantly, these results are consistent across ten low-resource languages, English, and four pre-trained models.
Responsible Urban Intelligence: Towards a Research Agenda
Venue: arXiv cs.CY 2023
Year: 2023

Acceleration of urbanisation is posing great challenges to sustainable development. Growing accessibility to big data and artificial intelligence (AI) technologies have revolutionised many fields and offered great potential for addressing pressing urban problems. However, using these technologies without explicitly considering responsibilities would bring new societal and environmental issues. To maximise the benefits of big data and AI while minimising potential issues, we envisage a conceptual framework of Responsible Urban Intelligence (RUI) and advocate an agenda for action. We first define RUI as consisting of three major components including urban problems, enabling technologies, and responsibilities; then introduce transparency, fairness, and eco-friendliness as the three dimensions of responsibilities which naturally link with the human, space, and time dimensions of cities; and further develop a four-stage implementation framework for responsibilities as consisting of solution design, data preparation, model building, and practical application; and finally present a research agenda for RUI addressing challenging issues including data and model transparency, tension between performance and fairness, and solving urban problems in an eco-friendly manner.
Mathematical Foundations of Complex Tonality
Venue: arXiv cs.SD 2023
Year: 2023

Equal temperament, in which semitones are tuned in the irrational ratio of $2^{1/12} : 1$, is best seen as a serviceable compromise, sacrificing purity for flexibility. Just intonation, in which intervals are given by products of powers of $2$, $3$, and $5$, is more natural, but of limited flexibility. We propose a new scheme in which ratios of Gaussian integers form the basis of an abstract tonal system. The tritone, so problematic in just temperament, given ambiguously by the ratios $\tfrac{45}{32}$, $\tfrac{64}{45}$, $\tfrac{36}{25}$, $\tfrac{25}{18}$, none satisfactory, is in our scheme represented by the complex ratio $1 + \rm{i} : 1$. The major and minor whole tones, given by intervals of $\tfrac{9}{8}$ and $\tfrac{10}{9}$, can each be factorized into products of complex semitones, giving us a major complex semitone $\tfrac{3}{4}(1 + \rm{i})$ and a minor complex semitone $\tfrac{1}{3}(3 + \rm{i})$. The perfect third, given by the interval $\tfrac{5}{4}$, factorizes into the product of a complex whole tone $\tfrac{1}{2}(1 + 2\rm{i})$ and its complex conjugate. Augmented with these supplementary tones, the resulting scheme of complex intervals based on products of low powers of Gaussian primes leads to the construction of a complete system of major and minor scales in all keys.
Reducing Exploitability with Population Based Training
Venue: arXiv cs.LG 2023
Year: 2023

Self-play reinforcement learning has achieved state-of-the-art, and often superhuman, performance in a variety of zero-sum games. Yet prior work has found that policies that are highly capable against regular opponents can fail catastrophically against adversarial policies: an opponent trained explicitly against the victim. Prior defenses using adversarial training were able to make the victim robust to a specific adversary, but the victim remained vulnerable to new ones. We conjecture this limitation was due to insufficient diversity of adversaries seen during training. We analyze a defense using population based training to pit the victim against a diverse set of opponents. We evaluate this defense's robustness against new adversaries in two low-dimensional environments. This defense increases robustness against adversaries, as measured by the number of attacker training timesteps to exploit the victim. Furthermore, we show that robustness is correlated with the size of the opponent population.
NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation
Venue: arXiv cs.LG 2023
Year: 2023

Test-time adaptation (TTA) is an emerging paradigm that addresses distributional shifts between training and testing phases without additional data acquisition or labeling cost; only unlabeled test data streams are used for continual model adaptation. Previous TTA schemes assume that the test samples are independent and identically distributed (i.i.d.), even though they are often temporally correlated (non-i.i.d.) in application scenarios, e.g., autonomous driving. We discover that most existing TTA methods fail dramatically under such scenarios. Motivated by this, we present a new test-time adaptation scheme that is robust against non-i.i.d. test data streams. Our novelty is mainly two-fold: (a) Instance-Aware Batch Normalization (IABN) that corrects normalization for out-of-distribution samples, and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d. data stream from non-i.i.d. stream in a class-balanced manner. Our evaluation with various datasets, including real-world non-i.i.d. streams, demonstrates that the proposed robust TTA not only outperforms state-of-the-art TTA algorithms in the non-i.i.d. setting, but also achieves comparable performance to those algorithms under the i.i.d. assumption. Code is available at https://github.com/TaesikGong/NOTE.