OpenRaiser/Intern-Atlas huggingface dataset:
Detecting Text Formality: A Study of Text Classification Approaches
Venue: arXiv cs.CL 2023
Year: 2023

Formality is one of the important characteristics of text documents. The automatic detection of the formality level of a text is potentially beneficial for various natural language processing tasks. Before, two large-scale datasets were introduced for multiple languages featuring formality annotation -- GYAFC and X-FORMAL. However, they were primarily used for the training of style transfer models. At the same time, the detection of text formality on its own may also be a useful application. This work proposes the first to our knowledge systematic study of formality detection methods based on statistical, neural-based, and Transformer-based machine learning methods and delivers the best-performing models for public usage. We conducted three types of experiments -- monolingual, multilingual, and cross-lingual. The study shows the overcome of Char BiLSTM model over Transformer-based ones for the monolingual and multilingual formality classification task, while Transformer-based classifiers are more stable to cross-lingual knowledge transfer.
Lengths of Cycles in Generalized Pancake Graphs
Venue: arXiv cs.DM 2023
Year: 2023

In this paper, we consider the lengths of cycles that can be embedded on the edges of the generalized pancake graphs which are the Cayley graph of the generalized symmetric group $S(m,n)$, generated by prefix reversals. The generalized symmetric group $S(m,n)$ is the wreath product of the cyclic group of order $m$ and the symmetric group of order $n!$. Our main focus is the underlying \emph{undirected} graphs, denoted by $\mathbb{P}_m(n)$. In the cases when the cyclic group has one or two elements, these graphs are isomorphic to the pancake graphs and burnt pancake graphs, respectively. We prove that when the cyclic group has three elements, $\mathbb{P}_3(n)$ has cycles of all possible lengths, thus resembling a similar property of pancake graphs and burnt pancake graphs. Moreover, $\mathbb{P}_4(n)$ has all the even-length cycles. We utilize these results as base cases and show that if $m>2$ is even, $\mathbb{P}_m(n)$ has all cycles of even length starting from its girth to a Hamiltonian cycle. Moreover, when $m>2$ is odd, $\mathbb{P}_m(n)$ has cycles of all lengths starting from its girth to a Hamiltonian cycle. We furthermore show that the girth of $\mathbb{P}_m(n)$ is $\min\{m,6\}$ if $m\geq3$, thus complementing the known results for $m=1,2.$
A Note on the Regularity of Images Generated by Convolutional Neural Networks
Venue: arXiv cs.CV 2023
Year: 2023

The regularity of images generated by convolutional neural networks, such as the U-net, generative networks, or the deep image prior, is analyzed. In a resolution-independent, infinite dimensional setting, it is shown that such images, represented as functions, are always continuous and, in some circumstances, even continuously differentiable, contradicting the widely accepted modeling of sharp edges in images via jump discontinuities. While such statements require an infinite dimensional setting, the connection to (discretized) neural networks used in practice is made by considering the limit as the resolution approaches infinity. As practical consequence, the results of this paper in particular provide analytical evidence that basic L2 regularization of network weights might lead to over-smoothed outputs.
From Books to Knowledge Graphs
Venue: arXiv cs.DL 2023
Year: 2023

The digital transformation of the scientific publishing industry has led to dramatic improvements in content discoverability and information analytics. Unfortunately, these improvements have not been uniform across research areas. The scientific literature in the arts, humanities and social sciences (AHSS) still lags behind, in part due to the scale of analog backlogs, the persisting importance of national languages, and a publisher ecosystem made of many, small or medium enterprises. We propose a bottom-up approach to support publishers in creating and maintaining their own publication knowledge graphs in the open domain. We do so by releasing a pipeline able to extract structured information from the bibliographies and indexes of AHSS publications, disambiguate, normalize and export it as linked data. We test the proposed pipeline on Brill's Classics collection, and release an implementation in open source for further use and improvement.
Predicting highway lane-changing maneuvers: A benchmark analysis of machine and ensemble learning algorithms
Venue: arXiv cs.LG 2023
Year: 2023

Understanding and predicting highway lane-change maneuvers is essential for driving modeling and its automation. The development of data-based lane-changing decision-making algorithms is nowadays in full expansion. We compare empirically in this article different machine and ensemble learning classification techniques to the MOBIL rule-based model using trajectory data of European two-lane highways. The analysis relies on instantaneous measurements of up to twenty-four spatial-temporal variables with the four neighboring vehicles on current and adjacent lanes. Preliminary descriptive investigations by principal component and logistic analyses allow identifying main variables intending a driver to change lanes. We predict two types of discretionary lane-change maneuvers: overtaking (from the slow to the fast lane) and fold-down (from the fast to the slow lane). The prediction accuracy is quantified using total, lane-changing and lane-keeping errors and associated receiver operating characteristic curves. The benchmark analysis includes logistic model, linear discriminant, decision tree, naïve Bayes classifier, support vector machine, neural network machine learning algorithms, and up to ten bagging and stacking ensemble learning meta-heuristics. If the rule-based model provides limited predicting accuracy, the data-based algorithms, devoid of modeling bias, allow significant prediction improvements. Cross validations show that selected neural networks and stacking algorithms allow predicting from a single observation both fold-down and overtaking maneuvers up to four seconds in advance with high accuracy.
On the Information Theoretic Secure Aggregation with Uncoded Groupwise Keys
Venue: arXiv cs.IT 2023
Year: 2023

Secure aggregation, which is a core component of federated learning, aggregates locally trained models from distributed users at a central server. The ``secure'' nature of such aggregation consists of the fact that no information about the local users' data must be leaked to the server except the aggregated local models. In order to guarantee security, some keys may be shared among the users. After the key sharing phase, each user masks its trained model which is then sent to the server. This paper follows the information theoretic secure aggregation problem originally formulated by Zhao and Sun, with the objective to characterize the minimum communication cost from the K users in the model aggregation phase. Due to user dropouts, the server may not receive all messages from the users. A secure aggregation scheme should tolerate the dropouts of at most $K-U$ users. The optimal communication cost is characterized by Zhao and Sun, but with the assumption that the keys stored by the users could be any random variables with arbitrary dependency. On the motivation that uncoded groupwise keys are more convenient to be shared and could be used in large range of applications besides federated learning, in this paper we assume the key variables are mutually independent and each key is shared by a group of S users. To the best of our knowledge, all existing secure aggregation schemes assign coded keys to the users. We show that if $S> K-U$, a new secure aggregation scheme with uncoded groupwise keys can achieve the same optimal communication cost as the best scheme with coded keys; if $S \leq K-U$, uncoded groupwise key sharing is strictly sub-optimal. Finally, we also implement our proposed secure aggregation scheme into Amazon EC2, which are then compared with the existing secure aggregation schemes with offline key sharing.
Properly learning monotone functions via local reconstruction
Venue: arXiv cs.DS 2023
Year: 2023

We give a $2^{\tilde{O}(\sqrt{n}/ε)}$-time algorithm for properly learning monotone Boolean functions under the uniform distribution over $\{0,1\}^n$. Our algorithm is robust to adversarial label noise and has a running time nearly matching that of the state-of-the-art improper learning algorithm of Bshouty and Tamon (JACM '96) and an information-theoretic lower bound of Blais et al (RANDOM '15). Prior to this work, no proper learning algorithm with running time smaller than $2^{Ω(n)}$ was known to exist. The core of our proper learner is a \emph{local computation algorithm} for sorting binary labels on a poset. Our algorithm is built on a body of work on distributed greedy graph algorithms; specifically we rely on a recent work of Ghaffari (FOCS'22), which gives an efficient algorithm for computing maximal matchings in a graph in the LCA model of Rubinfeld et al and Alon et al (ICS'11, SODA'12). The applications of our local sorting algorithm extend beyond learning on the Boolean cube: we also give a tolerant tester for Boolean functions over general posets that distinguishes functions that are $ε/3$-close to monotone from those that are $ε$-far. Previous tolerant testers for the Boolean cube only distinguished between $ε/Ω(\sqrt{n})$-close and $ε$-far.
Exact Wirelength of Embedding 3-Ary n-Cubes into certain Cylinders and Trees
Venue: arXiv cs.DM 2023
Year: 2023

Graph embeddings play a significant role in the design and analysis of parallel algorithms. It is a mapping of the topological structure of a guest graph G into a host graph H, which is represented as a one-to-one mapping from the vertex set of the guest graph to the vertex set of the host graph. In multiprocessing systems the interconnection networks enhance the efficient communication between the components in the system. Obtaining minimum wirelength in embedding problems is significant in the designing of network and simulating one architecture by another. In this paper, we determine the wirelength of embedding 3-ary n-cubes into cylinders and certain trees.
Adaptive Split-Fusion Transformer
Venue: arXiv cs.CV 2023
Year: 2023

Neural networks for visual content understanding have recently evolved from convolutional ones (CNNs) to transformers. The prior (CNN) relies on small-windowed kernels to capture the regional clues, demonstrating solid local expressiveness. On the contrary, the latter (transformer) establishes long-range global connections between localities for holistic learning. Inspired by this complementary nature, there is a growing interest in designing hybrid models to best utilize each technique. Current hybrids merely replace convolutions as simple approximations of linear projection or juxtapose a convolution branch with attention, without concerning the importance of local/global modeling. To tackle this, we propose a new hybrid named Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Specifically, an ASF-former encoder equally splits feature channels into half to fit dual-path inputs. Then, the outputs of dual-path are fused with weighting scalars calculated from visual cues. We also design the convolutional path compactly for efficiency concerns. Extensive experiments on standard benchmarks, such as ImageNet-1K, CIFAR-10, and CIFAR-100, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy (83.9% on ImageNet-1K), under similar conditions (12.9G MACs/56.7M Params, without large-scale pre-training). The code is available at: https://github.com/szx503045266/ASF-former.
Federated Progressive Sparsification (Purge, Merge, Tune)+
Venue: arXiv cs.LG 2023
Year: 2023

To improve federated training of neural networks, we develop FedSparsify, a sparsification strategy based on progressive weight magnitude pruning. Our method has several benefits. First, since the size of the network becomes increasingly smaller, computation and communication costs during training are reduced. Second, the models are incrementally constrained to a smaller set of parameters, which facilitates alignment/merging of the local models and improved learning performance at high sparsification rates. Third, the final sparsified model is significantly smaller, which improves inference efficiency and optimizes operations latency during encrypted communication. We show experimentally that FedSparsify learns a subnetwork of both high sparsity and learning performance. Our sparse models can reach a tenth of the size of the original model with the same or better accuracy compared to existing pruning and nonpruning baselines.
A Thorough Examination on Zero-shot Dense Retrieval
Venue: arXiv cs.CL 2023
Year: 2023

Recent years have witnessed the significant advance in dense retrieval (DR) based on powerful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they are shown to be not as competitive as traditional sparse retrieval models (e.g., BM25) in a zero-shot retrieval setting. However, in the related literature, there still lacks a detailed and comprehensive study on zero-shot retrieval. In this paper, we present the first thorough examination of the zero-shot capability of DR models. We aim to identify the key factors and analyze how they affect zero-shot retrieval performance. In particular, we discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zero-shot DR models. Our findings provide important evidence to better understand and develop zero-shot DR models.
3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective
Venue: arXiv cs.CV 2023
Year: 2023

This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with existing methods, we observe that three primary challenges remain: (1) 3D ground-truth meshes of clothing are usually inaccessible due to annotation difficulties and time costs; (2) Conventional template-based methods are limited to modeling non-rigid objects, e.g., handbags and dresses, which are common in fashion images; (3) The inherent ambiguity compromises the model training, such as the dilemma between a large shape with a remote camera or a small shape with a close camera. In an attempt to address the above limitations, we propose a causality-aware self-supervised learning method to adaptively reconstruct 3D non-rigid objects from 2D images without 3D annotations. In particular, to solve the inherent ambiguity among four implicit variables, i.e., camera position, shape, texture, and illumination, we introduce an explainable structural causal map (SCM) to build our model. The proposed model structure follows the spirit of the causal map, which explicitly considers the prior template in the camera estimation and shape prediction. When optimization, the causality intervention tool, i.e., two expectation-maximization loops, is deeply embedded in our algorithm to (1) disentangle four encoders and (2) facilitate the prior template. Extensive experiments on two 2D fashion benchmarks (ATR and Market-HQ) show that the proposed method could yield high-fidelity 3D reconstruction. Furthermore, we also verify the scalability of the proposed method on a fine-grained bird dataset, i.e., CUB. The code is available at https://github.com/layumi/ 3D-Magic-Mirror .
A Soft-Bodied Aerial Robot for Collision Resilience and Contact-Reactive Perching
Venue: arXiv cs.RO 2023
Year: 2023

Current aerial robots demonstrate limited interaction capabilities in unstructured environments when compared with their biological counterparts. Some examples include their inability to tolerate collisions and to successfully land or perch on objects of unknown shapes, sizes, and texture. Efforts to include compliance have introduced designs that incorporate external mechanical impact protection at the cost of reduced agility and flight time due to the added weight. In this work, we propose and develop a light-weight, inflatable, soft-bodied aerial robot (SoBAR) that can pneumatically vary its body stiffness to achieve intrinsic collision resilience. Unlike the conventional rigid aerial robots, SoBAR successfully demonstrates its ability to repeatedly endure and recover from collisions in various directions, not only limited to in-plane ones. Furthermore, we exploit its capabilities to demonstrate perching where the 3D collision resilience helps in improving the perching success rates. We also augment SoBAR with a novel hybrid fabric-based, bistable (HFB) grasper that can utilize impact energies to perform contact-reactive grasping through rapid shape conforming abilities. We exhaustively study and offer insights into the collision resilience, impact absorption, and manipulation capabilities of SoBAR with the HFB grasper. Finally, we compare the performance of conventional aerial robots with the SoBAR through collision characterizations, grasping identifications, and experimental validations of collision resilience and perching in various scenarios and on differently shaped objects.
AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation
Venue: arXiv cs.LG 2023
Year: 2023

In Federated Learning (FL), a number of clients or devices collaborate to train a model without sharing their data. Models are optimized locally at each client and further communicated to a central hub for aggregation. While FL is an appealing decentralized training paradigm, heterogeneity among data from different clients can cause the local optimization to drift away from the global objective. In order to estimate and therefore remove this drift, variance reduction techniques have been incorporated into FL optimization recently. However, these approaches inaccurately estimate the clients' drift and ultimately fail to remove it properly. In this work, we propose an adaptive algorithm that accurately estimates drift across clients. In comparison to previous works, our approach necessitates less storage and communication bandwidth, as well as lower compute costs. Additionally, our proposed methodology induces stability by constraining the norm of estimates for client drift, making it more practical for large scale FL. Experimental findings demonstrate that the proposed algorithm converges significantly faster and achieves higher accuracy than the baselines across various FL benchmarks.
Self-supervised Contrastive Learning for Audio-Visual Action Recognition
Venue: arXiv cs.CV 2023
Year: 2023

The underlying correlation between audio and visual modalities can be utilized to learn supervised information for unlabeled videos. In this paper, we propose an end-to-end self-supervised framework named Audio-Visual Contrastive Learning (AVCL), to learn discriminative audio-visual representations for action recognition. Specifically, we design an attention based multi-modal fusion module (AMFM) to fuse audio and visual modalities. To align heterogeneous audio-visual modalities, we construct a novel co-correlation guided representation alignment module (CGRA). To learn supervised information from unlabeled videos, we propose a novel self-supervised contrastive learning module (SelfCL). Furthermore, we build a new audio-visual action recognition dataset named Kinetics-Sounds100. Experimental results on Kinetics-Sounds32 and Kinetics-Sounds100 datasets demonstrate the superiority of our AVCL over the state-of-the-art methods on large-scale action recognition benchmark.
A suite of diagnostic metrics for characterizing selection schemes
Venue: arXiv cs.NE 2023
Year: 2023

Benchmark suites are crucial for assessing the performance of evolutionary algorithms, but the constituent problems are often too complex to provide clear intuition about an algorithm's strengths and weaknesses. To address this gap, we introduce DOSSIER ("Diagnostic Overview of Selection Schemes In Evolutionary Runs"), a diagnostic suite initially composed of eight handcrafted metrics. These metrics are designed to empirically measure specific capacities for exploitation, exploration, and their interactions. We consider exploitation both with and without constraints, and we divide exploration into two aspects: diversity exploration (the ability to simultaneously explore multiple pathways) and valley-crossing exploration (the ability to cross wider and wider fitness valleys). We apply DOSSIER to six popular selection schemes: truncation, tournament, fitness sharing, lexicase, nondominated sorting, and novelty search. Our results confirm that simple schemes (e.g., tournament and truncation) emphasized exploitation. For more sophisticated schemes, however, our diagnostics revealed interesting dynamics. Lexicase selection performed moderately well across all diagnostics that did not incorporate valley crossing, but faltered dramatically whenever valleys were present, performing worse than even random search. Fitness sharing was the only scheme to effectively contend with valley crossing but it struggled with the other diagnostics. Our study highlights the utility of using diagnostics to gain nuanced insights into selection scheme characteristics, which can inform the design of new selection methods.
Heterogeneous Graph Neural Networks using Self-supervised Reciprocally Contrastive Learning
Venue: arXiv cs.LG 2023
Year: 2023

Heterogeneous graph neural network (HGNN) is a very popular technique for the modeling and analysis of heterogeneous graphs. Most existing HGNN-based approaches are supervised or semi-supervised learning methods requiring graphs to be annotated, which is costly and time-consuming. Self-supervised contrastive learning has been proposed to address the problem of requiring annotated data by mining intrinsic information hidden within the given data. However, the existing contrastive learning methods are inadequate for heterogeneous graphs because they construct contrastive views only based on data perturbation or pre-defined structural properties (e.g., meta-path) in graph data while ignore the noises that may exist in both node attributes and graph topologies. We develop for the first time a novel and robust heterogeneous graph contrastive learning approach, namely HGCL, which introduces two views on respective guidance of node attributes and graph topologies and integrates and enhances them by reciprocally contrastive mechanism to better model heterogeneous graphs. In this new approach, we adopt distinct but most suitable attribute and topology fusion mechanisms in the two views, which are conducive to mining relevant information in attributes and topologies separately. We further use both attribute similarity and topological correlation to construct high-quality contrastive samples. Extensive experiments on three large real-world heterogeneous graphs demonstrate the superiority and robustness of HGCL over state-of-the-art methods.
EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing
Venue: arXiv cs.CL 2023
Year: 2023

The success of Pre-Trained Models (PTMs) has reshaped the development of Natural Language Processing (NLP). Yet, it is not easy to obtain high-performing models and deploy them online for industrial practitioners. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale PTMs, and provides a unified framework of model training, inference and deployment for real-world applications. Currently, EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. The source code of our EasyNLP toolkit is released at GitHub (https://github.com/alibaba/EasyNLP).