OpenRaiser/Intern-Atlas huggingface dataset:
Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-wise Distributed Data
Venue: arXiv cs.LG 2022
Year: 2022

Despite enormous research interest and rapid application of federated learning (FL) to various areas, existing studies mostly focus on supervised federated learning under the horizontally partitioned local dataset setting. This paper will study the unsupervised FL under the vertically partitioned dataset setting. Accordingly, we propose the federated principal component analysis for vertically partitioned dataset (VFedPCA) method, which reduces the dimensionality across the joint datasets over all the clients and extracts the principal component feature information for downstream data analysis. We further take advantage of the nonlinear dimensionality reduction and propose the vertical federated advanced kernel principal component analysis (VFedAKPCA) method, which can effectively and collaboratively model the nonlinear nature existing in many real datasets. In addition, we study two communication topologies. The first is a server-client topology where a semi-trusted server coordinates the federated training, while the second is the fully-decentralized topology which further eliminates the requirement of the server by allowing clients themselves to communicate with their neighbors. Extensive experiments conducted on five types of real-world datasets corroborate the efficacy of VFedPCA and VFedAKPCA under the vertically partitioned FL setting. Code is available at: https://github.com/juyongjiang/VFedPCA-VFedAKPCA
An Effective Graph Learning based Approach for Temporal Link Prediction: The First Place of WSDM Cup 2022
Venue: arXiv cs.SI 2022
Year: 2022

Temporal link prediction, as one of the most crucial work in temporal graphs, has attracted lots of attention from the research area. The WSDM Cup 2022 seeks for solutions that predict the existence probabilities of edges within time spans over temporal graph. This paper introduces the solution of AntGraph, which wins the 1st place in the competition. We first analysis the theoretical upper-bound of the performance by removing temporal information, which implies that only structure and attribute information on the graph could achieve great performance. Based on this hypothesis, then we introduce several well-designed features. Finally, experiments conducted on the competition datasets show the superiority of our proposal, which achieved AUC score of 0.666 on dataset A and 0.902 on dataset B, the ablation studies also prove the efficiency of each feature. Code is publicly available at https://github.com/im0qianqian/WSDM2022TGP-AntGraph.
Mobile device users' susceptibility to phishing attacks
Venue: arXiv cs.CR 2022
Year: 2022

The mobile device is one of the fasted growing technologies that is widely used in a diversifying sector. Mobile devices are used for everyday life, such as personal information exchange - chatting, email, shopping, and mobile banking, contributing to information security threats. Users' behavior can influence information security threats. More research is needed to understand users' threat avoidance behavior and motivation. Using Technology threat avoidance theory (TTAT), this study assessed factors that influenced mobile device users' threat avoidance motivations and behaviors as it relates to phishing attacks. From the data collected from 137 mobile device users using a questionnaire, the findings indicate that (1) mobile device users' perceived susceptibility and severity of phishing attacks have a significant correlation with a users' perception of the threat; (2) mobile device users' motivation to avoid a threat is correlated to a users' behavior in avoiding threat; and (3) a mobile device user's susceptibility to phishing attacks can be reduced by their perception of the threat. These findings reveal that a user's perception of threat increases if they perceive that the consequence of such threat to their mobile devices will be severe, thereby increasing a user's motivation and behavior to avoid phishing attack threats. This study is beneficial to mobile device users in personal and organizational settings.
Thermodynamics-informed graph neural networks
Venue: arXiv cs.LG 2022
Year: 2022

In this paper we present a deep learning method to predict the temporal evolution of dissipative dynamic systems. We propose using both geometric and thermodynamic inductive biases to improve accuracy and generalization of the resulting integration scheme. The first is achieved with Graph Neural Networks, which induces a non-Euclidean geometrical prior with permutation invariant node and edge update functions. The second bias is forced by learning the GENERIC structure of the problem, an extension of the Hamiltonian formalism, to model more general non-conservative dynamics. Several examples are provided in both Eulerian and Lagrangian description in the context of fluid and solid mechanics respectively, achieving relative mean errors of less than 3% in all the tested examples. Two ablation studies are provided based on recent works in both physics-informed and geometric deep learning.
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
Venue: arXiv cs.LG 2022
Year: 2022

The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making. As a result, the research fields of interpretable and explainable AI have gained traction within AI communities as well as interdisciplinary scientists seeking to apply AI in their subject areas. In this paper, we focus on advancing the state-of-the-art in interpreting multimodal models - a class of machine learning methods that tackle core challenges in representing and capturing interactions between heterogeneous data sources such as images, text, audio, and time-series data. Multimodal models have proliferated numerous real-world applications across healthcare, robotics, multimedia, affective computing, and human-computer interaction. By performing model disentanglement into unimodal contributions (UC) and multimodal interactions (MI), our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models while maintaining generality across arbitrary modalities, model architectures, and tasks. Through a comprehensive suite of experiments on both synthetic and real-world multimodal tasks, we show that DIME generates accurate disentangled explanations, helps users of multimodal models gain a deeper understanding of model behavior, and presents a step towards debugging and improving these models for real-world deployment. Code for our experiments can be found at https://github.com/lvyiwei1/DIME.
Provable and Efficient Continual Representation Learning
Venue: arXiv cs.LG 2022
Year: 2022

In continual learning (CL), the goal is to design models that can learn a sequence of tasks without catastrophic forgetting. While there is a rich set of techniques for CL, relatively little understanding exists on how representations built by previous tasks benefit new tasks that are added to the network. To address this, we study the problem of continual representation learning (CRL) where we learn an evolving representation as new tasks arrive. Focusing on zero-forgetting methods where tasks are embedded in subnetworks (e.g., PackNet), we first provide experiments demonstrating CRL can significantly boost sample efficiency when learning new tasks. To explain this, we establish theoretical guarantees for CRL by providing sample complexity and generalization error bounds for new tasks by formalizing the statistical benefits of previously-learned representations. Our analysis and experiments also highlight the importance of the order in which we learn the tasks. Specifically, we show that CL benefits if the initial tasks have large sample size and high "representation diversity". Diversity ensures that adding new tasks incurs small representation mismatch and can be learned with few samples while training only few additional nonzero weights. Finally, we ask whether one can ensure each task subnetwork to be efficient during inference time while retaining the benefits of representation learning. To this end, we propose an inference-efficient variation of PackNet called Efficient Sparse PackNet (ESPN) which employs joint channel & weight pruning. ESPN embeds tasks in channel-sparse subnets requiring up to 80% less FLOPs to compute while approximately retaining accuracy and is very competitive with a variety of baselines. In summary, this work takes a step towards data and compute-efficient CL with a representation learning perspective. GitHub page: https://github.com/ucr-optml/CtRL
Towards Mining OSS Skills from GitHub Activity
Venue: arXiv cs.SE 2022
Year: 2022

Open source software (OSS) development relies on diverse skill sets. However, to our knowledge, there are no tools which detect OSS-related skills. In this paper, we present a novel method to detect OSS skills and prototype it in a tool called Disko. Our approach relies on identifying relevant signals, which are measurable activities or cues associated with a skill. Our tool detects how contributors 1) teach others to be involved in OSS projects, 2) show commitment towards an OSS project, 3) have knowledge in specific programming languages, and 4) are familiar with OSS practices. We then evaluate the tool by administering a survey to 455 OSS contributors. We demonstrate that Disko yields promising results: it detects the presence of these skills with precision scores between 77% to 97%. We also find that over 54% of participants would display their high-proficiency skills. Our approach can be used to transform existing OSS experiences, such as identifying collaborators, matching mentors to mentees, and assigning project roles. Given the positive results and potential impact of our approach, we outline future research opportunities in interpreting and sharing OSS skills.
Baba is Y'all 2.0: Design and Investigation of a Collaborative Mixed-Initiative System
Venue: arXiv cs.HC 2022
Year: 2022

This paper describes a new version of the mixed-initiative collaborative level designing system: Baba is Y'all, as well as the results of a user study on the system. Baba is Y'all is a prototype for AI-assisted game design in collaboration with others. The updated version includes a more user-friendly interface, a better level-evolver and recommendation system, and extended site features. The system was evaluated via a user study where participants were required to play a previously submitted level from the site and then create their own levels using the editor. They reported on their individual process creating the level and their overall experience interacting with the site. The results have shown both the benefits and limitations of this mixed-initiative system and how it can help with creating a diversity of `Baba is You' levels that are both human and AI designed while maintaining their quality.
A Unified Spatially Coupled Code Design: Threshold, Cycles, and Locality
Venue: arXiv cs.IT 2022
Year: 2022

Spatially-Coupled (SC)-LDPC codes are known to have outstanding error-correction performance and low decoding latency. Whereas previous works on LDPC and SC-LDPC codes mostly take either an asymptotic or a finite-length design approach, in this paper we present a unified framework for jointly optimizing the codes' thresholds and cycle counts to address both regimes. The framework is based on efficient traversal and pruning of the code search space, building on the fact that the performance of a protograph-based SC-LDPC code depends on some characteristics of the code's partitioning matrix, which by itself is much smaller than the code's full parity-check matrix. We then propose an algorithm that traverses all nonequivalent partitioning matrices, and outputs a list of codes, each offering an attractive point on the trade-off between asymptotic and finite-length performance. We further extend the framework to designing SC-LDPC codes with sub-block locality, which is a recently introduced feature offering fast access to sub-blocks within the code block. Our simulations show that our framework results in SC-LDPC codes that outperform the state-of-the-art constructions, and that it offers the flexibility to choose low-SNR, high-SNR, or in-between SNR region as the primary design target.
X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback
Venue: arXiv cs.HC 2022
Year: 2022

We aim to help users communicate their intent to machines using flexible, adaptive interfaces that translate arbitrary user input into desired actions. In this work, we focus on assistive typing applications in which a user cannot operate a keyboard, but can instead supply other inputs, such as webcam images that capture eye gaze or neural activity measured by a brain implant. Standard methods train a model on a fixed dataset of user inputs, then deploy a static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging. We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learning from user feedback on the accuracy of the interface's actions. In the typing domain, we leverage backspaces as feedback that the interface did not perform the desired action. We propose an algorithm called x-to-text (X2T) that trains a predictive model of this feedback signal, and uses this model to fine-tune any existing, default interface for translating user input into actions that select words or characters. We evaluate X2T through a small-scale online user study with 12 participants who type sentences by gazing at their desired words, a large-scale observational study on handwriting samples from 60 users, and a pilot study with one participant using an electrocorticography-based brain-computer interface. The results show that X2T learns to outperform a non-adaptive default interface, stimulates user co-adaptation to the interface, personalizes the interface to individual users, and can leverage offline data collected from the default interface to improve its initial performance and accelerate online learning.
Weighted Simultaneous Algebra Reconstruction Technique (wSART) for Additive Light Field Synthesis
Venue: arXiv cs.GR 2022
Year: 2022

We apply an iterative weighting scheme for additive light field synthesis. Unlike previous work optimizing additive light field evenly over viewpoints, we constrain the optimization to deliver a reconstructed light field of high image quality for viewpoints of large weight.
HV-Net: Hypervolume Approximation based on DeepSets
Venue: arXiv cs.NE 2022
Year: 2022

In this letter, we propose HV-Net, a new method for hypervolume approximation in evolutionary multi-objective optimization. The basic idea of HV-Net is to use DeepSets, a deep neural network with permutation invariant property, to approximate the hypervolume of a non-dominated solution set. The input of HV-Net is a non-dominated solution set in the objective space, and the output is an approximated hypervolume value of this solution set. The performance of HV-Net is evaluated through computational experiments by comparing it with two commonly-used hypervolume approximation methods (i.e., point-based method and line-based method). Our experimental results show that HV-Net outperforms the other two methods in terms of both the approximation error and the runtime, which shows the potential of using deep learning technique for hypervolume approximation.
Freeform Body Motion Generation from Speech
Venue: arXiv cs.CV 2022
Year: 2022

People naturally conduct spontaneous body motions to enhance their speeches while giving talks. Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. Most existing works map speech to motion in a deterministic way by conditioning on certain styles, leading to sub-optimal results. Motivated by studies in linguistics, we decompose the co-speech motion into two complementary parts: pose modes and rhythmic dynamics. Accordingly, we introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture, i.e., a pose mode branch for primary posture generation, and a rhythmic motion branch for rhythmic dynamics synthesis. On one hand, diverse pose modes are generated by conditional sampling in a latent space, guided by speech semantics. On the other hand, rhythmic dynamics are synced with the speech prosody. Extensive experiments demonstrate the superior performance against several baselines, in terms of motion diversity, quality and syncing with speech. Code and pre-trained models will be publicly available through https://github.com/TheTempAccount/Co-Speech-Motion-Generation.
Intrinsically-Motivated Reinforcement Learning: A Brief Introduction
Venue: arXiv cs.LG 2022
Year: 2022

Reinforcement learning (RL) is one of the three basic paradigms of machine learning. It has demonstrated impressive performance in many complex tasks like Go and StarCraft, which is increasingly involved in smart manufacturing and autonomous driving. However, RL consistently suffers from the exploration-exploitation dilemma. In this paper, we investigated the problem of improving exploration in RL and introduced the intrinsically-motivated RL. In sharp contrast to the classic exploration strategies, intrinsically-motivated RL utilizes the intrinsic learning motivation to provide sustainable exploration incentives. We carefully classified the existing intrinsic reward methods and analyzed their practical drawbacks. Moreover, we proposed a new intrinsic reward method via Rényi state entropy maximization, which overcomes the drawbacks of the preceding methods and provides powerful exploration incentives. Finally, extensive simulation demonstrated that the proposed module achieve superior performance with higher efficiency and robustness.
Differentiable Causal Discovery Under Latent Interventions
Venue: arXiv cs.LG 2022
Year: 2022

Recent work has shown promising results in causal discovery by leveraging interventional data with gradient-based methods, even when the intervened variables are unknown. However, previous work assumes that the correspondence between samples and interventions is known, which is often unrealistic. We envision a scenario with an extensive dataset sampled from multiple intervention distributions and one observation distribution, but where we do not know which distribution originated each sample and how the intervention affected the system, \textit{i.e.}, interventions are entirely latent. We propose a method based on neural networks and variational inference that addresses this scenario by framing it as learning a shared causal graph among an infinite mixture (under a Dirichlet process prior) of intervention structural causal models. Experiments with synthetic and real data show that our approach and its semi-supervised variant are able to discover causal relations in this challenging scenario.
ViT-P: Rethinking Data-efficient Vision Transformers from Locality
Venue: arXiv cs.CV 2022
Year: 2022

Recent advances of Transformers have brought new trust to computer vision tasks. However, on small dataset, Transformers is hard to train and has lower performance than convolutional neural networks. We make vision transformers as data-efficient as convolutional neural networks by introducing multi-focal attention bias. Inspired by the attention distance in a well-trained ViT, we constrain the self-attention of ViT to have multi-scale localized receptive field. The size of receptive field is adaptable during training so that optimal configuration can be learned. We provide empirical evidence that proper constrain of receptive field can reduce the amount of training data for vision transformers. On Cifar100, our ViT-P Base model achieves the state-of-the-art accuracy (83.16%) trained from scratch. We also perform analysis on ImageNet to show our method does not lose accuracy on large data sets.
How human-robot collaboration impacts construction productivity: an agent-based multi-fidelity modeling approach
Venue: arXiv cs.RO 2022
Year: 2022

Though construction robots have drawn attention in research and practice for decades, human-robot collaboration (HRC) remains important to conduct complex construction tasks. Considering its complexity and uniqueness, it is still unclear how HRC process will impact construction productivity. To this end, an agent-based (AB) multi-fidelity modeling approach is introduced to simulate and evaluate how HRC influences construction productivity. A high-fidelity model is first proposed for a scenario with one robot. Then, a low-fidelity model is established to extract key parameters that capture the inner relationship among scenarios. The multi-fidelity models work together to simulate complex scenarios. Simulation and experiements show that: 1) the proposed approach is feasible and flexible for simulation of complex HRC processes, and can cover multiple collaboration and interaction modes; 2) the influence of the supplement strategy is simple when there is only one robot, where lower Check Interval (CI) and higher Supplement Limit (SL) will improve productivity. But the influence becomes much more complicated when there are more robots due to the internal competition among robots for the limited time of workers; 3) the productivity per robot improves when there are more robots and workers, even if the human-robot ratio remains the same; 4) introducing proactive interaction between robots and workers could improve productivity significantly, up to 22% in our experiments, which further depends on the supplement strategy and the human-robot ratio. Overall, this research contributes an integrated approach to simulate and evaluate HRC's impacts on productivity as well as valuable insights on how to optimize HRC for better performance.
Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving
Venue: arXiv cs.RO 2022
Year: 2022

Guaranteeing safety of perception-based learning systems is challenging due to the absence of ground-truth state information unlike in state-aware control scenarios. In this paper, we introduce a safety guaranteed learning framework for vision-based end-to-end autonomous driving. To this end, we design a learning system equipped with differentiable control barrier functions (dCBFs) that is trained end-to-end by gradient descent. Our models are composed of conventional neural network architectures and dCBFs. They are interpretable at scale, achieve great test performance under limited training data, and are safety guaranteed in a series of autonomous driving scenarios such as lane keeping and obstacle avoidance. We evaluated our framework in a sim-to-real environment, and tested on a real autonomous car, achieving safe lane following and obstacle avoidance via Augmented Reality (AR) and real parked vehicles.
R-GCN: The R Could Stand for Random
Venue: arXiv cs.LG 2022
Year: 2022

The inception of the Relational Graph Convolutional Network (R-GCN) marked a milestone in the Semantic Web domain as a widely cited method that generalises end-to-end hierarchical representation learning to Knowledge Graphs (KGs). R-GCNs generate representations for nodes of interest by repeatedly aggregating parameterised, relation-specific transformations of their neighbours. However, in this paper, we argue that the the R-GCN's main contribution lies in this "message passing" paradigm, rather than the learned weights. To this end, we introduce the "Random Relational Graph Convolutional Network" (RR-GCN), which leaves all parameters untrained and thus constructs node embeddings by aggregating randomly transformed random representations from neighbours, i.e., with no learned parameters. We empirically show that RR-GCNs can compete with fully trained R-GCNs in both node classification and link prediction settings.