OpenRaiser/Intern-Atlas huggingface dataset:
An interface-enriched generalized finite element formulation for locking-free coupling of non-conforming discretizations and contact
Venue: arXiv cs.CE 2021
Year: 2021

We propose an enriched finite element formulation to address the computational modeling of contact problems and the coupling of non-conforming discretizations in the small deformation setting. The displacement field is augmented by enriched terms that are associated with generalized degrees of freedom collocated along non-conforming interfaces or contact surfaces. The enrichment strategy effectively produces an enriched node-to-node discretization that can be used with any constraint enforcement criterion; this is demonstrated with both multiple-point constraints and Lagrange multipliers, the latter in a generalized Newton implementation where both primal and Lagrange multiplier fields are updated simultaneously. The method's ability to ensure continuity of the displacement field -- without locking -- in mesh coupling problems, and to transfer fairly accurate tractions at contact interfaces -- without the need for contact stabilization -- is demonstrated by means of several examples. In addition, we show that the formulation is stable with respect to the condition number of the stiffness matrix by using a simple Jacobi-like diagonal preconditioner.
An AGM Approach to Revising Preferences
Venue: arXiv cs.AI 2021
Year: 2021

We look at preference change arising out of an interaction between two elements: the first is an initial preference ranking encoding a pre-existing attitude; the second element is new preference information signaling input from an authoritative source, which may come into conflict with the initial preference. The aim is to adjust the initial preference and bring it in line with the new preference, without having to give up more information than necessary. We model this process using the formal machinery of belief change, along the lines of the well-known AGM approach. We propose a set of fundamental rationality postulates, and derive the main results of the paper: a set of representation theorems showing that preference change according to these postulates can be rationalized as a choice function guided by a ranking on the comparisons in the initial preference order. We conclude by presenting operators satisfying our proposed postulates. Our approach thus allows us to situate preference revision within the larger family of belief change operators.
Session-based Recommendation with Hypergraph Attention Networks
Venue: arXiv cs.IR 2021
Year: 2021

Session-based recommender systems aim to improve recommendations in short-term sessions that can be found across many platforms. A critical challenge is to accurately model user intent with only limited evidence in these short sessions. For example, is a flower bouquet being viewed meant as part of a wedding purchase or for home decoration? Such different perspectives greatly impact what should be recommended next. Hence, this paper proposes a novel session-based recommendation system empowered by hypergraph attention networks. Three unique properties of the proposed approach are: (i) it constructs a hypergraph for each session to model the item correlations defined by various contextual windows in the session simultaneously, to uncover item meanings; (ii) it is equipped with hypergraph attention layers to generate item embeddings by flexibly aggregating the contextual information from correlated items in the session; and (iii) it aggregates the dynamic item representations for each session to infer the general purpose and current need, which is decoded to infer the next interesting item in the session. Through experiments on three benchmark datasets, we find the proposed model is effective in generating informative dynamic item embeddings and providing more accurate recommendations compared to the state-of-the-art.
Harmonic Grassmannian codes
Venue: arXiv cs.IT 2021
Year: 2021

An equi-isoclinic tight fusion frame (EITFF) is a type of Grassmannian code, being a sequence of subspaces of a finite-dimensional Hilbert space of a given dimension with the property that the smallest spectral distance between any pair of them is as large as possible. EITFFs arise in compressed sensing, yielding dictionaries with minimal block coherence. Their existence remains poorly characterized. Most known EITFFs have parameters that match those of one that arose from an equiangular tight frame (ETF) in a rudimentary, direct-sum-based way. In this paper, we construct new infinite families of non-"tensor-sized" EITFFs in a way that generalizes the one previously known infinite family of them as well as the celebrated equivalence between harmonic ETFs and difference sets for finite abelian groups. In particular, we construct EITFFs consisting of $Q$ planes in $\mathbb{C}^Q$ for each prime power $Q\geq 4$, of $Q-1$ planes in $\mathbb{C}^Q$ for each odd prime power $Q$, and of $11$ three-dimensional subspaces in $\mathbb{R}^{11}$. The key idea is that every harmonic EITFF -- one that is the orbit of a single subspace under the action of a unitary representation of a finite abelian group -- arises from a smaller tight fusion frame with a nicely behaved "Fourier transform." Our particular constructions of harmonic EITFFs exploit the properties of Gauss sums over finite fields.
Feature-context driven Federated Meta-Learning for Rare Disease Prediction
Venue: arXiv cs.LG 2021
Year: 2021

Millions of patients suffer from rare diseases around the world. However, the samples of rare diseases are much smaller than those of common diseases. In addition, due to the sensitivity of medical data, hospitals are usually reluctant to share patient information for data fusion citing privacy concerns. These challenges make it difficult for traditional AI models to extract rare disease features for the purpose of disease prediction. In this paper, we overcome this limitation by proposing a novel approach for rare disease prediction based on federated meta-learning. To improve the prediction accuracy of rare diseases, we design an attention-based meta-learning (ATML) approach which dynamically adjusts the attention to different tasks according to the measured training effect of base learners. Additionally, a dynamic-weight based fusion strategy is proposed to further improve the accuracy of federated learning, which dynamically selects clients based on the accuracy of each local model. Experiments show that with as few as five shots, our approach out-performs the original federated meta-learning algorithm in accuracy and speed. Compared with each hospital's local model, the proposed model's average prediction accuracy increased by 13.28%.
Background-aware Classification Activation Map for Weakly Supervised Object Localization
Venue: arXiv cs.CV 2021
Year: 2021

Weakly supervised object localization (WSOL) relaxes the requirement of dense annotations for object localization by using image-level classification masks to supervise its learning process. However, current WSOL methods suffer from excessive activation of background locations and need post-processing to obtain the localization mask. This paper attributes these issues to the unawareness of background cues, and propose the background-aware classification activation map (B-CAM) to simultaneously learn localization scores of both object and background with only image-level labels. In our B-CAM, two image-level features, aggregated by pixel-level features of potential background and object locations, are used to purify the object feature from the object-related background and to represent the feature of the pure-background sample, respectively. Then based on these two features, both the object classifier and the background classifier are learned to determine the binary object localization mask. Our B-CAM can be trained in end-to-end manner based on a proposed stagger classification loss, which not only improves the objects localization but also suppresses the background activation. Experiments show that our B-CAM outperforms one-stage WSOL methods on the CUB-200, OpenImages and VOC2012 datasets.
Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification
Venue: arXiv cs.CV 2021
Year: 2021

We address the overlooked unbiasedness in existing long-tailed classification methods: we find that their overall improvement is mostly attributed to the biased preference of tail over head, as the test distribution is assumed to be balanced; however, when the test is as imbalanced as the long-tailed training data -- let the test respect Zipf's law of nature -- the tail bias is no longer beneficial overall because it hurts the head majorities. In this paper, we propose Cross-Domain Empirical Risk Minimization (xERM) for training an unbiased model to achieve strong performances on both test distributions, which empirically demonstrates that xERM fundamentally improves the classification by learning better feature representation rather than the head vs. tail game. Based on causality, we further theoretically explain why xERM achieves unbiasedness: the bias caused by the domain selection is removed by adjusting the empirical risks on the imbalanced domain and the balanced but unseen domain. Codes are available at https://github.com/BeierZhu/xERM.
DP-FP: Differentially Private Forward Propagation for Large Models
Venue: arXiv cs.LG 2021
Year: 2021

When applied to large-scale learning problems, the conventional wisdom on privacy-preserving deep learning, known as Differential Private Stochastic Gradient Descent (DP-SGD), has met with limited success due to significant performance degradation and high memory overhead when compared to the non-privacy counterpart. We show how to mitigate the performance drop by replacing the DP-SGD with a novel DP Forward-Propagation (DP-FP) followed by an off-the-shelf non-DP optimizer. Our DP-FP employs novel (1) representation clipping followed by noise addition in the forward propagation stage, as well as (2) micro-batch construction via subsampling to achieve DP amplification and reduce noise power to $1/M$, where $M$ is the number of micro-batch in a step. When training a classification model, our DP-FP with all of the privacy-preserving operations on the representation is innately free of gradient bias, total noise proportionally to model size, and memory issues in DP-SGD. As a result, our DP-FP outperforms cutting-edge DP-SGD while retaining the same level of privacy, and it approaches non-private baselines and significantly outperforms state-of-the-art DP-SGD variants. When applied to RoBERTa-large on four downstream tasks, for example, DP-FP achieves an average accuracy of 91.34\% with privacy budgets less than 3, representing a 3.81\% performance improvement over the state-of-the-art DP-SGD and only a 0.9\% loss compared to the non-private baseline but with a significantly lower privacy leakage risk.
Fully Distributed Informative Planning for Environmental Learning with Multi-Robot Systems
Venue: arXiv cs.RO 2021
Year: 2021

This paper proposes a cooperative environmental learning algorithm working in a fully distributed manner. A multi-robot system is more effective for exploration tasks than a single robot, but it involves the following challenges: 1) online distributed learning of environmental map using multiple robots; 2) generation of safe and efficient exploration path based on the learned map; and 3) maintenance of the scalability with respect to the number of robots. To this end, we divide the entire process into two stages of environmental learning and path planning. Distributed algorithms are applied in each stage and combined through communication between adjacent robots. The environmental learning algorithm uses a distributed Gaussian process, and the path planning algorithm uses a distributed Monte Carlo tree search. As a result, we build a scalable system without the constraint on the number of robots. Simulation results demonstrate the performance and scalability of the proposed system. Moreover, a real-world-dataset-based simulation validates the utility of our algorithm in a more realistic scenario.
Monte Carlo EM for Deep Time Series Anomaly Detection
Venue: arXiv cs.LG 2021
Year: 2021

Time series data are often corrupted by outliers or other kinds of anomalies. Identifying the anomalous points can be a goal on its own (anomaly detection), or a means to improving performance of other time series tasks (e.g. forecasting). Recent deep-learning-based approaches to anomaly detection and forecasting commonly assume that the proportion of anomalies in the training data is small enough to ignore, and treat the unlabeled data as coming from the nominal data distribution. We present a simple yet effective technique for augmenting existing time series models so that they explicitly account for anomalies in the training data. By augmenting the training data with a latent anomaly indicator variable whose distribution is inferred while training the underlying model using Monte Carlo EM, our method simultaneously infers anomalous points while improving model performance on nominal data. We demonstrate the effectiveness of the approach by combining it with a simple feed-forward forecasting model. We investigate how anomalies in the train set affect the training of forecasting models, which are commonly used for time series anomaly detection, and show that our method improves the training of the model.
Temporal Attention Augmented Transformer Hawkes Process
Venue: arXiv cs.LG 2021
Year: 2021

In recent years, mining the knowledge from asynchronous sequences by Hawkes process is a subject worthy of continued attention, and Hawkes processes based on the neural network have gradually become the most hotly researched fields, especially based on the recurrence neural network (RNN). However, these models still contain some inherent shortcomings of RNN, such as vanishing and exploding gradient and long-term dependency problems. Meanwhile, Transformer based on self-attention has achieved great success in sequential modeling like text processing and speech recognition. Although the Transformer Hawkes process (THP) has gained huge performance improvement, THPs do not effectively utilize the temporal information in the asynchronous events, for these asynchronous sequences, the event occurrence instants are as important as the types of events, while conventional THPs simply convert temporal information into position encoding and add them as the input of transformer. With this in mind, we come up with a new kind of Transformer-based Hawkes process model, Temporal Attention Augmented Transformer Hawkes Process (TAA-THP), we modify the traditional dot-product attention structure, and introduce the temporal encoding into attention structure. We conduct numerous experiments on a wide range of synthetic and real-life datasets to validate the performance of our proposed TAA-THP model, significantly improvement compared with existing baseline models on the different measurements is achieved, including log-likelihood on the test dataset, and prediction accuracies of event types and occurrence times. In addition, through the ablation studies, we vividly demonstrate the merit of introducing additional temporal attention by comparing the performance of the model with and without temporal attention.
Lyapunov Exponents for Diversity in Differentiable Games
Venue: arXiv cs.GT 2021
Year: 2021

Ridge Rider (RR) is an algorithm for finding diverse solutions to optimization problems by following eigenvectors of the Hessian ("ridges"). RR is designed for conservative gradient systems (i.e., settings involving a single loss function), where it branches at saddles - easy-to-find bifurcation points. We generalize this idea to non-conservative, multi-agent gradient systems by proposing a method - denoted Generalized Ridge Rider (GRR) - for finding arbitrary bifurcation points. We give theoretical motivation for our method by leveraging machinery from the field of dynamical systems. We construct novel toy problems where we can visualize new phenomena while giving insight into high-dimensional problems of interest. Finally, we empirically evaluate our method by finding diverse solutions in the iterated prisoners' dilemma and relevant machine learning problems including generative adversarial networks.
Bayesian Compressive Channel Estimation for Hybrid Full-Dimensional MIMO Communications
Venue: arXiv cs.IT 2021
Year: 2021

Efficient channel estimation is challenging in full-dimensional multiple-input multiple-output communication systems, particularly in those with hybrid digital-analog architectures. Under a compressive sensing framework, this letter first designs a uniform dictionary based on a spherical Fibonacci grid to represent channels in a sparse domain, yielding smaller angular errors in three-dimensional beamspace than traditional dictionaries. Then, a Bayesian inference-aided greedy pursuit algorithm is developed to estimate channels in the frequency domain. Finally, simulation results demonstrate that both the designed dictionary and the proposed Bayesian channel estimation outperform the benchmark schemes and attain a lower normalized mean squared error of channel estimation.
Efficient Data Exchange in Unmanned Aerial Vehicle Networks Utilizing Unsupervised Learning-Based Clustering
Venue: arXiv cs.IT 2021
Year: 2021

An unmanned aerial vehicle (UAV) network can serve as an aerial relay to periodically receive packets from macro base stations (BSs). Severe packet loss may happen especially when UAVs have bad wireless connections to a BS. In this paper, a data exchange scheme is proposed utilizing unsupervised learning to enable efficient lost packet retrieval through reliable wireless transmissions between UAVs instead of through retransmissions of macro BSs with a longer delay and higher overhead. With the proposed scheme, all UAVs are assigned to multiple clusters and a UAV can only request its lost packets to UAVs in the same cluster. By this way, UAVs in different clusters could carry out their lost packets retrieval processes simultaneously to expedite data exchange. The agglomerative hierarchical clustering, a type of unsupervised learning, is used to conduct clustering, guaranteeing that UAVs clustered together could supply and supplement each other's lost packets. To further enhance data exchange efficiency, a data exchange mechanism is designed, where the priority of UAVs performing data exchange is determined by the number of their lost packets or the number of requested packets that they can provide. The introduced data exchange mechanism can make each request-reply process maximally beneficial to other UAVs' lost packet retrieval in the same cluster. A new random backoff procedure based on the carrier sense multiple access with collision avoidance (CSMA/CA) is designed to support the data exchange mechanism. Finally, simulation studies are conducted to demonstrate the effectiveness and superiority of our proposed data exchange scheme.
Optimal Weighted Load Balancing in TCAMs
Venue: arXiv cs.NI 2021
Year: 2021

Traffic splitting is a required functionality in networks, for example for load balancing over multiple paths or among different servers. The capacities of the servers determine the partition by which traffic should be split. A recent approach implements traffic splitting within the ternary content addressable memory (TCAM), which is often available in switches. It is important to reduce the amount of memory allocated for this task since TCAMs are power consuming and are often also required for other tasks such as classification and routing. Previous work showed how to compute the smallest prefix-matching TCAM necessary to implement a given partition exactly. In this paper we solve the more practical case, where at most $n$ prefix-matching TCAM rules are available, restricting the ability to implement exactly the desired partition. We give simple and efficient algorithms to find $n$ rules that generate a partition closest in $L_\infty$ to the desired one. We do the same for a one-sided version of $L_\infty$ which equals to the maximum overload on a server and for a relative version of it. We use our algorithms to evaluate how the expected error changes as a function of the number of rules, the number of servers, and the width of the TCAM.
Frame invariance and scalability of neural operators for partial differential equations
Venue: arXiv cs.LG 2021
Year: 2021

Partial differential equations (PDEs) play a dominant role in the mathematical modeling of many complex dynamical processes. Solving these PDEs often requires prohibitively high computational costs, especially when multiple evaluations must be made for different parameters or conditions. After training, neural operators can provide PDEs solutions significantly faster than traditional PDE solvers. In this work, invariance properties and computational complexity of two neural operators are examined for transport PDE of a scalar quantity. Neural operator based on graph kernel network (GKN) operates on graph-structured data to incorporate nonlocal dependencies. Here we propose a modified formulation of GKN to achieve frame invariance. Vector cloud neural network (VCNN) is an alternate neural operator with embedded frame invariance which operates on point cloud data. GKN-based neural operator demonstrates slightly better predictive performance compared to VCNN. However, GKN requires an excessively high computational cost that increases quadratically with the increasing number of discretized objects as compared to a linear increase for VCNN.
Working mechanism of Eternalblue and its application in ransomworm
Venue: arXiv cs.CR 2021
Year: 2021

After the leaking of exploit Eternalblue, some ransomworms utilizing this exploit have been developed to sweep over the world in recent years. Ransomworm is a global growing threat as it blocks users' access to their files unless a ransom is paid by victims. Wannacry and Notpetya are two of those ransomworms which are responsible for the loss of millions of dollar, from crippling U.K. national systems to shutting down a Honda Motor Company in Japan. Many dynamic analytic papers on Wannacry were published, however, static analytic papers about Wannacry were limited. Our aim is to present readers an systematic knowledge about exploit Eternalblue, from a high\textendash leveled semantic view to the code details. Specifically, the working mechanism of Eternalblue, the reverse engineering analysis of Eternalblue in Wannacry, and the comparison with the Metasploit's Eternalblue exploit are presented. The key finding of our analysis is that the code remains almost the same when Eternalblue is transplanted into Wannacry, which indicates its potential for signatures and thus detection.
Recent Trends in Artificial Intelligence-inspired Electronic Thermal Management
Venue: arXiv cs.LG 2021
Year: 2021

The rise of computation-based methods in thermal management has gained immense attention in recent years due to the ability of deep learning to solve complex 'physics' problems, which are otherwise difficult to be approached using conventional techniques. Thermal management is required in electronic systems to keep them from overheating and burning, enhancing their efficiency and lifespan. For a long time, numerical techniques have been employed to aid in the thermal management of electronics. However, they come with some limitations. To increase the effectiveness of traditional numerical approaches and address the drawbacks faced in conventional approaches, researchers have looked at using artificial intelligence at various stages of the thermal management process. The present study discusses in detail, the current uses of deep learning in the domain of 'electronic' thermal management.