OpenRaiser/Intern-Atlas huggingface dataset:
Learning a Model-Driven Variational Network for Deformable Image Registration
Venue: arXiv cs.CV 2021
Year: 2021

Data-driven deep learning approaches to image registration can be less accurate than conventional iterative approaches, especially when training data is limited. To address this whilst retaining the fast inference speed of deep learning, we propose VR-Net, a novel cascaded variational network for unsupervised deformable image registration. Using the variable splitting optimization scheme, we first convert the image registration problem, established in a generic variational framework, into two sub-problems, one with a point-wise, closed-form solution while the other one is a denoising problem. We then propose two neural layers (i.e. warping layer and intensity consistency layer) to model the analytical solution and a residual U-Net to formulate the denoising problem (i.e. generalized denoising layer). Finally, we cascade the warping layer, intensity consistency layer, and generalized denoising layer to form the VR-Net. Extensive experiments on three (two 2D and one 3D) cardiac magnetic resonance imaging datasets show that VR-Net outperforms state-of-the-art deep learning methods on registration accuracy, while maintains the fast inference speed of deep learning and the data-efficiency of variational model.
Database Workload Characterization with Query Plan Encoders
Venue: arXiv cs.DB 2021
Year: 2021

Smart databases are adopting artificial intelligence (AI) technologies to achieve {\em instance optimality}, and in the future, databases will come with prepackaged AI models within their core components. The reason is that every database runs on different workloads, demands specific resources, and settings to achieve optimal performance. It prompts the necessity to understand workloads running in the system along with their features comprehensively, which we dub as workload characterization. To address this workload characterization problem, we propose our query plan encoders that learn essential features and their correlations from query plans. Our pretrained encoders capture the {\em structural} and the {\em computational performance} of queries independently. We show that our pretrained encoders are adaptable to workloads that expedite the transfer learning process. We performed independent assessments of structural encoder and performance encoders with multiple downstream tasks. For the overall evaluation of our query plan encoders, we architect two downstream tasks (i) query latency prediction and (ii) query classification. These tasks show the importance of feature-based workload characterization. We also performed extensive experiments on individual encoders to verify the effectiveness of representation learning and domain adaptability.
Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking
Venue: arXiv cs.CL 2021
Year: 2021

Chinese Spell Checking (CSC) aims to detect and correct erroneous characters for user-generated text in the Chinese language. Most of the Chinese spelling errors are misused semantically, phonetically or graphically similar characters. Previous attempts noticed this phenomenon and try to use the similarity for this task. However, these methods use either heuristics or handcrafted confusion sets to predict the correct character. In this paper, we propose a Chinese spell checker called ReaLiSe, by directly leveraging the multimodal information of the Chinese characters. The ReaLiSe model tackles the CSC task by (1) capturing the semantic, phonetic and graphic information of the input characters, and (2) selectively mixing the information in these modalities to predict the correct output. Experiments on the SIGHAN benchmarks show that the proposed model outperforms strong baselines by a large margin.
Optimal Transport Based Refinement of Physics-Informed Neural Networks
Venue: arXiv cs.CE 2021
Year: 2021

In this paper, we propose a refinement strategy to the well-known Physics-Informed Neural Networks (PINNs) for solving partial differential equations (PDEs) based on the concept of Optimal Transport (OT). Conventional black-box PINNs solvers have been found to suffer from a host of issues: spectral bias in fully-connected architectures, unstable gradient pathologies, as well as difficulties with convergence and accuracy. Current network training strategies are agnostic to dimension sizes and rely on the availability of powerful computing resources to optimize through a large number of collocation points. This is particularly challenging when studying stochastic dynamical systems with the Fokker-Planck-Kolmogorov Equation (FPKE), a second-order PDE which is typically solved in high-dimensional state space. While we focus exclusively on the stationary form of the FPKE, positivity and normalization constraints on its solution make it all the more unfavorable to solve directly using standard PINNs approaches. To mitigate the above challenges, we present a novel training strategy for solving the FPKE using OT-based sampling to supplement the existing PINNs framework. It is an iterative approach that induces a network trained on a small dataset to add samples to its training dataset from regions where it nominally makes the most error. The new samples are found by solving a linear programming problem at every iteration. The paper is complemented by an experimental evaluation of the proposed method showing its applicability on a variety of stochastic systems with nonlinear dynamics.
Runtime Monitoring for Markov Decision Processes
Venue: arXiv cs.LO 2021
Year: 2021

We investigate the problem of monitoring partially observable systems with nondeterministic and probabilistic dynamics. In such systems, every state may be associated with a risk, e.g., the probability of an imminent crash. During runtime, we obtain partial information about the system state in form of observations. The monitor uses this information to estimate the risk of the (unobservable) current system state. Our results are threefold. First, we show that extensions of state estimation approaches do not scale due the combination of nondeterminism and probabilities. While convex hull algorithms improve the practical runtime, they do not prevent an exponential memory blowup. Second, we present a tractable algorithm based on model checking conditional reachability probabilities. Third, we provide prototypical implementations and manifest the applicability of our algorithms to a range of benchmarks. The results highlight the possibilities and boundaries of our novel algorithms.
Multiple Domain Experts Collaborative Learning: Multi-Source Domain Generalization For Person Re-Identification
Venue: arXiv cs.CV 2021
Year: 2021

Recent years have witnessed significant progress in person re-identification (ReID). However, current ReID approaches still suffer from considerable performance degradation when unseen testing domains exhibit different characteristics from the source training ones, known as the domain generalization problem. Given multiple source training domains, previous Domain Generalizable ReID (DG-ReID) methods usually learn all domains together using a shared network, which can't learn sufficient knowledge from each domain. In this paper, we propose a novel Multiple Domain Experts Collaborative Learning (MECL) framework for better exploiting all training domains, which benefits from the proposed Domain-Domain Collaborative Learning (DDCL) and Universal-Domain Collaborative Learning (UDCL). DDCL utilizes domain-specific experts for fully exploiting each domain, and prevents experts from over-fitting the corresponding domain using a meta-learning strategy. In UDCL, a universal expert supervises the learning of domain experts and continuously gathers knowledge from all domain experts. Note, only the universal expert will be used for inference. Extensive experiments on DG-ReID benchmarks demonstrate the effectiveness of DDCL and UDCL, and show that the whole MECL framework significantly outperforms state-of-the-arts. Experimental results on DG-classification benchmarks also reveal the great potential of applying MECL to other DG tasks.
Using the Overlapping Score to Improve Corruption Benchmarks
Venue: arXiv cs.LG 2021
Year: 2021

Neural Networks are sensitive to various corruptions that usually occur in real-world applications such as blurs, noises, low-lighting conditions, etc. To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark. Unfortunately, no objective criterion exists to determine whether a benchmark is representative of a large diversity of independent corruptions. In this paper, we propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. Two corruptions overlap when the robustnesses of neural networks to these corruptions are correlated. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.
How to Calibrate Your Event Camera
Venue: arXiv cs.CV 2021
Year: 2021

We propose a generic event camera calibration framework using image reconstruction. Instead of relying on blinking LED patterns or external screens, we show that neural-network-based image reconstruction is well suited for the task of intrinsic and extrinsic calibration of event cameras. The advantage of our proposed approach is that we can use standard calibration patterns that do not rely on active illumination. Furthermore, our approach enables the possibility to perform extrinsic calibration between frame-based and event-based sensors without additional complexity. Both simulation and real-world experiments indicate that calibration through image reconstruction is accurate under common distortion models and a wide variety of distortion parameters
Joint Optimization of Tokenization and Downstream Model
Venue: arXiv cs.CL 2021
Year: 2021

Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies imply that the appropriate tokenization improves the performance. In this paper, we propose a novel method to find an appropriate tokenization to a given downstream model by jointly optimizing a tokenizer and the model. The proposed method has no restriction except for using loss values computed by the downstream model to train the tokenizer, and thus, we can apply the proposed method to any NLP task. Moreover, the proposed method can be used to explore the appropriate tokenization for an already trained model as post-processing. Therefore, the proposed method is applicable to various situations. We evaluated whether our method contributes to improving performance on text classification in three languages and machine translation in eight language pairs. Experimental results show that our proposed method improves the performance by determining appropriate tokenizations.
Deep Repulsive Prototypes for Adversarial Robustness
Venue: arXiv cs.LG 2021
Year: 2021

While many defences against adversarial examples have been proposed, finding robust machine learning models is still an open problem. The most compelling defence to date is adversarial training and consists of complementing the training data set with adversarial examples. Yet adversarial training severely impacts training time and depends on finding representative adversarial samples. In this paper we propose to train models on output spaces with large class separation in order to gain robustness without adversarial training. We introduce a method to partition the output space into class prototypes with large separation and train models to preserve it. Experimental results shows that models trained with these prototypes -- which we call deep repulsive prototypes -- gain robustness competitive with adversarial training, while also preserving more accuracy on natural samples. Moreover, the models are more resilient to large perturbation sizes. For example, we obtained over 50% robustness for CIFAR-10, with 92% accuracy on natural samples and over 20% robustness for CIFAR-100, with 71% accuracy on natural samples without adversarial training. For both data sets, the models preserved robustness against large perturbations better than adversarially trained models.
The Evaluation Case Study of Online Course During Pandemic Period in Mongolia
Venue: arXiv cs.CY 2021
Year: 2021

This paper describes a test and case study of self-evaluation of online courses during the pandemic time. Due to the Covid-19, the whole world needs to sit on lockdown in different periods. Many things need to be done in all kinds of business including the education sector of countries. To sustain the education development teaching methods had to switch from traditional face-to-face teaching to online courses. The government made decisions in a short time and educational institutions had no time to prepare the materials for the online teaching. All courses of the Mongolian University of Pharmaceutical Sciences switched to online lessons. Challenges were raised before professors and tutors during online teaching. Our university did not have a specific learning management system for online teaching and e-learning. Therefore professors used different platforms for their online teaching such as Zoom, Microsoft teams for instance. Moreover, different social networking platforms played an active role in communication between students and professors. The situation is very difficult for professors and students. To measure the quality of online courses and to figure out the positive and weak points of online teaching we need an evaluation of e-learning. The focus of this paper is to share the evaluation process of e-learning based on a structure-oriented evaluation model.
How Do Users Interact with an Error-Prone In-Air Gesture Recognizer?
Venue: arXiv cs.HC 2021
Year: 2021

We present results of two pilot studies that investigated human error behaviours with an error prone in-air gesture recognizer. During the studies, users performed a small set of simple in-air gestures. In the first study, these gestures were abstract. The second study associated concrete tasks with each gesture. Interestingly, the error patterns observed in the two studies were substantially different.
World War III Analysis using Signed Social Networks
Venue: arXiv cs.SI 2021
Year: 2021

In the recent period of time with a lot of social platforms emerging, the relationships among various units can be framed with respect to either positive, negative or no relation. These units can be individuals, countries or others that form the basic structural component of a signed network. These signed networks picture a dynamic characteristic of the graph so formed allowing only few combinations of signs that brings the structural balance theorem in picture. Structural balance theory affirms that signed social networks tend to be organized so as to avoid conflictual situations, corresponding to cycles of unstable relations. The aim of structural balance in networks is to find proper partitions of nodes that guarantee equilibrium in the system allowing only few combination triangles with signed edges to be permitted in graph. Most of the works in this field of networking have either explained the importance of signed graph or have applied the balance theorem and tried to solve problems. Following the recent time trends with each nation emerging to be superior and competing to be the best, the probable doubt of happening of WW-III(World War-III) comes into every individuals mind. Nevertheless, our paper aims at answering some of the interesting questions on World War-III. In this project we have worked with the creation of a signed graph picturing the World War-III participating countries as nodes and have predicted the best possible coalition of countries that will be formed during war. Also, we have visually depicted the number of communities that will be formed in this war and the participating countries in each communities.
TreeBERT: A Tree-Based Pre-Trained Model for Programming Language
Venue: arXiv cs.LG 2021
Year: 2021

Source code can be parsed into the abstract syntax tree (AST) based on defined syntax rules. However, in pre-training, little work has considered the incorporation of tree structure into the learning process. In this paper, we present TreeBERT, a tree-based pre-trained model for improving programming language-oriented generation tasks. To utilize tree structure, TreeBERT represents the AST corresponding to the code as a set of composition paths and introduces node position embedding. The model is trained by tree masked language modeling (TMLM) and node order prediction (NOP) with a hybrid objective. TMLM uses a novel masking strategy designed according to the tree's characteristics to help the model understand the AST and infer the missing semantics of the AST. With NOP, TreeBERT extracts the syntactical structure by learning the order constraints of nodes in AST. We pre-trained TreeBERT on datasets covering multiple programming languages. On code summarization and code documentation tasks, TreeBERT outperforms other pre-trained models and state-of-the-art models designed for these tasks. Furthermore, TreeBERT performs well when transferred to the pre-trained unseen programming language.
Understanding the Role of Digital Technology in the Transitions of Refugee Families with Young Children into A New Culture: A Case Study of Scotland
Venue: arXiv cs.HC 2021
Year: 2021

The worldwide refugee crisis is a major current challenge, affecting the health and education of millions of families with children due to displacement. Despite the various challenges and risks of migration practices, numerous refugee families have access to interactive technologies during these processes. The aim of this ongoing study is to explore the role of technologies in the transitions of refugee families in Scotland. Based on Tudge's ecocultural theory, a qualitative case-study approach has been adopted. Semi-structured interviews have been conducted with volunteers who work with refugee families in a big city in Scotland, and proxy observations of young children were facilitated remotely by their refugee parents. A preliminary overview of the participants' insights of the use and role of technology for transitioning into a new culture is provided here.
Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

We propose a successive convex approximation based off-policy optimization (SCAOPO) algorithm to solve the general constrained reinforcement learning problem, which is formulated as a constrained Markov decision process (CMDP) in the context of average cost. The SCAOPO is based on solving a sequence of convex objective/feasibility optimization problems obtained by replacing the objective and constraint functions in the original problems with convex surrogate functions. At each iteration, the convex surrogate problem can be efficiently solved by Lagrange dual method even the policy is parameterized by a high-dimensional function. Moreover, the SCAOPO enables to reuse old experiences from previous updates, thereby significantly reducing the implementation cost when deployed in the real-world engineering systems that need to online learn the environment. In spite of the time-varying state distribution and the stochastic bias incurred by the off-policy learning, the SCAOPO with a feasible initial point can still provably converge to a Karush-Kuhn-Tucker (KKT) point of the original problem almost surely.
TEMPEST -- Synthesis Tool for Reactive Systems and Shields in Probabilistic Environments
Venue: arXiv cs.LO 2021
Year: 2021

We present Tempest, a synthesis tool to automatically create correct-by-construction reactive systems and shields from qualitative or quantitative specifications in probabilistic environments. A shield is a special type of reactive system used for run-time enforcement; i.e., a shield enforces a given qualitative or quantitative specification of a running system while interfering with its operation as little as possible. Shields that enforce a qualitative or quantitative specification are called safety-shields or optimal-shields, respectively. Safety-shields can be implemented as pre-shields or as post-shields, optimal-shields are implemented as post-shields. Pre-shields are placed before the system and restrict the choices of the system. Post-shields are implemented after the system and are able to overwrite the system's output. Tempest is based on the probabilistic model checker Storm, adding model checking algorithms for stochastic games with safety and mean-payoff objectives. To the best of our knowledge, Tempest is the only synthesis tool able to solve 2-1/2-player games with mean-payoff objectives without restrictions on the state space. Furthermore, Tempest adds the functionality to synthesize safe and optimal strategies that implement reactive systems and shields
Measuring information exchange and brokerage capacity of healthcare teams
Venue: arXiv cs.SI 2021
Year: 2021

Purpose: The purpose of this paper is to explore possible factors impacting team performance in healthcare, by focusing on information exchange within and across hospital's boundaries. Design/methodology/approach: Through a web-survey and group interviews, the authors collected data on the communication networks of 31 members of four interdisciplinary healthcare teams involved in a system redesign initiative within a large US children's hospital. The authors mapped their internal and external social networks based on management advice, technical support and knowledge dissemination within and across departments, studying interaction patterns that involved more than 700 actors. The authors then compared team performance and social network metrics such as degree, closeness and betweenness centrality, and computed cross ties and constraint levels for each team. Findings: The results indicate that highly effective teams were more inwardly focused and less connected to outside members. Moreover, highly recognized teams communicated frequently but, overall, less intensely than the others. Originality/value: Mapping knowledge flows and balancing internal focus and outward connectivity of interdisciplinary teams may help healthcare decision makers in their attempt to achieve high value for patients, families and employees.
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
Venue: arXiv cs.LG 2021
Year: 2021

Tremendous success of machine learning (ML) and the unabated growth in ML model complexity motivated many ML-specific designs in both CPU and accelerator architectures to speed up the model inference. While these architectures are diverse, highly optimized low-precision arithmetic is a component shared by most. Impressive compute throughputs are indeed often exhibited by these architectures on benchmark ML models. Nevertheless, production models such as recommendation systems important to Facebook's personalization services are demanding and complex: These systems must serve billions of users per month responsively with low latency while maintaining high prediction accuracy, notwithstanding computations with many tens of billions parameters per inference. Do these low-precision architectures work well with our production recommendation systems? They do. But not without significant effort. We share in this paper our search strategies to adapt reference recommendation models to low-precision hardware, our optimization of low-precision compute kernels, and the design and development of tool chain so as to maintain our models' accuracy throughout their lifespan during which topic trends and users' interests inevitably evolve. Practicing these low-precision technologies helped us save datacenter capacities while deploying models with up to 5X complexity that would otherwise not be deployed on traditional general-purpose CPUs. We believe these lessons from the trenches promote better co-design between hardware architecture and software engineering and advance the state of the art of ML in industry.