Miracle-dz/UniFAR-FTUs huggingface dataset:
An Architecture for Memory Centric Active Storage (MCAS)

The advent of CPU-attached persistent memory technology, such as Intel's Optane Persistent Memory Modules (PMM), has brought with it new opportunities for storage. In 2018, IBM Research Almaden began investigating and developing a new enterprise-grade storage solution directly aimed at this emerging technology. MCAS (Memory Centric Active Storage) defines an evolved network-attached key-value store that offers both near-data compute and the ability to layer enterprise-grade data management services on shared persistent memory. As a converged memory-storage tier, MCAS moves towards eliminating the traditional separation of compute and storage, and thereby unifying the data space. This paper provides an in-depth review of the MCAS architecture and implementation, as well as general performance results.
Transformer in Transformer

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as"visual sentences"and present to further divide them into smaller patches (e.g., 4$\times$4) as"visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
A Minimax Probability Machine for Nondecomposable Performance Measures

Imbalanced classification tasks are widespread in many real-world applications. For such classification tasks, in comparison with the accuracy rate (AR), it is usually much more appropriate to use nondecomposable performance measures such as the area under the receiver operating characteristic curve (AUC) and the <inline-formula> <tex-math notation="LaTeX">$F_\beta $ </tex-math></inline-formula> measure as the classification criterion since the label class is imbalanced. On the other hand, the minimax probability machine is a popular method for binary classification problems and aims at learning a linear classifier by maximizing the AR, which makes it unsuitable to deal with imbalanced classification tasks. The purpose of this article is to develop a new minimax probability machine for the <inline-formula> <tex-math notation="LaTeX">$F_\beta $ </tex-math></inline-formula> measure, called minimax probability machine for the <inline-formula> <tex-math notation="LaTeX">$F_\beta $ </tex-math></inline-formula>-measures (MPMF), which can be used to deal with imbalanced classification tasks. A brief discussion is also given on how to extend the MPMF model for several other nondecomposable performance measures listed in the article. To solve the MPMF model effectively, we derive its equivalent form which can then be solved by an alternating descent method to learn a linear classifier. Further, the kernel trick is employed to derive a nonlinear MPMF model to learn a nonlinear classifier. Several experiments on real-world benchmark datasets demonstrate the effectiveness of our new model.
Deep Active Shape Model for Face Alignment and Pose Estimation in Mobile Environment

Active Shape Model (ASM) is a statistical model of object shapes that represents a target structure. ASM can guide machine learning algorithms to ﬁt a set of points representing an object (e.g., face) onto an image. This paper presents a lightweight Convolutional Neural Network (CNN) architecture with a loss function being assisted by ASM for face alignment and estimating head pose in the wild. We use ASM to ﬁrst guide the network towards learning the smoother distribution of the facial landmark points. Then, during the training process, inspired by the transfer learning, we gradually harden the regression problem and lead the network towards learning the original landmark points distribution. We deﬁne multi-tasks in our loss function that are responsible for detecting facial landmark points, as well as estimating face pose. Learning multiple correlated tasks simultaneously builds synergy and improves the performance of individual tasks. We compare the performance of our proposed CNN, ASMNet with MobileNetV2 (which is about 2 times bigger ASMNet) in both face alignment and pose estimation tasks. Experimental results on challenging datasets show that by using the proposed ASM assisted loss function, ASMNet performance is comparable with MobileNetV2 in face alignment task. Besides, for face pose estimation, ASMNet performs much better than MobileNetV2. Moreover, overall ASMNet achieves an acceptable performance for facial landmark points detection and pose estimation while having a signiﬁcantly smaller number of parameters and ﬂoating-point operations comparing to many CNN-based proposed models.
Local tail statistics of heavy-tailed random matrix ensembles with unitary invariance

We study heavy-tailed Hermitian random matrices that are unitarily invariant. The invariance implies that the eigenvalue and eigenvector statistics are decoupled. The motivating question has been whether a freely stable random matrix has stable eigenvalue statistics for the largest eigenvalues in the tail. We investigate this question through the use of both numerical and analytical means, the latter of which makes use of the supersymmetry method. A surprising behaviour is uncovered in that a freely stable random matrix does not necessarily yield stable statistics and if it does then it might exhibit Poisson or Poisson-like statistics. The Poisson statistics have been already observed for heavy-tailed Wigner matrices. We conclude with two conjectures on this peculiar behaviour.
A new tool GSEQ-FRC for two-dimensional field-reversed configuration equilibrium

A new tool (GSEQ-FRC) is developed for solving two-dimensional equilibrium of the field-reversed configuration (FRC) based on fixed boundary and free boundary conditions that includes external coils. Benefiting from the two-parameter modified rigid rotor radial equilibrium model and the numerical approaches presented (Ma et al 2021 Nucl. Fusion 61 036046) in this work, the GSEQ-FRC code is used to study the equilibrium properties of FRC quantitatively and will be used for fast FRC equilibrium reconstruction. In GSEQ-FRC, the FRC equilibrium is determined by two parameters, i.e. the ratio between thermal pressure and magnetic pressure at the separatrix β s , and the normalized scrape of layer width δ s . Examples with fixed and free boundary conditions are given to demonstrate the capability of GSEQ-FRC for equilibrium calculations. This new tool is used to quantitatively study the factors affecting the shape of the FRC separatrix, which reveals how the FRC changes from racetrack-like to ellipse-like.
Adversarial Reciprocal Points Learning for Open Set Recognition

Open set recognition (OSR), aiming to simultaneously classify the seen classes and identify the unseen classes as ’unknown’, is essential for reliable machine learning. The key challenge of OSR is how to reduce the empirical classification risk on the labeled known data and the open space risk on the potential unknown data simultaneously. To handle the challenge, we formulate the open space risk problem from the perspective of multi-class integration, and model the unexploited extra-class space with a novel concept Reciprocal Point. Follow this, a novel learning framework, termed Adversarial Reciprocal Point Learning (ARPL), is proposed to minimize the overlap of known distribution and unknown distributions without loss of known classification accuracy. Specifically, each reciprocal point is learned by the extra-class space with the corresponding known category, and the confrontation among multiple known categories are employed to reduce the empirical classification risk. Then, an adversarial margin constraint is proposed to reduce the open space risk by limiting the latent open space constructed by reciprocal points. To further estimate the unknown distribution from open space, an instantiated adversarial enhancement method is designed to generate diverse and confusing training samples, based on the adversarial mechanism between the reciprocal points and known classes. This can effectively enhance the model distinguishability to the unknown classes. Extensive experimental results on various benchmark datasets indicate that the proposed method is significantly superior to other existing approaches and achieves state-of-the-art performance. The code is released on github.com/iCGY96/ARPL.
High-Performance Training by Exploiting Hot-Embeddings in Recommendation Systems

—Recommendation models are commonly used learning models that suggest relevant items to a user for e-commerce and online advertisement-based applications. Current recommendation models include deep-learning based (DLRM) and time-based sequence (TBSM) models. These models use massive embedding tables to store numerical representation of item’s and user’s categorical variables (memory bound) while also using neural networks to generate outputs (compute bound). Due to these conﬂicting compute and memory requirements, the training process for recommendation models is divided across CPU and GPU for embedding and neural network executions, respectively. Such a training process naively assigns same level of importance to each embedding entry. This paper observes that some training inputs and their accesses into the embedding tables are heavily skewed with certain entries being accessed up to 10000 × more. This paper tries to leverage skewed embedded table accesses to efﬁciently use the GPU resources during training. To this end, this paper proposes a Frequently Accessed Embeddings (FAE) framework that provides three key features. First, it exposes a dynamic knob to the software based on the GPU memory capacity and the input popularity index, to efﬁciently estimate and vary the size of the hot portions of the embedding tables. These hot embedding tables can then be stored locally on each GPU. FAE uses statistical techniques to determine the knob, which is a threshold on embedding accesses, without proﬁling the entire input dataset. Second, FAE pre-processes the inputs to segregate hot inputs (which only access hot embedding entries) and cold inputs into a collection of hot and cold mini-batches. This ensures that a training mini-batch is either entirely hot or cold to obtain most of beneﬁts. Third, at runtime FAE generates a dynamic schedule for the hot and cold training mini-batches that minimizes data transfer latency between CPU and GPU executions while maintaining the model accuracy. The framework execution then uses the GPU(s) for hot input mini-batches and a baseline CPU-GPU mode for cold input mini-batches. Overall, our framework speeds-up the training of the recommendation models on Kaggle, Terabyte, and Alibaba datasets by 2.34 × as compared to a baseline that uses Intel-Xeon CPUs and Nvidia Tesla-V100 GPUs, while maintaining accuracy.
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

Automatic program repair (APR) is crucial to improve software reliability. Recently, neural machine translation (NMT) techniques have been used to automatically fix software bugs. While promising, these approaches have two major limitations. Their search space often does not contain the correct fix, and their search strategy ignores software knowledge such as strict code syntax. Due to these limitations, existing NMT-based techniques underperform the best template-based approaches. We propose CURE, a new NMT-based APR technique with three major novelties. First, CURE pre-trains a programming language (PL) model on a large software codebase to learn developer-like source code before the APR task. Second, CURE designs a new code-aware search strategy that finds more correct fixes by focusing on searching for compilable patches and patches that are close in length to the buggy code. Finally, CURE uses a subword tokenization technique to generate a smaller search space that contains more correct fixes. Our evaluation on two widely-used benchmarks shows that CURE correctly fixes 57 Defects4J bugs and 26 QuixBugs bugs, outperforming all existing APR techniques on both benchmarks.
Statistical Inference for Local Granger Causality

Granger causality has been employed to investigate causality relations between components of stationary multiple time series. We generalize this concept by developing statistical inference for local Granger causality for multivariate locally stationary processes. Our proposed local Granger causality approach captures time-evolving causality relationships in nonstationary processes. The proposed local Granger causality is well represented in the frequency domain and estimated based on the parametric time-varying spectral density matrix using the local Whittle likelihood. Under regularity conditions, we demonstrate that the estimators converge to multivariate normal in distribution. Additionally, the test statistic for the local Granger causality is shown The author gratefully acknowledge JSPS Grant-in-Aid for Young Scientists (B) 17K12652 and JSPS Grant-in-Aid for Scientific Research (C) 20K11719 The author gratefully acknowledge JSPS Grant-in-Aid for Scientific Research (S) 18H05290 The author gratefully acknowledge the KAUST Research Fund 1 to be asymptotically distributed as a quadratic form of a multivariate normal distribution. The finite sample performance is confirmed with several simulation studies for multivariate time-varying autoregressive models. For practical demonstration, the proposed local Granger causality method uncovered new functional connectivity relationships between channels in brain signals. Moreover, the method was able to identify structural changes in financial data.
Analysis, Prediction, and Control of Epidemics: A Survey from Scalar to Dynamic Network Models

During the ongoing COVID-19 pandemic, mathematical models of epidemic spreading have emerged as powerful tools to produce valuable predictions of the evolution of the pandemic, helping public health authorities decide which intervention policies should be implemented. The study of these modelsÑgrounded in the systems theory and often analyzed using control-theoretic toolsÑis an extremely important area for many researchers from different fields, including epidemiology, engineering, physics, mathematics, computer science, sociology, economics, and management. In this survey, we review the history and present the state of the art in the modeling, analysis, and control of epidemic dynamics. We discuss different approaches to epidemic modeling, either deterministic or stochastic, ranging from the first implementations of scalar systems of differential equations, which describe the epidemic spreading at the population level, to the most recent models on dynamic networks, which capture the spatial spread and the time-varying nature of human interactions.
Distilling Knowledge via Intermediate Classifier Heads

The crux of knowledge distillation—as a transfer-learning approach—is to effectively train a resource-limited student model with the guide of a pre-trained larger teacher model. However, when there is a large difference between the model complexities of teacher and student (i.e., capacity gap ), knowledge distillation loses its strength in transferring knowledge from the teacher to the student, thus training a weaker student. To mitigate the impact of the capacity gap, we introduce knowledge distillation via intermediate heads . By extending the intermediate layers of the teacher (at various depth) with classiﬁer heads, we cheaply acquire a cohort of heterogeneous pre-trained teachers. The intermediate classiﬁer heads can all together be efﬁciently learned while freezing the backbone of the pre-trained teacher. The cohort of teachers (including the original teacher) co-teach the student simultaneously. Our experiments on various teacher-student pairs and datasets have demonstrated that the proposed approach outperforms the canonical knowledge distillation approach and its extensions.
Silent versus modal multi-speaker speech recognition from ultrasound and video

We investigate multi-speaker speech recognition from ultrasound images of the tongue and video images of the lips. We train our systems on imaging data from modal speech, and evaluate on matched test sets of two speaking modes: silent and modal speech. We observe that silent speech recognition from imaging data underperforms compared to modal speech recognition, likely due to a speaking-mode mismatch between training and testing. We improve silent speech recognition performance using techniques that address the domain mismatch, such as fMLLR and unsupervised model adaptation. We also analyse the properties of silent and modal speech in terms of utterance duration and the size of the articulatory space. To estimate the articulatory space, we compute the convex hull of tongue splines, extracted from ultrasound tongue images. Overall, we observe that the duration of silent speech is longer than that of modal speech, and that silent speech covers a smaller articulatory space than modal speech. Although these two properties are statistically significant across speaking modes, they do not directly correlate with word error rates from speech recognition.
A causal and continuous interpretation of the quantum theory: About an original manuscript by David Bohm sent to Louis de Broglie in 1951

Aurélien Drezet 2 Univ. Grenoble Alpes, CNRS, Grenoble INP, Institut Neel, F-38000 Grenoble, France Benjamin Stock, 2 1, Le Chêne Brûlé, 28400 Arcisses, France Abstract The aim of this article is reproduce and analyze an original article of David Bohm sent to Louis de Broglie in 1951. This article is the older document of David Bohm about his well known hidden variable theory based on the pilot wave interpretation of Louis de Broglie. We analyse the chronology and the history of this fascinating document.
Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation

This paper presents a novel method, Zero-Reference Deep Curve Estimation (Zero-DCE), which formulates light enhancement as a task of image-specific curve estimation with a deep network. Our method trains a lightweight deep network, DCE-Net, to estimate pixel-wise and high-order curves for dynamic range adjustment of a given image. The curve estimation is specially designed, considering pixel value range, monotonicity, and differentiability. Zero-DCE is appealing in its relaxed assumption on reference images, i.e., it does not require any paired or even unpaired data during training. This is achieved through a set of carefully formulated non-reference loss functions, which implicitly measure the enhancement quality and drive the learning of the network. Despite its simplicity, we show that it generalizes well to diverse lighting conditions. Our method is efficient as image enhancement can be achieved by an intuitive and simple nonlinear curve mapping. We further present an accelerated and light version of Zero-DCE, called Zero-DCE++, that takes advantage of a tiny network with just 10K parameters. Zero-DCE++ has a fast inference speed (1000/11 FPS on a single GPU/CPU for an image of size 1200×900×3) while keeping the enhancement performance of Zero-DCE. Extensive experiments on various benchmarks demonstrate the advantages of our method over state-of-the-art methods qualitatively and quantitatively. Furthermore, the potential benefits of our method to face detection in the dark are discussed. The source code is made publicly available at https://li-chongyi.github. io/Proj_Zero-DCE++. html.
Embedded Knowledge Distillation in Depth-level Dynamic Neural Network

In real applications, different computation-resource devices need different-depth networks (e.g., ResNet-18/34/50) with high-accuracy. Usually, existing methods either design multiple networks and train them independently, or construct depth-level/width-level dynamic neural networks which is hard to prove the accuracy of each sub-net. In this article, we propose an elegant Depth-Level Dynamic Neural Network (DDNN) integrated different-depth sub-nets of similar architectures. To improve the generalization of sub-nets, we design the Embedded-Knowledge-Distillation (EKD) training mechanism for the DDNN to implement knowledge transfer from the teacher (full-net) to multiple students (sub-nets). Specifically, the Kullback-Leibler (KL) divergence is introduced to constrain the posterior class probability consistency between full-net and sub-nets, and self-attention distillation on the same resolution feature of different depth is addressed to drive more abundant feature representations of sub-nets. Thus, we can obtain multiple high-accuracy sub-nets simultaneously in a DDNN via the online knowledge distillation in each training iteration without extra computation cost. Extensive experiments on CIFAR-10/100, and ImageNet datasets demonstrate that sub-nets in DDNN with EKD training achieve better performance than individually training networks while preserving the original performance of full-nets.
Robust Stability Analysis of a Simple Data-Driven Model Predictive Control Approach

In this article, we provide a theoretical analysis of closed-loop properties of a simple data-driven model predictive control (MPC) scheme. The formulation does not involve any terminal ingredients, thus allowing for a simple implementation without (potential) feasibility issues. The proposed approach relies on an implicit description of linear time-invariant systems based on behavioral systems theory, which only requires one input–output trajectory of an unknown system. For the nominal case with noise-free data, we prove that the data-driven MPC scheme ensures exponential stability for the closed loop if the prediction horizon is sufficiently long. Moreover, we analyze the robust data-driven MPC scheme for noisy output measurements for which we prove closed-loop practical exponential stability. The advantages of the presented approach are illustrated with a numerical example.