Miracle-dz/UniFAR-FTUs huggingface dataset:
Improved Learning Rates for Stochastic Optimization: Two Theoretical Viewpoints

Generalization performance of stochastic optimization stands a central place in machine learning. In this paper, we investigate the excess risk performance and towards improved learning rates for two popular approaches of stochastic optimization: empirical risk minimization (ERM) and stochastic gradient descent (SGD). Although there exists plentiful generalization analysis of ERM and SGD for supervised learning, current theoretical understandings of ERM and SGD are either have stronger assumptions in convex learning, e.g., strong convexity condition, or show slow rates and less studied in nonconvex learning. Motivated by these problems, we aim to provide improved rates under milder assumptions in convex learning and derive faster rates in nonconvex learning. It is notable that our analysis span two popular theoretical viewpoints: stability and uniform convergence. To be specific, in stability regime, we present high probability rates of order O(1/n) w.r. t. the sample size n for ERM and SGD with milder assumptions in convex learning and similar high probability rates of order O(1/n) in nonconvex learning, rather than in expectation. Furthermore, this type of learning rate is improved to faster order O(1/n2) in uniform convergence regime. To the best of our knowledge, for ERM and SGD, the learning rates presented in this paper are all state-of-the-art.
MoP3SiO11
: A 
4d3
 honeycomb antiferromagnet with disconnected octahedra

Danis I. Badrtdinov, Lei Ding, 3 Clemens Ritter, Jan Hembacher, Niyaz Ahmed, 6 Yurii Skourski, and Alexander A. Tsirlin 1, ∗ Theoretical Physics and Applied Mathematics Department, Ural Federal University, 620002 Yekaterinburg, Russia ISIS Facility, Rutherford Appleton Laboratory, Harwell Oxford, Didcot, UK Institut Néel, CNRS and Université Joseph Fourier, 38042 Grenoble, France Institut Laue-Langevin, BP 156, F-38042 Grenoble, France Experimental Physics VI, Center for Electronic Correlations and Magnetism, Institute of Physics, University of Augsburg, 86135 Augsburg, Germany School of Physics, Indian Institute of Science Education and Research Thiruvananthapuram-695551, India Dresden High Magnetic Field Laboratory (HLD-EMFL), Helmholtz-Zentrum Dresden-Rossendorf, 01328 Dresden, Germany
Autonomy 2.0: Why is self-driving always 5 years away?

Despite the numerous successes of machine learning over the past decade (image recognition, decision-making, NLP, image synthesis), self-driving technology has not yet followed the same trend. In this paper, we study the history, composition, and development bottlenecks of the modern self-driving stack. We argue that the slow progress is caused by approaches that require too much hand-engineering, an over-reliance on road testing, and high fleet deployment costs. We observe that the classical stack has several bottlenecks that preclude the necessary scale needed to capture the long tail of rare events. To resolve these problems, we outline the principles of Autonomy 2.0, an ML-first approach to self-driving, as a viable alternative to the currently adopted state-of-the-art. This approach is based on (i) a fully differentiable AV stack trainable from human demonstrations, (ii) closed-loop data-driven reactive simulation, and (iii) large-scale, low-cost data collections as critical solutions towards scalability issues. We outline the general architecture, survey promising works in this direction and propose key challenges to be addressed by the community in the future.
A High-Performance Adaptive Quantization Approach for Edge CNN Applications

Recent convolutional neural network (CNN) development continues to advance the state-of-the-art model accuracy for various applications. However, the enhanced accuracy comes at the cost of substantial memory bandwidth and storage requirements and demanding computational resources. Although in the past the quantization methods have effectively reduced the deployment cost for edge devices, it suffers from significant information loss when processing the biased activations of contemporary CNNs. In this paper, we hence introduce an adaptive high-performance quantization method to resolve the issue of biased activation by dynamically adjusting the scaling and shifting factors based on the task loss. Our proposed method has been extensively evaluated on image classification models (ResNet-18/34/50, MobileNet-V2, EfficientNet-B0) with ImageNet dataset, object detection model (YOLO-V4) with COCO dataset, and language models with PTB dataset. The results show that our 4-bit integer (INT4) quantization models achieve better accuracy than the state-of-the-art 4-bit models, and in some cases, even surpass the golden full-precision models. The final designs have been successfully deployed onto extremely resource-constrained edge devices for many practical applications.
Quantization-Aided Secrecy: FD C-RAN Communications With Untrusted Radios

In this work, we study a full-duplex (FD) cloud radio access network (C-RAN) from the aspects of infrastructure sharing and information secrecy, where the central unit utilizes FD remote radio units (RU)s belonging to the same operator, i.e., the trusted RUs, as well as the RUs belonging to other operators or private owners, i.e., the untrusted RUs. Furthermore, the communication takes place in the presence of untrusted external receivers, i.e., eavesdropper nodes. The communicated uplink (UL) and downlink (DL) waveforms are quantized in order to comply with the limited capacity of the fronthaul links. In order to provide information secrecy, we propose a novel utilization of the quantization noise shaping in the DL, such that it is simultaneously used to comply with the limited capacity of the fronthaul links, as well as to degrade decoding capability of the individual eavesdropper and the untrusted RUs for both the UL and DL communications. In this regard, expressions describing the achievable secrecy rates are obtained. An optimization problem for jointly designing the DL and UL quantization and precoding strategies are then formulated, with the purpose of maximizing the overall system weighted sum secrecy rate. Due to the intractability of the formulated problem, an iterative solution is proposed, following the successive inner approximation and semi-definite relaxation frameworks, with convergence to a stationary point. Numerical evaluations indicate a promising gain of the proposed approaches for providing information secrecy against the untrusted infrastructure nodes and/or external eavesdroppers in the context of FD C-RAN communications.
Nonparametric Finite Mixture Models with Possible Shape Constraints: A Cubic Newton Approach

We explore computational aspects of maximum likelihood estimation of the mixture proportions of a nonparametric finite mixture model -- a convex optimization problem with old roots in statistics and a key member of the modern data analysis toolkit. Motivated by problems in shape constrained inference, we consider structured variants of this problem with additional convex polyhedral constraints. We propose a new cubic regularized Newton method for this problem and present novel worst-case and local computational guarantees for our algorithm. We extend earlier work by Nesterov and Polyak to the case of a self-concordant objective with polyhedral constraints, such as the ones considered herein. We propose a Frank-Wolfe method to solve the cubic regularized Newton subproblem; and derive efficient solutions for the linear optimization oracles that may be of independent interest. In the particular case of Gaussian mixtures without shape constraints, we derive bounds on how well the finite mixture problem approximates the infinite-dimensional Kiefer-Wolfowitz maximum likelihood estimator. Experiments on synthetic and real datasets suggest that our proposed algorithms exhibit improved runtimes and scalability features over existing benchmarks.
Redshift drift cosmography for model-independent cosmological inference

We develop a cosmographic framework for analysing redshift drift signals of nearby sources modelindependently, i.e., without making assumptions about the metric description of the Universe. We show that the Friedmann-Lemâıtre-Robertson-Walker (FLRW) prediction is altered nontrivially by regional anisotropies and inhomogeneities. In particular, we find that the position drift of the sources is nontrivially linked to the redshift drift signal. The redshift drift signal for nearby sources might be formulated in terms of an effective deceleration parameter, which reduces to the FLRW deceleration parameter in the homogeneous and isotropic limit. The presented cosmographic framework can be used for model-independent data analysis, exploiting the fact that the exact anisotropic redshift drift signal at lowest order in redshift is given by a finite set of physically interpretable coefficients. We discuss physical limits of interest as well as challenges related to the framework.
Action Forecasting with Feature-wise Self-Attention

We present a new architecture for human action forecasting from videos. A temporal recurrent encoder captures temporal information of input videos while a self-attention model is used to attend on relevant feature dimensions of the input space. To handle temporal variations in observed video data, a feature masking techniques is employed. We classify observed actions accurately using an auxiliary classifier which helps to understand what has happened so far. Then the decoder generates actions for the future based on the output of the recurrent encoder and the self-attention model. Experimentally, we validate each component of our architecture where we see that the impact of self-attention to identify relevant feature dimensions, temporal masking, and observed auxiliary classifier. We evaluate our method on two standard action forecasting benchmarks and obtain state-of-the-art results.
A survey for variable young stars with small telescopes – IV. Rotation periods of YSOs in IC 5070

Studying rotational variability of young stars is enabling us to investigate a multitude of properties of young star-disc systems. We utilize high cadence, multiwavelength optical time series data from the Hunting Outbursting Young Stars citizen science project to identify periodic variables in the Pelican Nebula (IC 5070). A double blind study using nine different period-finding algorithms was conducted and a sample of 59 periodic variables was identified. We find that a combination of four period finding algorithms can achieve a completeness of 85 per cent and a contamination of 30 per cent in identifying periods in inhomogeneous data sets. The best performing methods are periodograms that rely on fitting a sine curve. Utilizing Gaia EDR3 data, we have identified an unbiased sample of 40 periodic young stellar objects (YSOs), without using any colour or magnitude selections. With a 98.9 per cent probability, we can exclude a homogeneous YSO period distribution. Instead, we find a bi-modal distribution with peaks at 3 and 8 d. The sample has a disc fraction of 50 per cent, and its statistical properties are in agreement with other similarly aged YSOs populations. In particular, we confirm that the presence of the disc is linked to predominantly slow rotation and find a probability of 4.8 × 10−3 that the observed relation between period and presence of a disc has occurred by chance. In our sample of periodic variables, we also find pulsating giants, an eclipsing binary, and potential YSOs in the foreground of IC 5070.
Support vector machines for learning reactive islands

We develop a machine learning framework that can be applied to data sets derived from the trajectories of Hamilton's equations. The goal is to learn the phase space structures that play the governing role for phase space transport relevant to particular applications. Our focus is on learning reactive islands in two degrees-of-freedom Hamiltonian systems. Reactive islands are constructed from the stable and unstable manifolds of unstable periodic orbits and play the role of quantifying transition dynamics. We show that the support vector machines are an appropriate machine learning framework for this purpose as it provides an approach for finding the boundaries between qualitatively distinct dynamical behaviors, which is in the spirit of the phase space transport framework. We show how our method allows us to find reactive islands directly in the sense that we do not have to first compute unstable periodic orbits and their stable and unstable manifolds. We apply our approach to the Hénon-Heiles Hamiltonian system, which is a benchmark system in the dynamical systems community. We discuss different sampling and learning approaches and their advantages and disadvantages.
Learning to Limit Data Collection via Scaling Laws: Data Minimization Compliance in Practice

Data minimization is a legal obligation deﬁned in the European Union’s General Data Protection Regulation (GDPR) as the responsibility to process an adequate, relevant, and limited amount of personal data in relation to a processing purpose. However, unlike fairness or transparency, the principle has not seen wide adoption for machine learning systems due to a lack of computational interpretation. In this paper, we build on literature in machine learning and law to propose the ﬁrst learning framework for limiting data collection based on an interpretation that ties the data collection purpose to system performance. We formalize a data minimization criterion based on performance curve derivatives and provide an effective and interpretable piecewise power law technique that models distinct stages of an algorithm’s performance throughout data collection. Results from our empirical investigation offer deeper insights into the relevant considerations when designing a data minimization framework, including the choice of feature acquisition algorithm, initialization conditions, as well as impacts on individuals that hint at tensions between data minimization and fairness.
Over-Parameterization and Generalization in Audio Classification

Convolutional Neural Networks (CNNs) have been dominating classification tasks in various domains, such as machine vision, machine listening, and natural language processing. In machine listening, while generally exhibiting very good generalization capabilities, CNNs are sensitive to the specific audio recording device used, which has been recognized as a substantial problem in the acoustic scene classification (DCASE) community. In this study, we investigate the relationship between over-parameterization of acoustic scene classification models, and their resulting generalization abilities. Our results indicate that increasing width improves generalization to unseen devices, even without an increase in the number of parameters.
On the Copying Behaviors of Pre-Training for Neural Machine Translation

Previous studies have shown that initializing neural machine translation (NMT) models with the pre-trained language models (LM) can speed up the model training and boost the model performance. In this work, we identify a critical side-effect of pre-training for NMT, which is due to the discrepancy between the training objectives of LM-based pre-training and NMT. Since the LM objective learns to reconstruct a few source tokens and copy most of them, the pre-training initialization would affect the copying behaviors of NMT models. We provide a quantitative analysis of copying behaviors by introducing a metric called copying ratio, which empirically shows that pre-training based NMT models have a larger copying ratio than the standard one. In response to this problem, we propose a simple and effective method named copying penalty to control the copying behaviors in decoding. Extensive experiments on both in-domain and out-of-domain benchmarks show that the copying penalty method consistently improves translation performance by controlling copying behaviors for pre-training based NMT models. Source code is freely available at https://github.com/SunbowLiu/CopyingPenalty.
YOLOX: Exceeding YOLO Series in 2021

In this report, we present some experienced improvements to YOLO series, forming a new high-performance detector -- YOLOX. We switch the YOLO detector to an anchor-free manner and conduct other advanced detection techniques, i.e., a decoupled head and the leading label assignment strategy SimOTA to achieve state-of-the-art results across a large scale range of models: For YOLO-Nano with only 0.91M parameters and 1.08G FLOPs, we get 25.3% AP on COCO, surpassing NanoDet by 1.8% AP; for YOLOv3, one of the most widely used detectors in industry, we boost it to 47.3% AP on COCO, outperforming the current best practice by 3.0% AP; for YOLOX-L with roughly the same amount of parameters as YOLOv4-CSP, YOLOv5-L, we achieve 50.0% AP on COCO at a speed of 68.9 FPS on Tesla V100, exceeding YOLOv5-L by 1.8% AP. Further, we won the 1st Place on Streaming Perception Challenge (Workshop on Autonomous Driving at CVPR 2021) using a single YOLOX-L model. We hope this report can provide useful experience for developers and researchers in practical scenes, and we also provide deploy versions with ONNX, TensorRT, NCNN, and Openvino supported. Source code is at https://github.com/Megvii-BaseDetection/YOLOX.
A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search

In this work, we propose a novel evolutionary algorithm for neural architecture search, applicable to global search spaces. The algorithm's architectural representation organizes the topology in multiple hierarchical modules, while the design process exploits this representation, in order to explore the search space. We also employ a curation system, which promotes the utilization of well performing sub-structures to subsequent generations. We apply our method to Fashion-MNIST and NAS-Bench101, achieving accuracies of $93.2\%$ and $94.8\%$ respectively in a relatively small number of generations.
High performance of generating and transporting hot carriers in plasmonic TiN: A first-principles study

Promoting performance of generation and transport of hot carriers in metal/semiconductor junctions is critical for harvesting energy of hot carriers. However, the low injection efficiency of hot carriers generated in the commonly used noble metals such as Au hinder the applications of hot-carrier devices. Here, we proposed that metallic TiN might be a better plasmonic material than the noble metals for generating and transporting hot carriers, based on first-principles calculations and Monte Carlo simulations. For the TiN/TiO2 junction, the concentration of hot carriers near the Fermi level is high, the lifetime and mean free path of the hot carriers are long, and the injection efficiency is large. The optimal injection efficiency could be achieved in a core/shell cylindrical TiN/TiO2 junction with the thickness of TiN ~ 5 nm and the incident photon energy ~ 0.6 eV.
Three-Way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement

Abstract When evaluating the clinical performance of a medical imaging device, a multi-reader multi-case (MRMC) analysis is usually applied to account for both case and reader variability. For a clinical task that equates to a quantitative measurement, an agreement analysis such as a limits of agreement (LOA) method can be used to compare different measurement methods. In this work, we introduce four types of comparisons; these types differ depending on whether the measurements are within or between readers and within or between modalities. A three-way mixed effect ANOVA model is applied to estimate the variances of individual differences, which is an essential step for estimating LOA. To verify the estimates of LOA, we propose a hierarchical model to simulate quantitative MRMC data. Two simulation studies were conducted to validate both the simulation and the LOA variance estimates. From the simulation results, we can conclude that our estimate of variance is unbiased, and the uncertainty of the estimation drops as the number of readers and cases increases and rises as the value of true variance increases.
UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition

Action recognition based on skeleton data has recently witnessed increasing attention and progress. State-of-the-art approaches adopting Graph Convolutional networks (GCNs) can effectively extract features on human skeletons relying on the pre-defined human topology. Despite associated progress, GCN-based methods have difficulties to generalize across domains, especially with different human topological structures. In this context, we introduce UNIK, a novel skeleton-based action recognition method that is not only effective to learn spatio-temporal features on human skeleton sequences but also able to generalize across datasets. This is achieved by learning an optimal dependency matrix from the uniform distribution based on a multi-head attention mechanism. Subsequently, to study the cross-domain generalizability of skeleton-based action recognition in real-world videos, we re-evaluate state-of-the-art approaches as well as the proposed UNIK in light of a novel Posetics dataset. This dataset is created from Kinetics-400 videos by estimating, refining and filtering poses. We provide an analysis on how much performance improves on smaller benchmark datasets after pre-training on Posetics for the action classification task. Experimental results show that the proposed UNIK, with pre-training on Posetics, generalizes well and outperforms state-of-the-art when transferred onto four target action classification datasets: Toyota Smarthome, Penn Action, NTU-RGB+D 60 and NTU-RGB+D 120.
A congested schedule-based dynamic transit passenger flow estimator using stop count data

A network-level dynamic transit passenger flow estimation model based on congested schedule-based transit equilibrium assignment is proposed using observations from stop count data. A solution algorithm is proposed for the mathematical program with schedule-based transit equilibrium constraints with proven quadratic space- and time-complexities. The error bound is proven to be linearly proportional to the number of measurements. Computational experiments are conducted first to verify the methodology with two synthetic data sets (one of which is Sioux Falls compared to a benchmark model), followed by a validation of the method using bus data from Qingpu District in Shanghai, China from 1 July 2016, with 4 bus lines, 120 segments, and 55 bus stops. The estimated average of segment flows is only 2.5% off from the average of the observed segment flows; relative errors among segments are 42.5%, which fares well compared with even less complex OD estimation methods in the literature.
Damped Dirac magnon in the metallic kagome antiferromagnet FeSn

The kagome lattice is a fertile platform to explore topological excitations with both Fermi-Dirac and Bose-Einstein statistics. While relativistic Dirac Fermions and flat-bands have been discovered in the electronic structure of kagome metals, the spin excitations have received less attention. Here we report inelastic neutron scattering studies of the prototypical kagome magnetic metal FeSn. The spectra display well-defined spin waves extending up to 120 meV. Above this energy, the spin waves become progressively broadened, reflecting interactions with the Stoner continuum. Using linear spin wave theory, we determine an effective spin Hamiltonian that reproduces the measured dispersion. This analysis indicates that the Dirac magnon at the K-point remarkably occurs on the brink of a region where well-defined spin waves become unobservable. Our results emphasize the influential role of itinerant carriers on the topological spin excitations of metallic kagome magnets.