Miracle-dz/UniFAR-FTUs huggingface dataset:
A Hybrid Spatial-temporal Sequence-to-one Neural Network Model for Lane Detection

: Reliable and accurate lane detection is of vital importance for the safe performance of Lane Keeping Assistance and Lane Departure Warning systems. However, under certain challenging peculiar circumstances (e.g., marking degradation, serious vehicle occlusion), it is difficult to get satisfactory performance in accurately detecting the lane markings from one single image which is often the case in current literature. Since road markings are continuous lines on the road, the lanes that are difficult to be accurately detected in the current image frame might potentially be better inferred out if information from previous frames is incorporated. For this, we propose a novel hybrid spatial-temporal sequence-to-one deep learning architecture making full use of the spatial-temporal information in multiple frames of a continuous sequence of images to detect lane markings in the very last current image frame. Specifically, the hybrid model integrates the spatial convolutional neural network (SCNN), which is powerful in extracting spatial features and relationships in one single image, with convolutional long-short term memory (ConvLSTM) neural network, which can capture the spatial-temporal correlations and time dependencies among the image sequences. With the proposed model architecture, the advantages of both SCNN and ConvLSTM are fully combined and the spatial-temporal information is fully exploited. Treating lane detection as the image segmentation problem, we applied encoder-decoder structures to make it work in an end-to-end way. Extensive experiments on two large-scale datasets reveal that our proposed model can effectively handle challenging driving scenes and outperforms previous state-of-the-art methods.
Neural Tangent Kernel Eigenvalues Accurately Predict Generalization

Finding a quantitative theory of neural network generalization has long been a central goal of deep learning research. We extend recent results to demonstrate that, by examining the eigensystem of a neural network’s “neural tangent kernel”, one can predict its generalization performance when learning arbitrary functions. Our theory accurately predicts not only test mean-squared-error but all ﬁrst-and second-order statistics of the network’s learned function. Furthermore, using a measure quantifying the “learnability” of a given target function, we prove a new “no-free-lunch” theorem characterizing a fundamental tradeoff in the inductive bias of wide neural networks: improving a network’s generalization for a given target function must worsen its generalization for orthogonal functions. We further demonstrate the utility of our theory by analytically predicting two surprising phenomena — worse-than-chance generalization on hard-to-learn functions and nonmonotonic error curves in the small data regime — which we subsequently observe in experiments. Though our theory is derived for inﬁnite-width architectures, we ﬁnd it agrees with networks as narrow as width 20, suggesting it is predictive of generalization in practical neural networks.
Contrastive Learning for Source Code with Structural and Functional Properties

Pre-trained transformer models have recently shown promises for understanding the source code. Most existing works expect to understand code from the textual features and limited structural knowledge of code. However, the program functionalities sometimes cannot be fully revealed by the code sequence, even with structure information. Programs can contain very different tokens and structures while sharing the same functionality
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efficiency in model convergence. In this paper, we propose a simple training strategy called"Pseudo-to-Real"for high-memory-footprint-required large models. Pseudo-to-Real is compatible with large models with architecture of sequential layers. We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.
Photodesorption of Acetonitrile CH3CN in UV-irradiated Regions of the Interstellar Medium: Experimental Evidence

Pure acetonitrile (CH3CN) and mixed CO:CH3CN and H2O:CH3CN ices have been irradiated at 15 K with vacuum ultraviolet (VUV) photons in the 7–13.6 eV range using synchrotron radiation. VUV photodesorption yields of CH3CN and of photoproducts have been derived as a function of the incident photon energy. The coadsorption of CH3CN with CO and H2O molecules, which are expected to be among the main constituents of interstellar ices, is found to have no significant influence on the VUV photodesorption spectra of CH3CN, CHCN•, HCN, CN•, and CH3•. Contrary to what has generally been evidenced for most of the condensed molecules, these findings point toward a desorption process for which the CH3CN molecule that absorbs the VUV photon is the one desorbing. It can be ejected in the gas phase as intact CH3CN or in the form of its photodissociation fragments. Astrophysical VUV photodesorption yields, applicable to different locations, are derived and can be incorporated into astrochemical modeling. They vary from 0.67(± 0.33) × 10−5 to 2.0(± 1.0) × 10−5 molecule photon−1 for CH3CN depending on the region considered, which is high compared to other organic molecules such as methanol. These results could explain the multiple detections of gas-phase CH3CN in different regions of the interstellar medium and are well correlated to astrophysical observations of the Horsehead nebula and of protoplanetary disks (such as TW Hya and HD 163296).
Learning Topic Models: Identifiability and Finite-Sample Analysis

Abstract Topic models provide a useful text-mining tool for learning, extracting, and discovering latent structures in large text corpora. Although a plethora of methods have been proposed for topic modeling, lacking in the literature is a formal theoretical investigation of the statistical identifiability and accuracy of latent topic estimation. In this article, we propose a maximum likelihood estimator (MLE) of latent topics based on a specific integrated likelihood that is naturally connected to the concept, in computational geometry, of volume minimization. Our theory introduces a new set of geometric conditions for topic model identifiability, conditions that are weaker than conventional separability conditions, which typically rely on the existence of pure topic documents or of anchor words. Weaker conditions allow a wider and thus potentially more fruitful investigation. We conduct finite-sample error analysis for the proposed estimator and discuss connections between our results and those of previous investigations. We conclude with empirical studies employing both simulated and real datasets. Supplementary materials for this article are available online.
Cognitive Coding Of Speech

We propose an approach for cognitive coding of speech by unsupervised extraction of contextual representations in two hierarchical levels of abstraction. Speech attributes such as phoneme identity that last one hundred milliseconds or less are captured in the lower level of abstraction, while speech attributes such as speaker identity and emotion that persist up to one second are captured in the higher level of abstraction. This decomposition is achieved by a two-stage neural net-work, with a lower and an upper stage operating at different time scales. Both stages are trained to predict the content of the signal in their respective latent spaces. A top-down path-way between stages further improves the predictive capability of the network. With an application in speech compression in mind, we investigate the effect of dimensionality reduction and low bitrate quantization on the extracted representations. The performance measured on the LibriSpeech and EmoV-DB datasets reaches, and for some speech attributes even exceeds, that of state-of-the-art approaches.
Reconstructing the Pleiades with Gaia EDR3

We search through an eight million cubic parsec volume surrounding the Pleiades star cluster and the Sun to identify both the current and past members of the Pleiades cluster within the Gaia EDR3 data set. We find nearly 1300 current cluster members and 289 former cluster candidates. Many of these candidates lie well in front of or behind the cluster from our point of view, so formerly they were considered cluster members, but their parallaxes put them more than 10 pc from the center of the cluster today. Over the past 100 Myr we estimate that the cluster has lost twenty percent of its mass including two massive white dwarf stars and the α 2 Canum Venaticorum type variable star, 41 Tau. All three white dwarfs associated with the cluster are massive (1.01–1.06 M ⊙) and have progenitors with main-sequence masses of about six solar masses. Although we did not associate any giant stars with the cluster, the cooling time of the oldest white dwarf of 60 Myr gives a firm lower limit on the age of the cluster.
Entanglement Swapping in Quantum Switches: Protocol Design and Stability Analysis

Quantum switches are critical components in quantum networks, distributing maximally entangled pairs among end nodes by entanglement swapping. In this work, we design protocols that schedule entanglement swapping operations in quantum switches. Entanglement requests randomly arrive at the switch, and the goal of an entanglement swapping protocol is to stabilize the quantum switch so that the number of unfinished entanglement requests is bounded with a high probability. We determine the capacity region for the rates of entanglement requests and develop entanglement swapping protocols to stabilize the switch. Among these protocols, the on-demand protocols are not only computationally efficient, but also achieve high fidelity and low latency demonstrated by results obtained using a quantum network discrete event simulator.
Dataset Condensation with Distribution Matching

Computational cost of training state-of-the-art deep models in many learning problems is rapidly increasing due to more sophisticated models and larger datasets. A recent promising direction for reducing training cost is dataset condensation that aims to replace the original large training set with a significantly smaller learned synthetic set while preserving the original information. While training deep models on the small set of condensed images can be extremely fast, their synthesis remains computationally expensive due to the complex bi-level optimization and secondorder derivative computation. In this work, we propose a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces. Our method significantly reduces the synthesis cost while achieving comparable or better performance. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures and obtain a significant performance boost1. We also show promising practical benefits of our method in continual learning and neural architecture search.
Photodynamic Antimicrobial Cellulosic Material Through Covalent Linkage of Protoporphyrin IX onto Lyocell Fibers

Here, we report the preparation of porphyrin-functionalized Lyocell fibers according to an azide-alkyne click concept. First, azido-modified Lyocell fibers and alkynylated protoporphyrin building blocks were prepared through alkoxysilane chemistry and Steglich esterification, respectively. Lyocell fibers were pre-activated by swelling in organic solvents in order to increase the accessibility of hydroxyl groups in the subsequent silanization process. The azide-equipped cellulosic matrix reacted with the propargyl groups of the protoporphyrin IX derivative in a copper-catalyzed azide-alkyne cycloaddition reaction (CuAAC), by which protoporphyrin IX was introduced onto the surface of the Lyocell fibers. The modified building blocks and the final functionalized cellulosic materials were comprehensively characterized by FTIR, NMR, UV/Vis spectroscopy and elemental analysis. Photo-bactericidal activity of modified fibers against 2 gram-positive bacteria strains, including Staphylococcus aureus and Bacillus subtilis, were investigated and compared to those of unmodified and azido-modified Lyocell fibers. The results confirmed the photo-antibacterial activity of the synthesized fibers against both bacteria strains.
Taming Sparsely Activated Transformer with Stochastic Experts

Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts. In this paper, we propose a new expert-based model, THOR (Transformer witH StOchastic ExpeRts). Unlike classic expert-based models, such as the Switch Transformer, experts in THOR are randomly activated for each input during training and inference. THOR models are trained using a consistency regularized loss, where experts learn not only from training data but also from other experts as teachers, such that all the experts make consistent predictions. We validate the effectiveness of THOR on machine translation tasks. Results show that THOR models are more parameter efficient in that they significantly outperform the Transformer and MoE models across various settings. For example, in multilingual translation, THOR outperforms the Switch Transformer by 2 BLEU scores, and obtains the same BLEU score as that of a state-of-the-art MoE model that is 18 times larger. Our code is publicly available at: https://github.com/microsoft/Stochastic-Mixture-of-Experts.
Speeding up Deep Model Training by Sharing Weights and Then Unsharing

We propose a simple and efficient approach for training the BERT model. Our approach exploits the special structure of BERT that contains a stack of repeated modules (i.e., transformer encoders). Our proposed approach first trains BERT with the weights shared across all the repeated modules till some point. This is for learning the commonly shared component of weights across all repeated layers. We then stop weight sharing and continue training until convergence. We present theoretic insights for training by sharing weights then unsharing with analysis for simplified models. Empirical experiments on the BERT model show that our method yields better performance of trained models, and significantly reduces the number of training iterations.
Critical quantum metrology with fully-connected models: from Heisenberg to Kibble–Zurek scaling

Phase transitions represent a compelling tool for classical and quantum sensing applications. It has been demonstrated that quantum sensors can in principle saturate the Heisenberg scaling, the ultimate precision bound allowed by quantum mechanics, in the limit of large probe number and long measurement time. Due to the critical slowing down, the protocol duration time is of utmost relevance in critical quantum metrology. However, how the long-time limit is reached remains in general an open question. So far, only two dichotomic approaches have been considered, based on either static or dynamical properties of critical quantum systems. Here, we provide a comprehensive analysis of the scaling of the quantum Fisher information for different families of protocols that create a continuous connection between static and dynamical approaches. In particular, we consider fully-connected models, a broad class of quantum critical systems of high experimental relevance. Our analysis unveils the existence of universal precision-scaling regimes. These regimes remain valid even for finite-time protocols and finite-size systems. We also frame these results in a general theoretical perspective, by deriving a precision bound for arbitrary time-dependent quadratic Hamiltonians.
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: https://github.com/cheerss/CrossFormer.
Observations on K-Image Expansion of Image-Mixing Augmentation

Image-mixing augmentations (e.g., Mixup and CutMix), which typically involve mix ing two images, have become the de-facto training techniques for image classification. Despite their huge success in image classification, the number of images to be mix ed has not been elucidated in the literature: only the naive K-image expansion has been shown to lead to performance degradation. This study derives a new K-image mixing augmentation based on the stick-breaking process under Dirichlet prior distribution. We demonstrate the superiority of our K-image expansion augmentation over conventional two-image mixing augmentation methods through extensive experiments and analyses: 1) more robust and generalized classifiers; 2) a more desirable loss landscape shape; 3) better adversarial robustness. Moreover, we show that our probabilistic model can measure the sample-wise uncertainty and boost the efficiency for network architecture search by achieving a 7-fold reduction in the search time. Code will be available at https://github.com/yjyoo3312/DCutMix-PyTorch.git.
IHOP: Improved Statistical Query Recovery against Searchable Symmetric Encryption through Quadratic Optimization

Effective query recovery attacks against Searchable Symmetric Encryption (SSE) schemes typically rely on auxiliary ground-truth information about the queries or dataset. Query recovery is also possible under the weaker statistical auxiliary information assumption, although statistical-based attacks achieve lower accuracy and are not considered a serious threat. In this work we present IHOP, a statistical-based query recovery attack that formulates query recovery as a quadratic optimization problem and reaches a solution by iterating over linear assignment problems. We perform an extensive evaluation with five real datasets, and show that IHOP outperforms all other statistical-based query recovery attacks under different parameter and leakage configurations, including the case where the client uses some access-pattern obfuscation defenses. In some cases, our attack achieves almost perfect query recovery accuracy. Finally, we use IHOP in a frequency-only leakage setting where the client's queries are correlated, and show that our attack can exploit query dependencies even when PANCAKE, a recent frequency-hiding defense by Grubbs et al., is applied. Our findings indicate that statistical query recovery attacks pose a severe threat to privacy-preserving SSE schemes.
Adjustment of force-gradient operator in symplectic methods

Many force–gradient explicit symplectic integration algorithms have been designed for the Hamiltonian H=T(p)+V(q) with kinetic energy T(p)=p2/2 in the existing references. When a force–gradient operator is appropriately adjusted as a new operator, it is still suitable for a class of Hamiltonian problems H=K(p,q)+V(q) with integrable part K(p,q)=∑i=1n∑j=1naijpipj+∑i=1nbipi, where aij=aij(q) and bi=bi(q) are functions of coordinates q. The newly adjusted operator is not a force–gradient operator but is similar to the momentum-version operator associated to the potential V. The newly extended (or adjusted) algorithms are no longer solvers of the original Hamiltonian, but are solvers of slightly modified Hamiltonians. They are explicit symplectic integrators with symmetry or time reversibility. Numerical tests show that the standard symplectic integrators without the new operator are generally poorer than the corresponding extended methods with the new operator in computational accuracies and efficiencies. The optimized methods have better accuracies than the corresponding non-optimized counterparts. Among the tested symplectic methods, the two extended optimized seven-stage fourth-order methods of Omelyan, Mryglod and Folk exhibit the best numerical performance. As a result, one of the two optimized algorithms is used to study the orbital dynamical features of a modified Hénon–Heiles system and a spring pendulum. These extended integrators allow for integrations in Hamiltonian problems, such as the spiral structure in self-consistent models of rotating galaxies and the spiral arms in galaxies.
Field-tuned ferroquadrupolar quantum phase transition in the insulator TmVO4

Significance The behavior of metals close to an electronic nematic quantum critical point (QCP) is not well understood. Here, we study the simpler case of an insulator that undergoes a ferroquadrupolar phase transition, which is a specific realization of Ising-nematic order. The ferroquadrupolar transition can be tuned continuously toward a QCP via application of a magnetic field. We find that close to the QCP, the phase boundary deviates from semiclassical predictions, consistent with expectations of quantum fluctuations. The observed power-law dependence is different from that of magnetic Ising systems due to coupling of order-parameter fluctuations to the crystal-lattice fluctuations (phonons). These results provide a well-understood starting point to approach more complex cases of nematic quantum criticality in metallic and disordered materials.