Miracle-dz/UniFAR-FTUs huggingface dataset:
Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the"geometry of learning dynamics"in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify"kinetic symmetry breaking"(KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether's theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge:"Noether's Learning Dynamics"(NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of"implicit adaptive optimization", establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.
WKB approximation with conformable operator

In this paper, the Wentzel–Kramers–Brillouin (WKB) method is extended to be applicable for conformable Hamiltonian systems, where the concept of conformable operator with fractional order [Formula: see text] is involved. The WKB approximation for the [Formula: see text]-wave function is derived for potentials which slowly vary in space. Some illustrative examples to demonstrate the method are presented. The quantities of the conformable form are found to be in exact agreement with the corresponding traditional quantities when [Formula: see text].
Learning To Generate Piano Music With Sustain Pedals

Recent years have witnessed a growing interest in research related to the detection of piano pedals from audio signals in the music information retrieval community. However, to our best knowledge, recent generative models for symbolic music have rarely taken piano pedals into account. In this work, we employ the transcription model proposed by Kong et al. to get pedal information from the audio recordings of piano performance in the AILabs1k7 dataset, and then modify the Compound Word Transformer proposed by Hsiao et al. to build a Transformer decoder that generates pedal-related tokens along with other musical tokens. While the work is done by using inferred sustain pedal information as training data, the result shows hope for further improvement and the importance of the involvement of sustain pedal in tasks of piano performance generations.
Classifying Approximation Algorithms: Understanding the APX Complexity Class

We are interested in the intersection of approximation algorithms and complexity theory, in particular focusing on the complexity class APX. Informally, APX $\subseteq$ NPO is the complexity class comprising optimization problems where the ratio $\frac{OPT(I)}{ALG(I)} \leq c$ for all instances I. We will do a deep dive into studying APX as a complexity class, in particular, investigating how researchers have defined PTAS and L reductions, as well as the notion of APX-completeness, thereby clarifying where APX lies on the polynomial hierarchy. We will discuss the relationship of this class with FPTAS, PTAS, APX, log-APX and poly-APX). We will sketch the proof that Max 3-SAT is APX-hard, and compare this complexity class in relation to $BPP$, $ZPP$ to elucidate whether randomization is powerful enough to achieve certain approximation guarantees and introduce techniques that complement the design of approximation algorithms such as through \textit{primal-dual} analysis, \textit{local search} and \textit{semi-definite programming}. Through the PCP theorem, we will explore the fundamental relationship between hardness of approximation and randomness, and will recast the way we look at the complexity class NP. We will finish by looking at the \textit{"real world"} applications of this material in Economics. Finally, we will touch upon recent breakthroughs in the Metric Travelling Salesman and asymmetric travelling salesman problem, as well original directions for future research, such as quantifying the amount of additional compute power that access to an APX oracle provides, elucidating fundamental combinatorial properties of log-APX problems and unique ways to attack the problem of whether the minimum set-cover problem is self-improvable.
Realistic galaxy image simulation via score-based generative models

We show that a Denoising Diffusion Probabilistic Model (DDPM), a class of score-based generative model, can be used to produce realistic mock images that mimic observations of galaxies. Our method is tested with Dark Energy Spectroscopic Instrument grz imaging of galaxies from the Photometry and Rotation curve OBservations from Extragalactic Surveys (PROBES) sample and galaxies selected from the Sloan Digital Sky Survey. Subjectively, the generated galaxies are highly realistic when compared with samples from the real dataset. We quantify the similarity by borrowing from the deep generative learning literature, using the ‘Fréchet Inception Distance’ to test for subjective and morphological similarity. We also introduce the ‘Synthetic Galaxy Distance’ metric to compare the emergent physical properties (such as total magnitude, colour and half light radius) of a ground truth parent and synthesised child dataset. We argue that the DDPM approach produces sharper and more realistic images than other generative methods such as Adversarial Networks (with the downside of more costly inference), and could be used to produce large samples of synthetic observations tailored to a specific imaging survey. We demonstrate two potential uses of the DDPM: (1) accurate in-painting of occluded data, such as satellite trails, and (2) domain transfer, where new input images can be processed to mimic the properties of the DDPM training set. Here we ‘DESI-fy’ cartoon images as a proof of concept for domain transfer. Finally, we suggest potential applications for score-based approaches that could motivate further research on this topic within the astronomical community.
iCallee: Recovering Call Graphs for Binaries

—Recovering programs’ call graphs is crucial for inter-procedural analysis tasks and applications based on them. The core challenge is recognizing targets of indirect calls (i.e., indirect callees). It becomes more challenging if target programs are in binary forms, due to information loss in binaries. Existing indirect callee recognition solutions for binaries all have high false positives and negatives, making call graphs inaccurate. In this paper, we propose a new solution I C ALLEE based on the Siamese Neural Network, inspired by the advances in question-answering applications. The key insight is that, neural networks can learn to answer whether a callee function is a potential target of an indirect callsite by comprehending their contexts, i.e., instructions nearby callsites and of callees. Following this insight, we ﬁrst preprocess target binaries to extract contexts of callsites and callees. Then, we build a customized Natural Language Processing (NLP) model applicable to assembly language. Further, we collect abundant pairs of callsites and callees, and embed their contexts with the NLP model, then train a Siamese network and a classiﬁer to answer the callsite-callee question. We have implemented a prototype of I C ALLEE and evaluated it on several groups of targets. Evaluation results showed that, our solution could match callsites to callees with an F1-Measure of 93.7%, recall of 93.8%, and precision of 93.5%, much better than state-of-the-art solutions. To show its usefulness, we apply I C ALLEE to two speciﬁc applications - binary code similarity detection and binary program hardening, and found that it could greatly improve state-of-the-art solutions.
COLDz: Probing Cosmic Star Formation With Radio Free–Free Emission

Radio free–free emission is considered to be one of the most reliable tracers of star formation in galaxies. However, as it constitutes the faintest part of the radio spectrum—being roughly an order of magnitude less luminous than radio synchrotron emission at the GHz frequencies typically targeted in radio surveys—the usage of free–free emission as a star formation rate tracer has mostly remained limited to the local universe. Here, we perform a multifrequency radio stacking analysis using deep Karl G. Jansky Very Large Array observations at 1.4, 3, 5, 10, and 34 GHz in the COSMOS and GOODS-North fields to probe free–free emission in typical galaxies at the peak of cosmic star formation. We find that z ∼ 0.5–3 star-forming galaxies exhibit radio emission at rest-frame frequencies of ∼65–90 GHz that is ∼1.5–2 times fainter than would be expected from a simple combination of free–free and synchrotron emission, as in the prototypical starburst galaxy M82. We interpret this as a deficit in high-frequency synchrotron emission, while the level of free–free emission is as expected from M82. We additionally provide the first constraints on the cosmic star formation history using free–free emission at 0.5 ≲ z ≲ 3, which are in good agreement with more established tracers at high redshift. In the future, deep multifrequency radio surveys will be crucial in order to accurately determine the shape of the radio spectrum of faint star-forming galaxies, and to further establish radio free–free emission as a tracer of high-redshift star formation.
Superdense beaming of axion dark matter in the vicinity of the light cylinder of pulsars

In this article we treat the non-adiabatic photon-to-axion resonant conversion of curvature radiation, synchrotron emission and inverse Compton scattering dominating the spectral density function of pulsars. First, we introduce emission models and benchmark observational data. We adopt a state-of-the-art density profile that relieves tension with the quantum electrodynamics vacuum polarization effect in highly magnetic stars, leading to efficient mixing. Then, we estimate the dark matter flux induced by photon-axion oscillation across the light cylinder of the neutron star. We find that pulsars might produce axion overdensities many orders of magnitude over the occupation number of dark matter in the Galactic halo within a broad parameter space. We point out possible new methods for axion detection derived from these results and other future lines of work.
Recurrent neural network models for working memory of continuous variables: activity manifolds, connectivity patterns, and dynamic codes

Many daily activities and psychophysical experiments involve keeping multiple items in working memory. When items take continuous values (e.g., orientation, contrast, length, loudness) they must be stored in a continuous structure of appropriate dimensions. We investigate how this structure is represented in neural circuits by training recurrent networks to report two previously shown stimulus orientations. We find the activity manifold for the two orientations resembles a Clifford torus. Although a Clifford and standard torus (the surface of a donut) are topologically equivalent, they have important functional differences. A Clifford torus treats the two orientations equally and keeps them in orthogonal subspaces, as demanded by the task, whereas a standard torus does not. We find and characterize the connectivity patterns that support the Clifford torus. Moreover, in addition to attractors that store information via persistent activity, our networks also use a dynamic code where units change their tuning to prevent new sensory input from overwriting the previously stored one. We argue that such dynamic codes are generally required whenever multiple inputs enter a memory system via shared connections. Finally, we apply our framework to a human psychophysics experiment in which subjects reported two remembered orientations. By varying the training conditions of the RNNs, we test and support the hypothesis that human behavior is a product of both neural noise and reliance on the more stable and behaviorally relevant memory of the ordinal relationship between the two orientations. This suggests that suitable inductive biases in RNNs are important for uncovering how the human brain implements working memory. Together, these results offer an understanding of the neural computations underlying a class of visual decoding tasks, bridging the scales from human behavior to synaptic connectivity.
New Spectroscopic Confirmations of Lyα Emitters at Z ∼ 7 from the LAGER Survey

We report spectroscopic confirmations of 15 Lyα galaxies at z ∼ 7, implying a spectroscopic confirmation rate of ∼80% on candidates selected from the Lyα Galaxies in the Epoch of Reionization (LAGER), which is the largest (24 deg2) survey aimed at finding Lyα emitters (LAEs) at z ∼ 7 and uses deep narrowband imaging from the Dark Energy Camera at CTIO. LAEs at high redshifts are sensitive probes of cosmic reionization, and narrowband imaging is a robust and effective method for selecting a large number of LAEs. In this work, we present results from the spectroscopic follow-up of LAE candidates in two LAGER fields, COSMOS and WIDE-12, using observations from Keck/LRIS. We report the successful detection of Lyα emission in 15 candidates. Three of these in COSMOS have matching confirmations from a previous spectroscopic follow-up and are part of the overdense region, LAGER-z7OD1. Two other candidates that were not detected with LRIS have prior spectroscopic confirmations from Magellan. Including these, we obtain a spectroscopic confirmation success rate of ∼80% for LAGER LAE candidates. Thorough checks were performed to reject the possibility of these detections being foreground emission resulting with a probability of, at most, one contaminant. We do not detect any other UV nebular lines in our LRIS spectra, apart from Lyα. We estimate a 2σ upper limit for the ratio of N v/Lyα, f NV/f Lyα ≲ 0.27. Including confirmations from this work, a total of 33 LAE sources from LAGER are now spectroscopically confirmed. LAGER has more than doubled the sample of spectroscopically confirmed LAE sources at z ∼ 7.
Minimax Optimization: The Case of Convex-Submodular

Minimax optimization has been central in addressing various applications in machine learning, game theory, and control theory. Prior literature has thus far mainly focused on studying such problems in the continuous domain, e.g., convex-concave minimax optimization is now understood to a significant extent. Nevertheless, minimax problems extend far beyond the continuous domain to mixed continuous-discrete domains or even fully discrete domains. In this paper, we study mixed continuous-discrete minimax problems where the minimization is over a continuous variable belonging to Euclidean space and the maximization is over subsets of a given ground set. We introduce the class of convex-submodular minimax problems, where the objective is convex with respect to the continuous variable and submodular with respect to the discrete variable. Even though such problems appear frequently in machine learning applications, little is known about how to address them from algorithmic and theoretical perspectives. For such problems, we first show that obtaining saddle points are hard up to any approximation, and thus introduce new notions of (near-) optimality. We then provide several algorithmic procedures for solving convex and monotone-submodular minimax problems and characterize their convergence rates, computational complexity, and quality of the final solution according to our notions of optimally. Our proposed algorithms are iterative and combine tools from both discrete and continuous optimization. Finally, we provide numerical experiments to showcase the effectiveness of our purposed methods.
Human Attention in Fine-grained Classification

The way humans attend to, process and classify a given image has the potential to vastly benefit the performance of deep learning models. Exploiting where humans are focusing can rectify models when they are deviating from essential features for correct decisions. To validate that human attention contains valuable information for decision-making processes such as fine-grained classification, we compare human attention and model explanations in discovering important features. Towards this goal, we collect human gaze data for the fine-grained classification dataset CUB and build a dataset named CUB-GHA (Gaze-based Human Attention). Furthermore, we propose the Gaze Augmentation Training (GAT) and Knowledge Fusion Network (KFN) to integrate human gaze knowledge into classification models. We implement our proposals in CUB-GHA and the recently released medical dataset CXR-Eye of chest X-ray images, which includes gaze data collected from a radiologist. Our result reveals that integrating human attention knowledge benefits classification effectively, e.g. improving the baseline by 4.38% on CXR. Hence, our work provides not only valuable insights into understanding human attention in fine-grained classification, but also contributes to future research in integrating human gaze with computer vision tasks. CUB-GHA and code are available at https://github.com/yaorong0921/CUB-GHA.
Comparison Study of Paraquat Herbicide Adsorption by Xerogels: Application of Experimental Design

This study dealt with the Paraquat dichloride herbicide (PQT) adsorption on xerogels. The xerogels were synthesized with tetra methoxy orthosilicate (TMOS) and tetra ethoxy orthosilicate (TEOS) as a precursor. Whereas xerogels have free silanol groups on their surface and PQT has a positive charge as an ion, xerogels have shown a high affinity for the PQT adsorption in aqueous solution. Scanning Electron Microscopy, Fourier Transform‐IR, and BET analysis were characterized xerogels. The effect of four process factors such as PQT dosage, pH, a dose of adsorbent, and time was evaluated by the Box‐Behnken design. Between four parameters, pH was more effective. The second order equation (quadratic model) was also used for the analysis of the experimental data. Additionally, the optimum points for parameters were determined by desirability function in the Experimental Design. Removal Percentages of PQT at the optimum process parameters were found to be 98.05 and 92.83%, for TMOS and TEOS xerogels, respectively. In addition, the isotherm study showed that the Freundlich isotherm indicated better harmonic between experimental and theoretical results. Finally, in the kinetic study, Pseudo‐second order model had a better correlation than Pseudo‐first order model. © 2018 American Institute of Chemical Engineers Environ Prog, 38: S128–S138, 2019
Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model

The enormous amount of data being generated on the web and social media has increased the demand for detecting online hate speech. Detecting hate speech will reduce their negative impact and influence on others. A lot of effort in the Natural Language Processing (NLP) domain aimed to detect hate speech in general or detect specific hate speech such as religion, race, gender, or sexual orientation. Hate communities tend to use abbreviations, intentional spelling mistakes, and coded words in their communication to evade detection, adding more challenges to hate speech detection tasks. Thus, word representation will play an increasingly pivotal role in detecting hate speech. This paper investigates the feasibility of leveraging domain-specific word embedding in Bidirectional LSTM based deep model to automatically detect/classify hate speech. Furthermore, we investigate the use of the transfer learning language model (BERT) on hate speech problem as a binary classification task. The experiments showed that domainspecific word embedding with the Bidirectional LSTM based deep model achieved a 93% f1-score while BERT achieved up to 96% f1-score on a combined balanced dataset from available hate speech datasets.
In-situ dating of metamorphism in Adirondack anorthosite

Abstract The 3000 km2 Marcy anorthosite massif dominates the Adirondack Highlands (Grenville Province, New York). The Marcy massif was metamorphosed to granulite facies conditions, in places preserving igneous textures with metamorphic coronas and is most deformed near its margins. Historically, the relationship between anorthosite emplacement and metamorphism has been controversial, and many workers have argued that anorthosite emplacement coincided with metamorphism. Valley and O’Neil (1982) proposed that high-pressure metamorphic mineral assemblages in the anorthosite could not reflect the same event that formed wollastonite skarns adjacent to anorthosite, which have low δ18O and formed in the presence of meteoric water during shallow emplacement. This study presents new in-situ geochronology that constrains the timing of metamorphic mineral growth in Adirondack anorthosite to 1050–1035 Ma. The Zr source for metamorphic zircon growth was the breakdown of hemoilmenite and is texturally linked to high-pressure mineral assemblages. These data are consistent with previously determined ca. 1155 Ma magmatic ages and later granulite facies metamorphism during the 1090–1020 Ma Ottawan phase of the Grenvillian orogeny.
Trajectory Prediction with Graph-based Dual-scale Context Fusion

Motion prediction for traffic participants is essential for a safe and robust automated driving system, especially in cluttered urban environments. However, it is highly challenging due to the complex road topology as well as the uncertain intentions of the other agents. In this paper, we present a graph-based trajectory prediction network named the Dual Scale Predictor (DSP), which encodes both the static and dynamical driving context in a hierarchical manner. Different from methods based on a rasterized map or sparse lane graph, we consider the driving context as a graph with two layers, focusing on both geometrical and topological features. Graph neural networks (GNNs) are applied to extract features with different levels of granularity, and features are subsequently aggregated with attention-based inter-layer networks, realizing better local-global feature fusion. Following the recent goal-driven trajectory prediction pipeline, goal candidates with high likelihood for the target agent are extracted, and predicted trajectories are generated conditioned on these goals. Thanks to the proposed dual-scale context fusion network, our DSP is able to generate accurate and human-like multi-modal trajectories. We evaluate the proposed method on the large-scale Argoverse motion forecasting benchmark, and it achieves promising results, outperforming the recent state-of-the-art methods. We release the code on our project website. 11https://github.com/HKUST-Aerial-Robotics/DSP