Miracle-dz/UniFAR-FTUs huggingface dataset:
Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Renormalization Group

The success of deep learning in many real-world tasks has triggered an effort to theoretically understand the power and limitations of deep learning in training and generalization of complex tasks, so far with limited progress. In this work, we study the statistical mechanics of learning in Deep Linear Neural Networks (DLNNs) in which the input-output function of an individual unit is linear. Despite the linearity of the units, learning in DLNNs is highly nonlinear, hence studying its properties reveals some of the essential features of nonlinear Deep Neural Networks (DNNs). We solve exactly the network properties following supervised learning using an equilibrium Gibbs distribution in the weight space. To do this, we introduce the Back-Propagating Renormalization Group (BPRG) which allows for the incremental integration of the network weights layer by layer from the network output layer and progressing backward. This procedure allows us to evaluate important network properties such as its generalization error, the role of network width and depth, the impact of the size of the training set, and the effects of weight regularization and learning stochasticity. Furthermore, by performing partial integration of layers, BPRG allows us to compute the emergent properties of the neural representations across the different hidden layers. We have proposed a heuristic extension of the BPRG to nonlinear DNNs with rectified linear units (ReLU). Surprisingly, our numerical simulations reveal that despite the nonlinearity, the predictions of our theory are largely shared by ReLU networks with modest depth, in a wide regime of parameters. Our work is the first exact statistical mechanical study of learning in a family of Deep Neural Networks, and the first development of the Renormalization Group approach to the weight space of these systems.
Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization

Dimension reduction (DR) techniques such as t-SNE, UMAP, and TriMAP have demonstrated impressive visualization performance on many real world datasets. One tension that has always faced these methods is the trade-off between preservation of global structure and preservation of local structure: these methods can either handle one or the other, but not both. In this work, our main goal is to understand what aspects of DR methods are important for preserving both local and global structure: it is difficult to design a better method without a true understanding of the choices we make in our algorithms and their empirical impact on the lower-dimensional embeddings they produce. Towards the goal of local structure preservation, we provide several useful design principles for DR loss functions based on our new understanding of the mechanisms behind successful DR methods. Towards the goal of global structure preservation, our analysis illuminates that the choice of which components to preserve is important. We leverage these insights to design a new algorithm for DR, called Pairwise Controlled Manifold Approximation Projection (PaCMAP), which preserves both local and global structure. Our work provides several unexpected insights into what design choices both to make and avoid when constructing DR algorithms.
Learning Independent Instance Maps for Crowd Localization

Accurately locating each head's position in the crowd scenes is a crucial task in the field of crowd analysis. However, traditional density-based methods only predict coarse prediction, and segmentation/detection-based methods cannot handle extremely dense scenes and large-range scale-variations crowds. To this end, we propose an end-to-end and straightforward framework for crowd localization, named Independent Instance Map segmentation (IIM). Different from density maps and boxes regression, each instance in IIM is non-overlapped. By segmenting crowds into independent connected components, the positions and the crowd counts (the centers and the number of components, respectively) are obtained. Furthermore, to improve the segmentation quality for different density regions, we present a differentiable Binarization Module (BM) to output structured instance maps. BM brings two advantages into localization models: 1) adaptively learn a threshold map for different images to detect each instance more accurately; 2) directly train the model using loss on binary predictions and labels. Extensive experiments verify the proposed method is effective and outperforms the-state-of-the-art methods on the five popular crowd datasets. Significantly, IIM improves F1-measure by 10.4\% on the NWPU-Crowd Localization task. The source code and pre-trained models will be released at \url{https://github.com/taohan10200/IIM}.
The Relativistic Jet Orientation and Host Galaxy of the Peculiar Blazar PKS 1413+135

PKS 1413+135 is one of the most peculiar blazars known. Its strange properties led to the hypothesis almost four decades ago that it is gravitationally lensed by a mass concentration associated with an intervening galaxy. It exhibits symmetric achromatic variability, a rare form of variability that has been attributed to gravitational milli-lensing. It has been classified as a BL Lac object, and is one of the rare objects in this class with a visible counterjet. BL Lac objects have jet axes aligned close to the line of sight. It has also been classified as a compact symmetric object—objects that have jet axes not aligned close to the line of sight. Intensive efforts to understand this blazar have hitherto failed to resolve even the questions of the orientation of the relativistic jet and the host galaxy. Answering these two questions is important because they challenge our understanding of jets in active galactic nuclei and the classification schemes we use to describe them. We show that the jet axis is aligned close to the line of sight and PKS 1413+135 is almost certainly not located in the apparent host galaxy, but is a background object in the redshift range 0.247 < z < 0.5. The intervening spiral galaxy at z = 0.247 provides a natural host for the putative lens responsible for symmetric achromatic variability and is shown to be a Seyfert 2 galaxy. We also show that, as for the radio emission, a “multizone” model is needed to account for the high-energy emission.
Generalization bounds for deep learning

Generalization in deep learning has been the topic of much recent theoretical and empirical research. Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning. Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous. We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data. We review a wide range of existing approaches, from classical VC dimension to recent PAC-Bayesian bounds, commenting on how well they perform against the desiderata. We next use a function-based picture to derive a marginal-likelihood PAC-Bayesian bound. This bound is, by one definition, optimal up to a multiplicative constant in the asymptotic limit of large training sets, as long as the learning curve follows a power law, which is typically found in practice for deep learning problems. Extensive empirical analysis demonstrates that our marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results for 6 and 7 are promising, but not yet fully conclusive, while only desideratum 4 is currently beyond the scope of our bound. Finally, we comment on why this function-based bound performs significantly better than current parameter-based PAC-Bayes bounds.
Weak well-posedness of SDEs with drifts in critical spaces.

We prove the unique weak solvability of time-inhomogeneous stochastic differential equations with additive noises and drifts in critical Lebsgue space $L^q([0,T]; L^{p}(\mathbb{R}^d))$ with $d/p+2/q=1$. The weak uniqueness is obtained by solving corresponding Kolmogorov's backward equations in some second order Sobolev spaces, which is analytically interesting in itself.
On Cylindrical and non-Cylindrical $(p,q)$-Hardy potentials.

Let $k,N \in \mathbb{N}$ with $1\le k\le N$ and let $\Omega=\Omega_1 \times \Omega_2$ be an open set in $\mathbb{R}^k \times \mathbb{R}^{N-k}$. For $p\in (1,\infty)$ and $q \in (0,\infty),$ we consider the following Hardy-Sobolev type inequality: \begin{align} \int_{\Omega} g_1(y)g_2(z) |u(y,z)|^q \, dy \, dz \leq C \left( \int_{\Omega} | \nabla u(y,z) |^p \, dy \, dz \right)^{\frac{q}{p}}, \quad \forall \, u \in \mathcal{C}^1_c(\Omega), \end{align} for some $C>0$. Depending on the values of $N,k,p,q,$ we have identified various pairs of Lorentz spaces, Lorentz-Zygmund spaces and weighted Lebesgue spaces for $(g_1, g_2)$ so that the above inequality holds. Furthermore, we give a sufficient condition on $g_1,g_2$ so that the best constant in the above inequality is attained in the Beppo-Levi space $\mathcal{D}^{1,p}_0(\omega)$-the completion of $\mathcal{C}^1_c(\Omega)$ with respect to $\|\nabla u\|_{L^p(\Omega)}$.
Effects of co-fermentation with Candida stellata and Saccharomyces cerevisiae on the aroma and composition of Chardonnay wine

Effects of several inoculation protocols using Saccharomyces cerevisiae strain AWRI 838 (an isolate of Lalvin EC1118) and Candida stellata strain AWRI 1159 (CBS 2649) upon the aroma properties and chemical composition of Chardonnay wine were determined. An increase in the concentration of glycerol and acetic acid was observed when fermentation was performed with C. stellata AWRI 1159, which did not progress to dryness. Sensory descriptive analysis showed a substantial difference in aroma between the wines produced by monocultures of the two yeast species. The Candida stellata AWRI 1159 produced significantly more intense 'honey', 'apricot', and 'sauerkraut' aromas, and diminished the 'lime', 'banana' 'tropical fruit' and 'floral' aromas ascribed to S. cerevisiae AWRI 838. When C. stellata AWRI 1159 was co-inoculated at ten times the initial concentration of strain S. cerevisiae AWRI 838, the non-Saccharomyces yeast had only a minor impact upon wine aroma and composition despite maintaining a significant viable population of 5-10 X 106 colony forming units per mL throughout fermentation. Wine of a different aroma profile to either of the reference monoculture wines was produced by sequential fermentation, whereby C. stellata AWRI 1159 conducted the first half of fermentation, before inoculation with S. cerevisiae AWRI 838, and the subsequent completion of fermentation. This wine had 'floral', 'banana', 'lime', 'tropical fruit' and 'sauerkraut' aroma scores intermediate to the two reference monoculture wines, 'apricot' and 'honey' aroma ratings similar to the S. cerevisiae AWRI 838 wine, and 'ethyl acetate' aroma that exceeded that of both reference wines. These results suggest the potential of a reliable mixed culture fermentation strategy for exploiting unconventional, fermentation-impaired yeasts for producing greater flavour diversity in wine.
SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction With Self-Projection Optimization

The task of point cloud upsampling aims to acquire dense and uniform point sets from sparse and irregular point sets. Although significant progress has been made with deep learning models, state-of-the-art methods require ground-truth dense point sets as the supervision, which makes them limited to be trained under synthetic paired training data and not suitable to be under real-scanned sparse data. However, it is expensive and tedious to obtain large numbers of paired sparse-dense point sets as supervision from real-scanned sparse data. To address this problem, we propose a self-supervised point cloud upsampling network, named SPU-Net, to capture the inherent upsampling patterns of points lying on the underlying object surface. Specifically, we propose a coarse-to-fine reconstruction framework, which contains two main components: point feature extraction and point feature expansion, respectively. In the point feature extraction, we integrate the self-attention module with the graph convolution network (GCN) to capture context information inside and among local regions simultaneously. In the point feature expansion, we introduce a hierarchically learnable folding strategy to generate upsampled point sets with learnable 2D grids. Moreover, to further optimize the noisy points in the generated point sets, we propose a novel self-projection optimization associated with uniform and reconstruction terms as a joint loss to facilitate the self-supervised point cloud upsampling. We conduct various experiments on both synthetic and real-scanned datasets, and the results demonstrate that we achieve comparable performances to state-of-the-art supervised methods.
Texture Transform Attention for Realistic Image Inpainting

Over the last few years, the performance of inpainting to fill missing regions has shown significant improvements by using deep neural networks. Most of inpainting work create a visually plausible structure and texture, however, due to them often generating a blurry result, final outcomes appear unrealistic and make feel heterogeneity. In order to solve this problem, the existing methods have used a patch based solution with deep neural network, however, these methods also cannot transfer the texture properly. Motivated by these observation, we propose a patch based method. Texture Transform Attention network(TTA-Net) that better produces the missing region inpainting with fine details. The task is a single refinement network and takes the form of U-Net architecture that transfers fine texture features of encoder to coarse semantic features of decoder through skip-connection. Texture Transform Attention is used to create a new reassembled texture map using fine textures and coarse semantics that can efficiently transfer texture information as a result. To stabilize training process, we use a VGG feature layer of ground truth and patch discriminator. We evaluate our model end-to-end with the publicly available datasets CelebA-HQ and Places2 and demonstrate that images of higher quality can be obtained to the existing state-of-the-art approaches.
Interactions of a shock with a molecular cloud at various stages of its evolution due to thermal instability and gravity

Using the adaptive mesh refinement code mg, we perform hydrodynamic simulations of the interaction of a shock with a molecular cloud evolving due to thermal instability (TI) and gravity. To explore the relative importance of these processes, three case studies are presented. The first follows the formation of a molecular cloud out of an initially quiescent atomic medium due to the effects of TI and gravity. The second case introduces a shock whilst the cloud is still in the warm atomic phase, and the third scenario introduces a shock once the molecular cloud has formed. The shocks accelerate the global collapse of the clouds with both experiencing local gravitational collapse prior to this. When the cloud is still atomic, the evolution is shock dominated and structures form due to dynamical instabilities within a radiatively cooled shell. While the transmitted shock can potentially trigger the TI, this is prevented as material is shocked multiple times on the order of a cloud-crushing time-scale. When the cloud is molecular, the post-shock flow is directed via the pre-existing structure through low-density regions in the inter-clump medium. The clumps are accelerated and deformed as the flow induces clump–clump collisions and mergers that collapse under gravity. For a limited period, both shocked cases show a mixture of Kolmogorov and Burgers turbulence-like velocity and logarithmic density power spectra, and strongly varying density spectra. The clouds presented in this work provide realistic conditions that will be used in future feedback studies.
Redshift-space distortions in Lagrangian perturbation theory

We present the one-loop 2-point function of biased tracers in redshift space computed with Lagrangian perturbation theory, including a full resummation of both long-wavelength (infrared) displacements and associated velocities. The resulting model accurately predicts the power spectrum and correlation function of halos and mock galaxies from two different sets of N-body simulations at the percent level for quasi-linear scales, including the damping of the baryon acoustic oscillation signal due to the bulk motions of galaxies. We compare this full resummation with other, approximate, techniques including the moment expansion and Gaussian streaming model. We discuss infrared resummation in detail and compare our Lagrangian formulation with the Eulerian theory augmented by an infrared resummation based on splitting the input power spectrum into “wiggle” and “no-wiggle” components. We show that our model is able to recover unbiased cosmological parameters in mock data encompassing a volume much larger than what will be available to future galaxy surveys. We demonstrate how to efficiently compute the resulting expressions numerically, making available a fast Python code capable of rapidly computing these statistics in both configuration and Fourier space.
Parameter Efficient Multimodal Transformers for Video Representation Learning

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the weights of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters up to 80$\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips from Kinetics-700 and transfer it to audio-visual classification tasks.
Reinforcement potential of bleached sawdust kraft pulp in different mechanical pulp furnishes

Bleached unrefined and refined sawdust kraft pulps were added to bleached Norway spruce thermomechanical (TMP) and pressurised groundwood (PWG) pulps in different proportions. Handsheets were prepared and tested for physical properties. In addition, economic calculations were done to estimate the production costs of different bleached pulps in Finland. It was found that the addition of unrefined and refined sawdust kraft pulp improved drainability of the mechanical pulps. Tear strength of PGW furnishes was increased when either unrefined or refined sawdust pulp was added. Tear strength of TMP furnishes was not influenced when different sawdust kraft pulps were added. Up to 30 % of unrefined sawdust kraft pulp could be added and no significant negative effect was observed in TMP and PGW furnishes. On the other hand, when refined sawdust kraft pulp was added into the mechanical pulp furnishes, a clear improvement in the tensile strength was observed. According to the economic calculation the production cost of bleached sawdust kraft pulp is almost as low as the production cost of bleached mechanical pulp. We suggest that economically viable sawdust kraft pulp can be used as a substituent for expensive long fibre reinforcement kraft pulp in the production of mechanical pulp based papers.
Computing Fair Utilitarian Allocations of Indivisible Goods

We study the computational complexity of computing allocations that are both fair and maximize the utilitarian social welfare, i.e., the sum of utilities reported by the agents. We focus on two tractable fairness concepts: envy-freeness up to one item (EF1) and proportionality up to one item (PROP1). In particular, we consider the following two computational problems: (1) Among the utilitarian-maximal allocations, decide whether there exists one that is also fair according to either EF1 or PROP1; (2) among the fair allocations, compute one that maximizes the utilitarian welfare. We show that both problem (1) and problem (2) are strongly NP-hard when the number of agents is variable, and remain NP-hard for a fixed number of agents greater than two. Focusing on the special case of two agents, we find that problem (1) is polynomial-time solvable, whereas problem (2) remains NP-hard. Finally, for the case of fixed number of agents, we design pseudopolynomial-time algorithms for both problems.
Data Dependent Randomized Smoothing

Randomized smoothing is a recent technique that achieves state-of-art performance in training certifiably robust deep neural networks. While the smoothing family of distributions is often connected to the choice of the norm used for certification, the parameters of the distributions are always set as global hyper parameters independent of the input data on which a network is certified. In this work, we revisit Gaussian randomized smoothing where we show that the variance of the Gaussian distribution can be optimized at each input so as to maximize the certification radius for the construction of the smoothed classifier. This new approach is generic, parameter-free, and easy to implement. In fact, we show that our data dependent framework can be seamlessly incorporated into 3 randomized smoothing approaches, leading to consistent improved certified accuracy. When this framework is used in the training routine of these approaches followed by a data dependent certification, we get 9% and 6% improvement over the certified accuracy of the strongest baseline for a radius of 0.5 on CIFAR10 and ImageNet, respectively.
In situ visualization of tip undercooling and lamellar microstructure evolution of sea ice with manipulated orientation

Sea ice growth with lamellar microstructure containing brine channels has been extensively investigated. However, the quantitative growth information of sea ice remains lack due to the uncontrolled crystalline orientation in previous investigations. For the first time, we in-situ observed the unidirectional growth of lamellar sea ice with well-manipulated ice crystal orientation and visualized tip undercooling of sea ice. A semi-empirical model was proposed to quantitatively address the variation of tip undercooling with growth velocity and salinity and compared with a very recent analytical model. With the real-time observation, interesting phenomena of doublon tip in cellular ice growth and growth direction shift of ice dendritic tip were discovered for the first time, which are attributed to the complex solutal diffusion and anisotropic interface kinetics in sea ice growth. The quantitative experiment provides a clear micro scenario of sea ice growth, and will promote relevant investigations of sea ice in terms of the theoretical approach to describing the diffusion field around faceted ice dendritic tip.