Miracle-dz/UniFAR-FTUs huggingface dataset:
Classify and Generate Reciprocally: Simultaneous Positive-Unlabelled Learning and Conditional Generation with Extra Data

The scarcity of class-labeled data is a ubiquitous bottleneck in a wide range of machine learning problems. While abundant unlabeled data normally exist and provide a potential solution, it is extremely challenging to exploit them. In this paper, we address this problem by leveraging Positive-Unlabeled~(PU) classification and conditional generation with extra unlabeled data \emph{simultaneously}, both of which aim to make full use of agnostic unlabeled data to improve classification and generation performances. In particular, we present a novel training framework to jointly target both PU classification and conditional generation when exposing to extra data, especially out-of-distribution unlabeled data, by exploring the interplay between them: 1) enhancing the performance of PU classifiers with the assistance of a novel Conditional Generative Adversarial Network~(CGAN) that is robust to noisy labels, 2) leveraging extra data with predicted labels from a PU classifier to help the generation. Our key contribution is a Classifier-Noise-Invariant Conditional GAN~(CNI-CGAN) that can learn the clean data distribution from noisy labels predicted by a PU classifier. Theoretically, we proved the optimal condition of CNI-CGAN and experimentally, we conducted extensive evaluations on diverse datasets, verifying the simultaneous improvements on both classification and generation.
One Model Is Not Enough: Heterogeneity in Cryptocurrencies’ Multi-fractal Profiles

This paper studies of the multifractal dynamics in 84 cryptocurrencies. It fills an important gap in the literature, by studying this market using two alternative multi-scaling methodologies. We find compelling evidence that cryptocurrencies have different degree of long range dependence, and --more importantly -- follow different stochastic processes. Some of them follow models closer to monofractal fractional Gaussian noises, while others exhibit complex multifractal dynamics. Regarding the source of multifractality, our results are mixed. Time series shuffling produces a reduction in the level of multifractality, but not enough to offset it. We find an association of kurtosis with multifractality.
The loss surface and expressivity of deep convolutional neural networks

We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a "wide" layer which has more neurons than the number of training samples. This condition holds e.g. for the VGG network. Furthermore, we provide for such wide CNNs necessary and sufficient conditions for global minima with zero training error. For the case where the wide layer is followed by a fully connected layer, we show that almost every critical point of the empirical loss is a global minimum with zero training error. Our analysis suggests that both depth and width are very important in deep learning. While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufficiently wide network has a well-behaved loss surface with potentially no bad local minima.
Calibration of the von Wolffersdorff model using Genetic Algorithms

This article proposes an optimization framework, based on Genetic Algorithms (GA), to calibrate the constitutive law of von Wolffersdorff. This constitutive law is known as Sand Hypoplasticity (SH), and allows for robust and accurate modeling of the soil behavior but requires a complex calibration involving eight parameters. The proposed optimization can automatically fit these parameters from the results of an oedometric and a triaxial drained compression test, by combining the GA with a numerical solver that integrates the SH in the test conditions. By repeating the same calibration several times, the stochastic nature of the optimizer enables the uncertainty quantification of the calibration parameters and allows studying their relative importance on the model prediction. After validating the numerical solver on the ExCaliber-Laboratory software from the SoilModels' website, the GA calibration is tested on a synthetic dataset to analyze the convergence and the statistics of the results. In particular, a correlation analysis reveals that two couples of the eight model parameters are strongly correlated. Finally, the calibration procedure is tested on the results from von Wolffersdorff, 1996, and Herle & Gudehus, 1999, on the Hochstetten sand. The model parameters identified by the Genetic Algorithm optimization improves the matching with the experimental data and hence lead to a better calibration.
Unbiased Auxiliary Classifier GANs with MINE

Auxiliary Classifier GANs (AC-GANs) are widely used conditional generative models and are capable of generating high-quality images. Previous work has pointed out that AC-GAN learns a biased distribution. To remedy this, Twin Auxiliary Classifier GAN (TAC-GAN) introduces a twin classifier to the min-max game. However, it has been reported that using a twin auxiliary classifier may cause instability in training. To this end, we propose an Unbiased Auxiliary GANs (UAC-GAN) that utilizes the Mutual Information Neural Estimator (MINE) to estimate the mutual information between the generated data distribution and labels. To further improve the performance, we also propose a novel projection-based statistics network architecture for MINE. Experimental results on three datasets, including Mixture of Gaussian (MoG), MNIST and CIFAR10 datasets, show that our UAC-GAN performs better than AC-GAN and TAC-GAN. Code can be found on the project website.
Domain Adaptation and Image Classification via Deep Conditional Adaptation Network

Unsupervised domain adaptation aims to generalize the supervised model trained on a source domain to an unlabeled target domain. Marginal distribution alignment of feature spaces is widely used to reduce the domain discrepancy between the source and target domains. However, it assumes that the source and target domains share the same label distribution, which limits their application scope. In this paper, we consider a more general application scenario where the label distributions of the source and target domains are not the same. In this scenario, marginal distribution alignment-based methods will be vulnerable to negative transfer. To address this issue, we propose a novel unsupervised domain adaptation method, Deep Conditional Adaptation Network (DCAN), based on conditional distribution alignment of feature spaces. To be specific, we reduce the domain discrepancy by minimizing the Conditional Maximum Mean Discrepancy between the conditional distributions of deep features on the source and target domains, and extract the discriminant information from target domain by maximizing the mutual information between samples and the prediction labels. In addition, DCAN can be used to address a special scenario, Partial unsupervised domain adaptation, where the target domain category is a subset of the source domain category. Experiments on both unsupervised domain adaptation and Partial unsupervised domain adaptation show that DCAN achieves superior classification performance over state-of-the-art methods. In particular, DCAN achieves great improvement in the tasks with large difference in label distributions (6.1\% on SVHN to MNIST, 5.4\% in UDA tasks on Office-Home and 4.5\% in Partial UDA tasks on Office-Home).
Joint Adaptive Feature Smoothing and Topology Extraction via Generalized PageRank GNNs

In many important applications, the acquired graph-structured data includes both node features and graph topology information. Graph neural networks (GNNs) are able to accurately process both the feature signals and graph topology individually. Nevertheless, they face the problem of trading-off the benefits of a shallow network architecture and deep multi-step information propagation when attempting to optimize their learning performance using both data components. Most existing GNN implementations based on node feature propagation are shallow due to the fact that a large number of propagation steps leads to feature over-smoothing and hence diminishes their discriminative power. In contrast, when processing topological information, it is common to use label propagation and PageRank methods that require a large number of message passing steps. We address these two contradictory requirements by combining GNNs with an adaptive generalized PageRank (GPR) scheme in a model termed GPR-GNN. GPR-GNN is the first known architecture that not only provably mitigates feature over-smoothing but also adaptively learns the weights of the GPR model to optimize topological information extraction. Our theoretical analysis of the GPR-GNN method is facilitated by novel synthetic benchmark datasets generated by the contextual stochastic block model. We also compare the performance of our NN architecture with that of several state-of-the-art GNNs on the problem of node-classification, using nine well-known benchmark datasets. The results demonstrate that GPR-GNN offers significant performance improvement compared to existing techniques on both synthetic and benchmark data.
The Limit of the Batch Size

Large-batch training is an efficient approach for current distributed deep learning systems. It has enabled researchers to reduce the ImageNet/ResNet-50 training from 29 hours to around 1 minute. In this paper, we focus on studying the limit of the batch size. We think it may provide a guidance to AI supercomputer and algorithm designers. We provide detailed numerical optimization instructions for step-by-step comparison. Moreover, it is important to understand the generalization and optimization performance of huge batch training. Hoffer et al. introduced "ultra-slow diffusion" theory to large-batch training. However, our experiments show contradictory results with the conclusion of Hoffer et al. We provide comprehensive experimental results and detailed analysis to study the limitations of batch size scaling and "ultra-slow diffusion" theory. For the first time we scale the batch size on ImageNet to at least a magnitude larger than all previous work, and provide detailed studies on the performance of many state-of-the-art optimization schemes under this setting. We propose an optimization recipe that is able to improve the top-1 test accuracy by 18% compared to the baseline.
Quasar Sightline and Galaxy Evolution (QSAGE) survey – II. Galaxy overdensities around UV luminous quasars at z = 1–2

We demonstrate that the UV brightest quasars at z = 1–2 live in overdense environments. This is based on an analysis of deep Hubble Space Telescope WFC3 G141 grism spectroscopy of the galaxies along the lines-of-sight to UV luminous quasars in the redshift range z = 1–2. This constitutes some of the deepest grism spectroscopy performed by WFC3, with four roll angles spread over a year of observations to mitigate the effect of overlapping spectra. Of the 12 quasar fields studied, 8 display evidence for a galaxy overdensity at the redshift of the quasar. One of the overdensities, PG0117 + 213 at z = 1.50, has potentially 36 spectroscopically confirmed members, consisting of 19 with secure redshifts and 17 with single-line redshifts, within a cylinder of radius ∼700 kpc. Its halo mass is estimated to be log (M/M⊙) = 14.7. This demonstrates that spectroscopic and narrow-band observations around distant UV bright quasars may be an excellent route for discovering protoclusters. Our findings agree with previous hints from statistical observations of the quasar population and theoretical works, as feedback regulated black hole growth predicts a correlation between quasar luminosity and halo mass. We also present the high signal-to-noise rest-frame optical spectral and photometric properties of the quasars themselves.
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.
StackOverflow vs Kaggle: A Study of Developer Discussions About Data Science

Software developers are increasingly required to understand fundamental Data science (DS) concepts. Recently, the presence of machine learning (ML) and deep learning (DL) has dramatically increased in the development of user applications, whether they are leveraged through frameworks or implemented from scratch. These topics attract much discussion on online platforms. This paper conducts large-scale qualitative and quantitative experiments to study the characteristics of 197836 posts from StackOverflow and Kaggle. Latent Dirichlet Allocation topic modelling is used to extract twenty-four DS discussion topics. The main findings include that TensorFlow-related topics were most prevalent in StackOverflow, while meta discussion topics were the prevalent ones on Kaggle. StackOverflow tends to include lower-level troubleshooting, while Kaggle focuses on practicality and optimising leaderboard performance. In addition, across both communities, DS discussion is increasing at a dramatic rate. While TensorFlow discussion on StackOverflow is slowing, interest in Keras is rising. Finally, ensemble algorithms are the most mentioned ML/DL algorithms in Kaggle but are rarely discussed on StackOverflow. These findings can help educators and researchers to more effectively tailor and prioritise efforts in researching and communicating DS concepts towards different developer communities.
Evolution of binary black holes in AGN accretion discs: Disc-binary interaction and gravitational wave emission

Binary black hole (BBH) mergers are the primary sources of gravitational wave (GW) events detected by LIGO/Virgo. Binary black holes embedded in the accretion discs of active galactic nuclei (AGN) are possible candidates for such GW events. We have developed an idealised analytic model for the orbital evolution of BBHs in AGN accretion discs by combining the evolution equations of disc-binary interaction and GW inspiral. We investigated the coupled “disc+GW”-driven evolution of BBHs transitioning from the disc-driven regime at large orbital separations into the GW-driven regime at small separations. In this evolution channel, BBH mergers are accelerated by a combination of orbital decay and orbital eccentricity growth in the disc-dominated regime. We provide a quantification of the resulting merger timescale τmerger, and analyse its dependence on both the accretion disc and binary orbital parameters. By computing the evolution of the orbital eccentricity as a function of the GW frequency, we predict that most binaries in AGN discs should have significant residual eccentricities (e ∼ 0.01 − 0.1), potentially detectable by LISA. We further discuss the potentials and caveats of this particular BBH-in-AGN channel in the framework of binary evolutionary paths.
Hunting for Wandering Massive Black Holes

We investigate low-density accretion flows onto massive black holes (BHs) with masses of ≳105 orbiting around in the outskirts of their host galaxies, performing 3D hydrodynamical simulations. Those wandering BHs are populated via ejection from the galactic nuclei through multibody BH interactions and gravitational wave recoils associated with galaxy and BH coalescences. We find that when a wandering BH is fed with hot and diffuse plasma with density fluctuations, the mass accretion rate is limited at ∼10%–20% of the canonical Bondi–Hoyle–Littleton rate owing to a wide distribution of inflowing angular momentum. We further calculate radiation spectra from radiatively inefficient accretion flows onto the wandering BH using a semianalytical two-temperature disk model and find that the predicted spectra have a peak at the millimeter band, where the Atacama Large Millimeter/submillimeter Array (ALMA) has the highest sensitivity and spatial resolution. Millimeter observations with ALMA and future facilities such as the next-generation Very Large Array (ngVLA) will enable us to hunt for a population of wandering BHs and push the detectable mass limit down to M• ≃ 2 × 107 for massive nearby ellipticals, e.g., M87, and M• ≃ 105 for the Milky Way. This radiation spectral model, combined with numerical simulations, will be applied to give physical interpretations of off-nuclear BHs detected in dwarf galaxies, which may constrain BH seed formation scenarios.
Generalizing to unseen domains via distribution matching

Supervised learning results typically rely on assumptions of i.i. d. data. Unfortunately, those assumptions are commonly violated in practice. In this work, we tackle this problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions). Our work relies on a simple lemma: by minimizing a notion of discrepancy between all pairs from a set of given domains, we also minimize the discrepancy between any pairs of mixtures of domains. Using this result, we derive a generalization bound for our setting. We then show that low risk over unseen domains can be achieved by representing the data in a space where (i) the training distributions are indistinguishable, and (ii) relevant information for the task at hand is preserved. Minimizing the terms in our bound yields an adversarial formulation which estimates and minimizes pairwise discrepancies. We validate our proposed strategy on standard domain generalization benchmarks, outperforming a number of recently introduced methods. Notably, we tackle a real-world application where the underlying data corresponds to multi-channel electroencephalography time series from different subjects, each considered as a distinct domain.
ICAM: Interpretable Classification via Disentangled Representations and Feature Attribution Mapping

Feature attribution (FA), or the assignment of class-relevance to different locations in an image, is important for many classification problems but is particularly crucial within the neuroscience domain, where accurate mechanistic models of behaviours, or disease, require knowledge of all features discriminative of a trait. At the same time, predicting class relevance from brain images is challenging as phenotypes are typically heterogeneous, and changes occur against a background of significant natural variation. Here, we present a novel framework for creating class specific FA maps through image-to-image translation. We propose the use of a VAE-GAN to explicitly disentangle class relevance from background features for improved interpretability properties, which results in meaningful FA maps. We validate our method on 2D and 3D brain image datasets of dementia (ADNI dataset), ageing (UK Biobank), and (simulated) lesion detection. We show that FA maps generated by our method outperform baseline FA methods when validated against ground truth. More significantly, our approach is the first to use latent space sampling to support exploration of phenotype variation. Our code will be available online at this https URL.
Detangling robustness in high dimensions: Composite versus model-averaged estimation

Robust methods, though ubiquitous in practice, are yet to be fully understood in the context of regularized estimation and high dimensions. Even simple questions become challenging very quickly. For example, classical statistical theory identifies equivalence between model-averaged and composite quantile estimation. However, little to nothing is known about such equivalence between methods that encourage sparsity. This paper provides a toolbox to further study robustness in these settings and focuses on prediction. In particular, we study optimally weighted model-averaged as well as composite $l_1$-regularized estimation. Optimal weights are determined by minimizing the asymptotic mean squared error. This approach incorporates the effects of regularization, without the assumption of perfect selection, as is often used in practice. Such weights are then optimal for prediction quality. Through an extensive simulation study, we show that no single method systematically outperforms others. We find, however, that model-averaged and composite quantile estimators often outperform least-squares methods, even in the case of Gaussian model noise. Real data application witnesses the method's practical use through the reconstruction of compressed audio signals.
Combinatorial Group Testing and Sparse Recovery Schemes with Near-Optimal Decoding Time

In the long-studied problem of combinatorial group testing, one is asked to detect a set of $k$ defective items out of a population of size $n$, using $m\ll n$ disjunctive measurements. In the non-adaptive setting, the most widely used combinatorial objects are disjunct and list-disjunct matrices, which define incidence matrices of test schemes. Disjunct matrices allow the identification of the exact set of defectives, whereas list disjunct matrices identify a small superset of the defectives. Apart from the combinatorial guarantees, it is often of key interest to equip measurement designs with efficient decoding algorithms. The most efficient decoders should run in sublinear time in $n$, and ideally near-linear in the number of measurements $m$. In this work, we give several constructions with an optimal number of measurements and near-optimal decoding time for the most fundamental group testing tasks, as well as for central tasks in the compressed sensing and heavy hitters literature. For many of those tasks, the previous measurement-optimal constructions needed time either quadratic in the number of measurements or linear in the universe size. Among our results are the following: a construction of disjunct matrices matching the best-known construction in terms of the number of rows $m$, but achieving nearly linear decoding time in $m$; a construction of list disjunct matrices with the optimal $m=O(k\log(n/k)$ number of rows and nearly linear decoding time in $m$; error-tolerant variations of the above constructions; a non-adaptive group testing scheme for the “for-each” model with $m=O(k\log n)$ measurements and $O(m)$ decoding time; a streaming algorithm for the “for-all” version of the heavy hitters problem in the strict turnstile model with near-optimal query time, as well as a “list decoding” variant obtaining also near-optimal update time and $O(k\log(n/k))$ space usage; an $\ell_{2}/\ell_{2}$ weak identification system for compressed sensing with nearly optimal sample complexity and nearly linear decoding time in the sketch length. Most of our results are obtained via a clean and novel approach that avoids list-recoverable codes or related complex techniques that were present in almost every state-of-the-art work on efficiently decodable constructions of such objects.