Miracle-dz/UniFAR-FTUs huggingface dataset:
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers

This article does not propose any novel algorithm or new hardware for sparsity. Instead, it aims to serve the"common good"for the increasingly prosperous Sparse Neural Network (SNN) research community. We attempt to summarize some most common confusions in SNNs, that one may come across in various scenarios such as paper review/rebuttal and talks - many drawn from the authors' own bittersweet experiences! We feel that doing so is meaningful and timely, since the focus of SNN research is notably shifting from traditional pruning to more diverse and profound forms of sparsity before, during, and after training. The intricate relationships between their scopes, assumptions, and approaches lead to misunderstandings, for non-experts or even experts in SNNs. In response, we summarize ten Q\&As of SNNs from many key aspects, including dense vs. sparse, unstructured sparse vs. structured sparse, pruning vs. sparse training, dense-to-sparse training vs. sparse-to-sparse training, static sparsity vs. dynamic sparsity, before-training/during-training vs. post-training sparsity, and many more. We strive to provide proper and generically applicable answers to clarify those confusions to the best extent possible. We hope our summary provides useful general knowledge for people who want to enter and engage with this exciting community; and also provides some"mind of ease"convenience for SNN researchers to explain their work in the right contexts. At the very least (and perhaps as this article's most insignificant target functionality), if you are writing/planning to write a paper or rebuttal in the field of SNNs, we hope some of our answers could help you!
Random matching in 2D with exponent 2 for gaussian densities

We solve the Random Euclidean Matching problem with exponent 2 for the Gaussian distribution defined on the plane. Previous works by Ledoux and Talagrand determined the leading behavior of the average cost up to a multiplicative constant. We explicitly determine the constant, showing that the average cost is proportional to (log N)^2, where N is the number of points. Our approach relies on a geometric decomposition allowing an explicit computation of the constant. Our results illustrate the potential for exact solutions of random matching problems for many distributions defined on unbounded domains on the plane.
The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default

In recent years fairness in machine learning (ML) has emerged as a highly active area of research and development. Most define fairness in simple terms, where fairness means reducing gaps in performance or outcomes between demographic groups while preserving as much of the accuracy of the original system as possible. This oversimplification of equality through fairness measures is troubling. Many current fairness measures suffer from both fairness and performance degradation, or"levelling down,"where fairness is achieved by making every group worse off, or by bringing better performing groups down to the level of the worst off. When fairness can only be achieved by making everyone worse off in material or relational terms through injuries of stigma, loss of solidarity, unequal concern, and missed opportunities for substantive equality, something would appear to have gone wrong in translating the vague concept of 'fairness' into practice. This paper examines the causes and prevalence of levelling down across fairML, and explore possible justifications and criticisms based on philosophical and legal theories of equality and distributive justice, as well as equality law jurisprudence. We find that fairML does not currently engage in the type of measurement, reporting, or analysis necessary to justify levelling down in practice. We propose a first step towards substantive equality in fairML:"levelling up"systems by design through enforcement of minimum acceptable harm thresholds, or"minimum rate constraints,"as fairness constraints. We likewise propose an alternative harms-based framework to counter the oversimplified egalitarian framing currently dominant in the field and push future discussion more towards substantive equality opportunities and away from strict egalitarianism by default. N.B. Shortened abstract, see paper for full abstract.
multi-GPA-Tree: Statistical approach for pleiotropy informed and functional annotation tree guided prioritization of GWAS results

Genome-wide association studies (GWAS) have successfully identified over two hundred thousand genotype-trait associations. Yet some challenges remain. First, complex traits are often associated with many single nucleotide polymorphisms (SNPs), most with small or moderate effect sizes, making them difficult to detect. Second, many complex traits share a common genetic basis due to ‘pleiotropy’ and and though few methods consider it, leveraging pleiotropy can improve statistical power to detect genotype-trait associations with weaker effect sizes. Third, currently available statistical methods are limited in explaining the functional mechanisms through which genetic variants are associated with specific or multiple traits. We propose multi-GPA-Tree to address these challenges. The multi-GPA-Tree approach can identify risk SNPs associated with single as well as multiple traits while also identifying the combinations of functional annotations that can explain the mechanisms through which risk-associated SNPs are linked with the traits. First, we implemented simulation studies to evaluate the proposed multi-GPA-Tree method and compared its performance with existing statistical approaches. The results indicate that multi-GPA-Tree outperforms existing statistical approaches in detecting risk-associated SNPs for multiple traits. Second, we applied multi-GPA-Tree to a systemic lupus erythematosus (SLE) and rheumatoid arthritis (RA), and to a Crohn’s disease (CD) and ulcertive colitis (UC) GWAS, and functional annotation data including GenoSkyline and GenoSkylinePlus. Our results demonstrate that multi-GPA-Tree can be a powerful tool that improves association mapping while facilitating understanding of the underlying genetic architecture of complex traits and potential mechanisms linking risk-associated SNPs with complex traits.
Learning to Agree on Vision Attention for Visual Commonsense Reasoning

Visual Commonsense Reasoning (VCR) remains a significant yet challenging research problem in the realm of visual reasoning. A VCR model generally aims at answering a textual question regarding an image, followed by the rationale prediction for the preceding answering process. Though these two processes are sequential and intertwined, existing methods always consider them as two independent matching-based instances. They, therefore, ignore the pivotal relationship between the two processes, leading to sub-optimal model performance. This paper presents a novel visual attention alignment method to efficaciously handle these two processes in a unified framework. To achieve this, we first design a re-attention module for aggregating the vision attention map produced in each process. Thereafter, the resultant two sets of attention maps are carefully aligned to guide the two processes to make decisions based on the same image regions. We apply this method to both conventional attention and the recent Transformer models and carry out extensive experiments on the VCR benchmark dataset. The results demonstrate that with the attention alignment module, our method achieves a considerable improvement over the baseline methods, evidently revealing the feasibility of the coupling of the two processes as well as the effectiveness of the proposed method.
A Strong Baseline for Batch Imitation Learning

Imitation of expert behaviour is a highly desirable and safe approach to the problem of sequential decision making. We provide an easy-to-implement, novel algorithm for imitation learning under a strict data paradigm, in which the agent must learn solely from data collected a priori. This paradigm allows our algorithm to be used for environments in which safety or cost are of critical concern. Our algorithm requires no additional hyper-parameter tuning beyond any standard batch reinforcement learning (RL) algorithm, making it an ideal baseline for such data-strict regimes. Furthermore, we provide formal sample complexity guarantees for the algorithm in finite Markov Decision Problems. In doing so, we formally demonstrate an unproven claim from Kearns&Singh (1998). On the empirical side, our contribution is twofold. First, we develop a practical, robust and principled evaluation protocol for offline RL methods, making use of only the dataset provided for model selection. This stands in contrast to the vast majority of previous works in offline RL, which tune hyperparameters on the evaluation environment, limiting the practical applicability when deployed in new, cost-critical environments. As such, we establish precedent for the development and fair evaluation of offline RL algorithms. Second, we evaluate our own algorithm on challenging continuous control benchmarks, demonstrating its practical applicability and competitiveness with state-of-the-art performance, despite being a simpler algorithm.
Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs

Contrastively trained encoders have recently been proven to invert the data-generating process: they encode each input, e.g., an image, into the true latent vector that generated the image (Zimmermann et al., 2021). However, real-world observations often have inherent ambiguities. For instance, images may be blurred or only show a 2D view of a 3D object, so multiple latents could have generated them. This makes the true posterior for the latent vector probabilistic with heteroscedastic uncertainty. In this setup, we extend the common InfoNCE objective and encoders to predict latent distributions instead of points. We prove that these distributions recover the correct posteriors of the data-generating process, including its level of aleatoric uncertainty, up to a rotation of the latent space. In addition to providing calibrated uncertainty estimates, these posteriors allow the computation of credible intervals in image retrieval. They comprise images with the same latent as a given query, subject to its uncertainty. Code is available at https://github.com/mkirchhof/Probabilistic_Contrastive_Learning
Divide and Compose with Score Based Generative Models

While score based generative models, or diffusion models, have found success in image synthesis, they are often coupled with text data or image label to be able to manipulate and conditionally generate images. Even though manipulation of images by changing the text prompt is possible, our understanding of the text embedding and our ability to modify it to edit images is quite limited. Towards the direction of having more control over image manipulation and conditional generation, we propose to learn image components in an unsupervised manner so that we can compose those components to generate and manipulate images in informed manner. Taking inspiration from energy based models, we interpret different score components as the gradient of different energy functions. We show how score based learning allows us to learn interesting components and we can visualize them through generation. We also show how this novel decomposition allows us to compose, generate and modify images in interesting ways akin to dreaming. We make our code available at https://github.com/sandeshgh/Score-based-disentanglement
Clarifying trust of materials property predictions using neural networks with distribution-specific uncertainty quantification

It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the adsorption energy predictions, namely k-fold ensembling, Monte Carlo dropout, and evidential regression. The effectiveness of each UQ method is assessed based on accuracy, sharpness, dispersion, calibration, and tightness. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining tunable, competitively trustworthy UQ estimates for heterogeneous catalysis applications when using neural networks. Recalibration of model uncertainties is shown to be essential in practical screening applications of catalysts using uncertainties.
The effects of acrylamide and Vitamin E administration during pregnancy on adult rats testis

Thirty rats, with confirmed pregnancies by vaginal smear, were divided into five groups, each including six rats, as the Control, Corn Oil, Vitamin E, Acrylamide, Vitamin E + Acrylamide groups. The births were monitored on the 21st day to select the male rats, and the selected male rats were decapitated at the end of the 8th week. Oxidant–antioxidant parameters, serum hormone levels and histopathological examinations were performed on testis tissues of the rats. It was found that acrylamide (AA) negatively affected the serum hormone levels (Total Testosterone, Progesterone, FSH, LH, Estradiol), oxidant–antioxidant parameters in the tissues (MDA, GSH, NO, SOD, CAT, TAS, TOS) (p < 0.05) and the histological findings (the Johnson's score, seminiferous tubule diameter, histopathological images), and Vitamin E administration resulted with an increase in the total testosterone, progesterone, FSH, LH, GSH, TAS, NO, SOD, CAT levels (p < 0.05) and an improvement in histopathological findings. Currently, it is almost inevitable to be exposed to food‐induced AA toxicity and such toxicity is likely to cause lifelong damage. It was concluded that Vitamin E was able to present a protective effect in the testis tissue against AA toxicity; however, further studies are necessary.
Discrete tensor product BGG sequences: splines and finite elements

In this paper, we provide a systematic discretization of the Bernstein-Gelfand-Gelfand (BGG) diagrams and complexes over cubical meshes of arbitrary dimension via the use of tensor-product structures of one-dimensional piecewise-polynomial spaces, such as spline and finite element spaces. We demonstrate the construction of the Hessian, the elasticity, and the divdiv complexes as examples for our construction.
RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inference

The proliferation of deep learning (DL) has led to the emergence of privacy and security concerns. To address these issues, secure Two-party computation (2PC) has been proposed as a means of enabling privacy-preserving DL computation. However, in practice, 2PC methods often incur high computation and communication overhead, which can impede their use in large-scale systems. To address this challenge, we introduce RRNet, a systematic framework that aims to jointly reduce the overhead of MPC comparison protocols and accelerate computation through hardware acceleration. Our approach integrates the hardware latency of cryptographic building blocks into the DNN loss function, resulting in improved energy efficiency, accuracy, and security guarantees. Furthermore, we propose a cryptographic hardware scheduler and corresponding performance model for Field Programmable Gate Arrays (FPGAs) to further enhance the efficiency of our framework. Experiments show RRNet achieved a much higher ReLU reduction performance than all SOTA works on CIFAR-10 dataset.
Comparing Rural and Urban Medicare Advantage Beneficiary Characteristics.

Purpose The Medicare Advantage (MA) program allows Medicare beneficiaries to receive benefits from private plans rather than from traditional fee-for-service (FFS) Medicare. Little is known about the rural and urban differences in the populations that enroll in the MA program, and these differences may be important for setting policy. This brief uses data from the 2012-13 Medicare Current Beneficiary Survey (MCBS) to describe these differences, and combined with county-level data on MA issuer participation, this dataset also allows us to assess the degree to which issuers may engage in selective MA market entry on the basis of demographic characteristics. Key Findings (1) Rural and urban MA and FFS populations did not differ much on average by any characteristics reported in the data, including age, self-reported health status, cancer diagnosis, smoking status, Medicaid status, or by other variables assessing frailty and presence of chronic conditions. (2) Most measures of access were similar across rural and urban respondents. However, in terms of cost, urban enrollees were less likely to pay an additional premium (beyond Medicare Part A and B) to obtain MA coverage: 42 percent reported doing so in urban places, while 54 percent did so in rural places. (3) While rurality on its own was often a significant predictor of lower issuer participation in a county’s MA market, the addition of other demographic characteristics did not influence the prediction. In other words, we found no evidence, based upon MCBS data, that issuers exclude rural counties due to other demographics.
Whispering gallery mode aptasensors for detection of blood proteins

Whispering gallery mode resonators (WGMR), as silica microspheres, have been recently proposed as an efficient tool for the realisation of optical biosensors. In this work we present a functionalization procedure based on the DNA‐aptamer sequence immobilization on WGMR, able to recognize specifically thrombin or VEGF protein, preserving a high Q factor. The protein binding was optically characterized in terms of specificity in buffer solution or in 10% diluted human serum. Simulation of the protein flow was found in good agreement with experimental data. The aptasensor was also chemically regenerated and tested again, demonstrating the reusability of our system. (© 2013 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)
U-Clip: On-Average Unbiased Stochastic Gradient Clipping

U-Clip is a simple amendment to gradient clipping that can be applied to any iterative gradient optimization algorithm. Like regular clipping, U-Clip involves using gradients that are clipped to a prescribed size (e.g. with component wise or norm based clipping) but instead of discarding the clipped portion of the gradient, U-Clip maintains a buffer of these values that is added to the gradients on the next iteration (before clipping). We show that the cumulative bias of the U-Clip updates is bounded by a constant. This implies that the clipped updates are unbiased on average. Convergence follows via a lemma that guarantees convergence with updates $u_i$ as long as $\sum_{i=1}^t (u_i - g_i) = o(t)$ where $g_i$ are the gradients. Extensive experimental exploration is performed on CIFAR10 with further validation given on ImageNet.
Life beyond 30: Probing the −20 < M UV < −17 Luminosity Function at 8 < z < 13 with the NIRCam Parallel Field of the MIRI Deep Survey

We present the ultraviolet luminosity function and an estimate of the cosmic star formation rate density at 8 < z < 13 derived from deep NIRCam observations taken in parallel with the MIRI Deep Survey of the Hubble Ultra Deep Field (HUDF), NIRCam covering the parallel field 2. Our deep (40 hr) NIRCam observations reach an F277W magnitude of 30.8 (5σ), more than 2 mag deeper than JWST public data sets already analyzed to find high-redshift galaxies. We select a sample of 44 z > 8 galaxy candidates based on their dropout nature in the F115W and/or F150W filters, a high probability for their photometric redshifts, estimated with three different codes, being at z > 8, good fits based on χ 2 calculations, and predominant solutions compared to z < 8 alternatives. We find mild evolution in the luminosity function from z ∼ 13 to z ∼ 8, i.e., only a small increase in the average number density of ∼0.2 dex, while the faint-end slope and absolute magnitude of the knee remain approximately constant, with values α = − 2.2 ± 0.1, and M * = − 20.8 ± 0.2 mag. Comparing our results with the predictions of state-of-the-art galaxy evolution models, we find two main results: (1) a slower increase with time in the cosmic star formation rate density compared to a steeper rise predicted by models; (2) nearly a factor of 10 higher star formation activity concentrated in scales around 2 kpc in galaxies with stellar masses ∼108 M ⊙ during the first 350 Myr of the universe, z ∼ 12, with models matching better the luminosity density observational estimations ∼150 Myr later, by z ∼ 9.
Brownian loops on non-smooth surfaces and the Polyakov-Alvarez formula

Let $\rho$ be compactly supported on $D \subset \mathbb R^2$. Endow $\mathbb R^2$ with the metric $e^{\rho}(dx_1^2 + dx_2^2)$. As $\delta \to 0$ the set of Brownian loops centered in $D$ with length at least $\delta$ has measure $$\frac{\text{area}(D)}{2\pi \delta} + \frac{1}{48\pi}(\rho,\rho)_{\nabla}+ o(1). $$ When $\rho$ is smooth, this follows from the classical Polyakov-Alvarez formula. We show that the above also holds if $\rho$ is not smooth, e.g. if $\rho$ is only Lipschitz. This fact can alternatively be expressed in terms of heat kernel traces, eigenvalue asymptotics, or zeta regularized determinants. Variants of this statement apply to more general non-smooth manifolds on which one considers all loops (not only those centered in a domain $D$). We also show that the $o(1)$ error is uniform for any family of $\rho$ satisfying certain conditions. This implies that if we weight a measure $\nu$ on this family by the ($\delta$-truncated) Brownian loop soup partition function, and take the vague $\delta \to 0$ limit, we obtain a measure whose Radon-Nikodym derivative with respect to $\nu$ is $\exp\bigl( \frac{1}{48\pi}(\rho,\rho)_{\nabla}\bigr)$. When the measure is a certain regularized Liouville quantum gravity measure, a companion work [APPS20] shows that this weighting has the effect of changing the so-called central charge of the surface.
Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Time Series Classification and Extrinsic Regression are important and challenging machine learning tasks. Deep learning has revolutionized natural language processing and computer vision and holds great promise in other fields such as time series analysis where the relevant features must often be abstracted from the raw data but are not known a priori. This paper surveys the current state of the art in the fast-moving field of deep learning for time series classification and extrinsic regression. We review different network architectures and training methods used for these tasks and discuss the challenges and opportunities when applying deep learning to time series data. We also summarize two critical applications of time series classification and extrinsic regression, human activity recognition and satellite earth observation.