Miracle-dz/UniFAR-FTUs huggingface dataset:
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. Despite the practical benefits, task-agnostic distillation is challenging. Since the teacher model has a significantly larger capacity and stronger representation power than the student model, it is very difficult for the student to produce predictions that match the teacher's over a massive amount of open-domain training data. Such a large prediction discrepancy often diminishes the benefits of knowledge distillation. To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning. Specifically, we initialize the student model from the teacher model, and iteratively prune the student's neurons until the target width is reached. Such an approach maintains a small discrepancy between the teacher's and student's predictions throughout the distillation process, which ensures the effectiveness of knowledge transfer. Extensive experiments demonstrate that HomoDistil achieves significant improvements on existing baselines.
Control Flow Duplication for Columnar Arrays in a Dynamic Compiler

Columnar databases are an established way to speed up online analytical processing (OLAP) queries. Nowadays, data processing (e.g., storage, visualization, and analytics) is often performed at the programming language level, hence it is desirable to also adopt columnar data structures for common language runtimes. While there are frameworks, libraries, and APIs to enable columnar data stores in programming languages, their integration into applications typically requires developer interference. In prior work, researchers implemented an approach for *automated* transformation of arrays into columnar arrays in the GraalVM JavaScript runtime. However, this approach suffers from performance issues on smaller workloads as well as on more complex nested data structures. We find that the key to optimizing accesses to columnar arrays is to identify queries and apply specific optimizations to them. In this paper, we describe novel compiler optimizations in the GraalVM Compiler that optimize queries on columnar arrays. At JIT compile time, we identify loops that access potentially columnar arrays and duplicate them in order to specifically optimize accesses to columnar arrays. Additionally, we describe a new approach for creating columnar arrays from arrays consisting of complex objects by performing **multi-level storage transformation**. We demonstrate our approach via an implementation for JavaScript `Date` objects. [ full abstract at https://doi.org/10.22152/programming-journal.org/2023/7/9 ]
Certified private data release for sparse Lipschitz functions

As machine learning has become more relevant for everyday applications, a natural requirement is the protection of the privacy of the training data. When the relevant learning questions are unknown in advance, or hyper-parameter tuning plays a central role, one solution is to release a differentially private synthetic data set that leads to similar conclusions as the original training data. In this work, we introduce an algorithm that enjoys fast rates for the utility loss for sparse Lipschitz queries. Furthermore, we show how to obtain a certificate for the utility loss for a large class of algorithms.
Space weathering of asteroidal surfaces

Context. The surfaces of airless bodies in the Solar System are continuously altered by the bombardment of micrometeoroids and irradiation by solar wind, flares, and cosmic particles. Major effects of this process – space weathering – are darkening and “reddening” of the spectra of surface materials, as well as a “degrading” of absorption features. Aims. We studied the changes induced by energetic ion irradiation in the ultraviolet-visual-near-infrared (UV-Vis-NIR) (0.2–0.98 μm) reflectance spectra of targets selected to mimic the surfaces of airless bodies in the inner Solar System. Our chosen targets are olivine pellets, pure or covered by an organic polymer (polystyrene), which is transparent before irradiation. Polystyrene is used as a template for organic matter of low volatility that can be present on asteroidal surfaces. Moreover we measured the changes induced by ion irradiation in the absorption coefficient of the polymer. The purpose was to have a tool to better compare laboratory with observed spectra and distinguish between planetary objects with pure silicate surfaces and those whose surface is covered by organic matter exposed to cosmic ion bombardment. Methods. The samples were irradiated in vacuum, at room temperature, with 200 keV protons or 200–400 keV argon ions. Before, during, and after irradiation diffuse reflectance spectra were acquired. Polystyrene films were also deposited on quartz substrates and irradiated while transmittance spectra were recorded. Results. We measured the variations of the absorption coefficient of polystyrene as a function of ion fluence. We showed that after ion irradiation the diffuse reflectance spectra of the samples covered by organics exhibit a much more significant variation than those of pure silicates. The spectra of targets made of olivine plus polystyrene can be fitted by using the measured absorption coefficient of polystyrene. Conclusions. The results obtained for pure olivine extend to the UV the spectral range of previous experiments. The data concerning the absorption coefficient of polystyrene are available on our web site (http://web.ct.astro.it/weblab/dbindex.html) and can be used to compare laboratory with astronomical spectra. This will possibly allow us to obtain information about organic matter on the surface of a given object, as well as on the relevance of the exposure to cosmic ions (space weathering).
Matrix Product Study of Spin Fractionalization in the 1D Kondo Insulator

The Kondo lattice is one of the classic examples of strongly correlated electronic systems. We conduct a controlled study of the Kondo lattice in one dimension, highlighting the role of excitations created by the composite fermion operator. Using time-dependent matrix-product-state methods we compute various correlation functions and contrast them with both large-N mean-field theory and the strong-coupling expansion. We show that the composite fermion operator creates long-lived, charge-e and spin-1/2 excitations, which cover the low-lying single-particle excitation spectrum of the system. Furthermore, spin excitations can be thought to be composed of such fractionalized quasi-particles with a residual interaction which tend to disappear at weak Kondo coupling.
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of"semantic equivalence"-- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.
Inner-Outer Factorization for Non-Square Proper Functions with Infinite and Finite jω-Axis Zeros

The inner-outer factorization of a non-square proper function which has infinite and finite jomega -axis zeros and has no poles in the open right-half plane is studied in this paper via the concept of an infinite zero compensator. For a stabilizable and detectable realization of the above function with full normal column rank, the eigenstructures of the system matrix pencils of the function and its spectral density matrix are discussed based on Kronecker's theory, and a simple state-space solution to the inner-outer factorization is given via the solutions of the generalized eigenvalue problems in terms of the original realization of the function.
Cayley Sums and Minkowski Sums of Lattice Polytopes

In this paper, we discuss the integer decomposition property for Cayley sums and Minkowski sums of lattice polytopes. In fact, we characterize when Cayley sums have the integer decomposition property in terms of Minkowski sums. Moreover, by using this characterization, we consider when Cayley sums and Minkowski sums of $2$-convex-normal lattice polytopes have the integer decomposition property. Finally, we also discuss the level property for Minkowski sums and Cayley sums.
MultiViz: Towards Visualizing and Understanding Multimodal Models

The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models. However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics. How can we visualize the internal modeling of multimodal interactions in these models? Our paper aims to fill this gap by proposing MultiViz, a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance: how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions: how different modalities relate with each other, (3) multimodal representations: how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction: how decision-level features are composed to make a prediction. MultiViz is designed to operate on diverse modalities, models, tasks, and research areas. Through experiments on 8 trained models across 6 real-world tasks, we show that the complementary stages in MultiViz together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models. MultiViz is publicly available, will be regularly updated with new interpretation tools and metrics, and welcomes inputs from the community.
Fast Kernel Methods for Generic Lipschitz Losses via $p$-Sparsified Sketches

Kernel methods are learning algorithms that enjoy solid theoretical foundations while suffering from important computational limitations. Sketching, which consists in looking for solutions among a subspace of reduced dimension, is a well studied approach to alleviate these computational burdens. However, statistically-accurate sketches, such as the Gaussian one, usually contain few null entries, such that their application to kernel methods and their non-sparse Gram matrices remains slow in practice. In this paper, we show that sparsified Gaussian (and Rademacher) sketches still produce theoretically-valid approximations while allowing for important time and space savings thanks to an efficient \emph{decomposition trick}. To support our method, we derive excess risk bounds for both single and multiple output kernel problems, with generic Lipschitz losses, hereby providing new guarantees for a wide range of applications, from robust regression to multiple quantile regression. Our theoretical results are complemented with experiments showing the empirical superiority of our approach over SOTA sketching methods.
Quasi-Bayesian nonparametric density estimation via autoregressive predictive updates

Bayesian methods are a popular choice for statistical inference in small-data regimes due to the regularization effect induced by the prior. In the context of density estimation, the standard nonparametric Bayesian approach is to target the posterior predictive of the Dirichlet process mixture model. In general, direct estimation of the posterior predictive is intractable and so methods typically resort to approximating the posterior distribution as an intermediate step. The recent development of quasi-Bayesian predictive copula updates, however, has made it possible to perform tractable predictive density estimation without the need for posterior approximation. Although these estimators are computationally appealing, they tend to struggle on non-smooth data distributions. This is due to the comparatively restrictive form of the likelihood models from which the proposed copula updates were derived. To address this shortcoming, we consider a Bayesian nonparametric model with an autoregressive likelihood decomposition and a Gaussian process prior. While the predictive update of such a model is typically intractable, we derive a quasi-Bayesian predictive update that achieves state-of-the-art results in small-data regimes.
A Formal Logic for Formal Category Theory (Extended Version)

We present a domain-specific type theory for constructions and proofs in category theory. The type theory axiomatizes notions of category, functor, profunctor and a generalized form of natural transformations. The type theory imposes an ordered linear restriction on standard predicate logic, which guarantees that all functions between categories are functorial, all relations are profunctorial, and all transformations are natural by construction, with no separate proofs necessary. Important category-theoretic proofs such as the Yoneda lemma and Co-yoneda lemma become simple type-theoretic proofs about the relationship between unit, tensor and (ordered) function types, and can be seen to be ordered refinements of theorems in predicate logic. The type theory is sound and complete for a categorical model in virtual equipments, which model both internal and enriched category theory. While the proofs in our type theory look like standard set-based arguments, the syntactic discipline ensure that all proofs and constructions carry over to enriched and internal settings as well.
Energy Distribution of Heating Processes in the Quiet Solar Corona

We have determined the variations in the emission measure of the solar corona using EUV Imaging Telescope/Solar and Heliospheric Observatory observations of iron lines in a quiet region of the Sun. The emission measure is found to vary significantly in at least 85% of all the pixels within 42 minutes. The variations are interpreted as heating events that bring chromospheric material above the one million degree threshold of the observed lines and that cool the coronal plasma below that limit. A method to assess heating events has been developed. The thermal energy input by such microflares is calculated from the observed increases in emission measure and the derived temperature. Heating events have been found in the range from 8 × 1024 to 1.6 × 1026 ergs. The energy input by ≥3 σ events of the emission measure increase the amounts to about 16% of the average radiated power of the coronal plasma in the quiet corona. The frequency distribution of microflares is an approximate power law of the form f(E)=f0E-δ, with a power-law index δ between 2.3 and 2.6. Since the low-energy cutoff is due to sensitivity limitations and the power-law index is steeper than 2, these observations demonstrate the possibility that microflares dominate the energy input into the quiet corona. The observed power law would have to continue to about 3 × 1023 ergs in order to match the observed minimum heating requirement.
On the scaling of NSW HSC marks in mathematics and encouraging higher participation in calculus-based courses

There is widespread concern about the reduction in the number of students in Australia studying calculus-based mathematics courses in the final year of secondary education. Over the last 20 years in New South Wales, we have seen a dramatic increase in the number of students studying general mathematics in place of calculus courses. The concern arises because students graduating from high school with no calculus are often not well equipped for many aspects of university-level study. This paper explores one potential reason for this in New South Wales, namely the scaling algorithm used to derive the Australian Tertiary Admission Rank. At a time when we are encouraging students to study higher mathematics, this paper contributes to the discussion by illustrating some empirical evidence, based on an analysis of published output from the application of the scaling algorithm used to derive Australian Tertiary Admission Ranks, as to why students may be opting for lower level mathematics at the Year 12 level in great numbers.
Stability of model flocks in turbulent-like flow

We report numerical simulations of a simple model of flocking particles in the presence of an uncertain background environment. We consider two types of environmental perturbations: random noise applied separately to each particle, and spatiotemporally correlated ‘noise’ provided by a turbulent-like flow field. The effects of these two types of noise are very different; surprisingly, the applied flow field tends to destroy the global order of the flocking model even for vanishingly small flow amplitudes. Local order, however, is preserved in smaller sub-flocks, although their composition changes dynamically. Our results suggest that realistic perturbations must be considered in assessing the stability of models of collective animal behavior, and that random noise is not a sufficient proxy.
Control for coordination of linear systems

Control theory for distributed systems is motivated by control of large-scale engineering systems such as electrical power networks, motorway networks, underwater vehicles, and electric-mechanic machines. The control objectives of a control problem for such a system may be such that a tight interaction between the subsystems is necessary. The concept of a coordinated linear system is defined which contains a coordinator and two or more linear subsystems. The controller takes care of the coordination between the subsystems. A geometric condition, conditional linear independence of linear subspaces given another subspace and an invariance condition characterize coordinated linear systems. In case a distributed control system admits a particular representation then control synthesis separates into control synthesis for the coordinator and for each of the local subsystems.
A class of discrepancy principles for the simplified regularization of ill-posed problems

Abstract A class of discrepancy principles for the choice of parameters for the simplified regularization of ill-posed problems is proposed. This procedure does not require knowledge of the unknown solution, and if the smoothness of the unknown solution is known then the convergence rate obtained is optimal. The results of this paper include the Arcangeli's method considered by Groetsch and Guacaneme (1987) for which the convergence rate was not known and also of a result of Guacaneme (1988) for which there is a gap in the proof.
Likelihood analysis for errors-in-variables regression with replicate measurements

SUMMARY This paper advocates likelihood analysis for regression models with measurement errors in explanatory variables, for data problems in which the relevant distributions can be adequately modelled. Although computationally difficult, maximum likelihood estimates are more efficient than those based on first and second moment assumptions, and likelihood ratio inferences can be substantially better than those based on asymptotic normality of estimates. The EM algorithm is presented as a straightforward approach for likelihood analysis of normal linear regression with normal explanatory variables, and normal replicate measurements. Likelihood methods for regression analysis with explanatory variable measurement error have received little attention relative to methods based on first and second moment assumptions, largely because of computational difficulties, uncertainty about the robustness of likelihood methods, and the belief that the simpler methods perform just as well in practice. With improved computational tools for likelihood analysis, as discussed, for example, by Tanner (1993), it is worthwhile to reinvestigate these issues. In particular, the common belief that the simpler methods perform just as well as likelihood methods may be naive. First, the comparison is usually cast in terms of efficiency, without regard to test and confidence interval validity. Likelihood ratio tests and confidence intervals can be substantially better than tests and confidence intervals based on estimates and standard errors of the commonly-used method of moments approaches. The sampling distributions of the estimators are very often skewed, especially if the measurement errors are large. Furthermore, knowledge about the relative efficiency of the moment methods comes from a small number of efficiency comparisons from simulations of isolated situations, in papers promoting these simpler methods (Carroll, Rupert & Stefanski, 1995, ? 7.1). An important and often overlooked issue is that the moment methods may be especially inappropriate when the distribution of the true covariates is skewed (Pierce et al., 1992). Section 2 discusses computational approaches and difficulties associated with likelihood analysis in the measurement error problem generally. Section 3 illustrates the EM algorithm (Dempster, Laird & Rubin, 1977) for likelihood analysis of normal linear regression with a normally-distributed explanatory variable measured with normally-distributed error, when there are replicate measurements for at least some of the cases. Sections 5 and 6
Magnetotelluric strike rules

The distortion of the magnetotelluric impedance tensor by complex “near‐surface” structure leads to leakage between the elements of the tensor. The magnetotelluric impedance tensor for our principal model, which has both a local and a regional strike, can be written in the long‐period limit as a sum of the regional, undistorted impedance and a perturbed impedance. The latter can be written as a product of a local distortion (which can be regarded as thin‐sheet distortion in the long‐period range) tensor and the regional impedance. Local and regional strikes are found by rotating the impedance tensor into directions in which diagonal elements are proportional and column elements are proportional, respectively. The regional impedance tensor is calculated assuming that the strikes are uniquely defined. An example from a crystalline area with well conducting fracture zones illustrates the model concepts. A weighted least‐squares procedure is used for the estimation of distortion parameters.