Miracle-dz/UniFAR-FTUs huggingface dataset:
Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks

There is limited understanding of the information captured by deep spatiotemporal models in their intermediate representations. For example, while evidence suggests that action recognition algorithms are heavily influenced by visual appearance in single frames, no quantitative methodology exists for evaluating such static bias in the latent representation compared to bias toward dynamics. We tackle this challenge by proposing an approach for quantifying the static and dynamic biases of any spatiotemporal model, and apply our approach to three tasks, action recognition, automatic video object segmentation (AVOS) and video instance segmentation (VIS). Our key findings are: (i) Most examined models are biased toward static information. (ii) Some datasets that are assumed to be biased toward dynamics are actually biased toward static information. (iii) Individual channels in an architecture can be biased toward static, dynamic or a combination of the two. (iv) Most models converge to their culminating biases in the first half of training. We then explore how these biases affect performance on dynamically biased datasets. For action recognition, we propose StaticDropout, a semantically guided dropout that debiases a model from static information toward dynamics. For AVOS, we design a better combination of fusion and cross connection layers compared with previous architectures.
Bayesian Counterfactual Mean Embeddings and Off-Policy Evaluation

The counterfactual distribution models the effect of the treatment in the untreated group. While most of the work focuses on the expected values of the treatment effect, one may be interested in the whole counterfactual distribution or other quantities associated to it. Building on the framework of Bayesian conditional mean embeddings, we propose a Bayesian approach for modeling the counterfactual distribution, which leads to quantifying the epistemic uncertainty about the distribution. The framework naturally extends to the setting where one observes multiple treatment effects (e.g. an intermediate effect after an interim period, and an ultimate treatment effect which is of main interest) and allows for additionally modelling uncertainty about the relationship of these effects. For such goal, we present three novel Bayesian methods to estimate the expectation of the ultimate treatment effect, when only noisy samples of the dependence between intermediate and ultimate effects are provided. These methods differ on the source of uncertainty considered and allow for combining two sources of data. Moreover, we generalize these ideas to the off-policy evaluation framework, which can be seen as an extension of the counterfactual estimation problem. We empirically explore the calibration of the algorithms in two different experimental settings which require data fusion, and illustrate the value of considering the uncertainty stemming from the two sources of data.
PI is back! Switching Acquisition Functions in Bayesian Optimization

Bayesian Optimization (BO) is a powerful, sample-efficient technique to optimize expensive-to-evaluate functions. Each of the BO components, such as the surrogate model, the acquisition function (AF), or the initial design, is subject to a wide range of design choices. Selecting the right components for a given optimization task is a challenging task, which can have significant impact on the quality of the obtained results. In this work, we initiate the analysis of which AF to favor for which optimization scenarios. To this end, we benchmark SMAC3 using Expected Improvement (EI) and Probability of Improvement (PI) as acquisition functions on the 24 BBOB functions of the COCO environment. We compare their results with those of schedules switching between AFs. One schedule aims to use EI's explorative behavior in the early optimization steps, and then switches to PI for a better exploitation in the final steps. We also compare this to a random schedule and round-robin selection of EI and PI. We observe that dynamic schedules oftentimes outperform any single static one. Our results suggest that a schedule that allocates the first 25 % of the optimization budget to EI and the last 75 % to PI is a reliable default. However, we also observe considerable performance differences for the 24 functions, suggesting that a per-instance allocation, possibly learned on the fly, could offer significant improvement over the state-of-the-art BO designs.
Active CT Reconstruction with a Learned Sampling Policy

Computed tomography (CT) is a widely-used imaging technology that assists clinical decision-making with high-quality human body representations. To reduce the radiation dose posed by CT, sparse-view (SV) CT is developed with preserved image quality. However, these methods are still stuck with a fixed uniform SV (USV) sampling strategy, which inhibits the possibility of acquiring a better image with an even reduced dose. In this paper, we explore this possibility via learning an active SV (ASV) sampling policy that optimizes the sampling positions for regions of interest (RoI)-specific, high-quality reconstruction. To this end, we design an sampling agent for the recommendation of ASV sampling positions based on on-the-fly reconstruction with obtained sinograms in a progressive fashion. With such a design, we achieve better performances on the NIH-AAPM dataset over popular USV sampling, especially when the number of views is small. Finally, such a design enables the RoI-aware reconstruction with improved local quality within the RoI that are clinically important. Experiments on the VerSe dataset demonstrate the ability of the proposed sampling policy, which is difficult to achieve with USV sampling.
Query-based Instance Discrimination Network for Relational Triple Extraction

Joint entity and relation extraction has been a core task in the field of information extraction. Recent approaches usually consider the extraction of relational triples from a stereoscopic perspective, either learning a relation-specific tagger or separate classifiers for each relation type. However, they still suffer from error propagation, relation redundancy and lack of high-level connections between triples. To address these issues, we propose a novel query-based approach to construct instance-level representations for relational triples. By metric-based comparison between query embeddings and token embeddings, we can extract all types of triples in one step, thus eliminating the error propagation problem. In addition, we learn the instance-level representation of relational triples via contrastive learning. In this way, relational triples can not only enclose rich class-level semantics but also access to high-order global connections. Experimental results show that our proposed method achieves the state of the art on five widely used benchmarks.
Unlocking the potential of two-point cells for energy-efficient training of deep nets

—Context-sensitive two-point layer 5 pyramidal cells (L5PC) were discovered as long ago as 1999. However, the potential of this discovery to provide useful neural computation has yet to be demonstrated. Here we show for the ﬁrst time how a transformative L5PC-driven deep neural network (DNN), termed the multisensory cooperative computing (MCC) architecture, can effectively process large amounts of heterogeneous real-world audio-visual (AV) data, using far less energy compared to best available ‘point’ neuron-driven DNNs. A novel highly-distributed parallel implementation on a Xilinx UltraScale+ MPSoC device estimates energy savings up to 245759 × 50000 µ J (i.e., 62% less than the baseline model in a semi-supervised learning setup) where a single synapse consumes 8 e − 5 µ J. In a supervised learning setup, the energy-saving can potentially reach up to 1250x less (per feedforward transmission) than the baseline model. This remarkable performance in pilot experiments demonstrates the embodied neuromorphic intelligence of our proposed L5PC based MCC architecture that contextually selects the most salient and relevant information for onward transmission, from overwhelmingly large multimodal information utilised at the early stages of on-chip training. Our proposed approach opens new cross-disciplinary avenues for future on-chip DNN training implementations and posits a radical shift in current neuromorphic computing paradigms.
Entanglement complexity of the Rokhsar-Kivelson-sign wavefunctions

In this paper we study the transitions of entanglement complexity in an exemplary family of states - the Rokhsar-Kivelson-sign wavefunctions - whose degree of entanglement is controlled by a single parameter. This family of states is known to feature a transition between a phase exhibiting volume-law scaling of entanglement entropy and a phase with sub-extensive scaling of entanglement, reminiscent of the many-body-localization transition of disordered quantum Hamiltonians [Physical Review B 92, 214204 (2015)]. We study the singularities of the Rokhsar-Kivelson-sign wavefunctions and their entanglement complexity across the transition using several tools from quantum information theory: fidelity metric; entanglement spectrum statistics; entanglement entropy fluctuations; stabilizer R\'enyi Entropy; and the performance of a disentangling algorithm. Across the whole volume-law phase the states feature universal entanglement spectrum statistics. Yet a"super-universal"regime appears for small values of the control parameter in which all metrics become independent of the parameter itself; the entanglement entropy as well as the stabilizer R\'enyi entropy appear to approach their theoretical maximum; the entanglement fluctuations scale to zero as in output states of random universal circuits, and the disentangling algorithm has essentially null efficiency. All these indicators consistently reveal a complex pattern of entanglement. In the sub-volume-law phase, on the other hand, the entanglement spectrum statistics is no longer universal, entanglement fluctuations are larger and exhibiting a non-universal scaling; and the efficiency of the disentangling algorithm becomes finite. Our results, based on model wavefunctions, suggest that a similar combination of entanglement scaling properties and of entanglement complexity features may be found in high-energy Hamiltonian eigenstates.
A New Approach to Estimating Effective Resistances and Counting Spanning Trees in Expander Graphs

We demonstrate that for expander graphs, for all $\epsilon>0,$ there exists a data structure of size $\widetilde{O}(n\epsilon^{-1})$ which can be used to return $(1 + \epsilon)$-approximations to effective resistances in $\widetilde{O}(1)$ time per query. Short of storing all effective resistances, previous best approaches could achieve $\widetilde{O}(n\epsilon^{-2})$ size and $\widetilde{O}(\epsilon^{-2})$ time per query by storing Johnson-Lindenstrauss vectors for each vertex, or $\widetilde{O}(n\epsilon^{-1})$ size and $\widetilde{O}(n\epsilon^{-1})$ time per query by storing a spectral sketch. Our construction is based on two key ideas: 1) $\epsilon^{-1}$-sparse, $\epsilon$-additive approximations to $DL^+1_u$ for all $u,$ can be used to recover $(1 + \epsilon)$-approximations to the effective resistances, 2) In expander graphs, only $\widetilde{O}(\epsilon^{-1})$ coordinates of a vector similar to $DL^+1_u$ are larger than $\epsilon. $ We give an efficient construction for such a data structure in $\widetilde{O}(m + n\epsilon^{-2})$ time via random walks. This results in an algorithm for computing $(1+\epsilon)$-approximate effective resistances for $s$ vertex pairs in expanders that runs in $\widetilde{O}(m + n\epsilon^{-2} + s)$ time, improving over the previously best known running time of $m^{1 + o(1)} + (n + s)n^{o(1)}\epsilon^{-1.5}$ for $s = \omega(n\epsilon^{-0.5}). $ We employ the above algorithm to compute a $(1+\delta)$-approximation to the number of spanning trees in an expander graph, or equivalently, approximating the (pseudo)determinant of its Laplacian in $\widetilde{O}(m + n^{1.5}\delta^{-1})$ time. This improves on the previously best known result of $m^{1+o(1)} + n^{1.875+o(1)}\delta^{-1.75}$ time, and matches the best known size of determinant sparsifiers.
Assessing Resource-Performance Trade-off of Natural Language Models using Data Envelopment Analysis

Natural language models are often summarized through a high-dimensional set of de-scriptive metrics including training corpus size, training time, the number of trainable parameters, inference times, and evaluation statistics that assess performance across tasks. The high dimensional nature of these metrics yields challenges with regard to objectively comparing models; in particular it is challeng-ing to assess the trade-off models make between performance and resources (compute time, memory, etc.). We apply Data Envelopment Analysis (DEA) to this problem of assessing the resource-performance trade-off. DEA is a nonparamet-ric method that measures productive efﬁciency of abstract units that consume one or more inputs and yield at least one output . We recast natural language models as units suitable for DEA, and we show that DEA can be used to create an effective framework for quantifying model performance and efﬁciency. A central feature of DEA is that it identiﬁes a subset of models that live on an efﬁcient frontier of performance. DEA is also scalable, having been applied to problems with thousands of units. We report empirical results of DEA applied to 14 different language models that have a variety of architectures, and we show that DEA can be used to identify a subset of models that effectively balance resource demands against performance.
Competitive Kill-and-Restart and Preemptive Strategies for Non-Clairvoyant Scheduling

We study kill-and-restart and preemptive strategies for the fundamental scheduling problem of minimizing the sum of weighted completion times on a single machine in the non-clairvoyant setting. First, we show a lower bound of~$3$ for any deterministic non-clairvoyant kill-and-restart strategy. Then, we give for any $b>1$ a tight analysis for the natural $b$-scaling kill-and-restart strategy as well as for a randomized variant of it. In particular, we show a competitive ratio of $(1+3\sqrt{3})\approx 6.197$ for the deterministic and of $\approx 3.032$ for the randomized strategy, by making use of the largest eigenvalue of a Toeplitz matrix. In addition, we show that the preemptive Weighted Shortest Elapsed Time First (WSETF) rule is $2$-competitive when jobs are released online, matching the lower bound for the unit weight case with trivial release dates for any non-clairvoyant algorithm. Using this result as well as the competitiveness of round-robin for multiple machines, we prove performance guarantees smaller than $10$ for adaptions of the $b$-scaling strategy to online release dates and unweighted jobs on identical parallel machines.
ON THE DIRECTIONAL DERIVATIVE OF KEMENY’S CONSTANT

In a connected graph, Kemeny’s constant gives the expected time of a random walk from an arbitrary vertex x to reach a randomly-chosen vertex y . Because of this, Kemeny’s constant can be interpreted as a measure of how well a graph is connected. It is generally unknown how the addition or removal of edges aﬀects Kemeny’s constant. Inspired by the directional derivative of the normalized Laplacian, we derive the directional derivative of Kemeny’s constant for several graph families. In addition, we ﬁnd sharp bounds for the directional derivative of an eigenvalue of the normalized Laplacian and bounds for the directional derivative of Kemeny’s constant.
eXplainable AI for Quantum Machine Learning

Parametrized Quantum Circuits (PQCs) enable a novel method for machine learning (ML). How-ever, from a computational point of view they present a challenge to existing eXplainable AI (xAI) methods. On the one hand, measurements on quantum circuits introduce probabilistic errors which impact the convergence of these methods. On the other hand, the phase space of a quantum circuit expands exponentially with the number of qubits, complicating eﬀorts to execute xAI methods in polynomial time. In this paper we will discuss the performance of established xAI methods, such as Baseline SHAP and Integrated Gradients. Using the internal mechanics of PQCs we study ways to speed up their computation.
FUNCK: Information Funnels and Bottlenecks for Invariant Representation Learning

Learning invariant representations that remain useful for a downstream task is still a key challenge in machine learning. We investigate a set of related information funnels and bottleneck problems that claim to learn invariant representations from the data. We also propose a new element to this family of information-theoretic objectives: The Conditional Privacy Funnel with Side Information, which we investigate in fully and semi-supervised settings. Given the generally intractable objectives, we derive tractable approximations using amortized variational inference parameterized by neural networks and study the intrinsic trade-offs of these objectives. We describe empirically the proposed approach and show that with a few labels it is possible to learn fair classifiers and generate useful representations approximately invariant to unwanted sources of variation. Furthermore, we provide insights about the applicability of these methods in real-world scenarios with ordinary tabular datasets when the data is scarce.
Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh

Partial differential equations (PDEs) are often computationally challenging to solve, and in many settings many related PDEs must be be solved either at every timestep or for a variety of candidate boundary conditions, parameters, or geometric domains. We present a meta-learning based method which learns to rapidly solve problems from a distribution of related PDEs. We use meta-learning (MAML and LEAP) to identify initializations for a neural network representation of the PDE solution such that a residual of the PDE can be quickly minimized on a novel task. We apply our meta-solving approach to a nonlinear Poisson's equation, 1D Burgers' equation, and hyperelasticity equations with varying parameters, geometries, and boundary conditions. The resulting Meta-PDE method finds qualitatively accurate solutions to most problems within a few gradient steps; for the nonlinear Poisson and hyper-elasticity equation this results in an intermediate accuracy approximation up to an order of magnitude faster than a baseline finite element analysis (FEA) solver with equivalent accuracy. In comparison to other learned solvers and surrogate models, this meta-learning approach can be trained without supervision from expensive ground-truth data, does not require a mesh, and can even be used when the geometry and topology varies between tasks.
Statistical inference for scale mixture models via Mellin transform approach

This paper deals with statistical inference for the scale mixture models. We study an estimation approach based on the Mellin -- Stieltjes transform that can be applied to both discrete and absolute continuous mixing distributions. The accuracy of the corresponding estimate is analysed in terms of its expected pointwise error. As an important technical result, we prove the analogue of the Berry -- Esseen inequality for the Mellin transforms. The proposed statistical approach is illustrated by numerical examples.
Universality and its limits in non-Hermitian many-body quantum chaos using the Sachdev-Ye-Kitaev model

Spectral rigidity in Hermitian quantum chaotic systems signals the presence of dynamical universal features at timescales that can be much shorter than the Heisenberg time. We study the analog of this timescale in many-body non-Hermitian quantum chaos by a detailed analysis of long-range spectral correlators. For that purpose, we investigate the number variance and the spectral form factor of a non-Hermitian $q$-body Sachdev-Ye-Kitaev (nHSYK) model, which describes $N$ fermions in zero spatial dimensions. After an analytical and numerical analysis of these spectral observables for non-Hermitian random matrices, and a careful unfolding, we find good agreement with the nHSYK model for $q>2$ starting at a timescale that decreases sharply with $q$. The source of deviation from universality, identified analytically, is ensemble fluctuations not related to the quantum dynamics. For fixed $q$ and large enough $N$, these fluctuations become dominant up until after the Heisenberg time, so that the spectral form factor is no longer useful for the study of quantum chaos. In all cases, our results point to a weakened or vanishing spectral rigidity that effectively delays the observation of full quantum ergodicity. We also show that the number variance displays nonstationary spectral correlations for both the nHSYK model and random matrices. This nonstationarity, also not related to the quantum dynamics, points to intrinsic limitations of these observables to describe the quantum chaotic motion. On the other hand, we introduce the local spectral form factor, which is shown to be stationary and not affected by collective fluctuations, and propose it as an effective diagnostic of non-Hermitian quantum chaos. For $q = 2$, we find saturation to Poisson statistics at a timescale of $\log D$, compared to a scale of $\sqrt D$ for $ q>2$, with $D $ the total number of states.
Dormant Neural Trojans

We present a novel methodology for neural network backdoor attacks. Unlike existing training-time attacks where the Trojaned network would respond to the Trojan trigger after training, our approach inserts a Trojan that will remain dormant until it is activated. The activation is realized through a specific perturbation to the network's weight parameters only known to the attacker. Our analysis and the experimental results demonstrate that dormant Trojaned networks can effectively evade detection by state-of-the-art backdoor detection methods.
Multi-vehicle Conflict Resolution in Highly Constrained Spaces by Merging Optimal Control and Reinforcement Learning

We present a novel method to address the problem of multi-vehicle conflict resolution in highly constrained spaces. An optimal control problem is formulated to incorporate nonlinear, non-holonomic vehicle dynamics and exact collision avoidance constraints. A solution to the problem can be obtained by first learning configuration strategies with reinforcement learning (RL) in a simplified discrete environment, and then using these strategies to shape the constraint space of the original problem. Simulation results show that our method can explore efficient actions to resolve conflicts in confined space and generate dexterous maneuvers that are both collision-free and kinematically feasible.
Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration

AI agents designed to collaborate with people benefit from models that enable them to anticipate human behavior. However, realistic models tend to require vast amounts of human data, which is often hard to collect. A good prior or initialization could make for more data-efficient training, but what makes for a good prior on human behavior? Our work leverages a very simple assumption: people generally act closer to optimal than to random chance. We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient and able to generalize to new environments. Our intuition is that such a prior enables the training to focus one's precious real-world data on capturing the subtle nuances of human suboptimality, instead of on the basics of how to do the task in the first place. We also show that using these improved human models often leads to better human-AI collaboration performance compared to using models based on real human data alone.