nils-herrmann/clean_scirepeval huggingface dataset:
What Can Neural Networks Reason About?

Neural networks have succeeded in many reasoning tasks. Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, but less structured networks fail. Theoretically, there is limited understanding of why and when a network structure generalizes better than others, although they have equal expressive power. In this paper, we develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its computation structure aligns with the algorithmic structure of the relevant reasoning process. We formally define this algorithmic alignment and derive a sample complexity bound that decreases with better alignment. This framework offers an explanation for the empirical success of popular reasoning models, and suggests their limitations. As an example, we unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP). We show that GNNs align with DP and thus are expected to solve these tasks. On several reasoning tasks, our theory is supported by empirical results.
Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control

A major challenge in cognitive science and AI has been to understand how autonomous agents might acquire and predict behavioral and mental states of other agents in the course of complex social interactions. How does such an agent model the goals, beliefs, and actions of other agents it interacts with? What are the computational principles to model a Theory of Mind (ToM)? Deep learning approaches to address these questions fall short of a better understanding of the problem. In part, this is due to the black-box nature of deep networks, wherein computational mechanisms of ToM are not readily revealed. Here, we consider alternative hypotheses seeking to model how the brain might realize a ToM. In particular, we propose embodied and situated agent models based on distributed adaptive control theory to predict actions of other agents in five different game theoretic tasks (Harmony Game, Hawk-Dove, Stag-Hunt, Prisoner's Dilemma and Battle of the Exes). Our multi-layer control models implement top-down predictions from adaptive to reactive layers of control and bottom-up error feedback from reactive to adaptive layers. We test cooperative and competitive strategies among seven different agent models (cooperative, greedy, tit-for-tat, reinforcement-based, rational, predictive and other's-model agents). We show that, compared to pure reinforcement-based strategies, probabilistic learning agents modeled on rational, predictive and other's-model phenotypes perform better in game-theoretic metrics across tasks. Our autonomous multi-agent models capture systems-level processes underlying a ToM and highlight architectural principles of ToM from a control-theoretic perspective.
Private Hypothesis Selection

We provide a differentially private algorithm for hypothesis selection. Given samples from an unknown probability distribution P and a set of m probability distributions <inline-formula> <tex-math notation="LaTeX">$\mathcal {H}$ </tex-math></inline-formula>, the goal is to output, in a <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula>-differentially private manner, a distribution from <inline-formula> <tex-math notation="LaTeX">$\mathcal {H}$ </tex-math></inline-formula> whose total variation distance to P is comparable to that of the best such distribution (which we denote by <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>). The sample complexity of our basic algorithm is <inline-formula> <tex-math notation="LaTeX">$\text {O}\left ({\frac {\log \text {m}}{\alpha ^{2}} + \frac {\log \text {m}}{\alpha \varepsilon }}\right)$ </tex-math></inline-formula>, representing a minimal cost for privacy when compared to the non-private algorithm. We also can handle infinite hypothesis classes <inline-formula> <tex-math notation="LaTeX">$\mathcal {H}$ </tex-math></inline-formula> by relaxing to <inline-formula> <tex-math notation="LaTeX">$(\varepsilon,\delta)$ </tex-math></inline-formula>-differential privacy. We apply our hypothesis selection algorithm to give learning algorithms for a number of natural distribution classes, including Gaussians, product distributions, sums of independent random variables, piecewise polynomials, and mixture classes. Our hypothesis selection procedure allows us to generically convert a cover for a class to a learning algorithm, complementing known learning lower bounds which are in terms of the size of the packing number of the class. As the covering and packing numbers are often closely related, for constant <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>, our algorithms achieve the optimal sample complexity for many classes of interest. Finally, we describe an application to private distribution-free PAC learning.
horizon-AGN virtual observatory – 2. Template-free estimates of galaxy properties from colours

Using the horizon-AGN hydrodynamical simulation and self-organizing maps (SOMs), we show how to compress the complex, high-dimensional data structure of a simulation into a 2D grid, which greatly facilitates the analysis of how galaxy observables are connected to intrinsic properties. We first verify the tight correlation between the observed 0.3–5 μm broad-band colours of horizon-AGN galaxies and their high-resolution spectra. The correlation is found to extend to physical properties such as redshift, stellar mass, and star formation rate (SFR). This direct mapping from colour to physical parameter space still works after including photometric uncertainties that mimic the COSMOS survey. We then label the SOM grid with a simulated calibration sample to estimate redshift and SFR for COSMOS-like galaxies up to z ∼ 3. In comparison to state-of-the-art techniques based on synthetic templates, our method is comparable in performance but less biased at estimating redshifts, and significantly better at predicting SFRs. In particular, our ‘data-driven’ approach, in contrast to model libraries, intrinsically allows for the complexity of galaxy formation and can handle sample biases. We advocate that observations to calibrate this method should be one of the goals of next-generation galaxy surveys.
Ultra-diffuse galaxies without dark matter

I develop a high-velocity galaxy collision model to explain a rare but puzzling phenomenon, namely the apparent existence of ultra-diffuse galaxies with little dark matter. Predictions include simultaneous triggering of overpressured dense clouds to form luminous old globular clusters, a protogroup environment to generate high relative velocities of the initially gas-rich galaxies in the early universe, and spatially separated dark haloes, possibly detectable via gravitational lensing and containing relic low-metallicity stars with enhanced α/Fe at ultra-low surface brightness.
Enhanced cluster lensing models with measured galaxy kinematics

We present an improved determination of the total mass distribution of three massive clusters from the Cluster Lensing and Supernova Survey with Hubble and Hubble Frontier Fields, MACS J1206.2−0847 (z = 0.44), MACS J0416.1−2403 (z = 0.40), Abell S1063 (z = 0.35). We specifically reconstructed the sub-halo mass component with robust stellar kinematics information of cluster galaxies, in combination with precise strong lensing models based on large samples of spectroscopically identified multiple images. We used integral-field spectroscopy in the cluster cores, from the Multi Unit Spectroscopic Explorer on the Very Large Telescope, to measure the stellar velocity dispersion, σ, of 40−60 member galaxies per cluster, covering four to five magnitudes to mF160W ≃ 21.5. We verified the robustness and quantified the accuracy of the velocity dispersion measurements with extensive spectral simulations. With these data, we determined the normalization and slope of the galaxy L–σ Faber–Jackson relation in each cluster and used these parameters as a prior for the scaling relations of the sub-halo population in the mass distribution modeling. When compared to our previous lens models, the inclusion of member galaxies’ kinematics provides a similar precision in reproducing the positions of the multiple images. However, the inherent degeneracy between the central effective velocity dispersion, σ0, and truncation radius, rcut, of sub-halos is strongly reduced, thus significantly alleviating possible systematics in the measurements of sub-halo masses. The three independent determinations of the σ0 − rcut scaling relation in each cluster are found to be fully consistent, enabling a statistical determination of sub-halo sizes as a function of σ0, or halo masses. Finally, we derived the galaxy central velocity dispersion functions of the three clusters projected within 16% of their virial radius, finding that they are well in agreement with each other. We argue that such a methodology, when applied to high-quality kinematics and strong lensing data, allows the sub-halo mass functions to be determined and compared with those obtained from cosmological simulations.
SDN-based In-network Honeypot: Preemptively Disrupt and Mislead Attacks in IoT Networks

Detecting cyber attacks in the network environments used by Internet-of-things (IoT) and preventing them from causing physical perturbations play an important role in delivering dependable services. To achieve this goal, we propose in-network Honeypot based on Software-Defined Networking (SDN) to disrupt and mislead adversaries into exposures while they are in an early stage of preparing an attack. Different from traditional Honeypot requiring dedicated hardware setup, the in-network Honeypot directly reroutes traffic from suspicious nodes and intelligently spoofs the network traffic to them by adding misleading information into normal traffic. Preliminary evaluations on real networks demonstrate that the in-network Honeypot can have little impacts on the performance of IoT networks.
Large Scale Incremental Learning

Modern machine learning suffers from \textit{catastrophic forgetting} when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to \textbf{scale up to a large number of classes}. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1\% and 13.2\% respectively.
Using Metrics Suites to Improve the Measurement of Privacy in Graphs

Social graphs are widely used in research (e.g., epidemiology) and business (e.g., recommender systems). However, sharing these graphs poses privacy risks because they contain sensitive information about individuals. Graph anonymization techniques aim to protect individual users in a graph, while graph de-anonymization aims to re-identify users. The effectiveness of anonymization and de-anonymization algorithms is usually evaluated with privacy metrics. However, it is unclear how strong existing privacy metrics are when they are used in graph privacy. In this article, we study 26 privacy metrics for graph anonymization and de-anonymization and evaluate their strength in terms of three criteria: monotonicity indicates whether the metric indicates lower privacy for stronger adversaries; for within-scenario comparisons, evenness indicates whether metric values are spread evenly; and for between-scenario comparisons, shared value range indicates whether metrics use a consistent value range across scenarios. Our extensive experiments indicate that no single metric fulfills all three criteria perfectly. We therefore use methods from multi-criteria decision analysis to aggregate multiple metrics in a metrics suite, and we show that these metrics suites improve monotonicity compared to the best individual metric. This important result enables more monotonic, and thus more accurate, evaluations of new graph anonymization and de-anonymization algorithms.
General Dynamic Neural Networks for explainable PID parameter tuning in control engineering: An extensive comparison

Automation, the ability to run processes without human supervision, is one of the most important drivers of increased scalability and productivity. Modern automation largely relies on forms of closed loop control, wherein a controller interacts with a controlled process via actions, based on observations. Despite an increase in the use of machine learning for process control, most deployed controllers still are linear Proportional-Integral-Derivative (PID) controllers. PID controllers perform well on linear and near-linear systems but are not robust enough for more complex processes. As a main contribution of this paper, we examine the utility of extending standard PID controllers with General Dynamic Neural Networks (GDNN); we show that GDNN (neural) PID controllers perform well on a range of control systems and highlight what is needed to make them a stable, scalable, and interpretable option for control. To do so, we provide a comprehensive study using four different benchmark processes. All control environments are evaluated with and without noise as well as with and without disturbances. The neural PID controller performs better than standard PID control in 15 of 16 tasks and better than model-based control in 13 of 16 tasks. As a second contribution of this work, we address the Achilles heel that prevents neural networks from being used in real-world control processes so far: lack of interpretability. We use bounded-input bounded-output stability analysis to evaluate the parameters suggested by the neural network, thus making them understandable for human engineers. This combination of rigorous evaluation paired with better explainability is an important step towards the acceptance of neural-network-based control approaches for real-world systems. It is furthermore an important step towards explainable and safe applied artificial intelligence.
Quantum matrix diagonalization visualized

We show how to visualize the process of diagonalizing the Hamiltonian matrix to find the energy eigenvalues and eigenvectors of a generic one-dimensional quantum system. Starting in the familiar sine-wave basis of an embedding infinite square well, we display the Hamiltonian matrix graphically with the basis functions alongside. Each step in the diagonalization process consists of selecting a nonzero off-diagonal matrix element, then rotating the two corresponding basis vectors in their own subspace until this element is zero. We provide Mathematica code to display the effects of these rotations on both the matrix and the basis functions. As an electronic supplement we also provide a JavaScript web app to interactively carry out this process.
Parallel Algorithm for Non-Monotone DR-Submodular Maximization

In this work, we give a new parallel algorithm for the problem of maximizing a non-monotone diminishing returns submodular function subject to a cardinality constraint. For any desired accuracy $\epsilon$, our algorithm achieves a $1/e - \epsilon$ approximation using $O(\log{n} \log(1/\epsilon) / \epsilon^3)$ parallel rounds of function evaluations. The approximation guarantee nearly matches the best approximation guarantee known for the problem in the sequential setting and the number of parallel rounds is nearly-optimal for any constant $\epsilon$. Previous algorithms achieve worse approximation guarantees using $\Omega(\log^2{n})$ parallel rounds. Our experimental evaluation suggests that our algorithm obtains solutions whose objective value nearly matches the value obtained by the state of the art sequential algorithms, and it outperforms previous parallel algorithms in number of parallel rounds, iterations, and solution quality.
Efficient Covariance Estimation from Temporal Data

Estimating the covariance structure of multivariate time series is a fundamental problem with a wide-range of real-world applications -- from financial modeling to fMRI analysis. Despite significant recent advances, current state-of-the-art methods are still severely limited in terms of scalability, and do not work well in high-dimensional undersampled regimes. In this work we propose a novel method called Temporal Correlation Explanation, or T-CorEx, that (a) has linear time and memory complexity with respect to the number of variables, and can scale to very large temporal datasets that are not tractable with existing methods; (b) gives state-of-the-art results in highly undersampled regimes on both synthetic and real-world datasets; and (c) makes minimal assumptions about the character of the dynamics of the system. T-CorEx optimizes an information-theoretic objective function to learn a latent factor graphical model for each time period and applies two regularization techniques to induce temporal consistency of estimates. We perform extensive evaluation of T-Corex using both synthetic and real-world data and demonstrate that it can be used for detecting sudden changes in the underlying covariance matrix, capturing transient correlations and analyzing extremely high-dimensional complex multivariate time series such as high-resolution fMRI data.
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.
Labor Market Outcomes and Early Schooling: Evidence from School Entry Policies Using Exact Date of Birth

We use a rich, census-like Brazilian dataset containing information on spatial mobility, schooling, and income in which we can link children to parents to assess the impact of early education on several labor market outcomes. Brazilian public primary schools admit children up to one year younger than the national minimum age to enter school if their birthday is before an arbitrary threshold, causing an exogenous variation in schooling at adulthood. Using a Regression Discontinuity Design, we estimate one additional year of schooling increases labor income in 25.8% - almost twice as large as estimated using mincerian models. Around this cutoff there is also a gap of 9.6% on the probability of holding a college degree in adulthood, with which we estimate the college premium and find a 201% increase in labor income. We test the robustness of our estimates using placebo variables, alternative model specifcations and McCrary Density Tests.
Sum-of-squares meets square loss: Fast rates for agnostic tensor completion

We study tensor completion in the agnostic setting. In the classical tensor completion problem, we receive $n$ entries of an unknown rank-$r$ tensor and wish to exactly complete the remaining entries. In agnostic tensor completion, we make no assumption on the rank of the unknown tensor, but attempt to predict unknown entries as well as the best rank-$r$ tensor. 
For agnostic learning of third-order tensors with the square loss, we give the first polynomial time algorithm that obtains a "fast" (i.e., $O(1/n)$-type) rate improving over the rate obtained by reduction to matrix completion. Our prediction error rate to compete with the best $d\times{}d\times{}d$ tensor of rank-$r$ is $\tilde{O}(r^{2}d^{3/2}/n)$. We also obtain an exact oracle inequality that trades off estimation and approximation error. 
Our algorithm is based on the degree-six sum-of-squares relaxation of the tensor nuclear norm. The key feature of our analysis is to show that a certain characterization for the subgradient of the tensor nuclear norm can be encoded in the sum-of-squares proof system. This unlocks the standard toolbox for localization of empirical processes under the square loss, and allows us to establish restricted eigenvalue-type guarantees for various tensor regression models, with tensor completion as a special case. The new analysis of the relaxation complements Barak and Moitra (2016), who gave slow rates for agnostic tensor completion, and Potechin and Steurer (2017), who gave exact recovery guarantees for the noiseless setting. Our techniques are user-friendly, and we anticipate that they will find use elsewhere.
Identifying Classes Susceptible to Adversarial Attacks

Despite numerous attempts to defend deep learning based image classifiers, they remain susceptible to the adversarial attacks. This paper proposes a technique to identify susceptible classes, those classes that are more easily subverted. To identify the susceptible classes we use distance-based measures and apply them on a trained model. Based on the distance among original classes, we create mapping among original classes and adversarial classes that helps to reduce the randomness of a model to a significant amount in an adversarial setting. We analyze the high dimensional geometry among the feature classes and identify the k most susceptible target classes in an adversarial attack. We conduct experiments using MNIST, Fashion MNIST, CIFAR-10 (ImageNet and ResNet-32) datasets. Finally, we evaluate our techniques in order to determine which distance-based measure works best and how the randomness of a model changes with perturbation.