nils-herrmann/clean_scirepeval huggingface dataset:
Deep ReLU network approximation of functions on a manifold

Whereas recovery of the manifold from data is a well-studied topic, approximation rates for functions defined on manifolds are less known. In this work, we study a regression problem with inputs on a $d^*$-dimensional manifold that is embedded into a space with potentially much larger ambient dimension. It is shown that sparsely connected deep ReLU networks can approximate a H\"older function with smoothness index $\beta$ up to error $\epsilon$ using of the order of $\epsilon^{-d^*/\beta}\log(1/\epsilon)$ many non-zero network parameters. As an application, we derive statistical convergence rates for the estimator minimizing the empirical risk over all possible choices of bounded network parameters.
Calibrating the Learning Rate for Adaptive Gradient Methods to Improve Generalization Performance

Although adaptive gradient methods (AGMs) have fast speed in training deep neural networks, it is known to generalize worse than the stochastic gradient descent (SGD) or SGD with momentum (S-Momentum). Many works have attempted to modify AGMs so to close the gap in generalization performance between AGMs and S-Momentum, but they do not answer why there is such a gap. We identify that the anisotropic scale of the adaptive learning rate (A-LR) used by AGMs contributes to the generalization performance gap, and all existing modified AGMs actually represent efforts in revising the A-LR. Because the A-LR varies significantly across the dimensions of the problem over the optimization epochs (i.e., anisotropic scale), we propose a new AGM by calibrating the A-LR with a {\em softplus} function, resulting in the \textsc{Sadam} and \textsc{SAMSGrad} methods\footnote{Code is available at this https URL.}. These methods have better chance to not trap at sharp local minimizers, which helps them resume the dips in the generalization error curve observed with SGD and S-Momentum. We further provide a new way to analyze the convergence of AGMs (e.g., \textsc{Adam}, \textsc{Sadam}, and \textsc{SAMSGrad}) under the nonconvex, non-strongly convex, and Polyak-{\L}ojasiewicz conditions. We prove that the convergence rate of ADAM also depends on its hyper-parameter epsilon, which has been overlooked in prior convergence analysis. Empirical studies support our observation of the anisotropic A-LR and show that the proposed methods outperform existing AGMs and generalize even better than S-Momentum in multiple deep learning tasks.
Scale Matters: Temporal Scale Aggregation Network For Precise Action Localization In Untrimmed Videos

Temporal action localization is a recently-emerging task, aiming to localize video segments from untrimmed videos which contain specific actions. This work proposes a novel integrated temporal scale aggregation network (TSA-Net). Our main insight is that ensembling convolution filters with different dilation rates can effectively enlarge the receptive field with low computational cost, which inspires us to devise multi-dilation temporal convolution (MDC) block. Furthermore, to tackle video action instances with different durations, TSA-Net consists of multiple branches of sub-networks. Each of them adopts stacked MDC blocks with different dilation parameters, accomplishing a temporal receptive field specially optimized for specific-duration actions. We follow the formulation of boundary point detection, novelly detecting three kinds of critical points (i.e., starting / mid-point / ending) and pairing them for proposal generation. Comprehensive evaluations are conducted on THUMOS14. Our proposed TSA-Net demonstrates clear and consistent better performances and recalibrates new state-of-the-art on THUMOS14 benchmark.
L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention

Auto-encoder is an important architecture to understand point clouds in an encoding and decoding procedure of self reconstruction. Current auto-encoder mainly focuses on the learning of global structure by global shape reconstruction, while ignoring the learning of local structures. To resolve this issue, we propose Local-to-Global auto-encoder (L2G-AE) to simultaneously learn the local and global structure of point clouds by local to global reconstruction. Specifically, L2G-AE employs an encoder to encode the geometry information of multiple scales in a local region at the same time. In addition, we introduce a novel hierarchical self-attention mechanism to highlight the important points, scales and regions at different levels in the information aggregation of the encoder. Simultaneously, L2G-AE employs a recurrent neural network (RNN) as decoder to reconstruct a sequence of scales in a local region, based on which the global point cloud is incrementally reconstructed. Our outperforming results in shape classification, retrieval and upsampling show that L2G-AE can understand point clouds better than state-of-the-art methods.
Combining learned skills and reinforcement learning for robotic manipulations

Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision. The supervised approach of imitation learning can handle short tasks but suffers from compounding errors and the need of many demonstrations for longer and more complex tasks. Reinforcement learning (RL) can find solutions beyond demonstrations but requires tedious and task-specific reward engineering for multi-step problems. In this work we address the difficulties of both methods and explore their combination. To this end, we propose a RL policies operating on pre-trained skills, that can learn composite manipulations using no intermediate rewards and no demonstrations of full tasks. We also propose an efficient training of basic skills from few synthetic demonstrated trajectories by exploring recent CNN architectures and data augmentation. We show successful learning of policies for composite manipulation tasks such as making a simple breakfast. Notably, our method achieves high success rates on a real robot, while using synthetic training data only.
Universal Transforming Geometric Network

The recurrent geometric network (RGN), the first end-to-end differentiable neural architecture for protein structure prediction, is a competitive alternative to existing models. However, the RGN's use of recurrent neural networks (RNNs) as internal representations results in long training time and unstable gradients. And because of its sequential nature, it is less effective at learning global dependencies among amino acids than existing transformer architectures. We propose the Universal Transforming Geometric Network (UTGN), an end-to-end differentiable model that uses the encoder portion of the Universal Transformer architecture as an alternative for internal representations. Our experiments show that compared to RGN, UTGN achieve a $1.7$ \si{\angstrom} improvement on the free modeling portion and a $0.7$ \si{\angstrom} improvement on the template based modeling of the CASP12 competition.
Retrosynthesis with attention-based NMT model and chemical analysis of “wrong” predictions

We consider retrosynthesis to be a machine translation problem. Accordingly, we apply an attention-based and completely data-driven model named Tensor2Tensor to a data set comprising approximately 50 000 diverse reactions extracted from the United States patent literature. The model significantly outperforms the seq2seq model (37.4%), with top-1 accuracy reaching 54.1%. We also offer a novel insight into the causes of grammatically invalid SMILES, and conduct a test in which experienced chemists select and analyze the “wrong” predictions that may be chemically plausible but differ from the ground truth. The effectiveness of our model is found to be underestimated and the “true” top-1 accuracy reaches as high as 64.6%.
Efficient computation of counterfactual explanations of LVQ models

With the increasing use of machine learning in practice and because of legal regulations like EU's GDPR, it becomes indispensable to be able to explain the prediction and behavior of machine learning models. An example of easy to understand explanations of AI models are counterfactual explanations. However, for many models it is still an open research problem how to efficiently compute counterfactual explanations. 
We investigate how to efficiently compute counterfactual explanations of learning vector quantization models. In particular, we propose different types of convex and non-convex programs depending on the used learning vector quantization model.
Towards Learning How to Properly Play UNO with the iCub Robot

While interacting with another person, our reactions and behavior are much affected by the emotional changes within the temporal context of the interaction. Our intrinsic affective appraisal comprising perception, self-assessment, and the affective memories with similar social experiences will drive specific, and in most cases addressed as proper, reactions within the interaction. This paper proposes the roadmap for the development of multimodal research which aims to empower a robot with the capability to provide proper social responses in a Human-Robot Interaction (HRI) scenario.
Impact of substructure on local dark matter searches

Dark matter substructure can contribute significantly to local dark matter searches and may provide a large uncertainty in the interpretation of those experiments. For direct detection experiments, sub-halos give rise to an additional dark matter component on top of the smooth dark matter distribution of the host halo. In the case of dark matter capture in the Sun, sub-halo encounters temporarily increase the number of captured particles. Even if the encounter happened in the past, the number of dark matter particles captured by the Sun can still be enhanced today compared to expectations from the host halo as those enhancements decay over time. Using results from an analytical model of the sub-halo population of a Milky Way-like galaxy, valid for sub-halo masses between 10−5 M⊙ and 1011 M⊙, we assess the impact of sub-halos on direct dark matter searches in a probabilistic way. We find that the impact on direct detection can be sizable, with a probability of ∼ 10−3 to find an 𝒪(1) enhancement of the recoil rate. In the case of the capture rate in the Sun, we find that 𝒪(1) enhancements are very unlikely, with probability ≲ 10−5, and are even impossible for some dark matter masses.
A Visual Technique to Analyze Flow of Information in a Machine Learning System

Machine learning (ML) algorithms and machine learning based software systems implicitly or explicitly involve complex flow of information between various entities such as training data, feature space, validation set and results. Understanding the statistical distribution of such information and how they flow from one entity to another influence the operation and correctness of such systems, especially in large-scale applications that perform classification or prediction in real time. In this paper, we propose a visual approach to understand and analyze flow of information during model training and serving phases. We build the visualizations using a technique called Sankey Diagram - conventionally used to understand data flow among sets - to address various use cases of in a machine learning system. We demonstrate how the proposed technique, tweaked and twisted to suit a classification problem, can play a critical role in better understanding of the training data, the features, and the classifier performance. We also discuss how this technique enables diagnostic analysis of model predictions and comparative analysis of predictions from multiple classifiers. The proposed concept is illustrated with the example of categorization of millions of products in the e-commerce domain - a multi-class hierarchical classification problem.
Network with Sub-networks: Layer-wise Detachable Neural Network

We introduce network with sub-networks, a neural network which it’s weight layers could be detached into subneural networks during inference. To develop weights and biases which could be inserted in both base and subneural networks, firstly, the parameters are copied from sub-model to base-model. Each model is forwardpropagated separately. Gradients from a pair of networks are averaged and, used to update both networks. Our base model achieves the test-accuracy which is comparable to the regularly trained models, while the model maintains the ability to detach weight layers.
Independent Double Roman Domination on Block Graphs

Given a graph $G=(V,E)$, $f:V \rightarrow \{0,1,2 \}$ is the Italian dominating function of $G$ if $f$ satisfies $\sum_{u \in N(v)}f(u) \geq 2$ when $f(v)=0$. Denote $w(f)=\sum_{v \in V}f(v)$ as the weight of $f$. Let $V_i=\{v:f(v)=i\},i=0,1,2$, we call $f$ the independent Italian dominating function if $V_1 \cup V_2$ is an independent set. The independent Italian domination number of $G$ is the minimum weight of independent Italian dominating function $f$, denoted by $i_{I}(G)$. We equivalently transform the independent domination problem of the connected block graph $G$ to the induced independent domination problem of its block-cutpoint graph $T$, then a linear time algorithm is given to find $i_{I}(G)$ of any connected block graph $G$ based on dynamic programming.
Time-evolution of nonlinear optomechanical systems: interplay of mechanical squeezing and non-Gaussianity

We solve the time evolution of a nonlinear optomechanical Hamiltonian with arbitrary time-dependent mechanical displacement, mechanical single-mode squeezing and a time-dependent optomechanical coupling up to the solution of two second-order differential equations. The solution is based on identifying a minimal and finite Lie algebra that generates the time-evolution of the system. This reduces the problem to considering a finite set of coupled ordinary differential equations of real functions. To demonstrate the applicability of our method, we compute the degree of non-Gaussianity of the time-evolved state of the system by means of a measure based on the relative entropy of the non-Gaussian state and its closest Gaussian reference state. We find that the addition of a constant mechanical squeezing term to the standard optomechanical Hamiltonian generally decreases the overall non-Gaussian character of the state. For sinusoidally modulated squeezing, the two second-order differential equations mentioned above take the form of the Mathieu equation. We derive perturbative solutions for a small squeezing amplitude at parametric resonance and show that they correspond to the rotating-wave approximation at times larger than the scale set by the mechanical frequency. We find that the non-Gaussianity of the state increases with both time and the squeezing parameter in this specific regime.
Uncertainty Quantification in Computer-Aided Diagnosis: Make Your Model say "I don't know" for Ambiguous Cases

We evaluate two different methods for the integration of prediction uncertainty into diagnostic image classifiers to increase patient safety in deep learning. In the first method, Monte Carlo sampling is applied with dropout at test time to get a posterior distribution of the class labels (Bayesian ResNet). The second method extends ResNet to a probabilistic approach by predicting the parameters of the posterior distribution and sampling the final result from it (Variational ResNet).The variance of the posterior is used as metric for uncertainty.Both methods are trained on a data set of optical coherence tomography scans showing four different retinal conditions. Our results shown that cases in which the classifier predicts incorrectly correlate with a higher uncertainty. Mean uncertainty of incorrectly diagnosed cases was between 4.6 and 8.1 times higher than mean uncertainty of correctly diagnosed cases. Modeling of the prediction uncertainty in computer-aided diagnosis with deep learning yields more reliable results and is anticipated to increase patient safety.
A Full and Synthetic Model for Asset-Liability Management in Life Insurance, and Analysis of the SCR With the Standard Formula

The aim of this paper is to introduce a synthetic ALM model that catches the main specificity of life insurance contracts. First, it keeps track of both market and book values to apply the regulatory profit sharing rule. Second, it introduces a determination of the crediting rate to policyholders that is close to the practice and is a trade-off between the regulatory rate, a competitor rate and the available profits. Third, it considers an investment in bonds that enables to match a part of the cash outflow due to surrenders, while avoiding to store the trading history. We use this model to evaluate the Solvency Capital Requirement (SCR) with the standard formula, and show that the choice of the interest rate model is important to get a meaningful model after the regulatory shocks on the interest rate. We discuss the different values of the SCR modules first in a framework with moderate interest rates using the shocks of the present legislation, and then we consider a low interest framework with the latest recommandation of the EIOPA on the shocks. In both cases, we illustrate the importance of matching cash-flows and its impact on the SCR.
Large finite products of small fractions

Fix positive reals $a,b,c,d$, and let $h(x)$ be a real function behaving sort of like $\sin x$ near 0. Then, provided $m$ grows linearly with $n$. there exists a positive constant $C$ such that$$ 
\prod_{j=0}^m\frac{h\left((cj+a)\frac{d}{n}\right)}{h\left((cj+b)\frac{d}{n}\right)}\sim C n^{\frac{a-b}c}. 
$$
Voter model on networks partitioned into two cliques of arbitrary sizes

The voter model is an archetypal stochastic process that represents opinion dynamics. In each update, one agent is chosen uniformly at random. The selected agent then copies the current opinion of a randomly selected neighbour. We investigate the voter model on a network with an exogenous community structure: two cliques (i.e. complete subgraphs) randomly linked by X interclique edges. We show that, counterintuitively, the mean consensus time is typically not a monotonically decreasing function of X. Cliques of fixed proportions with opposite initial opinions reach a consensus, on average, most quickly if X scales as N3/2, where N is the number of agents in the network. Hence, to accelerate a consensus between cliques, agents should connect to more members in the other clique as N increases but not to the extent that cliques lose their identity as distinct communities. We support our numerical results with an equation-based analysis. By interpolating between two asymptotic heterogeneous mean-field approximations, we obtain an equation for the mean consensus time that is in excellent agreement with simulations for all values of X.
Distilling Knowledge From a Deep Pose Regressor Network

This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.
Scaling-up Simulations of Diffusion in Microporous Materials.

We introduce and demonstrate the coarse-graining of static and dynamical properties of host-guest systems constituted by methane in two different microporous materials. The reference systems are mapped to occupancy-based pore-scale lattice models. Each coarse-grained model is equipped with an appropriate coarse-grained potential and a local dynamical operator, which represents the probability of inter-pore molecular jumps between different cages. Both the coarse-grained thermodynamics and dynamics are defined based on small-scale atomistic simulations of the reference systems. We considered two host materials: the widely-studied ITQ-29 zeolite and the LTA-zeolite-templated carbon, which was recently theorized. Our method allows representing with satisfactory accuracy and a considerably reduced computational effort the reference systems while providing new interesting physical insights in terms of static and diffusive properties.
MarmoNet: a pipeline for automated projection mapping of the common marmoset brain from whole-brain serial two-photon tomography

Understanding the connectivity in the brain is an important prerequisite for understanding how the brain processes information. In the Brain/MINDS project, a connectivity study on marmoset brains uses two-photon microscopy fluorescence images of axonal projections to collect the neuron connectivity from defined brain regions at the mesoscopic scale. The processing of the images requires the detection and segmentation of the axonal tracer signal. The objective is to detect as much tracer signal as possible while not misclassifying other background structures as the signal. This can be challenging because of imaging noise, a cluttered image background, distortions or varying image contrast cause problems. 
We are developing MarmoNet, a pipeline that processes and analyzes tracer image data of the common marmoset brain. The pipeline incorporates state-of-the-art machine learning techniques based on artificial convolutional neural networks (CNN) and image registration techniques to extract and map all relevant information in a robust manner. The pipeline processes new images in a fully automated way. 
This report introduces the current state of the tracer signal analysis part of the pipeline.
Bayesian Persuasion with Sequential Games

We study an information-structure design problem (a.k.a. persuasion) with a single sender and multiple receivers with actions of a priori unknown types, independently drawn from action-specific marginal distributions. As in the standard Bayesian persuasion model, the sender has access to additional information regarding the action types, which she can exploit when committing to a (noisy) signaling scheme through which she sends a private signal to each receiver. The novelty of our model is in considering the case where the receivers interact in a sequential game with imperfect information, with utilities depending on the game outcome and the realized action types. After formalizing the notions of ex ante and ex interim persuasiveness (which differ in the time at which the receivers commit to following the sender's signaling scheme), we investigate the continuous optimization problem of computing a signaling scheme which maximizes the sender's expected revenue. We show that computing an optimal ex ante persuasive signaling scheme is NP-hard when there are three or more receivers. In contrast with previous hardness results for ex interim persuasion, we show that, for games with two receivers, an optimal ex ante persuasive signaling scheme can be computed in polynomial time thanks to a novel algorithm based on the ellipsoid method which we propose.