nils-herrmann/clean_scirepeval huggingface dataset:
Learning Activation Functions: A new paradigm of understanding Neural Networks

The scope of research in the domain of activation functions remains limited and centered around improving the ease of optimization or generalization quality of neural networks (NNs). However, to develop a deeper understanding of deep learning, it becomes important to look at the non linear component of NNs more carefully. In this paper, we aim to provide a generic form of activation function along with appropriate mathematical grounding so as to allow for insights into the working of NNs in future. We propose ``Self-Learnable Activation Functions'' (SLAF), which are learned during training and are capable of approximating most of the existing activation functions. SLAF is given as a weighted sum of pre-defined basis elements which can serve for a good approximation of the optimal activation function. The coefficients for these basis elements allow a search in the entire space of continuous functions (consisting of all the conventional activations). We propose various training routines which can be used to achieve performance with SLAF equipped neural networks (SLNNs). We prove that SLNNs can approximate any neural network with lipschitz continuous activations, to any arbitrary error highlighting their capacity and possible equivalence with standard NNs. Also, SLNNs can be completely represented as a collections of finite degree polynomial upto the very last layer obviating several hyper parameters like width and depth. Since the optimization of SLNNs is still a challenge, we show that using SLAF along with standard activations (like ReLU) can provide performance improvements with only a small increase in number of parameters.
Confidence Calibration for Convolutional Neural Networks Using Structured Dropout

In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dropout. Motivated by this, we explore the use of structured dropout to promote model diversity and improve confidence calibration. We use the SVHN, CIFAR-10 and CIFAR-100 datasets to empirically compare model diversity and confidence errors obtained using various dropout techniques. We also show the merit of structured dropout in a Bayesian active learning application.
Dark Sector Physics at the Belle II Experiment.

The Belle II experiment at the SuperKEKB energy-asymmetric $e^+ e^-$ collider is a substantial upgrade of the B factory facility at the Japanese KEK laboratory. The design luminosity of the machine is $8\times 10^{35}$ cm$^{-2}$s$^{-1}$ and the Belle II experiment aims to record 50 ab$^{-1}$ of data, a factor of 50 more than its predecessor. From February to July 2018, the machine has completed a commissioning run, achieved a peak luminosity of $5.5\times 10^{33}$ cm$^{-2}$s$^{-1}$, and Belle II has recorded a data sample of about 0.5 fb$^{-1}$. Main operation of SuperKEKB has started in March 2019. Already this early data set with specifically designed triggers offers the possibility to search for a large variety of dark sector particles in the GeV mass range complementary to the Large Hadron Collider (LHC) and dedicated low energy experiments; these searches will benefit from more data in the process of being accumulated. This talk will review the state of the dark sector searches at Belle II with a focus on the discovery potential of the early data, and show the first results.
Semi-Supervised Learning for Cancer Detection of Lymph Node Metastases

Pathologists find tedious to examine the status of the sentinel lymph node on a large number of pathological scans. The examination process of such lymph node which encompasses metastasized cancer cells is histopathologically organized. However, the task of finding metastatic tissues is gradual which is often challenging. In this work, we present our deep convolutional neural network based model validated on PatchCamelyon (PCam) benchmark dataset for fundamental machine learning research in histopathology diagnosis. We find that our proposed model trained with a semi-supervised learning approach by using pseudo labels on PCam-level significantly leads to better performances to strong CNN baseline on the AUC metric.
On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference

Our goal is for agents to optimize the right reward function, despite how difficult it is for us to specify what that is. Inverse Reinforcement Learning (IRL) enables us to infer reward functions from demonstrations, but it usually assumes that the expert is noisily optimal. Real people, on the other hand, often have systematic biases: risk-aversion, myopia, etc. One option is to try to characterize these biases and account for them explicitly during learning. But in the era of deep learning, a natural suggestion researchers make is to avoid mathematical models of human behavior that are fraught with specific assumptions, and instead use a purely data-driven approach. We decided to put this to the test -- rather than relying on assumptions about which specific bias the demonstrator has when planning, we instead learn the demonstrator's planning algorithm that they use to generate demonstrations, as a differentiable planner. Our exploration yielded mixed findings: on the one hand, learning the planner can lead to better reward inference than relying on the wrong assumption; on the other hand, this benefit is dwarfed by the loss we incur by going from an exact to a differentiable planner. This suggest that at least for the foreseeable future, agents need a middle ground between the flexibility of data-driven methods and the useful bias of known human biases. Code is available at this https URL.
Ranking Policy Gradient

Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency. Furthermore, the sample complexity of RPG does not depend on the dimension of state space, which enables RPG for large-scale problems. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models

We study several methods for full or partial sharing of the decoder parameters of multi-lingual NMT models. Using only the WMT 2019 shared task parallel datasets for training, we evaluate both fully supervised and zero-shot translation performance in 110 unique translation directions. We use additional test sets and re-purpose evaluation methods recently used for unsupervised MT in order to evaluate zero-shot translation performance for language pairs where no gold-standard parallel data is available. To our knowledge, this is the largest evaluation of multi-lingual translation yet conducted in terms of the total size of the training data we use, and in terms of the number of zero-shot translation pairs we evaluate. We conduct an in-depth evaluation of the translation performance of different models, highlighting the trade-offs between methods of sharing decoder parameters. We find that models which have task-specific decoder parameters outperform models where decoder parameters are fully shared across all tasks.
Singlet-doublet Dirac dark matter and neutrino masses

We examine an extension of the Standard Model that addresses the dark matter puzzle and generates Dirac neutrinos masses through the radiative seesaw mechanism. The new field content includes a scalar field that plays an important role in setting the relic abundance of dark matter. We analyze the phenomenology in the light of direct, indirect, and collider searches of dark matter. In this framework, the dark matter candidate is a Dirac particle that is a mixture of new singlet-doublet fields with mass $m_{\chi_1^0}\lesssim 1.1\,\text{TeV}$. We find that the allowed parameter space of this model is broader than the well-known Majorana dark matter scenario.
Quantum Teleportation in High Dimensions.

Quantum teleportation allows a "disembodied" transmission of unknown quantum states between distant quantum systems. Yet, all teleportation experiments to date were limited to a two-dimensional subspace of quantized multiple levels of the quantum systems. Here, we propose a scheme for teleportation of arbitrarily high-dimensional photonic quantum states and demonstrate an example of teleporting a qutrit. Measurements over a complete set of 12 qutrit states in mutually unbiased bases yield a teleportation fidelity of 0.75(1), which is well above both the optimal single-copy qutrit state-estimation limit of 1/2 and maximal qubit-qutrit overlap of 2/3, thus confirming a genuine and nonclassical three-dimensional teleportation. Our work will enable advanced quantum technologies in high dimensions, since teleportation plays a central role in quantum repeaters and quantum networks.
Social Contagion of Gift Exchange in Online Groups

Does receiving a gift encourage the recipient to send more gifts? Causal identification of behavioral contagion is very challenging, especially based on observational data from social networks. Given that the online monetary gift (also known as a "red packet") in WeChat groups is randomly split between group members, we are able to conduct a natural experiment to identify the causal effect of receiving online red packets on the contagion of gift exchange. Analyzing gifting behavior using a large-scale dataset of 3.4 million WeChat users, we find that recipients on average repay 18.29% of the amount they receive. Moreover, our analysis shows that recipients' contagion may be driven by different types of reciprocity. We further find that the "luckiest draw" recipients repay 1.5 times more than other recipients, and identify a group norm that luckiest draw recipients should send payment back to the group.
Limitations of zT as a figure of merit for nanostructured thermoelectric materials

A numerical study of thermoelectric nanocomposites is presented. Thermoelectric properties as a function of average grain size or nanoparticle density are examined by simulating the measurements as they would be done experimentally. In accordance with previous theory and experimental results, we find that the Seebeck coefficient, power factor and figure of merit, zT, can be increased by nanostructuring when energy barriers exist around the grain boundaries or embedded nanoparticles. When we simulate the performance of a thermoelectric cooler with the same material, however, we find that the maximum temperature difference is much less than expected from the given zT. The same nanocomposite without electrically active grain boundaries has a lower measured zT but a higher maximum cooling temperature. The physical reason for these results is explained. The results illustrate the limitations of zT as a figure of merit for nanocomposites with electrically active grain boundaries and suggest that the ability to enhance the electrical performance of thermoelectric materials by nanostructuring is limited.
On the Definition of Japanese Word

The annotation guidelines for Universal Dependencies (UD) stipulate that the basic units of dependency annotation are syntactic words, but it is not clear what are syntactic words in Japanese. Departing from the long tradition of using phrasal units called bunsetsu for dependency parsing, the current UD Japanese treebanks adopt the Short Unit Words. However, we argue that they are not syntactic word as specified by the annotation guidelines. Although we find non-mainstream attempts to linguistically define Japanese words, such definitions have never been applied to corpus annotation. We discuss the costs and benefits of adopting the rather unfamiliar criteria.
Decomposable Neural Paraphrase Generation

Paraphrasing exists at different granularity levels, such as lexical level, phrasal level and sentential level. This paper presents Decomposable Neural Paraphrase Generator (DNPG), a Transformer-based model that can learn and generate paraphrases of a sentence at different levels of granularity in a disentangled way. Specifically, the model is composed of multiple encoders and decoders with different structures, each of which corresponds to a specific granularity. The empirical study shows that the decomposition mechanism of DNPG makes paraphrase generation more interpretable and controllable. Based on DNPG, we further develop an unsupervised domain adaptation method for paraphrase generation. Experimental results show that the proposed model achieves competitive in-domain performance compared to state-of-the-art neural models, and significantly better performance when adapting to a new domain.
Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding

We propose a novel method for selecting coherent and diverse responses for a given dialogue context. The proposed method re-ranks response candidates generated from conversational models by using event causality relations between events in a dialogue history and response candidates (e.g., “be stressed out” precedes “relieve stress”). We use distributed event representation based on the Role Factored Tensor Model for a robust matching of event causality relations due to limited event causality knowledge of the system. Experimental results showed that the proposed method improved coherency and dialogue continuity of system responses.
Classification and Clustering of Arguments with Contextualized Word Embeddings

We experiment with two recent contextualized word embedding methods (ELMo and BERT) in the context of open-domain argument search. For the first time, we show how to leverage the power of contextualized word embeddings to classify and cluster topic-dependent arguments, achieving impressive results on both tasks and across multiple datasets. For argument classification, we improve the state-of-the-art for the UKP Sentential Argument Mining Corpus by 20.8 percentage points and for the IBM Debater - Evidence Sentences dataset by 7.4 percentage points. For the understudied task of argument clustering, we propose a pre-training step which improves by 7.8 percentage points over strong baselines on a novel dataset, and by 12.3 percentage points for the Argument Facet Similarity (AFS) Corpus.
Interactive Subspace Exploration on Generative Image Modelling

Generative image modeling techniques such as GAN demonstrate highly convincing image generation result. However, user interaction is often necessary to obtain the desired results. Existing attempts add interactivity but require either tailored architectures or extra data. We present a human-in-the-optimization method that allows users to directly explore and search the latent vector space of generative image modeling. Our system provides multiple candidates by sampling the latent vector space, and the user selects the best blending weights within the subspace using multiple sliders. In addition, the user can express their intention through image editing tools. The system samples latent vectors based on inputs and presents new candidates to the user iteratively. An advantage of our formulation is that one can apply our method to arbitrary pre-trained model without developing specialized architecture or data. We demonstrate our method with various generative image modeling applications, and show superior performance in a comparative user study with prior art iGAN \cite{iGAN2016}.
Multisensory cues facilitate coordination of stepping movements with a virtual reality avatar

The effectiveness of simple sensory cues for retraining gait have been demonstrated, yet the feasibility of humanoid avatars for entrainment have yet to be investigated. Here, we describe the development of a novel method of visually cued training, in the form of a virtual partner, and investigate its ability to provide movement guidance in the form of stepping. Real stepping movements were mapped onto an avatar using motion capture data. The trajectory of one of the avatar step cycles was then accelerated or decelerated by 15% to create a perturbation. Healthy participants were motion captured while instructed to step in time to the avatar's movements, as viewed through a virtual reality headset. Step onset times were used to measure the timing errors (asynchronies) between them. Participants completed either a visual-only condition, or auditory-visual with footstep sounds included. Participants' asynchronies exhibited slow drift in the Visual-Only condition, but became stable in the Auditory-Visual condition. Moreover, we observed a clear corrective response to the phase perturbation in both auditory-visual conditions. We conclude that an avatar's movements can be used to influence a person's own gait, but should include relevant auditory cues congruent with the movement to ensure a suitable accuracy is achieved.
TOI-150b and TOI-163b: two transiting hot Jupiters, one eccentric and one inflated, revealed by TESS near and at the edge of the JWST CVZ

We present the discovery of TYC9191-519-1b (TOI-150b, TIC 271893367) and HD271181b (TOI-163b, TIC 179317684), two hot Jupiters initially detected using 30-min cadence Transiting Exoplanet Survey Satellite (TESS) photometry from Sector 1 and thoroughly characterized through follow-up photometry (CHAT, Hazelwood, LCO/CTIO, El Sauce, TRAPPIST-S), high-resolution spectroscopy (FEROS, CORALIE), and speckle imaging (Gemini/DSSI), confirming the planetary nature of the two signals. A simultaneous joint fit of photometry and radial velocity using a new fitting package juliet reveals that TOI-150b is a $1.254\pm 0.016\ \rm {R}_ \rm{J}$, massive ($2.61^{+0.19}_{-0.12}\ \rm {M}_ \rm{J}$) hot Jupiter in a 5.857-d orbit, while TOI-163b is an inflated ($R_ \rm{P}$ = $1.478^{+0.022}_{-0.029} \,\mathrm{ R}_ \rm{J}$, $M_ \rm{P}$ = $1.219\pm 0.11 \, \rm{M}_ \rm{J}$) hot Jupiter on a P = 4.231-d orbit; both planets orbit F-type stars. A particularly interesting result is that TOI-150b shows an eccentric orbit ($e=0.262^{+0.045}_{-0.037}$), which is quite uncommon among hot Jupiters. We estimate that this is consistent, however, with the circularization time-scale, which is slightly larger than the age of the system. These two hot Jupiters are both prime candidates for further characterization – in particular, both are excellent candidates for determining spin-orbit alignments via the Rossiter–McLaughlin (RM) effect and for characterizing atmospheric thermal structures using secondary eclipse observations considering they are both located closely to the James Webb Space Telescope (JWST) Continuous Viewing Zone (CVZ).
Streaming Adaptation of Deep Forecasting Models using Adaptive Recurrent Units

We present ARU, an Adaptive Recurrent Unit for streaming adaptation of deep globally trained time-series forecasting models. The ARU combines the advantages of learning complex data transformations across multiple time series from deep global models, with per-series localization offered by closed-form linear models. Unlike existing methods of adaptation that are either memory-intensive or non-responsive after training, ARUs require only fixed sized state and adapt to streaming data via an easy RNN-like update operation. The core principle driving ARU is simple --- maintain sufficient statistics of conditional Gaussian distributions and use them to compute local parameters in closed form. Our contribution is in embedding such local linear models in globally trained deep models while allowing end-to-end training on the one hand, and easy RNN-like updates on the other. Across several datasets we show that ARU is more effective than recently proposed local adaptation methods that tax the global network to compute local parameters.
Dark matter as Planck relics without too exotic hypotheses

The idea that dark matter could be made of stable relics of microscopic black holes is not new. In this article, we revisit this hypothesis, focusing on the creation of black holes by the scattering of trans-planckian particles in the early universe. The only new physics required - and this can even be questioned - in this approach is an unusually high energy scale for inflation. We show that dark matter emerges naturally and we study the question of fine-tuning. We finally give some lines of thoughts for a possible detection.
DeepSTORM3D: dense three dimensional localization microscopy and point spread function design by deep learning

Localization microscopy is an imaging technique in which the positions of individual nanoscale point emitters (e.g. fluorescent molecules) are determined at high precision from their images. This is the key ingredient in single/multiple-particle-tracking and several super-resolution microscopy approaches. Localization in three-dimensions (3D) can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered using additional optical elements to vary distinctively with the depth of the point-source. However, localizing multiple adjacent emitters in 3D poses a significant algorithmic challenge, due to the lateral overlap of their PSFs. Here, we train a neural network to receive an image containing densely overlapping PSFs of multiple emitters over a large axial range and output a list of their 3D positions. Furthermore, we then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach numerically as well as experimentally by 3D STORM imaging of mitochondria, and volumetric imaging of dozens of fluorescently-labeled telomeres occupying a mammalian nucleus in a single snapshot.
Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming

We treat projective dependency trees as latent variables in our probabilistic model and induce them in such a way as to be beneficial for a downstream task, without relying on any direct tree supervision. Our approach relies on Gumbel perturbations and differentiable dynamic programming. Unlike previous approaches to latent tree learning, we stochastically sample global structures and our parser is fully differentiable. We illustrate its effectiveness on sentiment analysis and natural language inference tasks. We also study its properties on a synthetic structure induction task. Ablation studies emphasize the importance of both stochasticity and constraining latent structures to be projective trees.