nils-herrmann/clean_scirepeval huggingface dataset:
Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning

The recent success in deep learning has lead to various effective representation learning methods for videos. However, the current approaches for video representation require large amount of human labeled datasets for effective learning. We present an unsupervised representation learning framework to encode scene dynamics in videos captured from multiple viewpoints. The proposed framework has two main components: Representation Learning Network (RL-NET), which learns a representation with the help of Blending Network (BL-NET), and Video Rendering Network (VR-NET), which is used for video synthesis. The framework takes as input video clips from different viewpoints and time, learns an internal representation and uses this representation to render a video clip from an arbitrary given viewpoint and time. The ability of the proposed network to render video frames from arbitrary viewpoints and time enable it to learn a meaningful and robust representation of the scene dynamics. We demonstrate the effectiveness of the proposed method in rendering view-aware as well as time-aware video clips on two different real-world datasets including UCF-101 and NTU-RGB+D. To further validate the effectiveness of the learned representation, we use it for the task of view-invariant activity classification where we observe a significant improvement (~26%) in the performance on NTU-RGB+D dataset compared to the existing state-of-the art methods.
Combining High-Level Features of Raw Audio Waves and Mel-Spectrograms for Audio Tagging

In this paper, we describe our contribution to Task 2 of the DCASE 2018 Audio Challenge. While it has become ubiquitous to utilize an ensemble of machine learning methods for classification tasks to obtain better predictive performance, the majority of ensemble methods combine predictions rather than learned features. We propose a single-model method that combines learned high-level features computed from log-scaled mel-spectrograms and raw audio data. These features are learned separately by two Convolutional Neural Networks, one for each input type, and then combined by densely connected layers within a single network. This relatively simple approach along with data augmentation ranks among the best two percent in the Freesound General-Purpose Audio Tagging Challenge on Kaggle.
Evolving metric perturbations in dynamical Chern-Simons gravity

We present a well-posed constraint-preserving scheme for evolving first-order metric perturbations on an arbitrary background with arbitrary source. We use this scheme to evolve the leading-order metric perturbation in order-reduced dynamical Chern-Simons gravity (dCS) on a Kerr background. In particular we test the stability of stationary dCS data on a Kerr background with stationary first-order dCS scalar field source. We find that the leading-order metric perturbation numerically exhibits linear growth, but that the level of this growth converges to zero with numerical resolution. This analysis shows that spinning black holes in dCS gravity are numerically stable to leading-order perturbations in the metric.
IGNOR: Image-guided Neural Object Rendering

We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours \& sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.
Dissecting Latency in the Internet's Fiber Infrastructure

The recent publication of the `InterTubes' map of long-haul fiber-optic cables in the contiguous United States invites an exciting question: how much faster would the Internet be if routes were chosen to minimize latency? Previous measurement campaigns suggest the following rule of thumb for estimating Internet latency: multiply line-of-sight distance by 2.1, then divide by the speed of light in fiber. But a simple computation of shortest-path lengths through the conduits in the InterTubes map suggests that the conversion factor for all pairs of the 120 largest population centers in the U.S.\ could be reduced from 2.1 to 1.3, in the median, even using less than half of the links. To determine whether an overlay network could be used to provide shortest paths, and how well it would perform, we used the diverse server deployment of a CDN to measure latency across individual conduits. We were surprised to find, however, that latencies are sometimes much higher than would be predicted by conduit length alone. To understand why, we report findings from our analysis of network latency data from the backbones of two Tier-1 ISPs, two scientific and research networks, and the recently built fiber backbone of a CDN.
Mixture of Regression Experts in fMRI Encoding

fMRI semantic category understanding using linguistic encoding models attempt to learn a forward mapping that relates stimuli to the corresponding brain activation. Classical encoding models use linear multi-variate methods to predict the brain activation (all voxels) given the stimulus. However, these methods essentially assume multiple regions as one large uniform region or several independent regions, ignoring connections among them. In this paper, we present a mixture of experts-based model where a group of experts captures brain activity patterns related to particular regions of interest (ROI) and also show the discrimination across different experts. The model is trained word stimuli encoded as 25-dimensional feature vectors as input and the corresponding brain responses as output. Given a new word (25-dimensional feature vector), it predicts the entire brain activation as the linear combination of multiple experts brain activations. We argue that each expert learns a certain region of brain activations corresponding to its category of words, which solves the problem of identifying the regions with a simple encoding model. We showcase that proposed mixture of experts-based model indeed learns region-based experts to predict the brain activations with high spatial accuracy.
A Fully Sequential Methodology for Convolutional Neural Networks

Recent work has shown that the performance of convolutional neural networks could be significantly improved by increasing the depth of the representation. We propose a fully sequential methodology to construct and train extremely deep convolutional neural networks. 
We first introduce a novel sequential convolutional layer to construct the network. The proposed layer is capable of constructing trainable and highly efficient feedforward networks that consist of thousands of vanilla convolutional layers with rather limited number of parameters. The layer extracts each feature of the produced representation in sequence, allowing feature reuse within the layer. This form of feature reuse introduces in-layer hierarchy to the extracted features which greatly increases the depth of the representation, enabling richer structures to be explored. 
Furthermore, we employ the progressive growing training method to optimize each module of the network in sequence. This training manner progressively increases the network capacity allowing later modules to be optimized conditioning on prior knowledge from earlier modules. Thus, it encourages long term dependency to be established among each module of the network, which increases the effective depth of networks with skip connections, as well alleviates multiple optimization difficulties for deep networks.
Large-Scale Analysis of User Exposure to Online Advertising on Facebook

Online advertising is the major source of income for a large portion of the Internet Services. There exists a body of literature aiming at optimizing ads engagement, understanding the privacy and ethical implications of online advertising, and so on. However, to the best of our knowledge, no previous work analyzes, at large scale, the exposure of real users to online advertising. This paper performs a comprehensive analysis of the exposure of users to ads and advertisers using a dataset, including more than 7M ads from 140k unique advertisers delivered to more than 5k users, which was collected between October 2016 and May 2018. This paper focuses on Facebook, which is the second largest advertising platform next only to Google in terms of revenue and accounts for more than 2.2B monthly active users. Our analysis reveals that the Facebook users are exposed (in median) to 70 ads per week, which comes from 12 advertisers. Ads represent between 10% and 15% of all the information received in the users’ newsfeed. A small increment of 1% in the portion of ads in the newsfeed could roughly represent a revenue increase of 8.17M USD per week for Facebook. Finally, we also reveal that the Facebook users are overprofiled since, in the best case, only 23% of the active interests, Facebook assigns to users for the advertising purpose, are actually related to the ads these users receive.
HCqa: Hybrid and Complex Question Answering on Textual Corpus and Knowledge Graph

Answering (QA) systems provide easy access to the vast amount of knowledge without having to know the underlying complex structure of the knowledge. The research community has provided ad hoc solutions to the key QA tasks, including named entity recognition and disambiguation, relation extraction and query building. Furthermore, some have integrated and composed these components to implement many tasks automatically and efficiently. However, in general, the existing solutions are limited to simple and short questions and still do not address complex questions composed of several sub-questions. Exploiting the answer to complex questions is further challenged if it requires integrating knowledge from unstructured data sources, i.e., textual corpus, as well as structured data sources, i.e., knowledge graphs. In this paper, an approach (HCqa) is introduced for dealing with complex questions requiring federating knowledge from a hybrid of heterogeneous data sources (structured and unstructured). We contribute in developing (i) a decomposition mechanism which extracts sub-questions from potentially long and complex input questions, (ii) a novel comprehensive schema, first of its kind, for extracting and annotating relations, and (iii) an approach for executing and aggregating the answers of sub-questions. The evaluation of HCqa showed a superior accuracy in the fundamental tasks, such as relation extraction, as well as the federation task.
Fast Object Detection in Compressed Video

Object detection in videos has drawn increasing attention since it is more practical in real scenarios. Most of the deep learning methods use CNNs to process each decoded frame in a video stream individually. However, the free of charge yet valuable motion information already embedded in the video compression format is usually overlooked. In this paper, we propose a fast object detection method by taking advantage of this with a novel Motion aided Memory Network (MMNet). The MMNet has two major advantages: 1) It significantly accelerates the procedure of feature extraction for compressed videos. It only need to run a complete recognition network for I-frames, i.e. a few reference frames in a video, and it produces the features for the following P frames (predictive frames) with a light weight memory network, which runs fast; 2) Unlike existing methods that establish an additional network to model motion of frames, we take full advantage of both motion vectors and residual errors that are freely available in video streams. To our best knowledge, the MMNet is the first work that investigates a deep convolutional detector on compressed videos. Our method is evaluated on the large-scale ImageNet VID dataset, and the results show that it is 3x times faster than single image detector R-FCN and 10x times faster than high-performance detector MANet at a minor accuracy loss.
Learning State Representations in Complex Systems with Multimodal Data

Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the field still lacks a large-scale standard dataset for unified comparison. In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane. We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores. The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications.
Learning to Detect Dysarthria from Raw Speech

Speech classifiers of paralinguistic traits traditionally learn from diverse hand-crafted low-level features, by selecting the relevant information for the task at hand. We explore an alternative to this selection, by learning jointly the classifier, and the feature extraction. Recent work on speech recognition has shown improved performance over speech features by learning from the waveform. We extend this approach to paralinguistic classification and propose a neural network that can learn a filterbank, a normalization factor and a compression power from the raw speech, jointly with the rest of the architecture. We apply this model to dysarthria detection from sentence-level audio recordings. Starting from a strong attention-based baseline on which mel-filterbanks outperform standard low-level descriptors, we show that learning the filters or the normalization and compression improves over fixed features by 10% absolute accuracy. We also observe a gain over OpenSmile features by learning jointly the feature extraction, the normalization, and the compression factor with the architecture. This constitutes a first attempt at learning jointly all these operations from raw audio for a speech classification task.
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery

We propose FineGAN, a novel unsupervised GAN framework, which disentangles the background, object shape, and object appearance to hierarchically generate images of fine-grained object categories. To disentangle the factors without supervision, our key idea is to use information theory to associate each factor to a latent code, and to condition the relationships between the codes in a specific way to induce the desired hierarchy. Through extensive experiments, we show that FineGAN achieves the desired disentanglement to generate realistic and diverse images belonging to fine-grained classes of birds, dogs, and cars. Using FineGAN's automatically learned features, we also cluster real images as a first attempt at solving the novel problem of unsupervised fine-grained object category discovery. Our code/models/demo can be found at https://github.com/kkanshul/finegan
(P)reheating after minimal plateau inflation and constraints from CMB

In this paper we have investigated the preheating phase for a class of plateau inflationary model considering the four-legs interaction term (1/2)g2ϕ2χ2 between the inflaton (ϕ) and reheating field (χ). We specifically focus on the effects of our model parameter ϕ* which controls inflationary dynamics. For ϕ* < Mp, the departure of the inflaton potential from the usual power-law behavior ϕn significantly modifies the microscopic behavior of the preheating dynamics. We analyze and compare the efficiency of production, thermalization behavior and the final equation of states of the system for different values of n = 2,4,6 considering two different values of ϕ*. Most importantly as we increase n, or decrease ϕ*, the preheating occurs very efficiently with the final equation of state to be that of the radiation, w = 1/3. However, for n = 2, the final equation of state turned out to be w ≃ 0.2. In the non-perturbative framework complete decay of inflaton could not be achieved with the four-legs interaction for any model under consideration. Therefore, in order to complete the reheating process, we perform the perturbative analysis for the second stage of the reheating phase. With the appropriate initial condition set by the non-perturbative dynamics, we solved a set of homogeneous Boltzmann equations for both the fields supplemented by the constraints coming from the subsequent entropy conservation. In so doing, we calculated the reheating temperature which was otherwise ill-defined right after the end of preheating. The temperature can be uniquely fixed for a given inflaton decay constant and the CMB temperature. We also compare our results with the conventional reheating constraint analysis and discuss the limits on the inflaton decay constant from the field theory perspective.
Constraints on non-resonant photon-axion conversion from the Planck satellite data

The non-resonant conversion of Cosmic Microwave Background (CMB) photons into scalar as well as light pseudoscalar particles such as axion-like particles (ALPs) in the presence of turbulent magnetic fields can cause a unique, spatially fluctuating spectral distortion in the CMB. We use the publicly available Planck temperature maps for the frequency channels (70–545 GHz) to obtain the first ALP distortion map using 45% clean part of the sky. The 95th percentile upper limit on the RMS fluctuation of ALP distortions from the cleanest part of the CMB sky at 15 arcmin angular resolution is 18.5 × 10−6. The RMS fluctuation in the distortion map is also consistent with different combinations of frequency channels and sky-fractions.
Computing fast and reliable gravitational waveforms of binary neutron star merger remnants

Gravitational waves have been detected from the inspiral of a binary neutron-star, GW170817, which allowed constraints to be placed on the neutron star equation of state. The equation of state can be further constrained if gravitational waves from a post-merger remnant are detected. Post-merger waveforms are currently generated by numerical-relativity simulations, which are computationally expensive. Here we introduce a hierarchical model trained on numerical-relativity simulations, which can generate reliable post-merger spectra in a fraction of a second. Our spectra have mean fitting factors of 0.95, which compares to fitting factors of 0.76 and 0.85 between different numerical-relativity codes that simulate the same physical system. This method is the first step towards generating large template banks of spectra for use in post-merger detection and parameter estimation.
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.
Semantically-aware population health risk analyses

One primary task of population health analysis is the identification of risk factors that, for some subpopulation, have a significant association with some health condition. Examples include finding lifestyle factors associated with chronic diseases and finding genetic mutations associated with diseases in precision health. We develop a combined semantic and machine learning system that uses a health risk ontology and knowledge graph (KG) to dynamically discover risk factors and their associated subpopulations. Semantics and the novel supervised cadre model make our system explainable. Future population health studies are easily performed and documented with provenance by specifying additional input and output KG cartridges.
Cosmic Dawn II (CoDa II): a new radiation-hydrodynamics simulation of the self-consistent coupling of galaxy formation and reionization

Cosmic Dawn II (CoDa II) is a new, fully-coupled radiation-hydrodynamics simulation of cosmic reionization and galaxy formation and their mutual impact, to redshift z < 6. With $4096^3$ particles and cells in a 94 Mpc box, it is large enough to model global reionization and its feedback on galaxy formation while resolving all haloes above $10^8$ M$_\odot$. Using a constrained realization of {\Lambda}CDM constructed from galaxy survey data to reproduce the large-scale structure and familiar objects of the present-day Local Universe, CoDa II serves to model both global and local reionization, including its impact on the Local Group. Using the same hybrid CPU-GPU code RAMSES-CUDATON as CoDa I in Ocvirk et al. (2016), CoDa II modified and re-calibrated the subgrid star-formation algorithm, making reionization end earlier, at z $\gtrsim 6$, thereby better matching the observations of intergalactic Lyman-alpha opacity from quasar spectra and electron-scattering optical depth from cosmic microwave background fluctuations. The post-reionization UV background intensity is somewhat high, however, making the H I fraction after overlap lower than observed, a possible sign of missing bound-free opacity from unresolved substructure. CoDa II predicts a UV continuum luminosity function in good agreement with observations of high-z galaxies, especially at z = 6. The cosmic star formation rate density from M1600 < -17 galaxies is only 63% (18%) of the total in all haloes at z = 6 (z = 10). As in CoDa I, reionization feedback suppresses star formation in haloes below $\sim 2 \times 10^9 {\rm{M}}_\odot$ , though suppression here is less extreme, a possible consequence of modifying the star-formation algorithm. Suppression is environment-dependent, occurring earlier (later) in overdense (underdense) regions, in response to their local reionization times.