Reviewerly/exHarmony huggingface dataset:
Algorithm-dependent generalization bounds for overparameterized deep residual networks

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased generalization and stability of networks with this architecture, although there have been limited theoretical guarantees for this improved performance. In this work, we analyze overparameterized deep residual networks trained by gradient descent following random initialization, and demonstrate that (i) the class of networks learned by gradient descent constitutes a small subset of the entire neural network function class, and (ii) this subclass of networks is sufficiently large to guarantee small training error. By showing (i) we are able to demonstrate that deep residual networks trained with gradient descent have a small generalization gap between training and test error, and together with (ii) this guarantees that the test error will be small. Our optimization and generalization guarantees require overparameterization that is only logarithmic in the depth of the network, which helps explain why residual networks are preferable to fully connected ones.
Abstraction based Output Range Analysis for Neural Networks

In this paper, we consider the problem of output range analysis for feed-forward neural networks. The current approaches reduce the problem to satisfiability and optimization solving which are NP-hard problems, and whose computational complexity increases with the number of neurons in the network. We present a novel abstraction technique that constructs a simpler neural network with fewer neurons, albeit with interval weights called interval neural network (INN) which over-approximates the output range of the given neural network. We reduce the output range analysis on the INNs to solving a mixed integer linear programming problem. Our experimental results highlight the trade-off between the computation time and the precision of the computed output range.
Fast Efficient Hyperparameter Tuning for Policy Gradient Methods

The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedules for hyperparameters instead of fixed settings can yield better results, but are also sample inefficient and computationally expensive. In this paper, we propose Hyperparameter Optimisation on the Fly (HOOF), a gradient-free algorithm that requires no more than one training run to automatically adapt the hyperparameter that affect the policy update directly through the gradient. The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement. Our experimental results across multiple domains and algorithms show that using HOOF to learn these hyperparameter schedules leads to faster learning with improved performance.
NLNDE: The Neither-Language-Nor-Domain-Experts' Way of Spanish Medical Document De-Identification.

Natural language processing has huge potential in the medical domain which recently led to a lot of research in this field. However, a prerequisite of secure processing of medical documents, e.g., patient notes and clinical trials, is the proper de-identification of privacy-sensitive information. In this paper, we describe our NLNDE system, with which we participated in the MEDDOCAN competition, the medical document anonymization task of IberLEF 2019. We address the task of detecting and classifying protected health information from Spanish data as a sequence-labeling problem and investigate different embedding methods for our neural network. Despite dealing in a non-standard language and domain setting, the NLNDE system achieves promising results in the competition.
Meta-Curvature

We propose meta-curvature (MC), a framework to learn curvature information for better generalization and fast model adaptation. MC expands on the model-agnostic meta-learner (MAML) by learning to transform the gradients in the inner optimization such that the transformed gradients achieve better generalization performance to a new task. For training large scale neural networks, we decompose the curvature matrix into smaller matrices in a novel scheme where we capture the dependencies of the model's parameters with a series of tensor products. We demonstrate the effects of our proposed method on several few-shot learning tasks and datasets. Without any task specific techniques and architectures, the proposed method achieves substantial improvement upon previous MAML variants and outperforms the recent state-of-the-art methods. Furthermore, we observe faster convergence rates of the meta-training process. Finally, we present an analysis that explains better generalization performance with the meta-trained curvature.
Hyperparameter Learning via Distributional Transfer

Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial exploration even in cases where similar prior tasks have been solved. We propose to transfer information across tasks using learnt representations of training datasets used in those tasks. This results in a joint Gaussian process model on hyperparameters and data representations. Representations make use of the framework of distribution embeddings into reproducing kernel Hilbert spaces. The developed method has a faster convergence compared to existing baselines, in some cases requiring only a few evaluations of the target objective.
Improving ML Safety with Partial Specifications

Advanced autonomy features of vehicles are typically difficult or impossible to specify precisely and this has led to the rise of machine learning (ML) from examples as an alternative implementation approach to traditional programming. Developing software without specifications sacrifices the ability to effectively verify the software yet this is a key component of safety assurance. In this paper, we suggest that while complete specifications may not be possible, partial specifications typically are and these could be used with ML to strengthen safety assurance. We review the types of partial specifications that are applicable for these problems and discuss the places in the ML development workflow that they could be used to improve the safety of ML-based components.
How degenerate is the parametrization of neural networks with the ReLU activation function

Neural network training is usually accomplished by solving a non-convex optimization problem using stochastic gradient descent. Although one optimizes over the networks parameters, the main loss function generally only depends on the realization of the neural network, i.e. the function it computes. Studying the optimization problem over the space of realizations opens up new ways to understand neural network training. In particular, usual loss functions like mean squared error and categorical cross entropy are convex on spaces of neural network realizations, which themselves are non-convex. Approximation capabilities of neural networks can be used to deal with the latter non-convexity, which allows us to establish that for sufficiently large networks local minima of a regularized optimization problem on the realization space are almost optimal. Note, however, that each realization has many different, possibly degenerate, parametrizations. In particular, a local minimum in the parametrization space needs not correspond to a local minimum in the realization space. To establish such a connection, inverse stability of the realization map is required, meaning that proximity of realizations must imply proximity of corresponding parametrizations. We present pathologies which prevent inverse stability in general, and, for shallow networks, proceed to establish a restricted space of parametrizations on which we have inverse stability w.r.t. to a Sobolev norm. Furthermore, we show that by optimizing over such restricted sets, it is still possible to learn any function which can be learned by optimization over unrestricted sets.
Neural Shuffle-Exchange Networks - Sequence Processing in O(n log n) Time

A key requirement in sequence to sequence processing is the modeling of long range dependencies. To this end, a vast majority of the state-of-the-art models use attention mechanism which is of O(n^2) complexity that leads to slow execution for long sequences. We introduce a new Shuffle-Exchange neural network model for sequence to sequence tasks which have O(log n) depth and O(n log n) total complexity. We show that this model is powerful enough to infer efficient algorithms for common algorithmic benchmarks including sorting, addition and multiplication. We evaluate our architecture on the challenging LAMBADA question answering dataset and compare it with the state-of-the-art models which use attention. Our model achieves competitive accuracy and scales to sequences with more than a hundred thousand of elements. We are confident that the proposed model has the potential for building more efficient architectures for processing large interrelated data in language modeling, music generation and other application domains.
PAC-Bayes under potentially heavy tails

We derive PAC-Bayesian learning guarantees for heavy-tailed losses, and obtain a novel optimal Gibbs posterior which enjoys finite-sample excess risk bounds at logarithmic confidence. Our core technique itself makes use of PAC-Bayesian inequalities in order to derive a robust risk estimator, which by design is easy to compute. In particular, only assuming that the first three moments of the loss distribution are bounded, the learning algorithm derived from this estimator achieves nearly sub-Gaussian statistical error, up to the quality of the prior.
Structured Prediction with Projection Oracles

We propose in this paper a general framework for deriving loss functions for structured prediction. In our framework, the user chooses a convex set including the output space and provides an oracle for projecting onto that set. Given that oracle, our framework automatically generates a corresponding convex and smooth loss function. As we show, adding a projection as output layer provably makes the loss smaller. We identify the marginal polytope, the output space's convex hull, as the best convex set on which to project. However, because the projection onto the marginal polytope can sometimes be expensive to compute, we allow to use any convex superset instead, with potentially cheaper-to-compute projection. Since efficient projection algorithms are available for numerous convex sets, this allows us to construct loss functions for a variety of tasks. On the theoretical side, when combined with calibrated decoding, we prove that our loss functions can be used as a consistent surrogate for a (potentially non-convex) target loss function of interest. We demonstrate our losses on label ranking, ordinal regression and multilabel classification, confirming the improved accuracy enabled by projections.
Stitching swing arm profilometer test for large aperture aspherics

We implemented a stitching swing arm profilometer (SSAP) test for the inner and outer regions of a large aspheric surface with a short arm. The SSAP was more capable of improving sampling density of surface and was less sensitive to system error, like vibration noise and air-table noise. Firstly, a calculation model was built to evaluate the sampling density of the SSAP test. Then, sensitivity to system noise was tested when different lengths of arm were used. At the end, an experiment on a 3 m diameter aspheric mirror was implemented, where test efficiency was promoted, and high sampling density was achieved.
Animasi Interaktif Sebagai Media Pembelajaran Bahasa Inggris Menggunakan Metode ADDIE

English is an international standard language used in conversations between countries. Therefore, learning English is very important given to children from an early age. However, in its implementation there are still many children who have difficulty learning English using learning methods that are difficult, less interesting and difficult. This learning media only relies on stationery such as books, paper, and pencils so that children increasingly like the learning process. Nowadays there are various interesting ways to attract children's interest in learning. One step that can be taken is to use computer technology as a medium for learning English for communication material with audio and visual material. Interactive animation is one form of learning method that uses audio visual media. In the world of education, animation has a role as a learning media that is quite interesting. Animation can increase interest in learning and understanding of certain fields of science. In addition, animation can also facilitate teachers in the learning process and also delivery material for students.
Automated Theme Search in ICO Whitepapers

The authors explore how topic modeling can be used to automate the categorization of initial coin offerings (ICOs) into different topics (e.g., finance, media, information, professional services, health and social, natural resources) based solely on the content within the whitepapers. This tool has been developed by fitting a latent Dirichlet allocation (LDA) model to the text extracted from the ICO whitepapers. After evaluating the automated categorization of whitepapers using statistical and human judgment methods, it is determined that there is enough evidence to conclude that the LDA model appropriately categorizes the ICO whitepapers. The results from a two-population proportion test show a statistically significant difference between topics in the success of an ICO being funded, indicating that the topics are usefully differentiated and suggesting that the topic model could be used to help predict whether an ICO will be successful. <b>TOPICS:</b>Statistical methods, simulations, big data/machine learning <b>Key Findings</b> • Categorization of ICO whitepapers can be done via topic modeling with the latent Dirichlet allocation (LDA) model. • Statistical and human judgment methods confirms that there is enough evidence to conclude that the LDA model appropriately categorizes ICO whitepapers. • Statistical tests suggests that the categorization results from the LDA model provides useful information on predicting whether an ICO will be successful funded.
Human Age Classification System Using K-NN Classifier

The age classification system is used to categorize age into various groups determined by facial features. Many researchers have used different techniques for the age classification system, namely Geometric Ratio, Wrinkle Ratio, Local Gabor Pattern, Histograms of Oriented Gradients, Adaptive Neuro-Fuzzy Inference system, Texture feature, Support Vector Machine and Artificial neural network. The current system does not give great precision in light of the fact that these strategies are wasteful for classification. The proposed work represents the classification of images depending on the groups. This work classifies the age into old, adult and child groups. This work completes in three steps, i.e., preprocessing, facial aging feature extraction and classification. In preprocessing, the RGB image is converted into a grayscale level image; the facial aging feature extraction is performed by skin texture feature extraction and wrinkle analysis. LGBPH and the density of wrinkles accomplished the wrinkle analysis. LGBPH is a histogram of LGBP which is calculated by LBP. The LGBP feature is not dynamic for features like rotation, alignment, and the illumination of the image. The proposed work selected three different skins for skin texture feature and five different skin textures for wrinkle investigation. This work uses K-NN classifier due to its fast processing; classifies the testing dataset from training dataset. The LGBPH provides better results and estimates the accuracy as compared to the previous technique used by the researcher. This classification is a triumphant work as it gives an efficient classification using facial aging feature extraction.
mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis.

Multi-contrast MRI protocols increase the level of morphological information available for diagnosis. Yet, the number and quality of contrasts is limited in practice by various factors including scan time and patient motion. Synthesis of missing or corrupted contrasts can alleviate this limitation to improve clinical utility. Common approaches for multi-contrast MRI involve either one-to-one and many-to-one synthesis methods. One-to-one methods take as input a single source contrast, and they learn a latent representation sensitive to unique features of the source. Meanwhile, many-to-one methods receive multiple distinct sources, and they learn a shared latent representation more sensitive to common features across sources. For enhanced image synthesis, here we propose a multi-stream approach that aggregates information across multiple source images via a mixture of multiple one-to-one streams and a joint many-to-one stream. The shared feature maps generated in the many-to-one stream and the complementary feature maps generated in the one-to-one streams are combined with a fusion block. The location of the fusion block is adaptively modified to maximize task-specific performance. Qualitative and quantitative assessments on T1-, T2-, PD-weighted and FLAIR images clearly demonstrate the superior performance of the proposed method compared to previous state-of-the-art one-to-one and many-to-one methods.
Longest common substring made fully dynamic

Given two strings S and T, each of length at most n, the longest common substring (LCS) problem is to find a longest substring common to S and T. This is a classical problem in computer science with an O(n)-time solution. In the fully dynamic setting, edit operations are allowed in either of the two strings, and the problem is to find an LCS after each edit. We present the first solution to this problem requiring sublinear time in n per edit operation. In particular, we show how to find an LCS after each edit operation in O(n2/3) time, after O(n)-time and space preprocessing. 1 This line of research has been recently initiated in a somewhat restricted dynamic variant by Amir et al. [SPIRE 2017]. More specifically, they presented an O(n)-sized data structure that returns an LCS of the two strings after a single edit operation (that is reverted afterwards) in O(1) time. At CPM 2018, three papers (Abedin et al., Funakoshi et al., and Urabe et al.) studied analogously restricted dynamic variants of problems on strings. We show that the techniques we develop can be applied to obtain fully dynamic algorithms for all of these variants. The only previously known sublinear-time dynamic algorithms for problems on strings were for maintaining a dynamic collection of strings for comparison queries and for pattern matching, with the most recent advances made by Gawrychowski et al. [SODA 2018] and by Clifford et al. [STACS 2018]. As an intermediate problem we consider computing the solution for a string with a given set of k edits, which leads us, in particular, to answering internal queries on a string. The input to such a query is specified by a substring (or substrings) of a given string. Data structures for answering internal string queries that were proposed by Kociumaka et al. [SODA 2015] and by Gagie et al. [CCCG 2013] are used, along with new ones, based on ingredients such as the suffix tree, heavy-path decomposition, orthogonal range queries, difference covers, and string periodicity.
The Complexity of Finding Stationary Points with Stochastic Gradient Descent

We study the iteration complexity of stochastic gradient descent (SGD) for minimizing the gradient norm of smooth, possibly nonconvex functions. We provide several results, implying that the $\mathcal{O}(\epsilon^{-4})$ upper bound of Ghadimi and Lan~\cite{ghadimi2013stochastic} (for making the average gradient norm less than $\epsilon$) cannot be improved upon, unless a combination of additional assumptions is made. Notably, this holds even if we limit ourselves to convex quadratic functions. We also show that for nonconvex functions, the feasibility of minimizing gradients with SGD is surprisingly sensitive to the choice of optimality criteria.
A Verified LL(1) Parser Generator

An LL(1) parser is a recursive descent algorithm that uses a single token of lookahead to build a grammatical derivation for an input sequence. We present an LL(1) parser generator that, when applied to grammar G, produces an LL(1) parser for G if such a parser exists. We use the Coq Proof Assistant to verify that the generator and the parsers that it produces are sound and complete, and that they terminate on all inputs without using fuel parameters. As a case study, we extract the tool's source code and use it to generate a JSON parser. The generated parser runs in linear time; it is two to four times slower than an unverified parser for the same grammar.
Inductive Matrix Completion Based on Graph Neural Networks

We propose an inductive matrix completion model without using side information. By factorizing the (rating) matrix into the product of low-dimensional latent embeddings of rows (users) and columns (items), a majority of existing matrix completion methods are transductive, since the learned embeddings cannot generalize to unseen rows/columns or to new matrices. To make matrix completion inductive, most previous works use content (side information), such as user's age or movie's genre, to make predictions. However, high-quality content is not always available, and can be hard to extract. Under the extreme setting where not any side information is available other than the matrix to complete, can we still learn an inductive matrix completion model? In this paper, we propose an Inductive Graph-based Matrix Completion (IGMC) model to address this problem. IGMC trains a graph neural network (GNN) based purely on 1-hop subgraphs around (user, item) pairs generated from the rating matrix and maps these subgraphs to their corresponding ratings. It achieves highly competitive performance with state-of-the-art transductive baselines. In addition, IGMC is inductive -- it can generalize to users/items unseen during the training (given that their interactions exist), and can even transfer to new tasks. Our transfer learning experiments show that a model trained out of the MovieLens dataset can be directly used to predict Douban movie ratings with surprisingly good performance. Our work demonstrates that: 1) it is possible to train inductive matrix completion models without using side information while achieving similar or better performances than state-of-the-art transductive methods; 2) local graph patterns around a (user, item) pair are effective predictors of the rating this user gives to the item; and 3) Long-range dependencies might not be necessary for modeling recommender systems.
Reinforcement Learning Compensation based PD Control for a Double Inverted Pendulum

In this paper, we present a Control Algorithm based on Reinforcement Learning for a double inverted pendulum on a cart. By implementing the Q-Learning techniques in the PD control scheme, the second pendulum (top pendulum) is enabled to improve its performance. In a first step, Q-Learning is used so that the control can balance the second pendulum towards its inverted vertical position, while the first pendulum has no restrictions on its movement and also the car remains in a range of ±1 meter in its displacement. In a second step, we combine hybrid techniques of Q-Learning and PD control, in a system that has had changes in its parameters and in its initial conditions. Then, with the hybrid control, we obtain better results than using the controllers individually. Finally, the simulation results show the effectiveness of the proposed controller.