Reviewerly/exHarmony huggingface dataset:
How much progress have we made in neural network training? A New  Evaluation Protocol for Benchmarking Optimizers

Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance. In this work, we propose a new benchmarking protocol to evaluate both end-to-end efficiency (training a model from scratch without knowing the best hyperparameter) and data-addition training efficiency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data). For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy. A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search. For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift. We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining. Our results show that there is no clear winner across all the tasks.
The Effectiveness of Memory Replay in Large Scale Continual Learning

We study continual learning in the large scale setting where tasks in the input sequence are not limited to classification, and the outputs can be of high dimension. Among multiple state-of-the-art methods, we found vanilla experience replay (ER) still very competitive in terms of both performance and scalability, despite its simplicity. However, a degraded performance is observed for ER with small memory. A further visualization of the feature space reveals that the intermediate representation undergoes a distributional drift. While existing methods usually replay only the input-output pairs, we hypothesize that their regularization effect is inadequate for complex deep models and diverse tasks with small replay buffer size. Following this observation, we propose to replay the activation of the intermediate layers in addition to the input-output pairs. Considering that saving raw activation maps can dramatically increase memory and compute cost, we propose the Compressed Activation Replay technique, where compressed representations of layer activation are saved to the replay buffer. We show that this approach can achieve superior regularization effect while adding negligible memory overhead to replay method. Experiments on both the large-scale Taskonomy benchmark with a diverse set of tasks and standard common datasets (Split-CIFAR and Split-miniImageNet) demonstrate the effectiveness of the proposed method.
Improving Network Slimming with Nonconvex Regularization

Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes $\ell_1$ regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the $\ell_1$ penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate $\ell_p (0 < p < 1)$, transformed $\ell_1$ (T$\ell_1$), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on three neural network architectures -- VGG-19, DenseNet-40, and ResNet-164 -- on standard image classification datasets. Based on the numerical experiments, T$\ell_1$ preserves model accuracy against channel pruning, $\ell_{1/2, 3/4}$ yield better compressed models with similar accuracies after retraining as $\ell_1$, and MCP and SCAD provide more accurate models after retraining with similar compression as $\ell_1$. Network slimming with T$\ell_1$ regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.
on a conjecture on permutation rational functions over finite fields

Let $p$ be a prime and $n$ be a positive integer, and consider $f_b(X)=X+(X^p-X+b)^{-1}\in \Bbb F_p(X)$, where $b\in\Bbb F_{p^n}$ is such that $\text{Tr}_{p^n/p}(b)\ne 0$. It is known that (i) $f_b$ permutes $\Bbb F_{p^n}$ for $p=2,3$ and all $n\ge 1$; (ii) for $p>3$ and $n=2$, $f_b$ permutes $\Bbb F_{p^2}$ if and only if $\text{Tr}_{p^2/p}(b)=\pm 1$; and (iii) for $p>3$ and $n\ge 5$, $f_b$ does not permute $\Bbb F_{p^n}$. It has been conjectured that for $p>3$ and $n=3,4$, $f_b$ does not permute $\Bbb F_{p^n}$. We prove this conjecture for sufficiently large $p$.
Efficient multi port-based teleportation schemes

In this manuscript we analyse generalised port-based teleportation (PBT) schemes, allowing for transmitting more than one unknown quantum state (or a composite quantum state) in one go, where the state ends up in several ports at Bob's side. We investigate the efficiency of our scheme discussing both deterministic and probabilistic case, where parties share maximally entangled states. It turns out that the new scheme gives better performance than various variants of the optimal PBT protocol used for the same task. All the results are presented in group-theoretic manner depending on such quantities like dimensions and multiplicities of irreducible representations in the Schur-Weyl duality. The presented analysis was possible by considering the algebra of permutation operators acting on $n$ systems distorted by the action of partial transposition acting on more than one subsystem. Considering its action on the $n-$fold tensor product of the Hilbert space with finite dimension, we present construction of the respective irreducible matrix representations, which are in fact matrix irreducible representations of the Walled Brauer Algebra. I turns out that the introduced formalism, and symmetries beneath it, appears in many aspects of theoretical physics and mathematics - theory of anti ferromagnetism, aspects of gravity theory or in the problem of designing quantum circuits for special task like for example inverting an unknown unitary.
Derivation of Information-Theoretically Optimal Adversarial Attacks with  Applications to Robust Machine Learning

We consider the theoretical problem of designing an optimal adversarial attack on a decision system that maximally degrades the achievable performance of the system as measured by the mutual information between the degraded signal and the label of interest. This problem is motivated by the existence of adversarial examples for machine learning classifiers. By adopting an information theoretic perspective, we seek to identify conditions under which adversarial vulnerability is unavoidable i.e. even optimally designed classifiers will be vulnerable to small adversarial perturbations. We present derivations of the optimal adversarial attacks for discrete and continuous signals of interest, i.e., finding the optimal perturbation distributions to minimize the mutual information between the degraded signal and a signal following a continuous or discrete distribution. In addition, we show that it is much harder to achieve adversarial attacks for minimizing mutual information when multiple redundant copies of the input signal are available. This provides additional support to the recently proposed ``feature compression" hypothesis as an explanation for the adversarial vulnerability of deep learning classifiers. We also report on results from computational experiments to illustrate our theoretical results.
UVMBench: A Comprehensive Benchmark Suite for Researching Unified  Virtual Memory in GPUs

The recent introduction of Unified Virtual Memory (UVM) in GPUs offers a new programming model that allows GPUs and CPUs to share the same virtual memory space, which shifts the complex memory management from programmers to GPU driver/ hardware and enables kernel execution even when memory is oversubscribed. Meanwhile, UVM may also incur considerable performance overhead due to tracking and data migration along with special handling of page faults and page table walk. As UVM is attracting significant attention from the research community to develop innovative solutions to these problems, in this paper, we propose a comprehensive UVM benchmark suite named UVMBench to facilitate future research on this important topic. The proposed UVMBench consists of 32 representative benchmarks from a wide range of application domains. The suite also features unified programming implementation and diverse memory access patterns across benchmarks, thus allowing thorough evaluation and comparison with current state-of-the-art. A set of experiments have been conducted on real GPUs to verify and analyze the benchmark suite behaviors under various scenarios.
Simplification of Graph Convolutional Networks: A Matrix  Factorization-based Perspective

In recent years, substantial progress has been made on Graph Convolutional Networks (GCNs). However, the computing of GCN usually requires a large memory space for keeping the entire graph. In consequence, GCN is not flexible enough, especially for large scale graphs in complex real-world applications. Fortunately, methods based on Matrix Factorization (MF) naturally support constructing mini-batches, and thus are more friendly to distributed computing compared with GCN. Accordingly, in this paper, we analyze the connections between GCN and MF, and simplify GCN as matrix factorization with unitization and co-training. Furthermore, under the guidance of our analysis, we propose an alternative model to GCN named Unitized and Co-training Matrix Factorization (UCMF). Extensive experiments have been conducted on several real-world datasets. On the task of semi-supervised node classification, the experimental results illustrate that UCMF achieves similar or superior performances compared with GCN. Meanwhile, distributed UCMF significantly outperforms distributed GCN methods, which shows that UCMF can greatly benefit large scale and complex real-world applications. Moreover, we have also conducted experiments on a typical task of graph embedding, i.e., community detection, and the proposed UCMF model outperforms several representative graph embedding models.
Program Synthesis with Pragmatic Communication

Program synthesis techniques construct or infer programs from user-provided specifications, such as input-output examples. Yet most specifications, especially those given by end-users, leave the synthesis problem radically ill-posed, because many programs may simultaneously satisfy the specification. Prior work resolves this ambiguity by using various inductive biases, such as a preference for simpler programs. This work introduces a new inductive bias derived by modeling the program synthesis task as rational communication, drawing insights from recursive reasoning models of pragmatics. Given a specification, we score a candidate program both on its consistency with the specification, and also whether a rational speaker would chose this particular specification to communicate that program. We develop efficient algorithms for such an approach when learning from input-output examples, and build a pragmatic program synthesizer over a simple grid-like layout domain. A user study finds that end-user participants communicate more effectively with the pragmatic program synthesizer over a non-pragmatic one.
GOLD-NAS: Gradual, One-Level, Differentiable

There has been a large literature of neural architecture search, but most existing work made use of heuristic rules that largely constrained the search flexibility. In this paper, we first relax these manually designed constraints and enlarge the search space to contain more than $10^{160}$ candidates. In the new space, most existing differentiable search methods can fail dramatically. We then propose a novel algorithm named Gradual One-Level Differentiable Neural Architecture Search (GOLD-NAS) which introduces a variable resource constraint to one-level optimization so that the weak operators are gradually pruned out from the super-network. In standard image classification benchmarks, GOLD-NAS can find a series of Pareto-optimal architectures within a single search procedure. Most of the discovered architectures were never studied before, yet they achieve a nice tradeoff between recognition accuracy and model complexity. We believe the new space and search algorithm can advance the search of differentiable NAS.
WonderScope: Practical Near-surface AR Device for Museum Exhibits

Mobile augmented reality (AR) applications have become essential tools for delivering additional information to museum visitors. However, interacting through a mobile screen can potentially distract visitors from the exhibits. We propose WonderScope which is a peripheral system for mobile devices that enables practical near-surface AR interaction. Using a single small RFID tag on the exhibit as the origin, WonderScope can detect the position and orientation of the device on the surface of the exhibit. It performs on various surfaces of different materials based on the result of data fusion from two types of displacement sensors and an accelerometer of an inertial measurement unit (IMU). The mobile application utilizes the data for spatial registration of digital content on the exhibit’s surface, which make the users feel like seeing-through or magnifying the surface of exhibits.
The Alignment of Open Access with FAIR Principles in Musicological Publishing and Teaching

Open Access (OA) publishing and FAIR Principles both present opportunities to make music and music scholarship available to broader audiences and for innovative uses. This paper leverages findings from interviews conducted with music scholars about their perception of the opportunities, benefits, and disadvantages of OA publishing to explore how they might embrace FAIR Principles. While musicologists’ engagement of OA is generally passive, their perception of it is positive enough that OA could be used as a starting point for a deeper understanding of FAIR Principles.
Phantom Curves

We introduce phantom curves, a novel music-theoretical concept based on the discrete Fourier transform (DFT), and document the creative process that led to their discovery. In particular, we emphasize the importance of interactive web applications for music visualization and analysis. This is demonstrated using the example of the application midiVERTO which affords interactions with the pitch-class content of musical pieces encoded in MIDI format without requiring in-depth understanding of the underlying mathematics. We illustrate the analytical value of studying families of phantom curves by applying the concept to music from a Broadway musical, a video game, and a Hollywood movie. This process of discovery thus testifies to the fact that digital tools can bridge disciplinary boundaries between music theory and mathematics, and this interaction can generate new scientific knowledge.
Cell image augmentation for classification task using GANs on Pap smear dataset

One of the solutions to the problem of insufficiently large training datasets in image processing is data augmentation. This process artificially extends the size of training datasets to avoid overfitting. Generative Adversarial Networks yield that become increasingly difficult to differentiate from real images, until the differentiation is no longer possible. Thus, artificial images closely resembling original ones can be generated. Inclusion of artificial images contributes to improving the training process. Medical domain is one of the areas where data acquisition is burdened by many procedures, laws, and prohibitions. As a result the potential size of collected datasets is reduced. This article presents the results of training Convolutional Neural Networks on a artificially extended image datasets. The resulting classification accuracy on a cell classification task of models trained with images generated using the proposed method were increased by up to 12.9% in comparison to that of the model trained only with original dataset from the HErlev Pap smear dataset.
STUDENTS' PERCEPTIONS TOWARD THE USE OF BLENDED LEARNING IN TEACHING ENGLISH OF ENGLISH EDUCATION DEPARTMENT STUDENTS AT UIN ALAUDDIN MAKASSAR

The purpose of this research is to find out students’ perception toward the use of blended learning. The research question of this research was “what is the students’ perceptions toward the use of blended learning in teaching English at the English Education Students in UIN Alauddin Makassar?”. The research sample and population are the English Education Department students with total number of populations were 98 students.This research applied a quantitative method with survey design, using total sampling where the data collected with online questionnaires. The questionnaires were formed by the researcher, using theories from some experts such as a theory about the learners’ knowledge by (Macdonald, 2008), a theory about the effectiveness of blended learning and theories about the benefits of blended learning by (Debra and John, 2010) and (Brown in Buket and Meryem, 2008). The data was analyzed by using the frequencies of Likert Scale and finding the mean score. The findings showed that the highest percentage of students’ response on the students’ perception toward the use of blended learning in teaching English was in the statement 9 namely “I like to learn at my own phase, that is why blended learning is interesting” with 82.24% which is from the effectiveness of blended learning, one of the aspects of question that are included in the questionnaires. And the lowest percentage was in the statement 18 namely “There is no difference between blended learning and any other methods” with 34.69% which also is the same aspect of question as the highest percentage, but was a unfavorable statement. Based on the research findings, the majority of students gave positive responses toward the use of blended learning in teaching English from the aspect of the effectiveness of blended learning itself and the mean score of students’ perceptions toward the use of blended learning was 3.47 which came from 67 positive responses.
Development of Professional Creativity of Professional Teachers in Professional Courses on The Basis of E-Pedagogy Principle

This article describes the ways and methods of developing the professional competence of teachers of professional education in professional development courses on the basis of e-pedagogy. In addition, “Practical training in e-learning. Experiments in the development of education with the help of software "provides a technological map, which describes in detail the stages of application of technology" Design-thinking ".
Use of Technological Tools in Virtual Education for Educational Innovation in Amazon Public Universities

University education is in a process of change ready to take on the challenges of educational innovation by incorporating new technological tools to replace traditional ones; because, educational innovation must be reflected in educational practices that improve teaching learning. It is important to raise those technological tools in a requirement in virtual teaching. The professors Peruvian public universities of the Amazon present serious deficiencies to differentiate the digital and virtual tools of the traditional slate, multimedia projector and other resources that have supported teaching for many decades in the classroom. The objective of the research was: To determine the relationship between the use of technological tools in virtual education and educational innovation in teachers of the various faculties of San Martín National University. The study was of a correlational descriptive type. The sample consisted of 339 professors from the various faculties of the National University of San Martín. It is concluded that the use of technological tools in virtual education used by university teachers are: attendance, forum, grades, homework, etiquette, videoconference, polling, digital whiteboard, screen sharing and presenting videos, evidencing as a result, that the use of technological tools in virtual education is significantly related to educational innovation in professors of Peruvian public universities in the Amazon.
Modulation Format Identification Using Kmeans Clustering and Cluster Validity Index

A low complexity modulation format identification based on unsupervised Kmeans clustering algorithm and cluster validity index is proposed. Simulations show the scheme’s 100% identification accuracy and stable performance.
Graph Neural Networks Including Sparse Interpretability

Graph Neural Networks (GNNs) are versatile, powerful machine learning methods that enable graph structure and feature representation learning, and have applications across many domains. For applications critically requiring interpretation, attention-based GNNs have been leveraged. However, these approaches either rely on specific model architectures or lack a joint consideration of graph structure and node features in their interpretation. Here we present a model-agnostic framework for interpreting important graph structure and node features, Graph neural networks Including SparSe inTerpretability (GISST). With any GNN model, GISST combines an attention mechanism and sparsity regularization to yield an important subgraph and node feature subset related to any graph-based task. Through a single self-attention layer, a GISST model learns an importance probability for each node feature and edge in the input graph. By including these importance probabilities in the model loss function, the probabilities are optimized end-to-end and tied to the task-specific performance. Furthermore, GISST sparsifies these importance probabilities with entropy and L1 regularization to reduce noise in the input graph topology and node features. Our GISST models achieve superior node feature and edge explanation precision in synthetic datasets, as compared to alternative interpretation approaches. Moreover, our GISST models are able to identify important graph structure in real-world datasets. We demonstrate in theory that edge feature importance and multiple edge types can be considered by incorporating them into the GISST edge probability computation. By jointly accounting for topology, node features, and edge features, GISST inherently provides simple and relevant interpretations for any GNN models and tasks.
Image classification in frequency domain with 2SReLU: a second harmonics  superposition activation function

Deep Convolutional Neural Networks are able to identify complex patterns and perform tasks with super-human capabilities. However, besides the exceptional results, they are not completely understood and it is still impractical to hand-engineer similar solutions. In this work, an image classification Convolutional Neural Network and its building blocks are described from a frequency domain perspective. Some network layers have established counterparts in the frequency domain like the convolutional and pooling layers. We propose the 2SReLU layer, a novel non-linear activation function that preserves high frequency components in deep networks. It is demonstrated that in the frequency domain it is possible to achieve competitive results without using the computationally costly convolution operation. A source code implementation in PyTorch is provided at: https://gitlab.com/thomio/2srelu