OpenRaiser/Intern-Atlas huggingface dataset:
The Impact of a STEM-based Entrepreneurship Program on the Entrepreneurial Intention of Secondary School Female Students
Venue: arXiv cs.CY 2021
Year: 2021

Despite dedicated effort and research in the last two decades, the entrepreneurship field is still limited by little evidence-based knowledge of the impacts of entrepreneurship programs on the entrepreneurial intention of students in pre-university levels of study. Further, gender equity continues to be an issue in the entrepreneurial sector, particularly in STEM-focused entrepreneurship. In this context, this study was designed to explore the effects of a one-day female-focused STEM-based entrepreneurship program (for brevity, we call it the OzGirlsEntrepreneurship program) on the entrepreneurial intention of secondary school female students. The study collected data from two surveys completed by 193 secondary school female students, aged 14-16 years, who participated in the OzGirlsEntrepreneurship program. This program encouraged girls to develop and implement creative computational solutions to socially relevant problems, with an Internet of Things (IoT) component using the micro:bit device. The findings reveal that a key factor in the development of entrepreneurial attitudes in young female students is associated with soft-skills development, particularly in the areas of creative thinking, risk-taking, problem-solving, and leadership development. The importance of meaningful human connections, including positive role modelling and peer to peer learning were also important factors in fostering entrepreneurial intent. With these factors in mind, our findings highlight that the OzGirlsEntrepreneurship program substantially increased the entrepreneurial intention of secondary school female students. In addition, this study offers actionable implications and recommendations to develop and deliver entrepreneurship education programs for secondary school level students.
Certified Robustness of Nearest Neighbors against Data Poisoning and Backdoor Attacks
Venue: arXiv cs.CR 2021
Year: 2021

Data poisoning attacks and backdoor attacks aim to corrupt a machine learning classifier via modifying, adding, and/or removing some carefully selected training examples, such that the corrupted classifier makes incorrect predictions as the attacker desires. The key idea of state-of-the-art certified defenses against data poisoning attacks and backdoor attacks is to create a majority vote mechanism to predict the label of a testing example. Moreover, each voter is a base classifier trained on a subset of the training dataset. Classical simple learning algorithms such as k nearest neighbors (kNN) and radius nearest neighbors (rNN) have intrinsic majority vote mechanisms. In this work, we show that the intrinsic majority vote mechanisms in kNN and rNN already provide certified robustness guarantees against data poisoning attacks and backdoor attacks. Moreover, our evaluation results on MNIST and CIFAR10 show that the intrinsic certified robustness guarantees of kNN and rNN outperform those provided by state-of-the-art certified defenses. Our results serve as standard baselines for future certified defenses against data poisoning attacks and backdoor attacks.
Continuum Limit of Lipschitz Learning on Graphs
Venue: arXiv cs.LG 2021
Year: 2021

Tackling semi-supervised learning problems with graph-based methods has become a trend in recent years since graphs can represent all kinds of data and provide a suitable framework for studying continuum limits, e.g., of differential operators. A popular strategy here is $p$-Laplacian learning, which poses a smoothness condition on the sought inference function on the set of unlabeled data. For $p<\infty$ continuum limits of this approach were studied using tools from $Γ$-convergence. For the case $p=\infty$, which is referred to as Lipschitz learning, continuum limits of the related infinity-Laplacian equation were studied using the concept of viscosity solutions. In this work, we prove continuum limits of Lipschitz learning using $Γ$-convergence. In particular, we define a sequence of functionals which approximate the largest local Lipschitz constant of a graph function and prove $Γ$-convergence in the $L^\infty$-topology to the supremum norm of the gradient as the graph becomes denser. Furthermore, we show compactness of the functionals which implies convergence of minimizers. In our analysis we allow a varying set of labeled data which converges to a general closed set in the Hausdorff distance. We apply our results to nonlinear ground states, i.e., minimizers with constrained $L^p$-norm, and, as a by-product, prove convergence of graph distance functions to geodesic distance functions.
Counting ternary square-free words quickly
Venue: arXiv cs.FL 2021
Year: 2021

An efficient, when compared to exhaustive enumeration, algorithm for computing the number of square-free words of length $n$ over the alphabet $\{a, b, c\}$ is presented.
Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations
Venue: arXiv cs.CL 2021
Year: 2021

Intent detection is a key component of modern goal-oriented dialog systems that accomplish a user task by predicting the intent of users' text input. There are three primary challenges in designing robust and accurate intent detection models. First, typical intent detection models require a large amount of labeled data to achieve high accuracy. Unfortunately, in practical scenarios it is more common to find small, unbalanced, and noisy datasets. Secondly, even with large training data, the intent detection models can see a different distribution of test data when being deployed in the real world, leading to poor accuracy. Finally, a practical intent detection model must be computationally efficient in both training and single query inference so that it can be used continuously and re-trained frequently. We benchmark intent detection methods on a variety of datasets. Our results show that Watson Assistant's intent detection model outperforms other commercial solutions and is comparable to large pretrained language models while requiring only a fraction of computational resources and training data. Watson Assistant demonstrates a higher degree of robustness when the training and test distributions differ.
Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
Venue: arXiv cs.LG 2021
Year: 2021

The success of deep learning in many real-world tasks has triggered an intense effort to understand the power and limitations of deep learning in the training and generalization of complex tasks, so far with limited progress. In this work, we study the statistical mechanics of learning in Deep Linear Neural Networks (DLNNs) in which the input-output function of an individual unit is linear. Despite the linearity of the units, learning in DLNNs is nonlinear, hence studying its properties reveals some of the features of nonlinear Deep Neural Networks (DNNs). Importantly, we solve exactly the network properties following supervised learning using an equilibrium Gibbs distribution in the weight space. To do this, we introduce the Back-Propagating Kernel Renormalization (BPKR), which allows for the incremental integration of the network weights starting from the network output layer and progressing backward until the first layer's weights are integrated out. This procedure allows us to evaluate important network properties, such as its generalization error, the role of network width and depth, the impact of the size of the training set, and the effects of weight regularization and learning stochasticity. BPKR does not assume specific statistics of the input or the task's output. Furthermore, by performing partial integration of the layers, the BPKR allows us to compute the properties of the neural representations across the different hidden layers. We have proposed an extension of the BPKR to nonlinear DNNs with ReLU. Surprisingly, our numerical simulations reveal that despite the nonlinearity, the predictions of our theory are largely shared by ReLU networks in a wide regime of parameters. Our work is the first exact statistical mechanical study of learning in a family of DNNs, and the first successful theory of learning through successive integration of DoFs in the learned weight space.
Improved Swarm Engineering: Aligning Intuition and Analysis
Venue: arXiv cs.RO 2021
Year: 2021

We present a set of metrics intended to supplement designer intuitions when designing swarm-robotic systems, increase accuracy in extrapolating swarm behavior from algorithmic descriptions and small test experiments, and lead to faster and less costly design cycles. We build on previous works studying self-organizing behaviors in autonomous systems to derive a metric for swarm emergent self-organization. We utilize techniques from high performance computing, time series analysis, and queueing theory to derive metrics for swarm scalability, flexibility to changing external environments, and robustness to internal system stimuli such as sensor and actuator noise and robot failures. We demonstrate the utility of our metrics by analyzing four different control algorithms in two scenarios: an indoor warehouse object transport scenario with static objects and a spatially unconstrained outdoor search and rescue scenario with moving objects. In the spatially constrained warehouse scenario, efficient use of space is key to success so algorithms that use mechanisms for traffic regulation and congestion reduction are the most appropriate. In the search and rescue scenario, the same will happen with algorithms which can cope well with object motion through dynamic task allocation and randomized search trajectories. We show that our intuitions about comparative algorithm performance are well supported by the quantitative results obtained using our metrics, and that our metrics can be synergistically used together to predict collective behaviors based on previous results in some cases.
Efficient Estimation of Influence of a Training Instance
Venue: arXiv cs.LG 2021
Year: 2021

Understanding the influence of a training instance on a neural network model leads to improving interpretability. However, it is difficult and inefficient to evaluate the influence, which shows how a model's prediction would be changed if a training instance were not used. In this paper, we propose an efficient method for estimating the influence. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. By switching between dropout masks, we can use sub-networks that learned or did not learn each training instance and estimate its influence. Through experiments with BERT and VGGNet on classification datasets, we demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
Overcomplete Representations Against Adversarial Videos
Venue: arXiv cs.CV 2021
Year: 2021

Adversarial robustness of deep neural networks is an extensively studied problem in the literature and various methods have been proposed to defend against adversarial images. However, only a handful of defense methods have been developed for defending against attacked videos. In this paper, we propose a novel Over-and-Under complete restoration network for Defending against adversarial videos (OUDefend). Most restoration networks adopt an encoder-decoder architecture that first shrinks spatial dimension then expands it back. This approach learns undercomplete representations, which have large receptive fields to collect global information but overlooks local details. On the other hand, overcomplete representations have opposite properties. Hence, OUDefend is designed to balance local and global features by learning those two representations. We attach OUDefend to target video recognition models as a feature restoration block and train the entire network end-to-end. Experimental results show that the defenses focusing on images may be ineffective to videos, while OUDefend enhances robustness against different types of adversarial videos, ranging from additive attacks, multiplicative attacks to physically realizable attacks. Code: https://github.com/shaoyuanlo/OUDefend
Forecasting the Olympic medal distribution during a pandemic: a socio-economic machine learning model
Venue: arXiv cs.LG 2021
Year: 2021

Forecasting the number of Olympic medals for each nation is highly relevant for different stakeholders: Ex ante, sports betting companies can determine the odds while sponsors and media companies can allocate their resources to promising teams. Ex post, sports politicians and managers can benchmark the performance of their teams and evaluate the drivers of success. To significantly increase the Olympic medal forecasting accuracy, we apply machine learning, more specifically a two-staged Random Forest, thus outperforming more traditional naïve forecast for three previous Olympics held between 2008 and 2016 for the first time. Regarding the Tokyo 2020 Games in 2021, our model suggests that the United States will lead the Olympic medal table, winning 120 medals, followed by China (87) and Great Britain (74). Intriguingly, we predict that the current COVID-19 pandemic will not significantly alter the medal count as all countries suffer from the pandemic to some extent (data inherent) and limited historical data points on comparable diseases (model inherent).
Steady-State Rate-Optimal Power Adaptation in Energy Harvesting Opportunistic Cognitive Radios with Spectrum Sensing and Channel Estimation Errors
Venue: arXiv cs.IT 2021
Year: 2021

We consider an opportunistic cognitive radio network, consisting of Nu secondary users (SUs) and an access point (AP), that can access a spectrum band licensed to a primary user. Each SU is capable of harvesting energy, and is equipped with a finite size battery, for energy storage. The SUs operate under a time-slotted scheme, where each time slot consists of three non-overlapping phases: spectrum sensing phase, channel probing phase, and data transmission phase. The AP feeds back its estimates of fading coefficients of SUs-AP link to SUs. To strike a balance between the energy harvesting and the energy consumption, we propose a parameterized power control strategy that allows each SU to adapt its power, according to the feedback information and its stored energy. Modeling the randomly arriving energy packets during a time slot as a Poisson process, we establish a lower bound on the achievable sum-rate of SUs-AP links, in the presence of both spectrum sensing and channel estimation errors. We optimize the parameters of the proposed power control strategy, such that the derived sum-rate lower bound is maximized, subject to an interference constraint. Via simulations, we corroborate our analysis and explore spectrum sensing-channel probing-data transmission trade-offs.
Reconstruction of Backbone Curves for Snake Robots
Venue: arXiv cs.RO 2021
Year: 2021

Snake robots composed of alternating single-axis pitch and yaw joints have many internal degrees of freedom, which make them capable of versatile three-dimensional locomotion. In motion planning process, snake robot motions are often designed kinematically by a chronological sequence of continuous backbone curves that capture desired macroscopic shapes of the robot. However, as the geometric arrangement of single-axis rotary joints creates constraints on the rotations in the robot, it is challenging for the robot to reconstruct an arbitrary 3D curve. When the robot configuration does not accurately achieve the desired shapes defined by these backbone curves, the robot can have unexpected contacts with the environment, such that the robot does not achieve the desired motion. In this work, we propose a method for snake robots to reconstruct desired backbone curves by posing an optimization problem that exploits the robot's geometric structure. We verified that our method enables fast and accurate curve-configuration conversions through its applications to commonly used 3D gaits. We also demonstrated via robot experiments that 1) our method results in smooth locomotion on the robot; 2) our method allows the robot to approach the numerically predicted locomotive performance of a sequence of continuous backbone curve.
ESAD: End-to-end Deep Semi-supervised Anomaly Detection
Venue: arXiv cs.LG 2021
Year: 2021

This paper explores semi-supervised anomaly detection, a more practical setting for anomaly detection where a small additional set of labeled samples are provided. We propose a new KL-divergence based objective function for semi-supervised anomaly detection, and show that two factors: the mutual information between the data and latent representations, and the entropy of latent representations, constitute an integral objective function for anomaly detection. To resolve the contradiction in simultaneously optimizing the two factors, we propose a novel encoder-decoder-encoder structure, with the first encoder focusing on optimizing the mutual information and the second encoder focusing on optimizing the entropy. The two encoders are enforced to share similar encoding with a consistent constraint on their latent representations. Extensive experiments have revealed that the proposed method significantly outperforms several state-of-the-arts on multiple benchmark datasets, including medical diagnosis and several classic anomaly detection benchmarks.
Constructive Master's Thesis Work in Industry: Guidelines for Applying Design Science Research
Venue: arXiv cs.SE 2021
Year: 2021

Context: Software engineering researchers and practitioners rely on empirical evidence from the field. Thus, education of software engineers must include strong and applied education in empirical research methods. For most students, the master's thesis is the last, but also most applied form of this education in their studies. Problem: Especially thesis work in collaboration with industry requires that concerns of stakeholders from academia and practice are carefully balanced. It is possible, yet difficult to do high-impact empirical work within the timeframe of a typical thesis. In particular, if this research aims to provide practical value to industry, academic quality can suffer. Even though constructive research methods such as Design Science Research (DSR) exist, thesis projects repeatably struggle to apply them. Principle solution idea: DSR enables balancing such concerns by providing room both for knowledge questions and design work. Yet, only limited experience exists in our field on how to make this research method work within the context of a master's thesis. To enable running design science master's theses in collaboration with industry, we complement existing method descriptions and guidelines with our own experience and pragmatic advice to students, examiners, and supervisors in academia and industry. Method: This paper itself is based on DSR. Based on 12 design science theses over the last seven years, we collect common pitfalls and good practice from analysing the theses, the student-supervisor interaction, the supervisor-industry interaction, the examiner feedback, and, where available, reviewer comments on publications that are based on such theses. Results: We provide concrete advise for framing research questions, structuring a report, as well as for planning and conducting empirical work with practitioners.
Hybrid Quantum Computing -- Tabu Search Algorithm for Partitioning Problems: preliminary study on the Traveling Salesman Problem
Venue: arXiv cs.ET 2021
Year: 2021

Quantum Computing is considered as the next frontier in computing, and it is attracting a lot of attention from the current scientific community. This kind of computation provides to researchers with a revolutionary paradigm for addressing complex optimization problems, offering a significant speed advantage and an efficient search ability. Anyway, Quantum Computing is still in an incipient stage of development. For this reason, present architectures show certain limitations, which have motivated the carrying out of this paper. In this paper, we introduce a novel solving scheme coined as hybrid Quantum Computing - Tabu Search Algorithm. Main pillars of operation of the proposed method are a greater control over the access to quantum resources, and a considerable reduction of non-profitable accesses. To assess the quality of our method, we have used 7 different Traveling Salesman Problem instances as benchmarking set. The obtained outcomes support the preliminary conclusion that our algorithm is an approach which offers promising results for solving partitioning problems while it drastically reduces the access to quantum computing resources. We also contribute to the field of Transfer Optimization by developing an evolutionary multiform multitasking algorithm as initialization method.
Implicit Regularization in ReLU Networks with the Square Loss
Venue: arXiv cs.LG 2021
Year: 2021

Understanding the implicit regularization (or implicit bias) of gradient descent has recently been a very active research area. However, the implicit regularization in nonlinear neural networks is still poorly understood, especially for regression losses such as the square loss. Perhaps surprisingly, we prove that even for a single ReLU neuron, it is impossible to characterize the implicit regularization with the square loss by any explicit function of the model parameters (although on the positive side, we show it can be characterized approximately). For one hidden-layer networks, we prove a similar result, where in general it is impossible to characterize implicit regularization properties in this manner, except for the "balancedness" property identified in Du et al. [2018]. Our results suggest that a more general framework than the one considered so far may be needed to understand implicit regularization for nonlinear predictors, and provides some clues on what this framework should be.
MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos
Venue: arXiv cs.CV 2021
Year: 2021

Semantic aware reconstruction is more advantageous than geometric-only reconstruction for future robotic and AR/VR applications because it represents not only where things are, but also what things are. Object-centric mapping is a task to build an object-level reconstruction where objects are separate and meaningful entities that convey both geometry and semantic information. In this paper, we present MOLTR, a solution to object-centric mapping using only monocular image sequences and camera poses. It is able to localise, track, and reconstruct multiple objects in an online fashion when an RGB camera captures a video of the surrounding. Given a new RGB frame, MOLTR firstly applies a monocular 3D detector to localise objects of interest and extract their shape codes that represent the object shapes in a learned embedding space. Detections are then merged to existing objects in the map after data association. Motion state (i.e. kinematics and the motion status) of each object is tracked by a multiple model Bayesian filter and object shape is progressively refined by fusing multiple shape code. We evaluate localisation, tracking, and reconstruction on benchmarking datasets for indoor and outdoor scenes, and show superior performance over previous approaches.