Miracle-dz/UniFAR-FTUs huggingface dataset:
The symbiosis of scientific and technological innovation efficiency and economic efficiency in China — an analysis based on data envelopment analysis and logistic model

ABSTRACT In this study, data envelopment analysis (DEA) is used to estimate the scientific and technological innovation efficiency and economic efficiency of China’s provinces and cities from 2000 to 2016. Based on these two efficiency measures, a logistic model is used to calculate their symbiosis. The results show that, while the high scientific and technological innovation efficiency are concentrated in the eastern coastal areas, the high economic efficiency spread from the eastern coastal zones to the inland regions. Further, these efficiencies have gradually shifted from independent development to symbiotic development between 2000 and 2007. From 2008 to 2011, the symbiotic point largely spread, emphasising the difference between regions. After 2012, the symbiotic point gathered around 0 again, indicating that external influences greatly influenced the symbiotic patterns in all regions. Finally, we introduce relevant policy recommendations based on the symbiotic condition.
Re-conceptualizing technology adoption in informal settlements based on a Namibian application

Technology Adoption in underserved communities still presents a set of challenges, which despite adapting established technology adoption models and considering infrastructures has not been resolved. In a quest to address this issue, we engaged a group of inhabitants of an informal settlement in Windhoek to develop a localized technology adoption strategy. Over three cycles of action research, we have refined and validated the strategy, applied to a newly developed technology, which was successfully adopted. Our empirical work reveals the inadequacy of technology adoption models based on the Theory of Reasoned Action and Planned Behavior. Supported by psychological theories, we propose a novel technology adoption model, which includes enablers, social influence, and capabilities directly leading to system use.
Supporting entrepreneurship with digital services in emerging economies

This research project focuses on the ways the micro-entrepreneurs in the emerging economies can utilise the digital services in sustaining their livelihood. The project has instruments to bring digital services to new users via transportable solar power and mobile networks. The research uses the sustainable livelihoods approach in its analysis and different qualitative methods to reach its goals.
Using smartphones to take eye images for disease diagnosis in developing countries

Mobile health applications can increase access to healthcare in low and middle-income countries, where the penetration of smartphones is rapidly increasing. However, despite a large amount of clinical trials of mobile health applications, there is little proof of beneficial clinical outcomes. Recent studies suggest that one of the reasons is user experience issues. The aim of our work is to improve the user experience of acquiring medical eye images with a smartphone camera. We present EyeGuide, a mobile application that assists people in taking eye images. Possible application areas are the algorithmic detection of diseases and the support of clinicians during tele-medicine examinations.
DILAF: A framework for distributed analysis of large‐scale system logs for anomaly detection

System logs constitute a rich source of information for detection and prediction of anomalies. However, they can include a huge volume of data, which is usually unstructured or semistructured. We introduce DILAF, a framework for distributed analysis of large‐scale system logs for anomaly detection. DILAF is comprised of several processes to facilitate log parsing, feature extraction, and machine learning activities. It has two distinguishing features with respect to the existing tools. First, it does not require the availability of source code of the analyzed system. Second, it is designed to perform all the processes in a distributed manner to support scalable analysis in the context of large‐scale distributed systems. We discuss the software architecture of DILAF and we introduce an implementation of it. We conducted controlled experiments based on two datasets to evaluate the effectiveness of the framework. In particular, we evaluated the performance and scalability attributes under various degrees of parallelism. Results showed that DILAF can maintain the same accuracy levels while achieving more than 30% performance improvement on average as the system scales, compared to baseline approaches that do not employ fully distributed processing.
A non‐parametric statistical test method to detect significant cross‐outliers in spatial points

In spatial points that describe geographical events, outliers that deviate significantly from the global or local distribution indicate extraordinary geographical phenomena. Existing outlier detection methods cannot statistically identify significant outliers by considering the co‐occurrences of multiple categories of spatial points. Therefore, this study develops a non‐parametric statistical test method to detect significant cross‐outliers from two categories of spatial points divided into primary and reference ones. Firstly, the cross K function is employed to test the significance of co‐occurrences of the two categories of points, and the co‐occurrence intensity of each primary point is quantitatively defined as the number of reference points in its circular buffer. Then, the edge‐constrained Delaunay triangulation is utilized to construct a reasonable spatial neighborhood for each primary point. On this basis, the Monte Carlo method is utilized to simulate the distribution of reference points in the support domain of each primary point delineated using the α‐shape algorithm, and those primary points with significantly different local co‐occurrence intensity are determined as spatial cross‐outliers. Finally, those cross‐outliers obtained using different buffers are evaluated further by analysis of living intervals. Experiments on both simulated and real‐life datasets demonstrate the effectiveness and practicability of the proposed method.
Set Transformer

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.
EcoRNN: Efficient Computing of LSTM RNN Training on GPUs

The Long-Short-Term-Memory Recurrent Neural Network (LSTM RNN) is a state-of-the-art machine learning model for analyzing sequential data. It, however, has low training throughput and is limited by memory capacity when running on GPUs. Our results from profiling the LSTM RNN-based Neural Machine Translation (NMT) model reveal that the feature maps of the attention layers form the major memory bottleneck and the fully-connected layers form the runtime bottleneck when training on a modern GPU. We identify the key reason why the attention layers disproportionately consume so much GPU memory: their computation exhibits an O-shape (i.e. small input and output size, but large amounts of intermediate values). We observe that the repetitive nature of LSTM RNNs can be exploited to improve performance of the fully-connected layers by better utilizing the GPU memory hierarchy. Based on these observations, we propose EcoRNN that incorporates two optimizations that significantly reduce the memory footprint and runtime. (1) Partial forward propagation reduces the memory footprint by stashing the small input values of the attention layers and then recomputing the feature maps during the backward passes, instead of saving them in memory. (2) Data layout optimization boosts runtime performance of fully-connected layers by transposing the data between row-major and column-major layout. It is transparent to programmers since EcoRNN automatically selects the best implementation using model hyperparameters. We integrate EcoRNN into MXNet and evaluate it on the NMT and word-level language modeling task. On the NMT model, EcoRNN halves the memory consumption without losing any training accuracy or runtime performance. This memory footprint reduction can be converted into a 1.5× reduction in training time and energy consumption by training with a larger batch size. On the word-level language modeling task, EcoRNN improves the training throughput by up to 2× over the MXNet default implementation and 1.2× over cuDNN, the state-of-the-art deep learning library.
Teaching robots to imitate a human with no on-teacher sensors. What are the key challenges?

In this paper, we consider the problem of learning object manipulation tasks from human demonstration using RGB or RGB-D cameras. We highlight the key challenges in capturing sufficiently good data with no tracking devices - starting from sensor selection and accurate 6DoF pose estimation to natural language processing. In particular, we focus on two showcases: gluing task with a glue gun and simple block-stacking with variable blocks. Furthermore, we discuss how a linguistic description of the task could help to improve the accuracy of task description. We also present the whole architecture of our transfer of the imitated task to the simulated and real robot environment.
Semantic Matching by Weakly Supervised 2D Point Set Registration

In this paper we address the problem of establishing correspondences between different instances of the same object. The problem is posed as finding the geometric transformation that aligns a given image pair. We use a convolutional neural network (CNN) to directly regress the parameters of the transformation model. The alignment problem is defined in the setting where an unordered set of semantic key-points per image are available, but, without the correspondence information. To this end we propose a novel loss function based on cyclic consistency that solves this 2D point set registration problem by inferring the optimal geometric transformation model parameters. We train and test our approach on a standard benchmark dataset Proposal-Flow (PF-PASCAL). The proposed approach achieves state-of-the-art results demonstrating the effectiveness of the method. In addition, we show our approach further benefits from additional training samples in PF-PASCAL generated by using category level information.
Multi-fidelity Bayesian Optimization with Max-value Entropy Search

Bayesian optimization (BO) is an effective tool for black-box optimization in which objective function evaluation is usually quite expensive. In practice, lower fidelity approximations of the objective function are often available. Recently, multi-fidelity Bayesian optimization (MFBO) has attracted considerable attention because it can dramatically accelerate the optimization process by using those cheaper observations. We propose a novel information theoretic approach to MFBO. Information-based approaches are popular and empirically successful in BO, but existing studies for information-based MFBO are plagued by difficulty for accurately estimating the information gain. Our approach is based on a variant of information-based BO called max-value entropy search (MES), which greatly facilitates evaluation of the information gain in MFBO. In fact, computations of our acquisition function is written analytically except for one dimensional integral and sampling, which can be calculated efficiently and accurately. We demonstrate effectiveness of our approach by using synthetic and benchmark datasets, and further we show a real-world application to materials science data.
High Dimensional Robust Estimation of Sparse Models via Trimmed Hard Thresholding

We study the problem of sparsity constrained $M$-estimation with arbitrary corruptions to both {\em explanatory and response} variables in the high-dimensional regime, where the number of variables $d$ is larger than the sample size $n$. Our main contribution is a highly efficient gradient-based optimization algorithm that we call Trimmed Hard Thresholding -- a robust variant of Iterative Hard Thresholding (IHT) by using trimmed mean in gradient computations. Our algorithm can deal with a wide class of sparsity constrained $M$-estimation problems, and we can tolerate a nearly dimension independent fraction of arbitrarily corrupted samples. More specifically, when the corrupted fraction satisfies $\epsilon \lesssim {1} /\left({\sqrt{k} \log (nd)}\right)$, where $k$ is the sparsity of the parameter, we obtain accurate estimation and model selection guarantees with optimal sample complexity. Furthermore, we extend our algorithm to sparse Gaussian graphical model (precision matrix) estimation via a neighborhood selection approach. We demonstrate the effectiveness of robust estimation in sparse linear, logistic regression, and sparse precision matrix estimation on synthetic and real-world US equities data.
Distributed Learning of Decentralized Control Policies for Articulated Mobile Robots

State-of-the-art distributed algorithms for reinforcement learning rely on multiple independent agents, which simultaneously learn in parallel environments1 while asynchronously updating a common, shared policy. Moreover, decentralized control architectures (e.g., central pattern generators) can coordinate spatially distributed portions of an articulated robot to achieve system-level objectives. In this paper, we investigate the relationship between distributed learning and decentralized control by learning decentralized control policies for the locomotion of articulated robots in challenging environments. To this end, we present an approach that leverages the structure of the asynchronous advantage actor-critic (A3C) algorithm to provide a natural means of learning decentralized control policies on a single articulated robot. Our primary contribution shows individual agents in the A3C algorithm can be defined by independently controlled portions of the robot's body, thus enabling distributed learning on a single robot for efficient hardware implementation. We present results of closed-loop locomotion in unstructured terrains on a snake and a hexapod robot, using decentralized controllers learned offline and online, respectively, as a natural means to cover the different key applications of our approach. For the snake robot, we are optimizing the forward progression in unstructured environments, but for the hexapod robot, the goal is to maintain a stabilized body pose. Our results show that the proposed approach can be adapted to many different types of articulated robots by controlling some of their independent parts in a distributed manner, and the decentralized policy can be trained with high sample efficiency.
Really should we pruning after model be totally trained? Pruning based on a small amount of training

Pre-training of models in pruning algorithms plays an important role in pruning decision-making. We find that excessive pre-training is not necessary for pruning algorithms. According to this idea, we propose a pruning algorithm— Incremental pruning based on less training (IPLT). Compared with the traditional pruning algorithm based on a large number of pre-training, IPLT has competitive compression effect than the traditional pruning algorithm under the same simple pruning strategy. On the premise of ensuring accuracy, IPLT can achieve 8x-9x compression for VGG-19 on CIFAR-10 and only needs to pre-train few epochs. For VGG-19 on CIFAR-10, we can not only achieve 10 times test acceleration, but also about 10 times training acceleration. At present, the research mainly focuses on the compression and acceleration in the application stage of the model, while the compression and acceleration in the training stage are few. We newly proposed a pruning algorithm that can compress and accelerate in the training stage. It is novel to consider the amount of pre-training required by pruning algorithm. Our results have implications: Too much pre-training may be not necessary for pruning algorithms.
Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation

Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied. However, it is often the case that data are abundant in some domains but scarce in others. Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain. In general, this requires learning plausible mappings between domains. CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint. However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data. In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction. We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised. In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain. Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models. In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model. Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.
Fundamental Limits of Approximate Gradient Coding

In the distributed graident coding problem, it has been established that, to exactly recover the gradient under s slow machines, the mmimum computation load (number of stored data partitions) of each worker is at least linear ($s+1$), which incurs a large overhead when s is large~\citetandon2017gradient. In this paper, we focus on approximate gradient coding that aims to recover the gradient with bounded error ε. Theoretically, our main contributions are three-fold: (i) we analyze the structure of optimal gradient codes, and derive the information-theoretical lower bound of minimum computation load: $O(łog(n)/łog(n/s))$ for ε = 0$ and $d\geq O(łog(1/ε)/łog(n/s))$ for ε>0$, where d is the computation load, and ε is the error in the gradient computation; (ii) we design two approximate gradient coding schemes that exactly match such lower bounds based on random edge removal process; (iii) we implement our schemes and demonstrate the advantage of the approaches over the current fastest gradient coding strategies. The proposed schemes provide order-wise improvement over the state of the art in terms of computation load, and are also optimal in terms of both computation load and latency.
A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization

This paper aims to explore models based on the extreme gradient boosting (XGBoost) approach for business risk classification. Feature selection (FS) algorithms and hyper-parameter optimizations are simultaneously considered during model training. The five most commonly used FS methods including weight by Gini, weight by Chi-square, hierarchical variable clustering, weight by correlation, and weight by information are applied to alleviate the effect of redundant features. Two hyper-parameter optimization approaches, random search (RS) and Bayesian tree-structured Parzen Estimator (TPE), are applied in XGBoost. The effect of different FS and hyper-parameter optimization methods on the model performance are investigated by the Wilcoxon Signed Rank Test. The performance of XGBoost is compared to the traditionally utilized logistic regression (LR) model in terms of classification accuracy, area under the curve (AUC), recall, and F1 score obtained from the 10-fold cross validation. Results show that hierarchical clustering is the optimal FS method for LR while weight by Chi-square achieves the best performance in XG-Boost. Both TPE and RS optimization in XGBoost outperform LR significantly. TPE optimization shows a superiority over RS since it results in a significantly higher accuracy and a marginally higher AUC, recall and F1 score. Furthermore, XGBoost with TPE tuning shows a lower variability than the RS method. Finally, the ranking of feature importance based on XGBoost enhances the model interpretation. Therefore, XGBoost with Bayesian TPE hyper-parameter optimization serves as an operative while powerful approach for business risk modeling.
Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN

Network modeling is a critical component for building self-driving Software-Defined Networks, particularly to find optimal routing schemes that meet the goals set by administrators. However, existing modeling techniques do not meet the requirements to provide accurate estimations of relevant performance metrics such as delay and jitter. In this paper we propose a novel Graph Neural Network (GNN) model able to understand the complex relationship between topology, routing and input traffic to produce accurate estimates of the per-source/destination pair mean delay and jitter. GNN are tailored to learn and model information structured as graphs and as a result, our model is able to generalize over arbitrary topologies, routing schemes and variable traffic intensity. In the paper we show that our model provides accurate estimates of delay and jitter (worst case R2 = 0.86) when testing against topologies, routing and traffic not seen during training. In addition, we present the potential of the model for network operation by presenting several use-cases that show its effective use in per-source/destination pair delay/jitter routing optimization and its generalization capabilities by reasoning in topologies and routing schemes not seen during training.
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Recent works have cast some light on the mystery of why deep nets fit any data and generalize despite being very overparametrized. This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter characterization of training speed than recent papers, an explanation for why training a neural net with random labels leads to slower training, as originally observed in [Zhang et al. ICLR'17]. (ii) Generalization bound independent of network size, using a data-dependent complexity measure. Our measure distinguishes clearly between random labels and true labels on MNIST and CIFAR, as shown by experiments. Moreover, recent papers require sample complexity to increase (slowly) with the size, while our sample complexity is completely independent of the network size. (iii) Learnability of a broad class of smooth functions by 2-layer ReLU nets trained via gradient descent. The key idea is to track dynamics of training and generalization via properties of a related kernel.