OpenRaiser/Intern-Atlas huggingface dataset:
A Framework of Learning Through Empirical Gain Maximization
Venue: arXiv cs.LG 2021
Year: 2021

We develop in this paper a framework of empirical gain maximization (EGM) to address the robust regression problem where heavy-tailed noise or outliers may present in the response variable. The idea of EGM is to approximate the density function of the noise distribution instead of approximating the truth function directly as usual. Unlike the classical maximum likelihood estimation that encourages equal importance of all observations and could be problematic in the presence of abnormal observations, EGM schemes can be interpreted from a minimum distance estimation viewpoint and allow the ignorance of those observations. Furthermore, it is shown that several well-known robust nonconvex regression paradigms, such as Tukey regression and truncated least square regression, can be reformulated into this new framework. We then develop a learning theory for EGM, by means of which a unified analysis can be conducted for these well-established but not fully-understood regression approaches. Resulting from the new framework, a novel interpretation of existing bounded nonconvex loss functions can be concluded. Within this new framework, the two seemingly irrelevant terminologies, the well-known Tukey's biweight loss for robust regression and the triweight kernel for nonparametric smoothing, are closely related. More precisely, it is shown that the Tukey's biweight loss can be derived from the triweight kernel. Similarly, other frequently employed bounded nonconvex loss functions in machine learning such as the truncated square loss, the Geman-McClure loss, and the exponential squared loss can also be reformulated from certain smoothing kernels in statistics. In addition, the new framework enables us to devise new bounded nonconvex loss functions for robust learning.
An Overview on Optimal Flocking
Venue: arXiv cs.MA 2021
Year: 2021

The study of robotic flocking has received considerable attention in the past twenty years. As we begin to deploy flocking control algorithms on physical multi-agent and swarm systems, there is an increasing necessity for rigorous promises on safety and performance. In this paper, we present an overview the literature focusing on optimization approaches to achieve flocking behavior that provide strong safety guarantees. We separate the literature into cluster and line flocking, and categorize cluster flocking with respect to the system-level objective, which may be realized by a reactive or planning control algorithm. We also categorize the line flocking literature by the energy-saving mechanism that is exploited by the agents. We present several approaches aimed at minimizing the communication and computational requirements in real systems via neighbor filtering and event-driven planning, and conclude with our perspective on the outlook and future research direction of optimal flocking as a field.
AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators
Venue: arXiv cs.AR 2021
Year: 2021

Adopting FPGA as an accelerator in datacenters is becoming mainstream for customized computing, but the fact that FPGAs are hard to program creates a steep learning curve for software programmers. Even with the help of high-level synthesis (HLS), accelerator designers still have to manually perform code reconstruction and cumbersome parameter tuning to achieve the optimal performance. While many learning models have been leveraged by existing work to automate the design of efficient accelerators, the unpredictability of modern HLS tools becomes a major obstacle for them to maintain high accuracy. To address this problem, we propose an automated DSE framework-AutoDSE- that leverages a bottleneck-guided coordinate optimizer to systematically find a better design point. AutoDSE detects the bottleneck of the design in each step and focuses on high-impact parameters to overcome it. The experimental results show that AutoDSE is able to identify the design point that achieves, on the geometric mean, 19.9x speedup over one CPU core for Machsuite and Rodinia benchmarks. Compared to the manually optimized HLS vision kernels in Xilinx Vitis libraries, AutoDSE can reduce their optimization pragmas by 26.38x while achieving similar performance. With less than one optimization pragma per design on average, we are making progress towards democratizing customizable computing by enabling software programmers to design efficient FPGA accelerators.
Moment Generating Function of the AoI in a Two-Source System With Packet Management
Venue: arXiv cs.IT 2021
Year: 2021

We consider a status update system consisting of two independent sources and one server in which packets of each source are generated according to the Poisson process and packets are served according to an exponentially distributed service time. We derive the moment generating function (MGF) of the age of information (AoI) for each source in the system by using the stochastic hybrid systems (SHS) under two existing source-aware packet management policies which we term self-preemptive and non-preemptive policies. In the both policies, the system (i.e., the waiting queue and the server) can contain at most two packets, one packet of each source; when the server is busy and a new packet arrives, the possible packet of the same source in the waiting queue is replaced by the fresh packet. The main difference between the policies is that in the self-preemptive policy, the packet under service is replaced upon the arrival of a new packet from the same source, whereas in the non-preemptive policy, this new arriving packet is blocked and cleared. We use the derived MGF to find the first and second moments of the AoI and show the importance of higher moments.
OWL2Vec*: Embedding of OWL Ontologies
Venue: arXiv cs.AI 2021
Year: 2021

Semantic embedding of knowledge graphs has been widely studied and used for prediction and statistical analysis tasks across various domains such as Natural Language Processing and the Semantic Web. However, less attention has been paid to developing robust methods for embedding OWL (Web Ontology Language) ontologies which can express a much wider range of semantics than knowledge graphs and have been widely adopted in domains such as bioinformatics. In this paper, we propose a random walk and word embedding based ontology embedding method named OWL2Vec*, which encodes the semantics of an OWL ontology by taking into account its graph structure, lexical information and logical constructors. Our empirical evaluation with three real world datasets suggests that OWL2Vec* benefits from these three different aspects of an ontology in class membership prediction and class subsumption prediction tasks. Furthermore, OWL2Vec* often significantly outperforms the state-of-the-art methods in our experiments.
A Large Multi-Target Dataset of Common Bengali Handwritten Graphemes
Venue: arXiv cs.CV 2021
Year: 2021

Latin has historically led the state-of-the-art in handwritten optical character recognition (OCR) research. Adapting existing systems from Latin to alpha-syllabary languages is particularly challenging due to a sharp contrast between their orthographies. The segmentation of graphical constituents corresponding to characters becomes significantly hard due to a cursive writing system and frequent use of diacritics in the alpha-syllabary family of languages. We propose a labeling scheme based on graphemes (linguistic segments of word formation) that makes segmentation in-side alpha-syllabary words linear and present the first dataset of Bengali handwritten graphemes that are commonly used in an everyday context. The dataset contains 411k curated samples of 1295 unique commonly used Bengali graphemes. Additionally, the test set contains 900 uncommon Bengali graphemes for out of dictionary performance evaluation. The dataset is open-sourced as a part of a public Handwritten Grapheme Classification Challenge on Kaggle to benchmark vision algorithms for multi-target grapheme classification. The unique graphemes present in this dataset are selected based on commonality in the Google Bengali ASR corpus. From competition proceedings, we see that deep-learning methods can generalize to a large span of out of dictionary graphemes which are absent during training. Dataset and starter codes at www.kaggle.com/c/bengaliai-cv19.
Blind Federated Learning at the Wireless Edge with Low-Resolution ADC and DAC
Venue: arXiv cs.IT 2021
Year: 2021

We study collaborative machine learning systems where a massive dataset is distributed across independent workers which compute their local gradient estimates based on their own datasets. Workers send their estimates through a multipath fading multiple access channel with orthogonal frequency division multiplexing to mitigate the frequency selectivity of the channel. We assume that there is no channel state information (CSI) at the workers, and the parameter server (PS) employs multiple antennas to align the received signals. To reduce the power consumption and the hardware costs, we employ complex-valued low-resolution digital-to-analog converters (DACs) and analog-to-digital converters (ADCs), at the transmitter and the receiver sides, respectively, and study the effects of practical low-cost DACs and ADCs on the learning performance. Our theoretical analysis shows that the impairments caused by low-resolution DACs and ADCs, including those of one-bit DACs and ADCs, do not prevent the convergence of the federated learning algorithm, and the multipath channel effects vanish when a sufficient number of antennas are used at the PS. We also validate our theoretical results via simulations, and demonstrate that using low-resolution, even one-bit, DACs and ADCs causes only a slight decrease in the learning accuracy.
ReactiFi: Reactive Programming of Wi-Fi Firmware on Mobile Devices
Venue: arXiv cs.PL 2021
Year: 2021

Network programmability will be required to handle future increased network traffic and constantly changing application needs. However, there is currently no way of using a high-level, easy to use programming language to program Wi-Fi firmware. This impedes rapid prototyping and deployment of novel network services/applications and hinders continuous performance optimization in Wi-Fi networks, since expert knowledge is required for both the used hardware platforms and the Wi-Fi domain. In this paper, we present ReactiFi, a high-level reactive programming language to program Wi-Fi chips on mobile consumer devices. ReactiFi enables programmers to implement extensions of PHY, MAC, and IP layer mechanisms without requiring expert knowledge of Wi-Fi chips, allowing for novel applications and network protocols. ReactiFi programs are executed directly on the Wi-Fi chip, improving performance and power consumption compared to execution on the main CPU. ReactiFi is conceptually similar to functional reactive languages, but is dedicated to the domain-specific needs of Wi-Fi firmware. First, it handles low-level platform-specific details without interfering with the core functionality of Wi-Fi chips. Second, it supports static reasoning about memory usage of applications, which is important for typically memory-constrained Wi-Fi chips. Third, it limits dynamic changes of dependencies between computations to dynamic branching, in order to enable static reasoning about the order of computations. We evaluate ReactiFi empirically in two real-world case studies. Our results show that throughput, latency, and power consumption are significantly improved when executing applications on the Wi-Fi chip rather than in the operating system kernel or in user space. Moreover, we show that the high-level programming abstractions of ReactiFi have no performance overhead compared to manually written C code.
Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization
Venue: arXiv cs.LG 2021
Year: 2021

Momentum methods are now used pervasively within the machine learning community for training non-convex models such as deep neural networks. Empirically, they out perform traditional stochastic gradient descent (SGD) approaches. In this work we develop a Lyapunov analysis of SGD with momentum (SGD+M), by utilizing a equivalent rewriting of the method known as the stochastic primal averaging (SPA) form. This analysis is much tighter than previous theory in the non-convex case, and due to this we are able to give precise insights into when SGD+M may out-perform SGD, and what hyper-parameter schedules will work and why.
Evaluating Multilingual BERT for Estonian
Venue: arXiv cs.CL 2021
Year: 2021

Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet available. However, there exist several multilingual BERT models that can handle multiple languages simultaneously and that have been trained also on Estonian data. In this paper, we evaluate four multilingual models -- multilingual BERT, multilingual distilled BERT, XLM and XLM-RoBERTa -- on several NLP tasks including POS and morphological tagging, NER and text classification. Our aim is to establish a comparison between these multilingual BERT models and the existing baseline neural models for these tasks. Our results show that multilingual BERT models can generalise well on different Estonian NLP tasks outperforming all baselines models for POS and morphological tagging and text classification, and reaching the comparable level with the best baseline for NER, with XLM-RoBERTa achieving the highest results compared with other multilingual models.
From Handcrafted to Deep Features for Pedestrian Detection: A Survey
Venue: arXiv cs.CV 2021
Year: 2021

Pedestrian detection is an important but challenging problem in computer vision, especially in human-centric tasks. Over the past decade, significant improvement has been witnessed with the help of handcrafted features and deep features. Here we present a comprehensive survey on recent advances in pedestrian detection. First, we provide a detailed review of single-spectral pedestrian detection that includes handcrafted features based methods and deep features based approaches. For handcrafted features based methods, we present an extensive review of approaches and find that handcrafted features with large freedom degrees in shape and space have better performance. In the case of deep features based approaches, we split them into pure CNN based methods and those employing both handcrafted and CNN based features. We give the statistical analysis and tendency of these methods, where feature enhanced, part-aware, and post-processing methods have attracted main attention. In addition to single-spectral pedestrian detection, we also review multi-spectral pedestrian detection, which provides more robust features for illumination variance. Furthermore, we introduce some related datasets and evaluation metrics, and compare some representative methods. We conclude this survey by emphasizing open problems that need to be addressed and highlighting various future directions. Researchers can track an up-to-date list at https://github.com/JialeCao001/PedSurvey.
Physical Exercise Recommendation and Success Prediction Using Interconnected Recurrent Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Unhealthy behaviors, e.g., physical inactivity and unhealthful food choice, are the primary healthcare cost drivers in developed countries. Pervasive computational, sensing, and communication technology provided by smartphones and smartwatches have made it possible to support individuals in their everyday lives to develop healthier lifestyles. In this paper, we propose an exercise recommendation system that also predicts individual success rates. The system, consisting of two inter-connected recurrent neural networks (RNNs), uses the history of workouts to recommend the next workout activity for each individual. The system then predicts the probability of successful completion of the predicted activity by the individual. The prediction accuracy of this interconnected-RNN model is assessed on previously published data from a four-week mobile health experiment and is shown to improve upon previous predictions from a computational cognitive model.
Cell Complex Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Cell complexes are topological spaces constructed from simple blocks called cells. They generalize graphs, simplicial complexes, and polyhedral complexes that form important domains for practical applications. They also provide a combinatorial formalism that allows the inclusion of complicated relationships of restrictive structures such as graphs and meshes. In this paper, we propose \textbf{Cell Complexes Neural Networks (CXNs)}, a general, combinatorial and unifying construction for performing neural network-type computations on cell complexes. We introduce an inter-cellular message passing scheme on cell complexes that takes the topology of the underlying space into account and generalizes message passing scheme to graphs. Finally, we introduce a unified cell complex encoder-decoder framework that enables learning representation of cells for a given complex inside the Euclidean spaces. In particular, we show how our cell complex autoencoder construction can give, in the special case \textbf{cell2vec}, a generalization for node2vec.
Encoded Prior Sliced Wasserstein AutoEncoder for learning latent manifold representations
Venue: arXiv cs.LG 2021
Year: 2021

While variational autoencoders have been successful in several tasks, the use of conventional priors are limited in their ability to encode the underlying structure of input data. We introduce an Encoded Prior Sliced Wasserstein AutoEncoder wherein an additional prior-encoder network learns an embedding of the data manifold which preserves topological and geometric properties of the data, thus improving the structure of latent space. The autoencoder and prior-encoder networks are iteratively trained using the Sliced Wasserstein distance. The effectiveness of the learned manifold encoding is explored by traversing latent space through interpolations along geodesics which generate samples that lie on the data manifold and hence are more realistic compared to Euclidean interpolation. To this end, we introduce a graph-based algorithm for exploring the data manifold and interpolating along network-geodesics in latent space by maximizing the density of samples along the path while minimizing total energy. We use the 3D-spiral data to show that the prior encodes the geometry underlying the data unlike conventional autoencoders, and to demonstrate the exploration of the embedded data manifold through the network algorithm. We apply our framework to benchmarked image datasets to demonstrate the advantages of learning data representations in outlier generation, latent structure, and geodesic interpolation.
AMR:Autonomous Coin Mixer with Privacy Preserving Reward Distribution
Venue: arXiv cs.CR 2021
Year: 2021

It is well known that users on open blockchains are tracked by an industry providing services to governments, law enforcement, secret services, and alike. While most blockchains do not protect their users' privacy and allow external observers to link transactions and addresses, a growing research interest attempts to design add-on privacy solutions to help users regain their privacy on non-private blockchains. In this work, we propose to our knowledge the first censorship resilient mixer, which can reward its users in a privacy-preserving manner for participating in the system. Increasing the anonymity set size, and diversity of users, is, as we believe, an important endeavor to raise a mixer's contributed privacy in practice. The paid-out rewards can take the form of governance tokens to decentralize the voting on system parameters, similar to how popular "DeFi farming" protocols operate. Moreover, by leveraging existing "Defi" lending platforms, AMR is the first mixer design that allows participating clients to earn financial interests on their deposited funds. Our system AMR is autonomous as it does not rely on any external server or third party. The evaluation of our AMR implementation shows that the system supports today on Ethereum anonymity set sizes beyond thousands of users, and a capacity of over $66,000$ deposits per day, at constant system costs. We provide a formal specification of our zksnark-based AMR system, a privacy and security analysis, implementation, and evaluation with both the MiMC and Poseidon hash functions.
Data-Efficient Pretraining via Contrastive Self-Supervision
Venue: arXiv cs.CL 2021
Year: 2021

For natural language processing `text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on increasingly larger `task-external' data. Transfer learning from high-resource pretraining works well, but research has focused on settings with very large data and compute requirements, while the potential of efficient low-resource learning, without large `task-external' pretraining, remains under-explored. In this work, we evaluate against three core challenges for resource efficient learning. Namely, we analyze: (1) pretraining data ($X$) efficiency; (2) zero to few-shot label ($Y$) efficiency; and (3) long-tail generalization, since long-tail preservation has been linked to algorithmic fairness and because data in the tail is limited by definition. To address these challenges, we propose a data and compute efficient self-supervised, contrastive text encoder, pretrained on 60MB of `task-internal' text data, and compare it to RoBERTa, which was pretrained on 160GB of `task-external' text. We find our method outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's fine-tuning time.
REQIBA: Regression and Deep Q-Learning for Intelligent UAV Cellular User to Base Station Association
Venue: arXiv cs.IT 2021
Year: 2021

Unmanned Aerial Vehicles (UAVs) are emerging as important users of next-generation cellular networks. By operating in the sky, UAV users experience very different radio conditions than terrestrial users, due to factors such as strong Line-of-Sight (LoS) channels (and interference) and Base Station (BS) antenna misalignment. As a consequence, the UAVs may experience significant degradation to their received quality of service, particularly when they are moving and are subject to frequent handovers. The solution is to allow the UAV to be aware of its surrounding environment, and intelligently connect into the cellular network taking advantage of this awareness. In this paper we present REgression and deep Q-learning for Intelligent UAV cellular user to Base station Association (REQIBA), a solution that allows a UAV flying over an urban area to intelligently connect to underlying BSs, using information about the received signal powers, the BS locations, and the surrounding building topology. We demonstrate how REQIBA can as much as double the total UAV throughput, when compared to heuristic association schemes similar to those commonly used by terrestrial users. We also evaluate how environmental factors such as UAV height, building density, and throughput loss due to handovers impact the performance of our solution.