OpenRaiser/Intern-Atlas huggingface dataset:
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Venue: arXiv cs.CL 2022
Year: 2022

Knowledge distillation (KD) methods compress large models into smaller students with manually-designed student architectures given pre-specified computational cost. This requires several trials to find a viable student, and further repeating the process for each student or computational budget change. We use Neural Architecture Search (NAS) to automatically distill several compressed students with variable cost from a large model. Current works train a single SuperLM consisting of millions of subnetworks with weight-sharing, resulting in interference between subnetworks of different sizes. Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (K=3 for typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e.g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training. Fully task-agnostic training and search allow students to be reused for fine-tuning on any downstream task. Experiments on GLUE benchmark against state-of-the-art KD and NAS methods demonstrate AutoDistil to outperform leading compression techniques with upto 2.7x reduction in computational cost and negligible loss in task performance.
Composing a surrogate observation operator for sequential data assimilation
Venue: arXiv cs.LG 2022
Year: 2022

In data assimilation, state estimation is not straightforward when the observation operator is unknown. This study proposes a method for composing a surrogate operator when the true operator is unknown. A neural network is used to improve the surrogate model iteratively to decrease the difference between the observations and the results of the surrogate model. A twin experiment suggests that the proposed method outperforms approaches that tentatively use a specific operator throughout the data assimilation process.
Towards Fast and Accurate Federated Learning with non-IID Data for Cloud-Based IoT Applications
Venue: arXiv cs.LG 2022
Year: 2022

As a promising method of central model training on decentralized device data while securing user privacy, Federated Learning (FL)is becoming popular in Internet of Things (IoT) design. However, when the data collected by IoT devices are highly skewed in a non-independent and identically distributed (non-IID) manner, the accuracy of vanilla FL method cannot be guaranteed. Although there exist various solutions that try to address the bottleneck of FL with non-IID data, most of them suffer from extra intolerable communication overhead and low model accuracy. To enable fast and accurate FL, this paper proposes a novel data-based device grouping approach that can effectively reduce the disadvantages of weight divergence during the training of non-IID data. However, since our grouping method is based on the similarity of extracted feature maps from IoT devices, it may incur additional risks of privacy exposure. To solve this problem, we propose an improved version by exploiting similarity information using the Locality-Sensitive Hashing (LSH) algorithm without exposing extracted feature maps. Comprehensive experimental results on well-known benchmarks show that our approach can not only accelerate the convergence rate, but also improve the prediction accuracy for FL with non-IID data.
An Open Data and Geo-based Information Systems
Venue: arXiv cs.IR 2022
Year: 2022

Barangay is the smallest type of government in the Philippines, and it is driven and represented by its barangay authorities. The barangay officials are accountable for keeping the records of citizens health and crime incidents. It also the first-hand source of information of the national government to develop government programs, community services, and maintain peace and order. This paper presents a developed a web-based information system incorporating open data and geo-based features for a pilot community in the Philippines. This system serves as a platform for information collection and used for planning, analysis, decision-making and increase effectiveness and efficiency of government services in the community.
Transport Capacity Optimization for Resource Allocation in Tera-IoT Networks
Venue: arXiv cs.IT 2022
Year: 2022

We present a new adaptive resource optimization strategy that jointly allocates the subwindow and transmit power in multi-device terahertz (THz) band Internet of Things (Tera-IoT) networks. Unlike the prior studies focusing mostly on maximizing the sum distance, we incorporate both rate and transmission distance into the objective function of our problem formulation with key features of THz bands, including the spreading and molecular absorption losses. More specifically, as a performance metric of Tera-IoT networks, we adopt the transport capacity (TC), which is defined as the sum of the rate-distance products over all users. This metric has been widely adopted in large-scale ad hoc networks, and would also be appropriate for evaluating the performance of various Tera-IoT applications. We then formulate an optimization problem that aims at maximizing the TC. Moreover, motivated by the importance of the transmission distance that is very limited due to the high path loss in THz bands, our optimization problem is extended to the case of allocating the subwindow, transmit power, and transmission distance. We show how to solve our problems via an effective two-stage resource allocation strategy. We demonstrate the superiority of our adaptive solution over benchmark methods via intensive numerical evaluations for various environmental setups of large-scale Tera-IoT networks.
Joint Sensing and Communication-Rate Control for Energy Efficient Mobile Crowd Sensing
Venue: arXiv cs.IT 2022
Year: 2022

Driven by the rapid growth of Internet of Things applications, tremendous data need to be collected by sensors and uploaded to the servers for further process. As a promising solution, mobile crowd sensing enables controllable sensing and transmission processes for multiple types of data in a single device. In this paper, a typical user is considered that is required to sense and transmit data to a server, while it is assumed to remain busy and incapable of sensing data during an interval. An optimization problem is formulated to minimize the energy consumption of data sensing and transmission by controlling the sensing and transmission rates over time, subject to the constraints on the sensing data sizes, transmission data sizes, data casualty, and sensing busy time. This problem is highly challenging, due to the coupling between the rates as well as the existence of the busy time. To deal with this problem, we first show that it can be equivalently decomposed into two subproblems, corresponding to a search for the amount of data size that needs to be sensed before the busy time (referred to as the height), as well as the sensing and transmission rate control given the height. Next, we show that the latter problem can be efficiently solved by using the classical string-pulling method, while an efficient algorithm is proposed to progressively find the optimal height without the exhaustive search. Moreover, the solution approach is extended to a more complex scenario where there is a finite-size buffer at the server for receiving data. Last, simulations are conducted to evaluate the performance of the proposed design.
Design of Outdoor Autonomous Moble Robot
Venue: arXiv cs.RO 2022
Year: 2022

This study presents the design of a six-wheeled outdoor autonomous mobile robot. The main design goal of our robot is to increase its adaptability and flexibility when moving outdoors. This six-wheeled robot platform was equipped with some sensors, such as a global positioning system (GPS), high definition (HD) webcam, light detection and ranging (LiDAR), and rotary encoders. A personal mobile computer and 86Duino ONE microcontroller were used as the algorithm computing platform. In terms of control, the lateral offset and head angle offset of the robot were calculated using a differential GPS or a camera to detect structured and unstructured road boundaries. The lateral offset and head angle offset were fed to a fuzzy controller. The control input was designed by Q-learning of the differential speed between the left and right wheels. This made the robot track a reference route so that it could stay in its own lane. 2D LiDAR was also used to measure the relative distance from the front obstacle. The robot would immediately stop to avoid a collision when the distance between the robot and obstacle was less than a specific safety distance. A custom-designed rocker arm gave the robot the ability to climb a low step. Body balance could be maintained by controlling the angle of the rocker arm when the robot changed its pose. The autonomous mobile robot has been used for delivery service on our campus road by integrating the above system functionality.
Hand Gesture Recognition of Dumb Person Using one Against All Neural Network
Venue: arXiv cs.CV 2022
Year: 2022

We propose a new technique for recognition of dumb person hand gesture in real world environment. In this technique, the hand image containing the gesture is preprocessed and then hand region is segmented by convergent the RGB color image to L.a.b color space. Only few statistical features are used to classify the segmented image to different classes. Artificial Neural Network is trained in sequential manner using one against all. When the system gets trained, it becomes capable of recognition of each class in parallel manner. The result of proposed technique is much better than existing techniques.
Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity
Venue: arXiv cs.DC 2022
Year: 2022

More than 70% of cloud computing is paid for but sits idle. A large fraction of these idle compute are cheap CPUs with few cores that are not utilized during the less busy hours. This paper aims to enable those CPU cycles to train heavyweight AI models. Our goal is against mainstream frameworks, which focus on leveraging expensive specialized ultra-high bandwidth interconnect to address the communication bottleneck in distributed neural network training. This paper presents a distributed model-parallel training framework that enables training large neural networks on small CPU clusters with low Internet bandwidth. We build upon the adaptive sparse training framework introduced by the SLIDE algorithm. By carefully deploying sparsity over distributed nodes, we demonstrate several orders of magnitude faster model parallel training than Horovod, the main engine behind most commercial software. We show that with reduced communication, due to sparsity, we can train close to a billion parameter model on simple 4-16 core CPU nodes connected by basic low bandwidth interconnect. Moreover, the training time is at par with some of the best hardware accelerators.
Stochastic Neural Networks with Infinite Width are Deterministic
Venue: arXiv cs.LG 2022
Year: 2022

This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infinity, its predictive variance on the training set decreases to zero. Our theory justifies the common intuition that adding stochasticity to the model can help regularize the model by introducing an averaging effect. Two common examples that our theory can be relevant to are neural networks with dropout and Bayesian latent variable models in a special limit. Our result thus helps better understand how stochasticity affects the learning of neural networks and potentially design better architectures for practical problems.
Efficient Near-Optimal Codes for General Repeat Channels
Venue: arXiv cs.IT 2022
Year: 2022

Given a probability distribution $\mathcal{D}$ over the non-negative integers, a $\mathcal{D}$-repeat channel acts on an input symbol by repeating it a number of times distributed as $\mathcal{D}$. For example, the binary deletion channel ($\mathcal{D}=Bernoulli$) and the Poisson repeat channel ($\mathcal{D}=Poisson$) are special cases. We say a $\mathcal{D}$-repeat channel is square-integrable if $\mathcal{D}$ has finite first and second moments. In this paper, we construct explicit codes for all square-integrable $\mathcal{D}$-repeat channels with rate arbitrarily close to the capacity, that are encodable and decodable in linear and quasi-linear time, respectively. We also consider possible extensions to the repeat channel model, and illustrate how our construction can be extended to an even broader class of channels capturing insertions, deletions, and substitutions. Our work offers an alternative, simplified, and more general construction to the recent work of Rubinstein (arXiv:2111.00261), who attains similar results to ours in the cases of the deletion channel and the Poisson repeat channel. It also slightly improves the runtime and decoding failure probability of the polar codes constructions of Tal et al. (ISIT 2019) and of Pfister and Tal (arXiv:2102.02155) for the deletion channel and certain insertion/deletion/substitution channels. Our techniques follow closely the approaches of Guruswami and Li (IEEEToIT 2019) and Con and Shpilka (IEEEToIT 2020); what sets apart our work is that we show that a capacity-achieving code can be assumed to have an "approximate balance" in the frequency of zeros and ones of all sufficiently long substrings of all codewords. This allows us to attain near-capacity-achieving codes in a general setting. We consider this "approximate balance" result to be of independent interest, as it can be cast in much greater generality than repeat channels.
Co-Regularized Adversarial Learning for Multi-Domain Text Classification
Venue: arXiv cs.LG 2022
Year: 2022

Multi-domain text classification (MDTC) aims to leverage all available resources from multiple domains to learn a predictive model that can generalize well on these domains. Recently, many MDTC methods adopt adversarial learning, shared-private paradigm, and entropy minimization to yield state-of-the-art results. However, these approaches face three issues: (1) Minimizing domain divergence can not fully guarantee the success of domain alignment; (2) Aligning marginal feature distributions can not fully guarantee the discriminability of the learned features; (3) Standard entropy minimization may make the predictions on unlabeled data over-confident, deteriorating the discriminability of the learned features. In order to address the above issues, we propose a co-regularized adversarial learning (CRAL) mechanism for MDTC. This approach constructs two diverse shared latent spaces, performs domain alignment in each of them, and punishes the disagreements of these two alignments with respect to the predictions on unlabeled data. Moreover, virtual adversarial training (VAT) with entropy minimization is incorporated to impose consistency regularization to the CRAL method. Experiments show that our model outperforms state-of-the-art methods on two MDTC benchmarks.
A Brief Overview of Physics-inspired Metaheuristic Optimization Techniques
Venue: arXiv cs.AI 2022
Year: 2022

Metaheuristic algorithms are methods devised to efficiently solve computationally challenging optimization problems. Researchers have taken inspiration from various natural and physical processes alike to formulate meta-heuristics that have successfully provided near-optimal or optimal solutions to several engineering tasks. This chapter focuses on meta-heuristic algorithms modelled upon non-linear physical phenomena having a concrete optimization paradigm, having shown formidable exploration and exploitation abilities for such optimization problems. Specifically, this chapter focuses on several popular physics-based metaheuristics as well as describing the underlying unique physical processes associated with each algorithm.
Self-stabilizing Byzantine Fault-tolerant Repeated Reliable Broadcast
Venue: arXiv cs.DC 2022
Year: 2022

We study a well-known communication abstraction called Byzantine Reliable Broadcast (BRB). This abstraction is central in the design and implementation of fault-tolerant distributed systems, as many fault-tolerant distributed applications require communication with provable guarantees on message deliveries. Our study focuses on fault-tolerant implementations for message-passing systems that are prone to process-failures, such as crashes and malicious behavior. At PODC 1983, Bracha and Toueg, in short, BT, solved the BRB problem. BT has optimal resilience since it can deal with t < n/3 Byzantine processes, where n is the number of processes. The present work aims at the design of an even more robust solution than BT by expanding its fault-model with self-stabilization, a vigorous notion of fault-tolerance. In addition to tolerating Byzantine and communication failures, self-stabilizing systems can recover after the occurrence of arbitrary transient-faults. These faults represent any violation of the assumptions according to which the system was designed to operate (provided that the algorithm code remains intact). We propose, to the best of our knowledge, the first self-stabilizing Byzantine fault-tolerant (BFT) solution for repeated BRB in signature-free message-passing systems (that follows BT's problem specifications). Our contribution includes a self-stabilizing variation on a BT that solves a single-instance BRB for asynchronous systems. We also consider the problem of recycling instances of single-instance BRB. Our self-stabilizing BFT recycling for time-free systems facilitates the concurrent handling of a predefined number of BRB invocations and, by this way, can serve as the basis for self-stabilizing BFT consensus.
A Topological Centrality Measure for Directed Networks
Venue: arXiv cs.SI 2022
Year: 2022

Given a directed network $ G $, we are interested in studying the qualitative features of $ G $ which govern how perturbations propagate across $ G $. Various classical centrality measures have been already developed and proven useful to capture qualitative features and behaviors for undirected networks. In this paper, we use topological data analysis (TDA) to adapt measures of centrality to capture both directedness and non-local propagating behaviors in networks. We introduce a new metric for computing centrality in directed weighted networks, namely the quasi-centrality measure. We compute these metrics on trade networks to illustrate that our measure successfully captures propagating effects in the network and can also be used to identify sources of shocks that can disrupt the topology of directed networks. Moreover, we introduce a method that gives a hierarchical representation of the topological influences of nodes in a directed network.
Network Programming via Computable Products
Venue: arXiv cs.SE 2022
Year: 2022

The User Plane Function (UPF) aims to provide network services in the 3GPP 5G core network. These services need to be implemented on demand inexpensively with provable properties. Existing network dataplane programming languages are not up to the task. A new software paradigm is presented for the UPF. It is inspired by model checking a concurrent reactive system where conceptually each component of the system is modeled as an extended finite-state machine and their product is verified. We show how such a product can be computed for one example of a UPF and how its state invariants can be inferred, thereby eliminating the need to formally verify the product separately. Code can be generated from the product and regenerated on the fly to remain optimal for the probability distribution of network traffic the UPF must process.
Integrated Decision Control Approach for Cooperative Safety-Critical Payload Transport in a Cluttered Environment
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, the problem of coordinated transportation of heavy payload by a team of UAVs in a cluttered environment is addressed. The payload is modeled as a rigid body and is assumed to track a pre-computed global flight trajectory from a start point to a goal point. Due to the presence of local dynamic obstacles in the environment, the UAVs must ensure that there is no collision between the payload and these obstacles while ensuring that the payload oscillations are kept minimum. An Integrated Decision Controller (IDC) is proposed, that integrates the optimal tracking control law given by a centralized Model Predictive Controller with safety-critical constraints provided by the Exponential Control Barrier Functions. The entire payload-UAV system is enclosed by a safe convex hull boundary, and the IDC ensures that no obstacle enters this boundary. To evaluate the performance of the IDC, the results for a numerical simulation as well as a high-fidelity Gazebo simulation are presented. An ablation study is conducted to analyze the robustness of the proposed IDC against practical dubieties like noisy state values, relative obstacle safety margin, and payload mass uncertainty. The results clearly show that the IDC achieves both trajectory tracking and obstacle avoidance successfully while restricting the payload oscillations within a safe limit.
Lymphoma segmentation from 3D PET-CT images using a deep evidential network
Venue: arXiv cs.CV 2022
Year: 2022

An automatic evidential segmentation method based on Dempster-Shafer theory and deep learning is proposed to segment lymphomas from three-dimensional Positron Emission Tomography (PET) and Computed Tomography (CT) images. The architecture is composed of a deep feature-extraction module and an evidential layer. The feature extraction module uses an encoder-decoder framework to extract semantic feature vectors from 3D inputs. The evidential layer then uses prototypes in the feature space to compute a belief function at each voxel quantifying the uncertainty about the presence or absence of a lymphoma at this location. Two evidential layers are compared, based on different ways of using distances to prototypes for computing mass functions. The whole model is trained end-to-end by minimizing the Dice loss function. The proposed combination of deep feature extraction and evidential segmentation is shown to outperform the baseline UNet model as well as three other state-of-the-art models on a dataset of 173 patients.
Jet noise characterization for advanced pipeline leak detection
Venue: arXiv cs.LG 2022
Year: 2022

The detection of leaks in pipeline transportation systems is a matter of serious concern for operators, who pursue the integrity of their assets, the reduction of losses and the prevention of environmental hazards. Whenever a hole occurs in a pressurized pipeline, the corresponding fluid leakage is characterized by a turbulent flow and a peculiar acoustic noise, whose characteristics depend also on the size of the hole itself. This study shows that both the presence and the size of such a leaking hole can be successfully detected, by exploiting the acoustic noise (pressure transients) generated by the fluid exiting the pipe and recorded internally by hydrophones, or by considering the corresponding vibrations (e.g., acceleration signals) propagating along the external shell of the conduit. To this purpose, several experimental campaigns of acoustic noise generation have been performed using multiple calibrated nozzles on a 16 ID connection pipeline in a fuel tanks area. Detection and classification procedures are proposed to control the presence of leakages and to estimate the size of the hole, using pressure and vibration signals.