OpenRaiser/Intern-Atlas huggingface dataset:
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
Venue: arXiv cs.LG 2022
Year: 2022

1-bit gradient compression and local steps are two representative techniques that enable drastic communication reduction in distributed SGD. Their benefits, however, remain an open question on Adam-based large model pre-training (e.g. BERT and GPT). In this paper, we demonstrate the non-linearity in Adam causes slow convergence even when 1-bit compression or local steps are individually applied. To alleviate this limitation, we propose 0/1 Adam that linearizes each Adam step via approximating its optimizer states using their stale estimates and linear correlation. 0/1 Adam performs an Adam-like step to preserve the adaptivity, while its linearity allows utilizing 1-bit compression and local steps simultaneously for wall-clock time speed up. We provide convergence guarantee for 0/1 Adam on smooth non-convex objectives. On various large-scale benchmarks such as BERT-Base, BERT-Large, GPT-2 pre-training and ImageNet, we demonstrate on up to 128 GPUs that 0/1 Adam is able to reduce up to 87% of data volume, 54% of communication rounds, and achieve up to 2$\times$ higher training throughput and end-to-end training time reduction compared to the state-of-the-art baseline 1-bit Adam; while enjoying the same statistical convergence speed and end task model accuracy on GLUE dataset and ImageNet validation set.
Reflekt: a Library for Compile-Time Reflection in Kotlin
Venue: arXiv cs.SE 2022
Year: 2022

Reflection in Kotlin is a powerful mechanism to introspect program behavior during its execution at run-time. However, among the variety of practical tasks involving reflection, there are scenarios when the poor performance of run-time approaches becomes a significant disadvantage. This problem manifests itself in Kotless, a popular framework for developing serverless applications, because the faster the applications launch, the less their cloud infrastructure costs. In this paper, we present Reflekt - a compile-time reflection library which allows to perform the search among classes, object expressions (which in Kotlin are implemented as singleton classes), and functions in Kotlin code based on the given search query. It comes with a convenient DSL and better performance comparing to the existing run-time reflection approaches. Our experiments show that replacing run-time reflection calls with Reflekt in serverless applications created with Kotless resulted in a significant performance boost in start-up time of these applications.
Neural NID Rules
Venue: arXiv cs.LG 2022
Year: 2022

Abstract object properties and their relations are deeply rooted in human common sense, allowing people to predict the dynamics of the world even in situations that are novel but governed by familiar laws of physics. Standard machine learning models in model-based reinforcement learning are inadequate to generalize in this way. Inspired by the classic framework of noisy indeterministic deictic (NID) rules, we introduce here Neural NID, a method that learns abstract object properties and relations between objects with a suitably regularized graph neural network. We validate the greater generalization capability of Neural NID on simple benchmarks specifically designed to assess the transition dynamics learned by the model.
Hierarchical Aerial Computing for Internet of Things via Cooperation of HAPs and UAVs
Venue: arXiv cs.NI 2022
Year: 2022

With the explosive increment of computation requirements, the multi-access edge computing (MEC) paradigm appears as an effective mechanism. Besides, as for the Internet of Things (IoT) in disasters or remote areas requiring MEC services, unmanned aerial vehicles (UAVs) and high altitude platforms (HAPs) are available to provide aerial computing services for these IoT devices. In this paper, we develop the hierarchical aerial computing framework composed of HAPs and UAVs, to provide MEC services for various IoT applications. In particular, the problem is formulated to maximize the total IoT data computed by the aerial MEC platforms, restricted by the delay requirement of IoT and multiple resource constraints of UAVs and HAPs, which is an integer programming problem and intractable to solve. Due to the prohibitive complexity of exhaustive search, we handle the problem by presenting the matching game theory based algorithm to deal with the offloading decisions from IoT devices to UAVs, as well as a heuristic algorithm for the offloading decisions between UAVs and HAPs. The external effect affected by interplay of different IoT devices in the matching is tackled by the externality elimination mechanism. Besides, an adjustment algorithm is also proposed to make the best of aerial resources. The complexity of proposed algorithms is analyzed and extensive simulation results verify the efficiency of the proposed algorithms, and the system performances are also analyzed by the numerical results.
Local Differential Privacy for Federated Learning
Venue: arXiv cs.CR 2022
Year: 2022

Advanced adversarial attacks such as membership inference and model memorization can make federated learning (FL) vulnerable and potentially leak sensitive private data. Local differentially private (LDP) approaches are gaining more popularity due to stronger privacy notions and native support for data distribution compared to other differentially private (DP) solutions. However, DP approaches assume that the FL server (that aggregates the models) is honest (run the FL protocol honestly) or semi-honest (run the FL protocol honestly while also trying to learn as much information as possible). These assumptions make such approaches unrealistic and unreliable for real-world settings. Besides, in real-world industrial environments (e.g., healthcare), the distributed entities (e.g., hospitals) are already composed of locally running machine learning models (this setting is also referred to as the cross-silo setting). Existing approaches do not provide a scalable mechanism for privacy-preserving FL to be utilized under such settings, potentially with untrusted parties. This paper proposes a new local differentially private FL (named LDPFL) protocol for industrial settings. LDPFL can run in industrial settings with untrusted entities while enforcing stronger privacy guarantees than existing approaches. LDPFL shows high FL model performance (up to 98%) under small privacy budgets (e.g., epsilon = 0.5) in comparison to existing methods.
Predicting Pollution Level Using Random Forest: A Case Study of Marilao River in Bulacan Province, Philippines
Venue: arXiv cs.LG 2022
Year: 2022

This study aims to predict the pollution level that threatens the Marilao River, located in the province of Bulacan, Philippines. The inhabitants of this area are now being exposed to pollution. Contamination of this waterway comes from both formal and informal industries, such as a used lead-acid battery, open dumpsites metal refining, and other toxic metals. Using various water quality parameters like Dissolved Oxygen (DO), Potential of Hydrogen (pH), Biochemical Oxygen Demand (BOD) and Total Suspended Solids (TSS) were the basis for predicting the pollution level. This study used the Data Mining technique based on the sample data collected from January of 2013 to November of 2017. These were used as a training data and test results to predict the river condition with its corresponding pollution level classification indicated with the used of colors such as Green for Normal, Yellow for Average, Orange for Polluted and Red for Highly Polluted. The model got an accuracy of 91.75% with a Kappa value of 0.8115, interpreted as Strong in terms of the level of agreement.
Online Bayesian Recommendation with No Regret
Venue: arXiv cs.GT 2022
Year: 2022

We introduce and study the online Bayesian recommendation problem for a platform, who can observe a utility-relevant state of a product, repeatedly interacting with a population of myopic users through an online recommendation mechanism. This paradigm is common in a wide range of scenarios in the current Internet economy. For each user with her own private preference and belief, the platform commits to a recommendation strategy to utilize his information advantage on the product state to persuade the self-interested user to follow the recommendation. The platform does not know user's preferences and beliefs, and has to use an adaptive recommendation strategy to persuade with gradually learning user's preferences and beliefs in the process. We aim to design online learning policies with no Stackelberg regret for the platform, i.e., against the optimum policy in hindsight under the assumption that users will correspondingly adapt their behaviors to the benchmark policy. Our first result is an online policy that achieves double logarithm regret dependence on the number of rounds. We then present a hardness result showing that no adaptive online policy can achieve regret with better dependency on the number of rounds. Finally, by formulating the platform's problem as optimizing a linear program with membership oracle access, we present our second online policy that achieves regret with polynomial dependence on the number of states but logarithm dependence on the number of rounds.
Grasp Control of a Cable-Driven Robotic Hand Using a PVDF Slip Detection Sensor
Venue: arXiv cs.RO 2022
Year: 2022

Detecting and preventing slip is a major challenge in robotic hand operation, underpinning the robot's ability to perform safe and reliable grasps. Using the robotic hand design from the authors' earlier work, a sensing and control strategy is proposed here to prevent object slippage. The robotic hand is cable-driven, single-actuated, has five fingers, and is capable of replicating most human hand motions. The slip sensing approach utilizes a piezoelectric vibration sensor, namely, polyvinylidene fluoride (PVDF), which is a flexible, thin, cheap, and highly sensitive material. The power of the filtered PVDF signal is shown to exhibit identifiable signatures during slip, thus providing a suitable slip detection mechanism. Using the PVDF feedback, an integral controller is implemented to prevent the grasped object from falling and ensure a safe, powerful, and reliable grasp. The extension movement of the robotic hand is controlled using a bend sensor, through a proportional-integral (PI) controller. The robotic hand weights 338 gr. The functionality and robustness of the proposed slip-detection sensory system and control logic implementation are evaluated through experiments.
Study of Robust Adaptive Power allocation for the Downlink of Multiple-Antenna Systems
Venue: arXiv cs.IT 2022
Year: 2022

Multiple-input multiple-output (MIMO) systems greatly increase the overall throughput of wireless systems since they are capable of transmitting multiple streams employing the same time-frequency resources. However, this gain requires an appropriate precoder design and a power allocation technique. In general, precoders and power allocation schemes are designed assuming perfect channel estate information (CSI). Nonetheless, this is an optimistic assumption since real systems only possess partial or imperfect CSI at the transmitter (CSIT). The imperfect CSIT originates residual inter-user interference, which is detrimental for wireless systems. In this paper, two adaptive power allocation algorithms are proposed, which are more robust against CSIT imperfections than conventional techniques. Both techniques employ the mean square error as the objective function. Simulation results show that the proposed techniques obtain a higher performance in terms of sum-rate than conventional approaches.
Online Scheduling of Transmission and Processing for AoI Minimization with Edge Computing
Venue: arXiv cs.IT 2022
Year: 2022

Age of Information (AoI), which measures the time elapsed since the generation of the last received packet at the destination, is a new metric for real-time status update tracking applications. In this paper, we consider a status-update system in which a source node samples updates and sends them to an edge server over a delay channel. The received updates are processed by the server with an infinite buffer and then delivered to a destination. The channel can send only one update at a time, and the server can process one at a time as well. The source node applies generate-at-will model according to the state of the channel, the edge server, and the buffer. We aim to minimize the average AoI with \emph{independent and identically distributed} transmission time and processing time. We consider three online scheduling policies. The first one is the optimal long wait policy, under which the source node only transmits a new packet after the old one is delivered. Secondly, we propose a peak age threshold policy, under which the source node determines the sending time based on the estimated peak age of information (PAoI). Finally, we improve the peak age threshold policy by considering a postponed plan to reduce the waiting time in the buffer. The AoI performance under these policies is illustrated by numerical results with different parameters.
PQuAD: A Persian Question Answering Dataset
Venue: arXiv cs.CL 2022
Year: 2022

We present Persian Question Answering Dataset (PQuAD), a crowdsourced reading comprehension dataset on Persian Wikipedia articles. It includes 80,000 questions along with their answers, with 25% of the questions being adversarially unanswerable. We examine various properties of the dataset to show the diversity and the level of its difficulty as an MRC benchmark. By releasing this dataset, we aim to ease research on Persian reading comprehension and development of Persian question answering systems. Our experiments on different state-of-the-art pre-trained contextualized language models show 74.8% Exact Match (EM) and 87.6% F1-score that can be used as the baseline results for further research on Persian QA.
Robust Deepfake On Unrestricted Media: Generation And Detection
Venue: arXiv cs.CV 2022
Year: 2022

Recent advances in deep learning have led to substantial improvements in deepfake generation, resulting in fake media with a more realistic appearance. Although deepfake media have potential application in a wide range of areas and are drawing much attention from both the academic and industrial communities, it also leads to serious social and criminal concerns. This chapter explores the evolution of and challenges in deepfake generation and detection. It also discusses possible ways to improve the robustness of deepfake detection for a wide variety of media (e.g., in-the-wild images and videos). Finally, it suggests a focus for future fake media research.
A Tech Hybrid-Recommendation Engine and Personalized Notification: An integrated tool to assist users through Recommendations (Project ATHENA)
Venue: arXiv cs.IR 2022
Year: 2022

Project ATHENA aims to develop an application to address information overload, primarily focused on Recommendation Systems (RSs) with the personalization and user experience design of a modern system. Two machine learning (ML) algorithms were used: (1) TF-IDF for Content-based filtering (CBF); (2) Classification with Matrix Factorization- Singular Value Decomposition(SVD) applied with Collaborative filtering (CF) and mean (normalization) for prediction accuracy of the CF. Data sampling in academic Research and Development of Philippine Council for Agriculture, Aquatic, and Natural Resources Research and Development (PCAARRD) e-Library and Project SARAI publications plus simulated data used as training sets to generate a recommendation of items that uses the three RS filtering (CF, CBF, and personalized version of item recommendations). Series of Testing and TAM performed and discussed. Findings allow users to engage in online information and quickly evaluate retrieved items produced by the application. Compatibility-testing (CoT) shows the application is compatible with all major browsers and mobile-friendly. Performance-testing (PT) recommended v-parameter specs and TAM evaluations results indicate strongly associated with overall positive feedback, thoroughly enough to address the information-overload problem as the core of the paper. A modular architecture presented addressing the information overload, primarily focused on RSs with the personalization and design of modern systems. Developers utilized Two ML algorithms and prototyped a simplified version of the architecture. Series of testing (CoT and PT) and evaluations with TAM were performed and discussed. Project ATHENA added a UX feature design of a modern system.
Fine-Grained Population Mobility Data-Based Community-Level COVID-19 Prediction Model
Venue: arXiv cs.LG 2022
Year: 2022

Predicting the number of infections in the anti-epidemic process is extremely beneficial to the government in developing anti-epidemic strategies, especially in fine-grained geographic units. Previous works focus on low spatial resolution prediction, e.g., county-level, and preprocess data to the same geographic level, which loses some useful information. In this paper, we propose a fine-grained population mobility data-based model (FGC-COVID) utilizing data of two geographic levels for community-level COVID-19 prediction. We use the population mobility data between Census Block Groups (CBGs), which is a finer-grained geographic level than community, to build the graph and capture the dependencies between CBGs using graph neural networks (GNNs). To mine as finer-grained patterns as possible for prediction, a spatial weighted aggregation module is introduced to aggregate the embeddings of CBGs to community level based on their geographic affiliation and spatial autocorrelation. Extensive experiments on 300 days LA city COVID-19 data indicate our model outperforms existing forecasting models on community-level COVID-19 prediction.
Locating Charging Stations: Connected, Capacitated and Prize- Collecting
Venue: arXiv cs.DS 2022
Year: 2022

In this paper, we study locating charging station problem as facility location problem and its variants ($k$-Median, $k$-Facility location and $k$-center). We study the connectivity and the capacity constraints in these problem. Capacity and connectivity constraints have been studied in the literature separately for all these problems. We give first constant factor approximations when both the constraints are present. Extending/modifying the techniques used for connected variants of the problem to include capacities or for capacitated variants of problem to include connectivity is a tedious and challenging task. In this paper, we combine the two constraints by reducing the problem to underlying well studied problems, solving them as black box and combine the obtained solutions. We also, combine the two constraints in the prize-collection set up. In the prize-collecting set up, the problems are not even studied when one of the constraint is present. We present constant factor approximation for them as well.
I'm Hearing (Different) Voices: Anonymous Voices to Protect User Privacy
Venue: arXiv cs.CR 2022
Year: 2022

In this paper, we present AltVoice -- a system designed to help user's protect their privacy when using remotely accessed voice services. The system allows a user to conceal their true voice identity information with no cooperation from the remote voice service: AltVoice re-synthesizes user's spoken audio to sound as if it has been spoken by a different, private identity. The system converts audio to its textual representation at its midpoint, and thus removes any linkage between the user's voice and the generated private voices. We implement AltVoice and we propose six different methods to generate private voice identities, each is based on a user-known secret. We identify the system's trade-offs, and we investigate them for each of the proposed identity generation methods. Specifically, we investigate generated voices' diversity, word error rate, perceived speech quality and the success of attackers under privacy compromise and authentication compromise attack scenarios. Our results show that AltVoice-generated voices are not easily linked to original users, enabling users to protect themselves from voice data leakages and allowing for the revocability of (generated) voice data; akin to using passwords. However the results also show further work is needed on ensuring that the produced audio is natural, and that identities of private voices are distinct from one another. We discuss the future steps into improving AltVoice and the new implications that its existence has for the creations of remotely accessed voice services.
Zero-Reference Image Restoration for Under-Display Camera of UAV
Venue: arXiv cs.CV 2022
Year: 2022

The exposed cameras of UAV can shake, shift, or even malfunction under the influence of harsh weather, while the add-on devices (Dupont lines) are very vulnerable to damage. We can place a low-cost T-OLED overlay around the camera to protect it, but this would also introduce image degradation issues. In particular, the temperature variations in the atmosphere can create mist that adsorbs to the T-OLED, which can cause secondary disasters (i.e., more severe image degradation) during the UAV's filming process. To solve the image degradation problem caused by overlaying T-OLEDs, in this paper we propose a new method to enhance the visual experience by enhancing the texture and color of images. Specifically, our method trains a lightweight network to estimate a low-rank affine grid on the input image, and then utilizes the grid to enhance the input image at block granularity. The advantages of our method are that no reference image is required and the loss function is developed from visual experience. In addition, our model can perform high-quality recovery of images of arbitrary resolution in real time. In the end, the limitations of our model and the collected datasets (including the daytime and nighttime scenes) are discussed.
Learning Asymmetric Embedding for Attributed Networks via Convolutional Neural Network
Venue: arXiv cs.LG 2022
Year: 2022

Recently network embedding has gained increasing attention due to its advantages in facilitating network computation tasks such as link prediction, node classification and node clustering. The objective of network embedding is to represent network nodes in a low-dimensional vector space while retaining as much information as possible from the original network including structural, relational, and semantic information. However, asymmetric nature of directed networks poses many challenges as how to best preserve edge directions in the embedding process. Here, we propose a novel deep asymmetric attributed network embedding model based on convolutional graph neural network, called AAGCN. The main idea is to maximally preserve the asymmetric proximity and asymmetric similarity of directed attributed networks. AAGCN introduces two neighbourhood feature aggregation schemes to separately aggregate the features of a node with the features of its in- and out- neighbours. Then, it learns two embedding vectors for each node, one source embedding vector and one target embedding vector. The final representations are the results of concatenating source and target embedding vectors. We test the performance of AAGCN on three real-world networks for network reconstruction, link prediction, node classification and visualization tasks. The experimental results show the superiority of AAGCN against state-of-the-art embedding methods.