OpenRaiser/Intern-Atlas huggingface dataset:
DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search
Venue: arXiv cs.LG 2021
Year: 2021

Deep Learning (DL) has been successfully applied to a wide range of application domains, including safety-critical ones. Several DL testing approaches have been recently proposed in the literature but none of them aims to assess how different interpretable features of the generated inputs affect the system's behaviour. In this paper, we resort to Illumination Search to find the highest-performing test cases (i.e., misbehaving and closest to misbehaving), spread across the cells of a map representing the feature space of the system. We introduce a methodology that guides the users of our approach in the tasks of identifying and quantifying the dimensions of the feature space for a given domain. We developed DeepHyperion, a search-based tool for DL systems that illuminates, i.e., explores at large, the feature space, by providing developers with an interpretable feature map where automatically generated inputs are placed along with information about the exposed behaviours.
Diff-Net: Image Feature Difference based High-Definition Map Change Detection for Autonomous Driving
Venue: arXiv cs.CV 2021
Year: 2021

Up-to-date High-Definition (HD) maps are essential for self-driving cars. To achieve constantly updated HD maps, we present a deep neural network (DNN), Diff-Net, to detect changes in them. Compared to traditional methods based on object detectors, the essential design in our work is a parallel feature difference calculation structure that infers map changes by comparing features extracted from the camera and rasterized images. To generate these rasterized images, we project map elements onto images in the camera view, yielding meaningful map representations that can be consumed by a DNN accordingly. As we formulate the change detection task as an object detection problem, we leverage the anchor-based structure that predicts bounding boxes with different change status categories. To the best of our knowledge, the proposed method is the first end-to-end network that tackles the high-definition map change detection task, yielding a single stage solution. Furthermore, rather than relying on single frame input, we introduce a spatio-temporal fusion module that fuses features from history frames into the current, thus improving the overall performance. Finally, we comprehensively validate our method's effectiveness using freshly collected datasets. Results demonstrate that our Diff-Net achieves better performance than the baseline methods and is ready to be integrated into a map production pipeline maintaining an up-to-date HD map.
GGT: Graph-Guided Testing for Adversarial Sample Detection of Deep Neural Network
Venue: arXiv cs.LG 2021
Year: 2021

Deep Neural Networks (DNN) are known to be vulnerable to adversarial samples, the detection of which is crucial for the wide application of these DNN models. Recently, a number of deep testing methods in software engineering were proposed to find the vulnerability of DNN systems, and one of them, i.e., Model Mutation Testing (MMT), was used to successfully detect various adversarial samples generated by different kinds of adversarial attacks. However, the mutated models in MMT are always huge in number (e.g., over 100 models) and lack diversity (e.g., can be easily circumvented by high-confidence adversarial samples), which makes it less efficient in real applications and less effective in detecting high-confidence adversarial samples. In this study, we propose Graph-Guided Testing (GGT) for adversarial sample detection to overcome these aforementioned challenges. GGT generates pruned models with the guide of graph characteristics, each of them has only about 5% parameters of the mutated model in MMT, and graph guided models have higher diversity. The experiments on CIFAR10 and SVHN validate that GGT performs much better than MMT with respect to both effectiveness and efficiency.
Motor Imagery Classification based on CNN-GRU Network with Spatio-Temporal Feature Representation
Venue: arXiv cs.NE 2021
Year: 2021

Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.
Online Learning for Recommendations at Grubhub
Venue: arXiv cs.IR 2021
Year: 2021

We propose a method to easily modify existing offline Recommender Systems to run online using Transfer Learning. Online Learning for Recommender Systems has two main advantages: quality and scale. Like many Machine Learning algorithms in production if not regularly retrained will suffer from Concept Drift. A policy that is updated frequently online can adapt to drift faster than a batch system. This is especially true for user-interaction systems like recommenders where the underlying distribution can shift drastically to follow user behaviour. As a platform grows rapidly like Grubhub, the cost of running batch training jobs becomes material. A shift from stateless batch learning offline to stateful incremental learning online can recover, for example, at Grubhub, up to a 45x cost savings and a +20% metrics increase. There are a few challenges to overcome with the transition to online stateful learning, namely convergence, non-stationary embeddings and off-policy evaluation, which we explore from our experiences running this system in production.
Collision Avoidance Using Spherical Harmonics
Venue: arXiv cs.RO 2021
Year: 2021

In this paper, we propose a novel optimization-based trajectory planner that utilizes spherical harmonics to estimate the collision-free solution space around an agent. The space is estimated using a constrained over-determined least-squares estimator to determine the parameters that define a spherical harmonic approximation at a given time step. Since spherical harmonics produce star-convex shapes, the planner can consider all paths that are in line-of-sight for the agent within a given radius. This contrasts with other state-of-the-art planners that generate trajectories by estimating obstacle boundaries with rough approximations and using heuristic rules to prune a solution space into one that can be easily explored. Those methods cause the trajectory planner to be overly conservative in environments where an agent must get close to obstacles to accomplish a goal. Our method is shown to perform on-par with other path planners and surpass these planners in certain environments. It generates feasible trajectories while still running in real-time and guaranteeing safety when a valid solution exists.
Genetic CFL: Optimization of Hyper-Parameters in Clustered Federated Learning
Venue: arXiv cs.AI 2021
Year: 2021

Federated learning (FL) is a distributed model for deep learning that integrates client-server architecture, edge computing, and real-time intelligence. FL has the capability of revolutionizing machine learning (ML) but lacks in the practicality of implementation due to technological limitations, communication overhead, non-IID (independent and identically distributed) data, and privacy concerns. Training a ML model over heterogeneous non-IID data highly degrades the convergence rate and performance. The existing traditional and clustered FL algorithms exhibit two main limitations, including inefficient client training and static hyper-parameter utilization. To overcome these limitations, we propose a novel hybrid algorithm, namely genetic clustered FL (Genetic CFL), that clusters edge devices based on the training hyper-parameters and genetically modifies the parameters cluster-wise. Then, we introduce an algorithm that drastically increases the individual cluster accuracy by integrating the density-based clustering and genetic hyper-parameter optimization. The results are bench-marked using MNIST handwritten digit dataset and the CIFAR-10 dataset. The proposed genetic CFL shows significant improvements and works well with realistic cases of non-IID and ambiguous data.
Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy
Venue: arXiv cs.NE 2021
Year: 2021

Evolution-based neural architecture search requires high computational resources, resulting in long search time. In this work, we propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS, which achieves better results than previous evolution-based methods while reducing the search time significantly. The architectures are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. We used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of an individual architecture to reduce the search time. We also used an architecture-fitness table (AF table) for keeping record of the already evaluated architecture, thus further reducing the search time. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU. The top architectures from the searches on CIFAR-10 and CIFAR-100 were then transferred to ImageNet, achieving the top-5 accuracy of 92.6% and 92.1%, respectively.
Auto-detecting groups based on textual similarity for group recommendations
Venue: arXiv cs.IR 2021
Year: 2021

In general, recommender systems are designed to provide personalized items to a user. But in few cases, items are recommended for a group, and the challenge is to aggregate the individual user preferences to infer the recommendation to a group. It is also important to consider the similarity of characteristics among the members of a group to generate a better recommendation. Members of an automatically identified group will have similar characteristics, and reaching a consensus with a decision-making process is preferable in this case. It requires users-items and their rating interactions over a utility matrix to auto-detect the groups in group recommendations. We may not overlook other intrinsic information to form a group. The textual information also plays a pivotal role in user clustering. In this paper, we auto-detect the groups based on the textual similarity of the metadata (review texts). We consider the order in user preferences in our models. We have conducted extensive experiments over two real-world datasets to check the efficacy of the proposed models. We have also conducted a competitive comparison with a baseline model to show the improvements in the quality of recommendations.
The Art of the Meta Stream Protocol: Torrents of Streams
Venue: arXiv cs.PL 2021
Year: 2021

The rise of streaming libraries such as Akka Stream, Reactive Extensions, and LINQ popularized the declarative functional style of data processing. The stream paradigm offers concise syntax to write down processing pipelines to consume the vast amounts of real-time data available today. These libraries offer the programmer a domain specific language (DSL) embedded in the host language to describe data streams. These libraries however, all suffer from extensibility issues. The semantics of a stream is hard-coded into the DSL language and cannot be changed by the user of the library. We introduce an approach to modify the semantics of a streaming library by means of meta-programming at both run-time and compile-time, and showcase its generality. We show that the expressiveness of the meta-facilities is strong enough to enable push and pull semantics, error handling, parallelism, and operator fusion. We evaluate our work by implementing the identified shortcomings in terms of a novel stream meta-architecture and show that its design and architecture adhere to the design principles of a meta-level architecture. The state of the art offers plenty of choice to programmers regarding reactive stream processing libraries. Expressing reactive systems is otherwise difficult to do in general purpose languages. Extensibility and fine-tuning should be possible in these libraries to ensure a broad variety of applications can be expressed within this single DSL.
A multi-schematic classifier-independent oversampling approach for imbalanced datasets
Venue: arXiv cs.LG 2021
Year: 2021

Over 85 oversampling algorithms, mostly extensions of the SMOTE algorithm, have been built over the past two decades, to solve the problem of imbalanced datasets. However, it has been evident from previous studies that different oversampling algorithms have different degrees of efficiency with different classifiers. With numerous algorithms available, it is difficult to decide on an oversampling algorithm for a chosen classifier. Here, we overcome this problem with a multi-schematic and classifier-independent oversampling approach: ProWRAS(Proximity Weighted Random Affine Shadowsampling). ProWRAS integrates the Localized Random Affine Shadowsampling (LoRAS)algorithm and the Proximity Weighted Synthetic oversampling (ProWSyn) algorithm. By controlling the variance of the synthetic samples, as well as a proximity-weighted clustering system of the minority classdata, the ProWRAS algorithm improves performance, compared to algorithms that generate synthetic samples through modelling high dimensional convex spaces of the minority class. ProWRAS has four oversampling schemes, each of which has its unique way to model the variance of the generated data. Most importantly, the performance of ProWRAS with proper choice of oversampling schemes, is independent of the classifier used. We have benchmarked our newly developed ProWRAS algorithm against five sate-of-the-art oversampling models and four different classifiers on 20 publicly available datasets. ProWRAS outperforms other oversampling algorithms in a statistically significant way, in terms of both F1-score and Kappa-score. Moreover, we have introduced a novel measure for classifier independence I-score, and showed quantitatively that ProWRAS performs better, independent of the classifier used. In practice, ProWRAS customizes synthetic sample generation according to a classifier of choice and thereby reduces benchmarking efforts.
Using Cyber Digital Twins for Automated Automotive Cybersecurity Testing
Venue: arXiv cs.CR 2021
Year: 2021

Cybersecurity testing of automotive systems has become a practical necessity, with the wide adoption of advanced driving assistance functions and vehicular communications. These functionalities require the integration of information and communication technologies that not only allow for a plethora of on-the-fly configuration abilities, but also provide a huge surface for attacks. Theses circumstances have also been recognized by standardization and regulation bodies, making the need for not only proper cybersecurity engineering but also proving the effectiveness of security measures by verification and validation through testing also a formal necessity. In order to keep pace with the rapidly growing demand of neutral-party security testing of vehicular systems, novel approaches are needed. This paper therefore presents a methodology to create and execute cybersecurity test cases on the fly in a black box setting by using pattern matching-based binary analysis and translation mechanisms to formal attack descriptions as well as model-checking techniques. The approach is intended to generate meaningful attack vectors on a system with next-to-zero a priori knowledge.
A Fixed Version of Quadratic Program in Gradient Episodic Memory
Venue: arXiv cs.LG 2021
Year: 2021

Gradient Episodic Memory is indeed a novel method for continual learning, which solves new problems quickly without forgetting previously acquired knowledge. However, in the process of studying the paper, we found there were some problems in the proof of the dual problem of Quadratic Program, so here we give our fixed version for this problem.
Local Search for Weighted Tree Augmentation and Steiner Tree
Venue: arXiv cs.DS 2021
Year: 2021

We present a technique that allows for improving on some relative greedy procedures by well-chosen (non-oblivious) local search algorithms. Relative greedy procedures are a particular type of greedy algorithm that start with a simple, though weak, solution, and iteratively replace parts of this starting solution by stronger components. Some well-known applications of relative greedy algorithms include approximation algorithms for Steiner Tree and, more recently, for connectivity augmentation problems. The main application of our technique leads to a $(1.5+ε)$-approximation for Weighted Tree Augmentation, improving on a recent relative greedy based method with approximation factor $1+\ln 2 + ε\approx 1.69$. Furthermore, we show how our local search technique can be applied to Steiner Tree, leading to an alternative way to obtain the currently best known approximation factor of $\ln 4 + ε$. Contrary to prior methods, our approach is purely combinatorial without the need to solve an LP. Nevertheless, the solution value can still be bounded in terms of the well-known hypergraphic LP, leading to an alternative, and arguably simpler, technique to bound its integrality gap by $\ln 4$.
High carbon stock mapping at large scale with optical satellite imagery and spaceborne LIDAR
Venue: arXiv cs.CV 2021
Year: 2021

The increasing demand for commodities is leading to changes in land use worldwide. In the tropics, deforestation, which causes high carbon emissions and threatens biodiversity, is often linked to agricultural expansion. While the need for deforestation-free global supply chains is widely recognized, making progress in practice remains a challenge. Here, we propose an automated approach that aims to support conservation and sustainable land use planning decisions by mapping tropical landscapes at large scale and high spatial resolution following the High Carbon Stock (HCS) approach. A deep learning approach is developed that estimates canopy height for each 10 m Sentinel-2 pixel by learning from sparse GEDI LIDAR reference data, achieving an overall RMSE of 6.3 m. We show that these wall-to-wall maps of canopy top height are predictive for classifying HCS forests and degraded areas with an overall accuracy of 86 % and produce a first high carbon stock map for Indonesia, Malaysia, and the Philippines.
Data vs classifiers, who wins?
Venue: arXiv cs.LG 2021
Year: 2021

The experiments covered by Machine Learning (ML) must consider two important aspects to assess the performance of a model: datasets and algorithms. Robust benchmarks are needed to evaluate the best classifiers. For this, one can adopt gold standard benchmarks available in public repositories. However, it is common not to consider the complexity of the dataset when evaluating. This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players (e.g., chess). For each dataset in a benchmark, the IRT is used to estimate the ability of classifiers, where good classifiers have good predictions for the most difficult test instances. Tournaments are then run for each pair of classifiers so that Glicko-2 updates performance information such as rating value, rating deviation and volatility for each classifier. A case study was conducted hereby which adopted the OpenML-CC18 benchmark as the collection of datasets and pool of various classification algorithms for evaluation. Not all datasets were observed to be really useful for evaluating algorithms, where only 10% were considered really difficult. Furthermore, the existence of a subset containing only 50% of the original amount of OpenML-CC18 was verified, which is equally useful for algorithm evaluation. Regarding the algorithms, the methodology proposed herein identified the Random Forest as the algorithm with the best innate ability.
Amodal segmentation just like doing a jigsaw
Venue: arXiv cs.CV 2021
Year: 2021

Amodal segmentation is a new direction of instance segmentation while considering the segmentation of the visible and occluded parts of the instance. The existing state-of-the-art method uses multi-task branches to predict the amodal part and the visible part separately and subtract the visible part from the amodal part to obtain the occluded part. However, the amodal part contains visible information. Therefore, the separated prediction method will generate duplicate information. Different from this method, we propose a method of amodal segmentation based on the idea of the jigsaw. The method uses multi-task branches to predict the two naturally decoupled parts of visible and occluded, which is like getting two matching jigsaw pieces. Then put the two jigsaw pieces together to get the amodal part. This makes each branch focus on the modeling of the object. And we believe that there are certain rules in the occlusion relationship in the real world. This is a kind of occlusion context information. This jigsaw method can better model the occlusion relationship and use the occlusion context information, which is important for amodal segmentation. Experiments on two widely used amodally annotated datasets prove that our method exceeds existing state-of-the-art methods. The source code of this work will be made public soon.
Data Disclosure with Non-zero Leakage and Non-invertible Leakage Matrix
Venue: arXiv cs.IT 2021
Year: 2021

We study a statistical signal processing privacy problem, where an agent observes useful data $Y$ and wants to reveal the information to a user. Since the useful data is correlated with the private data $X$, the agent employs a privacy mechanism to generate data $U$ that can be released. We study the privacy mechanism design that maximizes the revealed information about $Y$ while satisfying a strong $\ell_1$-privacy criterion. When a sufficiently small leakage is allowed, we show that the optimizer vectors of the privacy mechanism design problem have a specific geometry, i.e., they are perturbations of fixed vector distributions. This geometrical structure allows us to use a local approximation of the conditional entropy. By using this approximation the original optimization problem can be reduced to a linear program so that an approximate solution for privacy mechanism can be easily obtained. The main contribution of this work is to consider non-zero leakage with a non-invertible leakage matrix. In an example inspired by water mark application, we first investigate the accuracy of the approximation. Then, we employ different measures for utility and privacy leakage to compare the privacy-utility trade-off using our approach with other methods. In particular, it has been shown that by allowing small leakage, significant utility can be achieved using our method compared to the case where no leakage is allowed.