OpenRaiser/Intern-Atlas huggingface dataset:
Self-Supervised Masked Digital Elevation Models Encoding for Low-Resource Downstream Tasks
Venue: arXiv cs.CV 2023
Year: 2023

The lack of quality labeled data is one of the main bottlenecks for training Deep Learning models. As the task increases in complexity, there is a higher penalty for overfitting and unstable learning. The typical paradigm employed today is Self-Supervised learning, where the model attempts to learn from a large corpus of unstructured and unlabeled data and then transfer that knowledge to the required task. Some notable examples of self-supervision in other modalities are BERT for Large Language Models, Wav2Vec for Speech Recognition, and the Masked AutoEncoder for Vision, which all utilize Transformers to solve a masked prediction task. GeoAI is uniquely poised to take advantage of the self-supervised methodology due to the decades of data collected, little of which is precisely and dependably annotated. Our goal is to extract building and road segmentations from Digital Elevation Models (DEM) that provide a detailed topography of the earths surface. The proposed architecture is the Masked Autoencoder pre-trained on ImageNet (with the limitation that there is a large domain discrepancy between ImageNet and DEM) with an UperNet Head for decoding segmentations. We tested this model with 450 and 50 training images only, utilizing roughly 5% and 0.5% of the original data respectively. On the building segmentation task, this model obtains an 82.1% Intersection over Union (IoU) with 450 Images and 69.1% IoU with only 50 images. On the more challenging road detection task the model obtains an 82.7% IoU with 450 images and 73.2% IoU with only 50 images. Any hand-labeled dataset made today about the earths surface will be immediately obsolete due to the constantly changing nature of the landscape. This motivates the clear necessity for data-efficient learners that can be used for a wide variety of downstream tasks.
This is How You Lose the Transient Execution War
Venue: arXiv cs.CR 2023
Year: 2023

A new class of vulnerabilities related to speculative and out-of-order execution, fault-injection, and microarchitectural side channels rose to attention in 2018. The techniques behind the transient execution vulnerabilities were not new, but the combined application of the techniques was more sophisticated, and the security impact more severe, than previously considered possible. Numerous mitigations have been proposed and implemented for variants of the transient execution vulnerabilities. While Meltdown-type exception-based transient execution vulnerabilities have proven to be tractable, Spectre-type vulnerabilities and other speculation-based transient execution vulnerabilities have been far more resistant to countermeasures. A few proposed mitigations have been widely adopted by hardware vendors and software developers, but combining those commonly deployed mitigations does not produce an effective and comprehensive solution, it only protects against a small subset of the variants. Over the years, newly proposed mitigations have been trending towards more effective and comprehensive approaches with better performance, and yet, older mitigations remain the most popular despite limited security benefits and prohibitive performance penalties. If we continue this way, we can look forward to many generations of hardware debilitated by performance penalties from increasing layers of mitigations as new variants are discovered, and yet still vulnerable to both known and future variants.
Identifying Defect-Inducing Changes in Visual Code
Venue: arXiv cs.SE 2023
Year: 2023

Defects, or bugs, often form during software development. Identifying the root cause of defects is essential to improve code quality, evaluate testing methods, and support defect prediction. Examples of defect-inducing changes can be found using the SZZ algorithm to trace the textual history of defect-fixing changes back to the defect-inducing changes that they fix in line-based code. The line-based approach of the SZZ method is ineffective for visual code that represents source code graphically rather than textually. In this paper we adapt SZZ for visual code and present the "SZZ Visual Code" (SZZ-VC) algorithm, that finds changes in visual code based on the differences of graphical elements rather than differences of lines to detect defect-inducing changes. We validated the algorithm for an industry-made AAA video game and 20 music visual programming defects across 12 open source projects. Our results show that SZZ-VC is feasible for detecting defects in visual code for 3 different visual programming languages.
Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project
Venue: arXiv cs.SE 2023
Year: 2023

Video game development increasingly relies on using visual programming languages as the primary way to build video game features. The aim of using visual programming is to move game logic into the hands of game designers, who may not be as well versed in textual coding. In this paper, we empirically observe that there are more defect-inducing commits containing visual code than textual code in a AAA video game project codebase. This indicates that the existing textual code Just-in-Time (JIT) defect prediction models under evaluation by Electronic Arts (EA) may be ineffective as they do not account for changes in visual code. Thus, we focus our research on constructing visual code defect prediction models that encompass visual code metrics and evaluate the models against defect prediction models that use language agnostic features, and textual code metrics. We test our models using features extracted from the historical codebase of a AAA video game project, as well as the historical codebases of 70 open source projects that use textual and visual code. We find that defect prediction models have better performance overall in terms of the area under the ROC curve (AUC), and Mathews Correlation Coefficient (MCC) when incorporating visual code features for projects that contain more commits with visual code than textual code.
The Power of Internet of Things (IoT): Connecting the Dots with Cloud, Edge, and Fog Computing
Venue: arXiv cs.DC 2023
Year: 2023

The Internet of Things (IoT) is regarded as an improved communication system that has revolutionized traditional lifestyles. To function successfully, IoT requires a combination of cloud, fog, and edge computing architectures. Few studies have addressed cloud, fog, and edge computing simultaneously, comparing them and their issues, although several studies have looked into ways of integrating IoT with either one or two computing systems. Thus, this review provides a thorough understanding of IoT integration with these three computing architectures, as well as their respective applications and limitations. It also highlights the advantages, unresolved issues, future opportunities and directions of IoT integration with the computing systems to advance the IoT. IoT can use the Cloud's almost limitless resources to overcome technology restrictions, such as data processing, storage, and transmission. While edge computing can outperform cloud computing in many circumstances, IoT and edge computing become increasingly integrated as IoT devices increase. Cloud computing also poses a few issues, including managing time-sensitive IoT applications like video gaming, simulation, and streaming, which can be addressed by fog computing integrated with IoT. Due to the proximity of fog computing resources to the edge, data transfers and communication delays to the cloud can be reduced as a result of combining the two. The integration of IoT with cloud, fog, and edge computing will create new business prototypes and opportunities. Since IoT has the potential to greatly enhance connectivity infrastructure as an inevitable component of the future internet, further study is needed before it can be fully integrated.
Machine Learning for Tangible Effects: Natural Language Processing for Uncovering the Illicit Massage Industry & Computer Vision for Tactile Sensing
Venue: arXiv cs.CL 2023
Year: 2023

I explore two questions in this thesis: how can computer science be used to fight human trafficking? And how can computer vision create a sense of touch? I use natural language processing (NLP) to monitor the United States illicit massage industry (IMI), a multi-billion dollar industry that offers not just therapeutic massages but also commercial sexual services. Employees of this industry are often immigrant women with few job opportunities, leaving them vulnerable to fraud, coercion, and other facets of human trafficking. Monitoring spatiotemporal trends helps prevent trafficking in the IMI. By creating datasets with three publicly-accessible websites: Google Places, Rubmaps, and AMPReviews, combined with NLP techniques such as bag-of-words and Word2Vec, I show how to derive insights into the labor pressures and language barriers that employees face, as well as the income, demographics, and societal pressures affecting sex buyers. I include a call-to-action to other researchers given these datasets. I also consider how to creating synthetic financial data, which can aid with counter-trafficking in the banking sector. I use an agent-based model to create both tabular and payee-recipient graph data. I then consider the role of computer vision in making tactile sensors. I report on a novel sensor, the Digger Finger, that adapts the Gelsight sensor to finding objects in granular media. Changes include using a wedge shape to facilitate digging, replacing the internal lighting LEDs with fluorescent paint, and adding a vibrator motor to counteract jamming. Finally, I also show how to use a webcam and a printed reference marker, or fiducial, to create a low-cost six-axis force-torque sensor. This sensor is up to a hundred times less expensive than commercial sensors, allowing for a wider range of applications. For this and earlier chapters I release design files and code as open source.
Perceptual Quality Assessment of 360$^\circ$ Images Based on Generative Scanpath Representation
Venue: arXiv cs.CV 2023
Year: 2023

Despite substantial efforts dedicated to the design of heuristic models for omnidirectional (i.e., 360$^\circ$) image quality assessment (OIQA), a conspicuous gap remains due to the lack of consideration for the diversity of viewing behaviors that leads to the varying perceptual quality of 360$^\circ$ images. Two critical aspects underline this oversight: the neglect of viewing conditions that significantly sway user gaze patterns and the overreliance on a single viewport sequence from the 360$^\circ$ image for quality inference. To address these issues, we introduce a unique generative scanpath representation (GSR) for effective quality inference of 360$^\circ$ images, which aggregates varied perceptual experiences of multi-hypothesis users under a predefined viewing condition. More specifically, given a viewing condition characterized by the starting point of viewing and exploration time, a set of scanpaths consisting of dynamic visual fixations can be produced using an apt scanpath generator. Following this vein, we use the scanpaths to convert the 360$^\circ$ image into the unique GSR, which provides a global overview of gazed-focused contents derived from scanpaths. As such, the quality inference of the 360$^\circ$ image is swiftly transformed to that of GSR. We then propose an efficient OIQA computational framework by learning the quality maps of GSR. Comprehensive experimental results validate that the predictions of the proposed framework are highly consistent with human perception in the spatiotemporal domain, especially in the challenging context of locally distorted 360$^\circ$ images under varied viewing conditions. The code will be released at https://github.com/xiangjieSui/GSR
Interactive, Iterative, Tooled, Rule-Based Migration of Microsoft Access to Web Technologies
Venue: arXiv cs.SE 2023
Year: 2023

In the context of a collaboration with Berger-Levrault, an IT company producing information systems, we are working on migrating Microsoft Access monolithic applications to the web front-end and microservices back-end. Like in most software migrations, developers must learn the target technology, and they will be in charge of the evolution of the migrated system in the future. To respond to this problem, we propose the developers take over the migration project. To enable the developers to drive the migration to the target systems, we propose an Interactive, Iterative, Tooled, Rule-Based Migration approach. The contributions of this article are (i) an iterative, interactive process to language, library, GUI and architectural migration; (ii) proposal of a set of artefacts required to support such an approach; (iii) three different validations of the approach: (a) library and paradigm usage migration to Java and Pharo, (b) tables and queries migration to Java and Typescript, (c) form migration to Java Springboot and Typescript Angular.
Topological fingerprints for audio identification
Venue: arXiv cs.SD 2023
Year: 2023

We present a topological audio fingerprinting approach for robustly identifying duplicate audio tracks. Our method applies persistent homology on local spectral decompositions of audio signals, using filtered cubical complexes computed from mel-spectrograms. By encoding the audio content in terms of local Betti curves, our topological audio fingerprints enable accurate detection of time-aligned audio matchings. Experimental results demonstrate the accuracy of our algorithm in the detection of tracks with the same audio content, even when subjected to various obfuscations. Our approach outperforms existing methods in scenarios involving topological distortions, such as time stretching and pitch shifting.
ReuNify: A Step Towards Whole Program Analysis for React Native Android Apps
Venue: arXiv cs.SE 2023
Year: 2023

React Native is a widely-used open-source framework that facilitates the development of cross-platform mobile apps. The framework enables JavaScript code to interact with native-side code, such as Objective-C/Swift for iOS and Java/Kotlin for Android, via a communication mechanism provided by React Native. However, previous research and tools have overlooked this mechanism, resulting in incomplete analysis of React Native app code. To address this limitation, we have developed REUNIFY, a prototype tool that integrates the JavaScript and native-side code of React Native apps into an intermediate language that can be processed by the Soot static analysis framework. By doing so, REUNIFY enables the generation of a comprehensive model of the app's behavior. Our evaluation indicates that, by leveraging REUNIFY, the Soot-based framework can improve its coverage of static analysis for the 1,007 most popular React Native Android apps, augmenting the number of lines of Jimple code by 70%. Additionally, we observed an average increase of 84% in new nodes reached in the callgraph for these apps, after integrating REUNIFY. When REUNIFY is used for taint flow analysis, an average of two additional privacy leaks were identified. Overall, our results demonstrate that REUNIFY significantly enhances the Soot-based framework's capability to analyze React Native Android apps.
Region Generation and Assessment Network for Occluded Person Re-Identification
Venue: arXiv cs.CV 2023
Year: 2023

Person Re-identification (ReID) plays a more and more crucial role in recent years with a wide range of applications. Existing ReID methods are suffering from the challenges of misalignment and occlusions, which degrade the performance dramatically. Most methods tackle such challenges by utilizing external tools to locate body parts or exploiting matching strategies. Nevertheless, the inevitable domain gap between the datasets utilized for external tools and the ReID datasets and the complicated matching process make these methods unreliable and sensitive to noises. In this paper, we propose a Region Generation and Assessment Network (RGANet) to effectively and efficiently detect the human body regions and highlight the important regions. In the proposed RGANet, we first devise a Region Generation Module (RGM) which utilizes the pre-trained CLIP to locate the human body regions using semantic prototypes extracted from text descriptions. Learnable prompt is designed to eliminate domain gap between CLIP datasets and ReID datasets. Then, to measure the importance of each generated region, we introduce a Region Assessment Module (RAM) that assigns confidence scores to different regions and reduces the negative impact of the occlusion regions by lower scores. The RAM consists of a discrimination-aware indicator and an invariance-aware indicator, where the former indicates the capability to distinguish from different identities and the latter represents consistency among the images of the same class of human body regions. Extensive experimental results for six widely-used benchmarks including three tasks (occluded, partial, and holistic) demonstrate the superiority of RGANet against state-of-the-art methods.
On Statman's Finite Completeness Theorem
Venue: arXiv cs.LO 2023
Year: 2023

We give a complete self-contained proof of Statman's finite completeness theorem and of a corollary of this theorem stating that the $λ$-definability conjecture implies the higher-order matching conjecture.
Enhancing 5G Radio Planning with Graph Representations and Deep Learning
Venue: arXiv cs.NI 2023
Year: 2023

The roll out of new mobile network generations poses hard challenges due to various factors such as cost-benefit tradeoffs, existing infrastructure, and new technology aspects. In particular, one of the main challenges for the 5G deployment lies in optimal 5G radio coverage while accounting for diverse service performance metrics. This paper introduces a Deep Learning-based approach to assist in 5G radio planning by utilizing data from previous-generation cells. Our solution relies on a custom graph representation to leverage the information available from existing cells, and employs a Graph Neural Network (GNN) model to process such data efficiently. In our evaluation, we test its potential to model the transition from 4G to 5G NSA using real-world data from a UK mobile network operator. The experimental results show that our solution achieves high accuracy in predicting key performance indicators in new 5G cells, with a Mean Absolute Percentage Error (MAPE)~<17\% when evaluated on samples from the same area where it was trained. Moreover, we test its generalization capability over various geographical areas not included in the training, achieving a MAPE~<19\%. This suggests beneficial properties for achieving robust solutions applicable to 5G planning in new areas without the need of retraining.
Formal Verification of Chase-Lev Deque in Concurrent Separation Logic
Venue: arXiv cs.LO 2023
Year: 2023

Chase-Lev deque is a concurrent data structure designed for efficient load balancing in multiprocessor scheduling. It employs a work-stealing strategy, where each thread possesses its own work-stealing deque to store tasks, and idle threads steal tasks from other threads. However, given the inherent risk of bugs in software, particularly in a multiprocessor environment, it is crucial to formally establish the correctness of programs and data structures. To our knowledge, no formal verification work for the Chase-Lev deque has met three key criteria: (1) utilizing a minimal trusted computing base, (2) using a realistic and unrestricted implementation, and (3) proving a strong specification. In this thesis, we address this gap by presenting the formal verification of the Chase-Lev deque using a concurrent separation logic. Our work is mechanized in the Coq proof assistant, and our verified implementation is both realistic and unbounded in terms of the number of tasks it can handle. Also, we adopt linearizability as the specification, as it is widely recognized as a strong specification for concurrent data structures. Consequently, our work satisfies all three aforementioned criteria for formal verification. Additionally, we extend our verification to support safe memory reclamation, and provide a basis for verifying the Chase-Lev deque in the relaxed memory model.
Learning from Limited Heterogeneous Training Data: Meta-Learning for Unsupervised Zero-Day Web Attack Detection across Web Domains
Venue: arXiv cs.CR 2023
Year: 2023

Recently unsupervised machine learning based systems have been developed to detect zero-day Web attacks, which can effectively enhance existing Web Application Firewalls (WAFs). However, prior arts only consider detecting attacks on specific domains by training particular detection models for the domains. These systems require a large amount of training data, which causes a long period of time for model training and deployment. In this paper, we propose RETSINA, a novel meta-learning based framework that enables zero-day Web attack detection across different domains in an organization with limited training data. Specifically, it utilizes meta-learning to share knowledge across these domains, e.g., the relationship between HTTP requests in heterogeneous domains, to efficiently train detection models. Moreover, we develop an adaptive preprocessing module to facilitate semantic analysis of Web requests across different domains and design a multi-domain representation method to capture semantic correlations between different domains for cross-domain model training. We conduct experiments using four real-world datasets on different domains with a total of 293M Web requests. The experimental results demonstrate that RETSINA outperforms the existing unsupervised Web attack detection methods with limited training data, e.g., RETSINA needs only 5-minute training data to achieve comparable detection performance to the existing methods that train separate models for different domains using 1-day training data. We also conduct real-world deployment in an Internet company. RETSINA captures on average 126 and 218 zero-day attack requests per day in two domains, respectively, in one month.
Detecting unknown HTTP-based malicious communication behavior via generated adversarial flows and hierarchical traffic features
Venue: arXiv cs.CR 2023
Year: 2023

Malicious communication behavior is the network communication behavior generated by malware (bot-net, spyware, etc.) after victim devices are infected. Experienced adversaries often hide malicious information in HTTP traffic to evade detection. However, related detection methods have inadequate generalization ability because they are usually based on artificial feature engineering and outmoded datasets. In this paper, we propose an HTTP-based Malicious Communication traffic Detection Model (HMCD-Model) based on generated adversarial flows and hierarchical traffic features. HMCD-Model consists of two parts. The first is a generation algorithm based on WGAN-GP to generate HTTP-based malicious communication traffic for data enhancement. The second is a hybrid neural network based on CNN and LSTM to extract hierarchical spatial-temporal features of traffic. In addition, we collect and publish a dataset, HMCT-2020, which consists of large-scale malicious and benign traffic during three years (2018-2020). Taking the data in HMCT-2020(18) as the training set and the data in other datasets as the test set, the experimental results show that the HMCD-Model can effectively detect unknown HTTP-based malicious communication traffic. It can reach F1 = 98.66% in the dataset HMCT-2020(19-20), F1 = 90.69% in the public dataset CIC-IDS-2017, and F1 = 83.66% in the real traffic, which is 20+% higher than other representative methods on average. This validates that HMCD-Model has the ability to discover unknown HTTP-based malicious communication behavior.