OpenRaiser/Intern-Atlas huggingface dataset:
Understanding URDF: A Survey Based on User Experience
Venue: arXiv cs.RO 2023
Year: 2023

With the increasing complexity of robot systems, it is necessary to simulate them before deployment. To do this, a model of the robot's kinematics or dynamics is required. One of the most commonly used formats for modeling robots is the Unified Robot Description Format (URDF). The goal of this article is to understand how URDF is currently used, what challenges people face when working with it, and how the community sees the future of URDF. The outcome can potentially be used to guide future research. This article presents the results from a survey based on 510 anonymous responses from robotic developers of different backgrounds and levels of experience. We find that 96.8% of the participants have simulated robots before, and of them 95.5% had used URDF. We identify a number of challenges and limitations that complicate the use of URDF, such as the inability to model parallel linkages and closed-chain systems, no real standard, lack of documentation, and a limited number of dynamic parameters to model the robot. Future perspectives for URDF are also determined, where 53.5% believe URDF will be more commonly used in the future, 12.2% believe other standards or tools will make URDF obsolete, and 34.4% are not sure what the future of URDF will be. Most participants agree that there is a need for better tooling to ensure URDF's future use.
Dynamic Resource Allocation for Metaverse Applications with Deep Reinforcement Learning
Venue: arXiv cs.NI 2023
Year: 2023

This work proposes a novel framework to dynamically and effectively manage and allocate different types of resources for Metaverse applications, which are forecasted to demand massive resources of various types that have never been seen before. Specifically, by studying functions of Metaverse applications, we first propose an effective solution to divide applications into groups, namely MetaInstances, where common functions can be shared among applications to enhance resource usage efficiency. Then, to capture the real-time, dynamic, and uncertain characteristics of request arrival and application departure processes, we develop a semi-Markov decision process-based framework and propose an intelligent algorithm that can gradually learn the optimal admission policy to maximize the revenue and resource usage efficiency for the Metaverse service provider and at the same time enhance the Quality-of-Service for Metaverse users. Extensive simulation results show that our proposed approach can achieve up to 120% greater revenue for the Metaverse service providers and up to 178.9% higher acceptance probability for Metaverse application requests than those of other baselines.
Optimizing Federated Learning in LEO Satellite Constellations via Intra-Plane Model Propagation and Sink Satellite Scheduling
Venue: arXiv cs.LG 2023
Year: 2023

The advances in satellite technology developments have recently seen a large number of small satellites being launched into space on Low Earth orbit (LEO) to collect massive data such as Earth observational imagery. The traditional way which downloads such data to a ground station (GS) to train a machine learning (ML) model is not desirable due to the bandwidth limitation and intermittent connectivity between LEO satellites and the GS. Satellite edge computing (SEC), on the other hand, allows each satellite to train an ML model onboard and uploads only the model to the GS which appears to be a promising concept. This paper proposes FedLEO, a novel federated learning (FL) framework that realizes the concept of SEC and overcomes the limitation (slow convergence) of existing FL-based solutions. FedLEO (1) augments the conventional FL's star topology with ``horizontal'' intra-plane communication pathways in which model propagation among satellites takes place; (2) optimally schedules communication between ``sink'' satellites and the GS by exploiting the predictability of satellite orbiting patterns. We evaluate FedLEO extensively and benchmark it with the state of the art. Our results show that FedLEO drastically expedites FL convergence, without sacrificing -- in fact it considerably increases -- the model accuracy.
Contextual adversarial attack against aerial detection in the physical world
Venue: arXiv cs.CV 2023
Year: 2023

Deep Neural Networks (DNNs) have been extensively utilized in aerial detection. However, DNNs' sensitivity and vulnerability to maliciously elaborated adversarial examples have progressively garnered attention. Recently, physical attacks have gradually become a hot issue due to they are more practical in the real world, which poses great threats to some security-critical applications. In this paper, we take the first attempt to perform physical attacks in contextual form against aerial detection in the physical world. We propose an innovative contextual attack method against aerial detection in real scenarios, which achieves powerful attack performance and transfers well between various aerial object detectors without smearing or blocking the interested objects to hide. Based on the findings that the targets' contextual information plays an important role in aerial detection by observing the detectors' attention maps, we propose to make full use of the contextual area of the interested targets to elaborate contextual perturbations for the uncovered attacks in real scenarios. Extensive proportionally scaled experiments are conducted to evaluate the effectiveness of the proposed contextual attack method, which demonstrates the proposed method's superiority in both attack efficacy and physical practicality.
Detection and Amelioration of Social Engineering Vulnerability in Contingency Table Data using an Orthogonalised Log-linear Analysis
Venue: arXiv cs.CR 2023
Year: 2023

Social Engineering has emerged as a significant threat in cyber security. In a dialog based attack, by having enough of a potential victim's personal data to be convincing, a social engineer impersonates the victim in order to manipulate the attack's target into revealing sufficient information for accessing the victim's accounts etc. We utilise the developing understanding of human information processing in the Information Sciences to characterise the vulnerability of the target to manipulation and to propose a form of countermeasure. Our focus is on the possibility of the social engineer being able to build the victim's profile by, in part, inferring personal attribute values from statistical information available either informally, from general knowledge, or, more formally, from some public database. We use an orthogonalised log linear analysis of data in the form of a contingence table to develop a measure of how susceptible particular subtables are to probabilistic inference as the basis for our proposed countermeasure. This is based on the observation that inference relies on a high degree of non-uniformity and exploits the orthogonality of the analysis to define the measure in terms of subspace projections.
Finding Support Examples for In-Context Learning
Venue: arXiv cs.CL 2023
Year: 2023

Additionally, the strong dependency among in-context examples makes it an NP-hard combinatorial optimization problem and enumerating all permutations is infeasible. Hence we propose LENS, a fiLter-thEN-Search method to tackle this challenge in two stages: First we filter the dataset to obtain informative in-context examples individually. Specifically, we propose a novel metric, InfoScore, to evaluate the example's in-context informativeness based on the language model's feedback, and further propose a progressive filtering process to filter out uninformative examples. Then we propose diversity-guided example search which iteratively refines and evaluates the selected example permutations, to find examples that fully depict the task. The experimental results show that LENS significantly outperforms a wide range of baselines.
OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion
Venue: arXiv cs.CV 2023
Year: 2023

3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) module is proposed to better fuse 3D depth-aware features by implicitly learning the correlation between stereo images. In particular, when the input are RGBD image, a virtual stereo images can be generated through original RGB image and depth map. Besides, the Occupancy Aware Depth (OAD) module is used to obtain geometry-aware 3D features by knowledge distillation using pre-trained depth models. In addition, a reformed TartanAir benchmark, named SemanticTartanAir, is provided in this paper for further testing our OccDepth method on SSC task. Compared with the state-of-the-art RGB-inferred SSC method, extensive experiments on SemanticKITTI show that our OccDepth method achieves superior performance with improving +4.82% mIoU, of which +2.49% mIoU comes from stereo images and +2.33% mIoU comes from our proposed depth-aware method. Our code and trained models are available at https://github.com/megvii-research/OccDepth.
Random-Order Enumeration for Self-Reducible NP-Problems
Venue: arXiv cs.DS 2023
Year: 2023

In plenty of data analysis tasks, a basic and time-consuming process is to produce a large number of solutions and feed them into downstream processing. Various enumeration algorithms have been developed for this purpose. An enumeration algorithm produces all solutions of a problem instance without repetition. To be a statistically meaningful representation of the solution space, solutions are required to be enumerated in uniformly random order. This paper studies a set of self-reducible NP-problems in three hierarchies, where the problems are polynomially countable ($Sr_{NP}^{FP}$), admit FPTAS ($Sr_{NP}^{FPTAS}$), and admit FPRAS ($Sr_{NP}^{FPRAS}$), respectively. The trivial algorithm based on a (almost) uniform generator is in fact inefficient. We provide a new insight that the (almost) uniform generator is not the end of the story. More efficient algorithmic frameworks are proposed to enumerate solutions in uniformly random order for problems in these three hierarchies. (1) For problems in $Sr_{NP}^{FP}$, we show a random-order enumeration algorithm with polynomial delay (PDREnum); (2) For problems in $Sr_{NP}^{FPTAS}$, we show a Las Vegas random-order enumeration algorithm with expected polynomial delay (PDLVREnum); (3) For problems in $Sr_{NP}^{FPRAS}$, we devise a fully polynomial delay Atlantic City random-order enumeration algorithm with expected delay polynomial in the input size and the given error probability $δ$ (FPACREnum), which has a probability of at least $1-δ$ becoming a Las Vegas random-order enumeration algorithm. Finally, to further improve the efficiency of the random-order enumeration algorithms, based on the master/slave paradigm, we present a parallelization with 1.5-optimal enumeration delay and running time, along with the theoretical analysis.
Invariant Layers for Graphs with Nodes of Different Types
Venue: arXiv cs.LG 2023
Year: 2023

Neural networks that satisfy invariance with respect to input permutations have been widely studied in machine learning literature. However, in many applications, only a subset of all input permutations is of interest. For heterogeneous graph data, one can focus on permutations that preserve node types. We fully characterize linear layers invariant to such permutations. We verify experimentally that implementing these layers in graph neural network architectures allows learning important node interactions more effectively than existing techniques. We show that the dimension of space of these layers is given by a generalization of Bell numbers, extending the work (Maron et al., 2019). We further narrow the invariant network design space by addressing a question about the sizes of tensor layers necessary for function approximation on graph data. Our findings suggest that function approximation on a graph with $n$ nodes can be done with tensors of sizes $\leq n$, which is tighter than the best-known bound $\leq n(n-1)/2$. For $d \times d$ image data with translation symmetry, our methods give a tight upper bound $2d - 1$ (instead of $d^{4}$) on sizes of invariant tensor generators via a surprising connection to Davenport constants.
A collection of the data coding means and event coding means multiplexed over the 1000BASE-T1, 10GBASE-T, 10GBASE-KR, MultiGBASE-T1, 25G/40GBASE-T, 2.5G/5GBASE-T, 100GBASE-R, 10G/25G-EPON, and 10GPASS-XR microframes
Venue: arXiv cs.NI 2023
Year: 2023

This collection bundles the following memos dedicated to Ethernet protocols featuring a very long binary transport word (microframe): [1] "Data Coding Means and Event Coding Means Multiplexed Over the 1000BASE-T1 PCS Payload" (extends the IEEE 802.3bp protocol, present at pp. 1-4 in the bundle); [2] "Data Coding Means and Event Coding Means Multiplexed Over the 10GBASE-T PCS Payload" (IEEE 802.3an, pp. 5-8); [3] "Data Coding Means and Event Coding Means Multiplexed Over the 10GBASE-KR PCS Payload" (IEEE 802.3ap, pp. 9-12); [4] "Data Coding Means and Event Coding Means Multiplexed Over the MultiGBASE-T1 PCS Payload" (IEEE 802.3ch, pp. 13-16); [5] "Data Coding Means and Event Coding Means Multiplexed Over the 25G/40GBASE-T FEC Input" (IEEE 802.3bq, pp. 17-20); [6] "Data Coding Means and Event Coding Means Multiplexed Over the 2.5G/5GBASE-T FEC Input" (IEEE 802.3bz, pp. 21-24); [7] "Data Coding Means and Event Coding Means Multiplexed Over the 100GBASE-R FEC Inputs" (IEEE 802.3bj, pp. 25-28); [8] "Data Coding Means and Event Coding Means Multiplexed Over the 10G/25G-EPON FEC Inputs" (IEEE 802.3av/ca, pp. 29-32); and [9] "Data Coding Means and Event Coding Means Multiplexed Over the 10GPASS-XR FEC Inputs" (IEEE 802.3bn, pp. 33-36).
The Role of Pre-training Data in Transfer Learning
Venue: arXiv cs.CV 2023
Year: 2023

The transfer learning paradigm of model pre-training and subsequent fine-tuning produces high-accuracy models. While most studies recommend scaling the pre-training size to benefit most from transfer learning, a question remains: what data and method should be used for pre-training? We investigate the impact of pre-training data distribution on the few-shot and full fine-tuning performance using 3 pre-training methods (supervised, contrastive language-image and image-image), 7 pre-training datasets, and 9 downstream datasets. Through extensive controlled experiments, we find that the choice of the pre-training data source is essential for the few-shot transfer, but its role decreases as more data is made available for fine-tuning. Additionally, we explore the role of data curation and examine the trade-offs between label noise and the size of the pre-training dataset. We find that using 2000X more pre-training data from LAION can match the performance of supervised ImageNet pre-training. Furthermore, we investigate the effect of pre-training methods, comparing language-image contrastive vs. image-image contrastive, and find that the latter leads to better downstream accuracy
Predicting the Performance of a Computing System with Deep Networks
Venue: arXiv cs.LG 2023
Year: 2023

Predicting the performance and energy consumption of computing hardware is critical for many modern applications. This will inform procurement decisions, deployment decisions, and autonomic scaling. Existing approaches to understanding the performance of hardware largely focus around benchmarking -- leveraging standardised workloads which seek to be representative of an end-user's needs. Two key challenges are present; benchmark workloads may not be representative of an end-user's workload, and benchmark scores are not easily obtained for all hardware. Within this paper, we demonstrate the potential to build Deep Learning models to predict benchmark scores for unseen hardware. We undertake our evaluation with the openly available SPEC 2017 benchmark results. We evaluate three different networks, one fully-connected network along with two Convolutional Neural Networks (one bespoke and one ResNet inspired) and demonstrate impressive $R^2$ scores of 0.96, 0.98 and 0.94 respectively.
Deimos: A Grammar of Dynamic Embodied Immersive Visualisation Morphs and Transitions
Venue: arXiv cs.HC 2023
Year: 2023

We present Deimos, a grammar for specifying dynamic embodied immersive visualisation morphs and transitions. A morph is a collection of animated transitions that are dynamically applied to immersive visualisations at runtime and is conceptually modelled as a state machine. It is comprised of state, transition, and signal specifications. States in a morph are used to generate animation keyframes, with transitions connecting two states together. A transition is controlled by signals, which are composable data streams that can be used to enable embodied interaction techniques. Morphs allow immersive representations of data to transform and change shape through user interaction, facilitating the embodied cognition process. We demonstrate the expressivity of Deimos in an example gallery and evaluate its usability in an expert user study of six immersive analytics researchers. Participants found the grammar to be powerful and expressive, and showed interest in drawing upon Deimos' concepts and ideas in their own research.
A Formal Analysis of RANKING
Venue: arXiv cs.LO 2023
Year: 2023

We describe a formal correctness proof of RANKING, an online algorithm for online bipartite matching. An outcome of our formalisation is that it shows that there is a gap in all combinatorial proofs of the algorithm. Filling that gap constituted the majority of the effort which went into this work. This is despite the algorithm being one of the most studied algorithms and a central result in theoretical computer science. This gap is an example of difficulties in formalising graphical arguments which are ubiquitous in the theory of computing.
Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness
Venue: arXiv cs.CL 2023
Year: 2023

Generative pre-trained language models (GPLMs) like ChatGPT encode in the model's parameters knowledge the models observe during the pre-training phase. This knowledge is then used at inference to address the task specified by the user in their prompt. For example, for the question-answering task, the GPLMs leverage the knowledge and linguistic patterns learned at training to produce an answer to a user question. Aside from the knowledge encoded in the model itself, answers produced by GPLMs can also leverage knowledge provided in the prompts. For example, a GPLM can be integrated into a retrieve-then-generate paradigm where a search engine is used to retrieve documents relevant to the question; the content of the documents is then transferred to the GPLM via the prompt. In this paper we study the differences in answer correctness generated by ChatGPT when leveraging the model's knowledge alone vs. in combination with the prompt knowledge. We study this in the context of consumers seeking health advice from the model. Aside from measuring the effectiveness of ChatGPT in this context, we show that the knowledge passed in the prompt can overturn the knowledge encoded in the model and this is, in our experiments, to the detriment of answer correctness. This work has important implications for the development of more robust and transparent question-answering systems based on generative pre-trained language models.
Differentially Private Diffusion Models Generate Useful Synthetic Images
Venue: arXiv cs.LG 2023
Year: 2023

The ability to generate privacy-preserving synthetic versions of sensitive image datasets could unlock numerous ML applications currently constrained by data availability. Due to their astonishing image generation quality, diffusion models are a prime candidate for generating high-quality synthetic data. However, recent studies have found that, by default, the outputs of some diffusion models do not preserve training data privacy. By privately fine-tuning ImageNet pre-trained diffusion models with more than 80M parameters, we obtain SOTA results on CIFAR-10 and Camelyon17 in terms of both FID and the accuracy of downstream classifiers trained on synthetic data. We decrease the SOTA FID on CIFAR-10 from 26.2 to 9.8, and increase the accuracy from 51.0% to 88.0%. On synthetic data from Camelyon17, we achieve a downstream accuracy of 91.1% which is close to the SOTA of 96.5% when training on the real data. We leverage the ability of generative models to create infinite amounts of data to maximise the downstream prediction performance, and further show how to use synthetic data for hyperparameter tuning. Our results demonstrate that diffusion models fine-tuned with differential privacy can produce useful and provably private synthetic data, even in applications with significant distribution shift between the pre-training and fine-tuning distributions.
Capstone: A Capability-based Foundation for Trustless Secure Memory Access (Extended Version)
Venue: arXiv cs.CR 2023
Year: 2023

Capability-based memory isolation is a promising new architectural primitive. Software can access low-level memory only via capability handles rather than raw pointers, which provides a natural interface to enforce security restrictions. Existing architectural capability designs such as CHERI provide spatial safety, but fail to extend to other memory models that security-sensitive software designs may desire. In this paper, we propose Capstone, a more expressive architectural capability design that supports multiple existing memory isolation models in a trustless setup, i.e., without relying on trusted software components. We show how Capstone is well-suited for environments where privilege boundaries are fluid (dynamically extensible), memory sharing/delegation are desired both temporally and spatially, and where such needs are to be balanced with availability concerns. Capstone can also be implemented efficiently. We present an implementation sketch and through evaluation show that its overhead is below 50% in common use cases. We also prototype a functional emulator for Capstone and use it to demonstrate the runnable implementations of six real-world memory models without trusted software components: three types of enclave-based TEEs, a thread scheduler, a memory allocator, and Rust-style memory safety -- all within the interface of Capstone.