OpenRaiser/Intern-Atlas huggingface dataset:
Context Query Simulation for Smart Carparking Scenarios in the Melbourne CDB
Venue: arXiv cs.DB 2023
Year: 2023

The rapid growth in Internet of Things (IoT) has ushered in the way for better context-awareness enabling more smarter applications. Although for the growth in the number of IoT devices, Context Management Platforms (CMPs) that integrate different domains of IoT to produce context information lacks scalability to cater to a high volume of context queries. Research in scalability and adaptation in CMPs are of significant importance due to this reason. However, there is limited methods to benchmarks and validate research in this area due to the lack of sizable sets of context queries that could simulate real-world situations, scenarios, and scenes. Commercially collected context query logs are not publicly accessible and deploying IoT devices, and context consumers in the real-world at scale is expensive and consumes a significant effort and time. Therefore, there is a need to develop a method to reliably generate and simulate context query loads that resembles real-world scenarios to test CMPs for scale. In this paper, we propose a context query simulator for the context-aware smart car parking scenario in Melbourne Central Business District in Australia. We present the process of generating context queries using multiple real-world datasets and publicly accessible reports, followed by the context query execution process. The context query generator matches the popularity of places with the different profiles of commuters, preferences, and traffic variations to produce a dataset of context query templates containing 898,050 records. The simulator is executable over a seven-day profile which far exceeds the simulation time of any IoT system simulator. The context query generation process is also generic and context query language independent.
Compressibility-Aware Quantum Algorithms on Strings
Venue: arXiv cs.DS 2023
Year: 2023

Sublinear time quantum algorithms have been established for many fundamental problems on strings. This work demonstrates that new, faster quantum algorithms can be designed when the string is highly compressible. We focus on two popular and theoretically significant compression algorithms -- the Lempel-Ziv77 algorithm (LZ77) and the Run-length-encoded Burrows-Wheeler Transform (RL-BWT), and obtain the results below. We first provide a quantum algorithm running in $\tilde{O}(\sqrt{zn})$ time for finding the LZ77 factorization of an input string $T[1..n]$ with $z$ factors. Combined with multiple existing results, this yields an $\tilde{O}(\sqrt{rn})$ time quantum algorithm for finding the RL-BWT encoding with $r$ BWT runs. Note that $r = \tildeΘ(z)$. We complement these results with lower bounds proving that our algorithms are optimal (up to polylog factors). Next, we study the problem of compressed indexing, where we provide a $\tilde{O}(\sqrt{rn})$ time quantum algorithm for constructing a recently designed $\tilde{O}(r)$ space structure with equivalent capabilities as the suffix tree. This data structure is then applied to numerous problems to obtain sublinear time quantum algorithms when the input is highly compressible. For example, we show that the longest common substring of two strings of total length $n$ can be computed in $\tilde{O}(\sqrt{zn})$ time, where $z$ is the number of factors in the LZ77 factorization of their concatenation. This beats the best known $\tilde{O}(n^\frac{2}{3})$ time quantum algorithm when $z$ is sufficiently small.
The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus
Venue: arXiv cs.LG 2023
Year: 2023

One of the unsolved challenges in the field of Explainable AI (XAI) is determining how to most reliably estimate the quality of an explanation method in the absence of ground truth explanation labels. Resolving this issue is of utmost importance as the evaluation outcomes generated by competing evaluation methods (or ''quality estimators''), which aim at measuring the same property of an explanation method, frequently present conflicting rankings. Such disagreements can be challenging for practitioners to interpret, thereby complicating their ability to select the best-performing explanation method. We address this problem through a meta-evaluation of different quality estimators in XAI, which we define as ''the process of evaluating the evaluation method''. Our novel framework, MetaQuantus, analyses two complementary performance characteristics of a quality estimator: its resilience to noise and reactivity to randomness, thus circumventing the need for ground truth labels. We demonstrate the effectiveness of our framework through a series of experiments, targeting various open questions in XAI such as the selection and hyperparameter optimisation of quality estimators. Our work is released under an open-source license (https://github.com/annahedstroem/MetaQuantus) to serve as a development tool for XAI- and Machine Learning (ML) practitioners to verify and benchmark newly constructed quality estimators in a given explainability context. With this work, we provide the community with clear and theoretically-grounded guidance for identifying reliable evaluation methods, thus facilitating reproducibility in the field.
Diminished Diversity-of-Thought in a Standard Large Language Model
Venue: arXiv cs.HC 2023
Year: 2023

We test whether Large Language Models (LLMs) can be used to simulate human participants in social-science studies. To do this, we run replications of 14 studies from the Many Labs 2 replication project with OpenAI's text-davinci-003 model, colloquially known as GPT3.5. Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results. However, we were unable to analyse the remaining six studies due to an unexpected phenomenon we call the "correct answer" effect. Different runs of GPT3.5 answered nuanced questions probing political orientation, economic preference, judgement, and moral philosophy with zero or near-zero variation in responses: with the supposedly "correct answer." In one exploratory follow-up study, we found that a "correct answer" was robust to changing the demographic details that precede the prompt. In another, we found that most but not all "correct answers" were robust to changing the order of answer choices. One of our most striking findings occurred in our replication of the Moral Foundations Theory survey results, where we found GPT3.5 identifying as a political conservative in 99.6% of the cases, and as a liberal in 99.3% of the cases in the reverse-order condition. However, both self-reported 'GPT conservatives' and 'GPT liberals' showed right-leaning moral foundations. Our results cast doubts on the validity of using LLMs as a general replacement for human participants in the social sciences. Our results also raise concerns that a hypothetical AI-led future may be subject to a diminished diversity-of-thought.
MSDA: Monocular Self-supervised Domain Adaptation for 6D Object Pose Estimation
Venue: arXiv cs.CV 2023
Year: 2023

Acquiring labeled 6D poses from real images is an expensive and time-consuming task. Though massive amounts of synthetic RGB images are easy to obtain, the models trained on them suffer from noticeable performance degradation due to the synthetic-to-real domain gap. To mitigate this degradation, we propose a practical self-supervised domain adaptation approach that takes advantage of real RGB(-D) data without needing real pose labels. We first pre-train the model with synthetic RGB images and then utilize real RGB(-D) images to fine-tune the pre-trained model. The fine-tuning process is self-supervised by the RGB-based pose-aware consistency and the depth-guided object distance pseudo-label, which does not require the time-consuming online differentiable rendering. We build our domain adaptation method based on the recent pose estimator SC6D and evaluate it on the YCB-Video dataset. We experimentally demonstrate that our method achieves comparable performance against its fully-supervised counterpart while outperforming existing state-of-the-art approaches.
Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
Venue: arXiv cs.HC 2023
Year: 2023

Artificial Intelligence (AI) brings advancements to support pathologists in navigating high-resolution tumor images to search for pathology patterns of interest. However, existing AI-assisted tools have not realized this promised potential due to a lack of insight into pathology and HCI considerations for pathologists' navigation workflows in practice. We first conducted a formative study with six medical professionals in pathology to capture their navigation strategies. By incorporating our observations along with the pathologists' domain knowledge, we designed NaviPath -- a human-AI collaborative navigation system. An evaluation study with 15 medical professionals in pathology indicated that: (i) compared to the manual navigation, participants saw more than twice the number of pathological patterns in unit time with NaviPath, and (ii) participants achieved higher precision and recall against the AI and the manual navigation on average. Further qualitative analysis revealed that navigation was more consistent with NaviPath, which can improve the overall examination quality.
Hybrid Spiking Neural Network Fine-tuning for Hippocampus Segmentation
Venue: arXiv cs.NE 2023
Year: 2023

Over the past decade, artificial neural networks (ANNs) have made tremendous advances, in part due to the increased availability of annotated data. However, ANNs typically require significant power and memory consumptions to reach their full potential. Spiking neural networks (SNNs) have recently emerged as a low-power alternative to ANNs due to their sparsity nature. SNN, however, are not as easy to train as ANNs. In this work, we propose a hybrid SNN training scheme and apply it to segment human hippocampi from magnetic resonance images. Our approach takes ANN-SNN conversion as an initialization step and relies on spike-based backpropagation to fine-tune the network. Compared with the conversion and direct training solutions, our method has advantages in both segmentation accuracy and training efficiency. Experiments demonstrate the effectiveness of our model in achieving the design goals.
Semi-Supervised Visual Tracking of Marine Animals using Autonomous Underwater Vehicles
Venue: arXiv cs.CV 2023
Year: 2023

In-situ visual observations of marine organisms is crucial to developing behavioural understandings and their relations to their surrounding ecosystem. Typically, these observations are collected via divers, tags, and remotely-operated or human-piloted vehicles. Recently, however, autonomous underwater vehicles equipped with cameras and embedded computers with GPU capabilities are being developed for a variety of applications, and in particular, can be used to supplement these existing data collection mechanisms where human operation or tags are more difficult. Existing approaches have focused on using fully-supervised tracking methods, but labelled data for many underwater species are severely lacking. Semi-supervised trackers may offer alternative tracking solutions because they require less data than fully-supervised counterparts. However, because there are not existing realistic underwater tracking datasets, the performance of semi-supervised tracking algorithms in the marine domain is not well understood. To better evaluate their performance and utility, in this paper we provide (1) a novel dataset specific to marine animals located at http://warp.whoi.edu/vmat/, (2) an evaluation of state-of-the-art semi-supervised algorithms in the context of underwater animal tracking, and (3) an evaluation of real-world performance through demonstrations using a semi-supervised algorithm on-board an autonomous underwater vehicle to track marine animals in the wild.
BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Pretrained Language Models (PLMs) harbor inherent social biases that can result in harmful real-world implications. Such social biases are measured through the probability values that PLMs output for different social groups and attributes appearing in a set of test sentences. However, bias testing is currently cumbersome since the test sentences are generated either from a limited set of manual templates or need expensive crowd-sourcing. We instead propose using ChatGPT for the controllable generation of test sentences, given any arbitrary user-specified combination of social groups and attributes appearing in the test sentences. When compared to template-based methods, our approach using ChatGPT for test sentence generation is superior in detecting social bias, especially in challenging settings such as intersectional biases. We present an open-source comprehensive bias testing framework (BiasTestGPT), hosted on HuggingFace, that can be plugged into any open-source PLM for bias testing. User testing with domain experts from various fields has shown their interest in being able to test modern AI for social biases. Our tool has significantly improved their awareness of such biases in PLMs, proving to be learnable and user-friendly. We thus enable seamless open-ended social bias testing of PLMs by domain experts through an automatic large-scale generation of diverse test sentences for any combination of social categories and attributes.
Two-sided convexity testing with certificates
Venue: arXiv cs.CG 2023
Year: 2023

We revisit the problem of property testing for convex position for point sets in $\mathbb{R}^d$. Our results draw from previous ideas of Czumaj, Sohler, and Ziegler (ESA 2000). First, the algorithm is redesigned and its analysis is revised for correctness. Second, its functionality is expanded by (i)~exhibiting both negative and positive certificates along with the convexity determination, and (ii)~significantly extending the input range for moderate and higher dimensions. The behavior of the randomized tester is as follows: (i)~if $P$ is in convex position, it accepts; (ii)~if $P$ is far from convex position, with probability at least $2/3$, it rejects and outputs a $(d+2)$-point witness of non-convexity as a negative certificate; (iiii)~if $P$ is close to convex position, with probability at least $2/3$, it accepts and outputs an approximation of the largest subset in convex position. The algorithm examines a sublinear number of points and runs in subquadratic time for every dimension $d$.
How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval
Venue: arXiv cs.IR 2023
Year: 2023

Various techniques have been developed in recent years to improve dense retrieval (DR), such as unsupervised contrastive learning and pseudo-query generation. Existing DRs, however, often suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, which some argue was due to the limited model capacity. We contradict this hypothesis and show that a generalizable DR can be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size. In particular, we systematically examine the contrastive learning of DRs, under the framework of Data Augmentation (DA). Our study shows that common DA practices such as query augmentation with generative models and pseudo-relevance label creation using a cross-encoder, are often inefficient and sub-optimal. We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As a result, DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction (ColBERTv2 and SPLADE++).
Advancing Software Citation Implementation (Software Citation Workshop 2022)
Venue: arXiv cs.DL 2023
Year: 2023

Software is foundationally important to scientific and social progress, however, traditional acknowledgment of the use of others' work has not adapted in step with the rapid development and use of software in research. This report outlines a series of collaborative discussions that brought together an international group of stakeholders and experts representing many communities, forms of labor, and expertise. Participants addressed specific challenges about software citation that have so far gone unresolved. The discussions took place in summer 2022 both online and in-person and involved a total of 51 participants. The activities described in this paper were intended to identify and prioritize specific software citation problems, develop (potential) interventions, and lay out a series of mutually supporting approaches to address them. The outcomes of this report will be useful for the GLAM (Galleries, Libraries, Archives, Museums) community, repository managers and curators, research software developers, and publishers.
Design, Performance, and Complexity of CRC-Aided List Decoding of Convolutional and Polar Codes for Short Messages
Venue: arXiv cs.IT 2023
Year: 2023

Motivated by the need to communicate short control messages in 5G and beyond, this paper carefully designs codes for cyclic redundancy check (CRC)-aided list decoding of tail-biting convolutional codes (TBCCs) and polar codes. Both codes send a 32-bit message using an 11-bit CRC and 512 transmitted bits. We aim to provide a careful, fair comparison of the error performance and decoding complexity of polar and TBCC techniques for a specific case. Specifically, a TBCC is designed to match the rate of a (512, 43) polar code, and optimal 11-bit CRCs for both codes are designed. The paper examines the distance spectra of the polar and TBCC codes, illuminating the different distance structures for the two code types. We consider both adaptive and non-adaptive CRC-aided list decoding schemes. For polar codes, an adaptive decoder must start with a larger list size to avoid an error floor. For rate-32/512 codes with an 11-bit CRC, the optimized CRC-TBCC design achieves a lower total failure rate than the optimized CRC-polar design. Simulations showed that the optimized CRC-TBCC design achieved significantly higher throughput than the optimized CRC-polar design, so that the TBCC solution achieved a lower total failure rate while requiring less computational complexity.
Feature-Enhanced Network with Hybrid Debiasing Strategies for Unbiased Learning to Rank
Venue: arXiv cs.IR 2023
Year: 2023

Unbiased learning to rank (ULTR) aims to mitigate various biases existing in user clicks, such as position bias, trust bias, presentation bias, and learn an effective ranker. In this paper, we introduce our winning approach for the "Unbiased Learning to Rank" task in WSDM Cup 2023. We find that the provided data is severely biased so neural models trained directly with the top 10 results with click information are unsatisfactory. So we extract multiple heuristic-based features for multi-fields of the results, adjust the click labels, add true negatives, and re-weight the samples during model training. Since the propensities learned by existing ULTR methods are not decreasing w.r.t. positions, we also calibrate the propensities according to the click ratios and ensemble the models trained in two different ways. Our method won the 3rd prize with a DCG@10 score of 9.80, which is 1.1% worse than the 2nd and 25.3% higher than the 4th.
On the Role of Preemption for Timing Metrics in Coded Multipath Communication
Venue: arXiv cs.NI 2023
Year: 2023

Recent trends in communication networks have focused on Quality of Service (QoS) requirements expressed through timing metrics such as latency or Age of Information (AoI). A possible way to achieve this is coded multipath communication: redundancy is added to a block of information through a robust packet-level code, transmitting across multiple independent channels to reduce the impact of blockages or rate fluctuation. The number of these links can grow significantly over traditional two-path schemes: in these scenarios, the optimization of the timing metrics is non-trivial, and latency and AoI might require different settings. In particular, packet preemption is often the optimal solution to optimize AoI in uncoded communication, but can significantly reduce the reliability of individual blocks. In this work, we model the multipath communication as a fork-join D/M/(K,N)/L queue, where K blocks of information are encoded into N>K redundant blocks. We derive the latency and Peak AoI (PAoI) distributions for different values of the queue size L. Our results show that preemption is not always the optimal choice, as dropping a late packet on one path might affect the reliability of the whole block, and that minimizing the PAoI leads to poor latency performance.
Vulnerability Analysis of Digital Banks' Mobile Applications
Venue: arXiv cs.CR 2023
Year: 2023

There is a rapid increase in the number of mobile banking applications' users due to an increase in smart mobile devices. Mobile banking is a financial transaction and service offered through mobile devices. Almost all financial institutions now provide mobile banking services to their customers. However, the security of mobile banking applications is of huge concern because of the amount of personal data and information they collect. If an attacker gets hold of personal information, they can access bank payment or card accounts. This research aims to analyze the vulnerability of the UK digital banks' applications to identify vulnerabilities in the apps and proffer countermeasures that can help improve the security of the bank applications. Androbugs, a vulnerability scanner, was used to analyze the vulnerability of six digital banks' android applications. Starling, Monese, Atom bank, Transferwise, Monzo, and Revolut were scanned. All the scanned digital banks' applications have vulnerabilities; however, some have more vulnerabilities than others. For example, Revolut's mobile application has the highest number of identified vulnerabilities. Therefore, there is need for more security in the digital banks' applications as well as other mobile banking applications.
Uncertainty-Estimation with Normalized Logits for Out-of-Distribution Detection
Venue: arXiv cs.LG 2023
Year: 2023

Out-of-distribution (OOD) detection is critical for preventing deep learning models from making incorrect predictions to ensure the safety of artificial intelligence systems. Especially in safety-critical applications such as medical diagnosis and autonomous driving, the cost of incorrect decisions is usually unbearable. However, neural networks often suffer from the overconfidence issue, making high confidence for OOD data which are never seen during training process and may be irrelevant to training data, namely in-distribution (ID) data. Determining the reliability of the prediction is still a difficult and challenging task. In this work, we propose Uncertainty-Estimation with Normalized Logits (UE-NL), a robust learning method for OOD detection, which has three main benefits. (1) Neural networks with UE-NL treat every ID sample equally by predicting the uncertainty score of input data and the uncertainty is added into softmax function to adjust the learning strength of easy and hard samples during training phase, making the model learn robustly and accurately. (2) UE-NL enforces a constant vector norm on the logits to decouple the effect of the increasing output norm from optimization process, which causes the overconfidence issue to some extent. (3) UE-NL provides a new metric, the magnitude of uncertainty score, to detect OOD data. Experiments demonstrate that UE-NL achieves top performance on common OOD benchmarks and is more robust to noisy ID data that may be misjudged as OOD data by other methods.