OpenRaiser/Intern-Atlas huggingface dataset:
3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V
Venue: arXiv cs.AI 2023
Year: 2023

In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.
A Comparative Evaluation of Additive Separability Tests for Physics-Informed Machine Learning
Venue: arXiv cs.LG 2023
Year: 2023

Many functions characterising physical systems are additively separable. This is the case, for instance, of mechanical Hamiltonian functions in physics, population growth equations in biology, and consumer preference and utility functions in economics. We consider the scenario in which a surrogate of a function is to be tested for additive separability. The detection that the surrogate is additively separable can be leveraged to improve further learning. Hence, it is beneficial to have the ability to test for such separability in surrogates. The mathematical approach is to test if the mixed partial derivative of the surrogate is zero; or empirically, lower than a threshold. We present and comparatively and empirically evaluate the eight methods to compute the mixed partial derivative of a surrogate function.
A merging interaction model explains human drivers' behaviour from input signals to decisions
Venue: arXiv cs.HC 2023
Year: 2023

One of the bottlenecks of automated driving technologies is safe and socially acceptable interactions with human-driven vehicles, for example during merging. Driver models that provide accurate predictions of joint and individual driver behaviour of high-level decisions, safety margins, and low-level control inputs are required to improve the interactive capabilities of automated driving. Existing driver models typically focus on one of these aspects. Unified models capturing all aspects are missing which hinders understanding of the principles that govern human traffic interactions. This in turn limits the ability of automated vehicles to resolve merging interactions. Here, we present a communication-enabled interaction model based on risk perception with the potential to capture merging interactions on all three levels. Our model accurately describes human behaviour in a simplified merging scenario, addressing both individual actions (such as velocity adjustments) and joint actions (such as the order of merging). Contrary to other interaction models, our model does not assume humans are rational and explicitly accounts for communication between drivers. Our results demonstrate that communication and risk-based decision-making explain observed human interactions on multiple levels. This explanation improves our understanding of the underlying mechanisms of human traffic interactions and poses a step towards interaction-aware automated driving.
Monoculture in Matching Markets
Venue: arXiv cs.GT 2023
Year: 2023

Algorithmic monoculture arises when many decision-makers rely on the same algorithm to evaluate applicants. An emerging body of work investigates possible harms of this kind of homogeneity, but has been limited by the challenge of incorporating market effects in which the preferences and behavior of many applicants and decision-makers jointly interact to determine outcomes. Addressing this challenge, we introduce a tractable theoretical model of algorithmic monoculture in a two-sided matching market with many participants. We use the model to analyze outcomes under monoculture (when decision-makers all evaluate applicants using a common algorithm) and under polyculture (when decision-makers evaluate applicants independently). All else equal, monoculture (1) selects less-preferred applicants when noise is well-behaved, (2) matches more applicants to their top choice, though individual applicants may be worse off depending on their value to decision-makers and risk tolerance, and (3) is more robust to disparities in the number of applications submitted.
Nonlinear In-situ Calibration of Strain-Gauge Force/Torque Sensors for Humanoid Robots
Venue: arXiv cs.RO 2023
Year: 2023

High force/torque (F/T) sensor calibration accuracy is crucial to achieving successful force estimation/control tasks with humanoid robots. State-of-the-art affine calibration models do not always approximate correctly the physical phenomenon of the sensor/transducer, resulting in inaccurate F/T measurements for specific applications such as thrust estimation of a jet-powered humanoid robot. This paper proposes and validates nonlinear polynomial models for F/T calibration, increasing the number of model coefficients to minimize the estimation residuals. The analysis of several models, based on the data collected from experiments with the iCub3 robot, shows a significant improvement in minimizing the force/torque estimation error when using higher-degree polynomials. In particular, when using a 4th-degree polynomial model, the Root Mean Square error (RMSE) decreased to 2.28N from the 4.58N obtained with an affine model, and the absolute error in the forces remained under 6N while it was reaching up to 16N with the affine model.
Accelerating Spatio-Textual Queries with Learned Indices
Venue: arXiv cs.DS 2023
Year: 2023

Efficiently computing spatio-textual queries has become increasingly important in various applications that need to quickly retrieve geolocated entities associated with textual information, such as in location-based services and social networks. To accelerate such queries, several works have proposed combining spatial and textual indices into hybrid index structures. Recently, the novel idea of replacing traditional indices with ML models has attracted a lot of attention. This includes works on learned spatial indices, where the main challenge is to address the lack of a total ordering among objects in a multidimensional space. In this work, we investigate how to extend this novel type of index design to the case of spatio-textual data. We study different design choices, based on either loose or tight coupling between the spatial and textual part, as well as a hybrid index that combines a traditional and a learned component. We also perform an experimental evaluation using several real-world datasets to assess the potential benefits of using a learned index for evaluating spatio-textual queries.
ChemTime: Rapid and Early Classification for Multivariate Time Series Classification of Chemical Sensors
Venue: arXiv cs.LG 2023
Year: 2023

Multivariate time series data are ubiquitous in the application of machine learning to problems in the physical sciences. Chemiresistive sensor arrays are highly promising in chemical detection tasks relevant to industrial, safety, and military applications. Sensor arrays are an inherently multivariate time series data collection tool which demand rapid and accurate classification of arbitrary chemical analytes. Previous research has benchmarked data-agnostic multivariate time series classifiers across diverse multivariate time series supervised tasks in order to find general-purpose classification algorithms. To our knowledge, there has yet to be an effort to survey machine learning and time series classification approaches to chemiresistive hardware sensor arrays for the detection of chemical analytes. In addition to benchmarking existing approaches to multivariate time series classifiers, we incorporate findings from a model survey to propose the novel \textit{ChemTime} approach to sensor array classification for chemical sensing. We design experiments addressing the unique challenges of hardware sensor arrays classification including the rapid classification ability of classifiers and minimization of inference time while maintaining performance for deployed lightweight hardware sensing devices. We find that \textit{ChemTime} is uniquely positioned for the chemical sensing task by combining rapid and early classification of time series with beneficial inference and high accuracy.
Simple Weak Coresets for Non-Decomposable Classification Measures
Venue: arXiv cs.LG 2023
Year: 2023

While coresets have been growing in terms of their application, barring few exceptions, they have mostly been limited to unsupervised settings. We consider supervised classification problems, and non-decomposable evaluation measures in such settings. We show that stratified uniform sampling based coresets have excellent empirical performance that are backed by theoretical guarantees too. We focus on the F1 score and Matthews Correlation Coefficient, two widely used non-decomposable objective functions that are nontrivial to optimize for and show that uniform coresets attain a lower bound for coreset size, and have good empirical performance, comparable with ``smarter'' coreset construction strategies.
Generative Context-aware Fine-tuning of Self-supervised Speech Models
Venue: arXiv cs.CL 2023
Year: 2023

When performing tasks like automatic speech recognition or spoken language understanding for a given utterance, access to preceding text or audio provides contextual information can improve performance. Considering the recent advances in generative large language models (LLM), we hypothesize that an LLM could generate useful context information using the preceding text. With appropriate prompts, LLM could generate a prediction of the next sentence or abstractive text like titles or topics. In this paper, we study the use of LLM-generated context information and propose an approach to distill the generated information during fine-tuning of self-supervised speech models, which we refer to as generative context-aware fine-tuning. This approach allows the fine-tuned model to make improved predictions without access to the true surrounding segments or to the LLM at inference time, while requiring only a very small additional context module. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks: automatic speech recognition, named entity recognition, and sentiment analysis. The results show that generative context-aware fine-tuning outperforms a context injection fine-tuning approach that accesses the ground-truth previous text, and is competitive with a generative context injection fine-tuning approach that requires the LLM at inference time.
Sample-Efficient Learning to Solve a Real-World Labyrinth Game Using Data-Augmented Model-Based Reinforcement Learning
Venue: arXiv cs.RO 2023
Year: 2023

Motivated by the challenge of achieving rapid learning in physical environments, this paper presents the development and training of a robotic system designed to navigate and solve a labyrinth game using model-based reinforcement learning techniques. The method involves extracting low-dimensional observations from camera images, along with a cropped and rectified image patch centered on the current position within the labyrinth, providing valuable information about the labyrinth layout. The learning of a control policy is performed purely on the physical system using model-based reinforcement learning, where the progress along the labyrinth's path serves as a reward signal. Additionally, we exploit the system's inherent symmetries to augment the training data. Consequently, our approach learns to successfully solve a popular real-world labyrinth game in record time, with only 5 hours of real-world training data.
Exploring Automatic Text Simplification of German Narrative Documents
Venue: arXiv cs.CL 2023
Year: 2023

In this paper, we apply transformer-based Natural Language Generation (NLG) techniques to the problem of text simplification. Currently, there are only a few German datasets available for text simplification, even fewer with larger and aligned documents, and not a single one with narrative texts. In this paper, we explore to which degree modern NLG techniques can be applied to German narrative text simplifications. We use Longformer attention and a pre-trained mBART model. Our findings indicate that the existing approaches for German are not able to solve the task properly. We conclude on a few directions for future research to address this problem.
TMP: Temporal Motion Propagation for Online Video Super-Resolution
Venue: arXiv cs.CV 2023
Year: 2023

Online video super-resolution (online-VSR) highly relies on an effective alignment module to aggregate temporal information, while the strict latency requirement makes accurate and efficient alignment very challenging. Though much progress has been achieved, most of the existing online-VSR methods estimate the motion fields of each frame separately to perform alignment, which is computationally redundant and ignores the fact that the motion fields of adjacent frames are correlated. In this work, we propose an efficient Temporal Motion Propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment among consecutive frames. Specifically, we first propagate the offsets from previous frames to the current frame, and then refine them in the neighborhood, which significantly reduces the matching space and speeds up the offset estimation process. Furthermore, to enhance the robustness of alignment, we perform spatial-wise weighting on the warped features, where the positions with more precise offsets are assigned higher importance. Experiments on benchmark datasets demonstrate that the proposed TMP method achieves leading online-VSR accuracy as well as inference speed. The source code of TMP can be found at https://github.com/xtudbxk/TMP.
Quantifying Device Usefulness -- How Useful is an Obsolete Device?
Venue: arXiv cs.HC 2023
Year: 2023

Obsolete devices add to the rising levels of electronic waste, a major environmental concern, and a contributing factor to climate change. In recent years, device manufacturers have established environmental commitments and launched initiatives such as supporting the recycling of obsolete devices by making more ways available for consumers to safely dispose of their old devices. However, little support is available for individuals who want to continue using legacy or 'end-of-life' devices and few studies have explored the usefulness of these older devices, the barriers to their continued use and the associated user experiences. With a human-computer interaction lens, this paper reflects on device usefulness as a function of utility and usability, and on the barriers to continued device use and app installation. Additionally, the paper contributes insights from a sequel study that extends on prior work evaluating app functionality of a 'vintage' Apple device with new empirical data on app downloadability and functionality for the same device when newly classified as 'obsolete'. A total of 230 apps, comprising the top 10 free App Store apps for each of 23 categories, were assessed for downloadability and functionality on an Apple iPad Mini tablet. Although only 20 apps (8.7%) could be downloaded directly onto the newly obsolete device, 143 apps (62.2%) could be downloaded with the use of a different non-legacy device. Of these 163 downloadable apps, 131 apps (com-prising 57% of all 230 apps and 80.4% of the downloadable apps) successfully installed, opened, and functioned. This was a decrease of only 4.3% in functional apps (of the 230 total apps) compared to the performance of the device when previously classified as 'vintage'.
Power Minimizing MEC Offloading with QoS Constraints over RIS-Empowered Communications
Venue: arXiv cs.IT 2023
Year: 2023

This work lies at the intersection of two cutting edge technologies envisioned to proliferate in future 6G wireless systems: Multi-access Edge Computing (MEC) and Reconfigurable Intelligent Surfaces (RISs). While the former will bring a powerful information technology environment at the wireless edge, the latter will enhance communication performance, thanks to the possibility of adapting wireless propagation as per end users' convenience, according to specific service requirements. We propose a joint optimization of radio, computing, and wireless environment reconfiguration through an RIS, with the goal of enabling low power computation offloading services with reliability guarantees. Going beyond previous works on this topic, multi-carrier frequency selective RIS elements' responses and wireless channels are considered. This opens new challenges in RIS optimization, accounting for frequency dependent RIS response profiles, which strongly affect RIS-aided wireless links and, as a consequence, MEC service performance. We formulate an optimization problem accounting for short and long-term constraints involving device transmit power allocation across multiple subcarriers and local computing resources, as well as RIS reconfiguration parameters according to a recently developed Lorentzian model. Besides a theoretical optimization framework, numerical results show the effectiveness of the proposed method in enabling low power reliable computation offloading over RIS-aided frequency selective channels.
Prompting Datasets: Data Discovery with Conversational Agents
Venue: arXiv cs.HC 2023
Year: 2023

Can large language models assist in data discovery? Data discovery predominantly happens via search on a data portal or the web, followed by assessment of the dataset to ensure it is fit for the intended purpose. The ability of conversational generative AI (CGAI) to support recommendations with reasoning implies it can suggest datasets to users, explain why it has done so, and provide information akin to documentation regarding the dataset in order to support a use decision. We hold 3 workshops with data users and find that, despite limitations around web capabilities, CGAIs are able to suggest relevant datasets and provide many of the required sensemaking activities, as well as support dataset analysis and manipulation. However, CGAIs may also suggest fictional datasets, and perform inaccurate analysis. We identify emerging practices in data discovery and present a model of these to inform future research directions and data prompt design.
Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping
Venue: arXiv cs.LG 2023
Year: 2023

Inverse reinforcement learning (IRL) is computationally challenging, with common approaches requiring the solution of multiple reinforcement learning (RL) sub-problems. This work motivates the use of potential-based reward shaping to reduce the computational burden of each RL sub-problem. This work serves as a proof-of-concept and we hope will inspire future developments towards computationally efficient IRL.
SAT-Based Algorithms for Regular Graph Pattern Matching
Venue: arXiv cs.AI 2023
Year: 2023

Graph matching is a fundamental problem in pattern recognition, with many applications such as software analysis and computational biology. One well-known type of graph matching problem is graph isomorphism, which consists of deciding if two graphs are identical. Despite its usefulness, the properties that one may check using graph isomorphism are rather limited, since it only allows strict equality checks between two graphs. For example, it does not allow one to check complex structural properties such as if the target graph is an arbitrary length sequence followed by an arbitrary size loop. We propose a generalization of graph isomorphism that allows one to check such properties through a declarative specification. This specification is given in the form of a Regular Graph Pattern (ReGaP), a special type of graph, inspired by regular expressions, that may contain wildcard nodes that represent arbitrary structures such as variable-sized sequences or subgraphs. We propose a SAT-based algorithm for checking if a target graph matches a given ReGaP. We also propose a preprocessing technique for improving the performance of the algorithm and evaluate it through an extensive experimental evaluation on benchmarks from the CodeSearchNet dataset.
One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
Venue: arXiv cs.AI 2023
Year: 2023

Abstract Visual Reasoning (AVR) comprises a wide selection of various problems similar to those used in human IQ tests. Recent years have brought dynamic progress in solving particular AVR tasks, however, in the contemporary literature AVR problems are largely dealt with in isolation, leading to highly specialized task-specific methods. With the aim of developing universal learning systems in the AVR domain, we propose the unified model for solving Single-Choice Abstract visual Reasoning tasks (SCAR), capable of solving various single-choice AVR tasks, without making any a priori assumptions about the task structure, in particular the number and location of panels. The proposed model relies on a novel Structure-Aware dynamic Layer (SAL), which adapts its weights to the structure of the considered AVR problem. Experiments conducted on Raven's Progressive Matrices, Visual Analogy Problems, and Odd One Out problems show that SCAR (SAL-based models, in general) effectively solves diverse AVR tasks, and its performance is on par with the state-of-the-art task-specific baselines. What is more, SCAR demonstrates effective knowledge reuse in multi-task and transfer learning settings. To our knowledge, this work is the first successful attempt to construct a general single-choice AVR solver relying on self-configurable architecture and unified solving method. With this work we aim to stimulate and foster progress on task-independent research paths in the AVR domain, with the long-term goal of development of a general AVR solver.
Challenges with unsupervised LLM knowledge discovery
Venue: arXiv cs.LG 2023
Year: 2023

We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent. The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a series of experiments showing settings in which unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. Conceptually, we hypothesise that the identification issues explored here, e.g. distinguishing a model's knowledge from that of a simulated character's, will persist for future unsupervised methods.