OpenRaiser/Intern-Atlas huggingface dataset:
Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
Venue: arXiv cs.CV 2023
Year: 2023

Brain decoding is a field of computational neuroscience that uses measurable brain activity to infer mental states or internal representations of perceptual inputs. Therefore, we propose a novel approach to brain decoding that also relies on semantic and contextual similarity. We employ an fMRI dataset of natural image vision and create a deep learning decoding pipeline inspired by the existence of both bottom-up and top-down processes in human vision. We train a linear brain-to-feature model to map fMRI activity features to visual stimuli features, assuming that the brain projects visual information onto a space that is homeomorphic to the latent space represented by the last convolutional layer of a pretrained convolutional neural network, which typically collects a variety of semantic features that summarize and highlight similarities and differences between concepts. These features are then categorized in the latent space using a nearest-neighbor strategy, and the results are used to condition a generative latent diffusion model to create novel images. From fMRI data only, we produce reconstructions of visual stimuli that match the original content very well on a semantic level, surpassing the state of the art in previous literature. We evaluate our work and obtain good results using a quantitative semantic metric (the Wu-Palmer similarity metric over the WordNet lexicon, which had an average value of 0.57) and perform a human evaluation experiment that resulted in correct evaluation, according to the multiplicity of human criteria in evaluating image similarity, in over 80% of the test set.
LidarCLIP or: How I Learned to Talk to Point Clouds
Venue: arXiv cs.CV 2023
Year: 2023

Research connecting text and images has recently seen several breakthroughs, with models like CLIP, DALL-E 2, and Stable Diffusion. However, the connection between text and other visual modalities, such as lidar data, has received less attention, prohibited by the lack of text-lidar datasets. In this work, we propose LidarCLIP, a mapping from automotive point clouds to a pre-existing CLIP embedding space. Using image-lidar pairs, we supervise a point cloud encoder with the image CLIP embeddings, effectively relating text and lidar data with the image domain as an intermediary. We show the effectiveness of LidarCLIP by demonstrating that lidar-based retrieval is generally on par with image-based retrieval, but with complementary strengths and weaknesses. By combining image and lidar features, we improve upon both single-modality methods and enable a targeted search for challenging detection scenarios under adverse sensor conditions. We also explore zero-shot classification and show that LidarCLIP outperforms existing attempts to use CLIP for point clouds by a large margin. Finally, we leverage our compatibility with CLIP to explore a range of applications, such as point cloud captioning and lidar-to-image generation, without any additional training. Code and pre-trained models are available at https://github.com/atonderski/lidarclip.
Mechanics of geodesics in Information geometry and Black Hole Thermodynamics
Venue: arXiv cs.IT 2023
Year: 2023

In this article we shall discuss the theory of geodesics in information geometry, and an application in astrophysics. We will study how gradient flows in information geometry describe geodesics, explore the related mechanics by introducing a constraint, and apply our theory to Gaussian model and black hole thermodynamics. Thus, we demonstrate how deformation of gradient flows leads to more general Randers-Finsler metrics, describe Hamiltonian mechanics that derive from a constraint, and prove duality via canonical transformation. We also verified our theories for a deformation of the Gaussian model, and described dynamical evolution of flat metrics for Kerr and Reissner-Nordström black holes.
API-Miner: an API-to-API Specification Recommendation Engine
Venue: arXiv cs.SE 2023
Year: 2023

When designing a new API for a large project, developers need to make smart design choices so that their code base can grow sustainably. To ensure that new API components are well designed, developers can learn from existing API components. However, the lack of standardized methods for comparing API designs makes this learning process time-consuming and difficult. To address this gap we developed API-Miner, to the best of our knowledge, one of the first API-to-API specification recommendation engines. API-Miner retrieves relevant specification components written in OpenAPI (a widely adopted language used to describe web APIs). API-miner presents several significant contributions, including: (1) novel methods of processing and extracting key information from OpenAPI specifications, (2) innovative feature extraction techniques that are optimized for the highly technical API specification domain, and (3) a novel log-linear probabilistic model that combines multiple signals to retrieve relevant and high quality OpenAPI specification components given a query specification. We evaluate API-Miner in both quantitative and qualitative tasks and achieve an overall of 91.7% recall@1 and 56.2% F1, which surpasses baseline performance by 15.4% in recall@1 and 3.2% in F1. Overall, API-Miner will allow developers to retrieve relevant OpenAPI specification components from a public or internal database in the early stages of the API development cycle, so that they can learn from existing established examples and potentially identify redundancies in their work. It provides the guidance developers need to accelerate development process and contribute thoughtfully designed APIs that promote code maintainability and quality. Code is available on GitHub at https://github.com/jpmorganchase/api-miner.
Gaussian Process Mapping of Uncertain Building Models with GMM as Prior
Venue: arXiv cs.RO 2023
Year: 2023

Mapping with uncertainty representation is required in many research domains, especially for localization. Although there are many investigations regarding the uncertainty of the pose estimation of an ego-robot with map information, the quality of the reference maps is often neglected. To avoid potential problems caused by the errors of maps and a lack of uncertainty quantification, an adequate uncertainty measure for the maps is required. In this letter, uncertain building models with abstract map surfaces using Gaussian Processes (GPs) are proposed to describe the map uncertainty in a probabilistic way. To reduce the redundant computation for simple planar objects, extracted facets from a Gaussian Mixture Model (GMM) are combined with an implicit GP map, also employing local GP-block techniques. The proposed method is evaluated on LiDAR point clouds of city buildings collected by a mobile mapping system. Compared to the performance of other methods such as OctoMap, GP Occupancy Map (GPOM), Bayesian Generalized Kernel OctoMap (BGKOctoMap), Local automatic relevance determination Hilbert map (LARD-HM) and Gaussian Implicit Surface map (GPIS), our method achieves a higher Precision-Recall AUC for the evaluated buildings.
Achievable Information Rates and Concatenated Codes for the DNA Nanopore Sequencing Channel
Venue: arXiv cs.IT 2023
Year: 2023

The errors occurring in DNA-based storage are correlated in nature, which is a direct consequence of the synthesis and sequencing processes. In this paper, we consider the memory-$k$ nanopore channel model recently introduced by Hamoum et al., which models the inherent memory of the channel. We derive the maximum a posteriori (MAP) decoder for this channel model. The derived MAP decoder allows us to compute achievable information rates for the true DNA storage channel assuming a mismatched decoder matched to the memory-$k$ nanopore channel model, and quantify the loss in performance assuming a small memory length--and hence limited decoding complexity. Furthermore, the derived MAP decoder can be used to design error-correcting codes tailored to the DNA storage channel. We show that a concatenated coding scheme with an outer low-density parity-check code and an inner convolutional code yields excellent performance.
Invariant Lipschitz Bandits: A Side Observation Approach
Venue: arXiv cs.LG 2023
Year: 2023

Symmetry arises in many optimization and decision-making problems, and has attracted considerable attention from the optimization community: By utilizing the existence of such symmetries, the process of searching for optimal solutions can be improved significantly. Despite its success in (offline) optimization, the utilization of symmetries has not been well examined within the online optimization settings, especially in the bandit literature. As such, in this paper we study the invariant Lipschitz bandit setting, a subclass of the Lipschitz bandits where the reward function and the set of arms are preserved under a group of transformations. We introduce an algorithm named \texttt{UniformMesh-N}, which naturally integrates side observations using group orbits into the \texttt{UniformMesh} algorithm (\cite{Kleinberg2005_UniformMesh}), which uniformly discretizes the set of arms. Using the side-observation approach, we prove an improved regret upper bound, which depends on the cardinality of the group, given that the group is finite. We also prove a matching regret's lower bound for the invariant Lipschitz bandit class (up to logarithmic factors). We hope that our work will ignite further investigation of symmetry in bandit theory and sequential decision-making theory in general.
Faster Maximum Inner Product Search in High Dimensions
Venue: arXiv cs.LG 2023
Year: 2023

Maximum Inner Product Search (MIPS) is a ubiquitous task in machine learning applications such as recommendation systems. Given a query vector and $n$ atom vectors in $d$-dimensional space, the goal of MIPS is to find the atom that has the highest inner product with the query vector. Existing MIPS algorithms scale at least as $O(\sqrt{d})$, which becomes computationally prohibitive in high-dimensional settings. In this work, we present BanditMIPS, a novel randomized MIPS algorithm whose complexity is independent of $d$. BanditMIPS estimates the inner product for each atom by subsampling coordinates and adaptively evaluates more coordinates for more promising atoms. The specific adaptive sampling strategy is motivated by multi-armed bandits. We provide theoretical guarantees that BanditMIPS returns the correct answer with high probability, while improving the complexity in $d$ from $O(\sqrt{d})$ to $O(1)$. We also perform experiments on four synthetic and real-world datasets and demonstrate that BanditMIPS outperforms prior state-of-the-art algorithms. For example, in the Movie Lens dataset ($n$=4,000, $d$=6,000), BanditMIPS is 20$\times$ faster than the next best algorithm while returning the same answer. BanditMIPS requires no preprocessing of the data and includes a hyperparameter that practitioners may use to trade off accuracy and runtime. We also propose a variant of our algorithm, named BanditMIPS-$α$, which achieves further speedups by employing non-uniform sampling across coordinates. Finally, we demonstrate how known preprocessing techniques can be used to further accelerate BanditMIPS, and discuss applications to Matching Pursuit and Fourier analysis.
DeepJoin: Joinable Table Discovery with Pre-trained Language Models
Venue: arXiv cs.DB 2023
Year: 2023

Due to the usefulness in data enrichment for data analysis tasks, joinable table discovery has become an important operation in data lake management. Existing approaches target equi-joins, the most common way of combining tables for creating a unified view, or semantic joins, which tolerate misspellings and different formats to deliver more join results. They are either exact solutions whose running time is linear in the sizes of query column and target table repository or approximate solutions lacking precision. In this paper, we propose Deepjoin, a deep learning model for accurate and efficient joinable table discovery. Our solution is an embedding-based retrieval, which employs a pre-trained language model (PLM) and is designed as one framework serving both equi- and semantic joins. We propose a set of contextualization options to transform column contents to a text sequence. The PLM reads the sequence and is fine-tuned to embed columns to vectors such that columns are expected to be joinable if they are close to each other in the vector space. Since the output of the PLM is fixed in length, the subsequent search procedure becomes independent of the column size. With a state-of-the-art approximate nearest neighbor search algorithm, the search time is logarithmic in the repository size. To train the model, we devise the techniques for preparing training data as well as data augmentation. The experiments on real datasets demonstrate that by training on a small subset of a corpus, Deepjoin generalizes to large datasets and its precision consistently outperforms other approximate solutions'. Deepjoin is even more accurate than an exact solution to semantic joins when evaluated with labels from experts. Moreover, when equipped with a GPU, Deepjoin is up to two orders of magnitude faster than existing solutions.
Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results
Venue: arXiv cs.NE 2023
Year: 2023

The potential of learned models for fundamental scientific research and discovery is drawing increasing attention worldwide. Physics-informed neural networks (PINNs), where the loss function directly embeds governing equations of scientific phenomena, is one of the key techniques at the forefront of recent advances. PINNs are typically trained using stochastic gradient descent methods, akin to their deep learning counterparts. However, analysis in this paper shows that PINNs' unique loss formulations lead to a high degree of complexity and ruggedness that may not be conducive for gradient descent. Unlike in standard deep learning, PINN training requires globally optimum parameter values that satisfy physical laws as closely as possible. Spurious local optimum, indicative of erroneous physics, must be avoided. Hence, neuroevolution algorithms, with their superior global search capacity, may be a better choice for PINNs relative to gradient descent methods. Here, we propose a set of five benchmark problems, with open-source codes, spanning diverse physical phenomena for novel neuroevolution algorithm development. Using this, we compare two neuroevolution algorithms against the commonly used stochastic gradient descent, and our baseline results support the claim that neuroevolution can surpass gradient descent, ensuring better physics compliance in the predicted outputs. %Furthermore, implementing neuroevolution with JAX leads to orders of magnitude speedup relative to standard implementations.
Adaptive Multi-Agent Continuous Learning System
Venue: arXiv cs.CV 2023
Year: 2023

We propose an adaptive multi-agent clustering recognition system that can be self-supervised driven, based on a temporal sequences continuous learning mechanism with adaptability. The system is designed to use some different functional agents to build up a connection structure to improve adaptability to cope with environmental diverse demands, by predicting the input of the agent to drive the agent to achieve the act of clustering recognition of sequences using the traditional algorithmic approach. Finally, the feasibility experiments of video behavior clustering demonstrate the feasibility of the system to cope with dynamic situations. Our work is placed here\footnote{https://github.com/qian-git/MAMMALS}.
CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Users often ask dialogue systems ambiguous questions that require clarification. We show that current language models rarely ask users to clarify ambiguous questions and instead provide incorrect answers. To address this, we introduce CLAM: a framework for getting language models to selectively ask for clarification about ambiguous user questions. In particular, we show that we can prompt language models to detect whether a given question is ambiguous, generate an appropriate clarifying question to ask the user, and give a final answer after receiving clarification. We also show that we can simulate users by providing language models with privileged information. This lets us automatically evaluate multi-turn clarification dialogues. Finally, CLAM significantly improves language models' accuracy on mixed ambiguous and unambiguous questions relative to SotA.
Temporal Saliency Detection Towards Explainable Transformer-based Timeseries Forecasting
Venue: arXiv cs.LG 2023
Year: 2023

Despite the notable advancements in numerous Transformer-based models, the task of long multi-horizon time series forecasting remains a persistent challenge, especially towards explainability. Focusing on commonly used saliency maps in explaining DNN in general, our quest is to build attention-based architecture that can automatically encode saliency-related temporal patterns by establishing connections with appropriate attention heads. Hence, this paper introduces Temporal Saliency Detection (TSD), an effective approach that builds upon the attention mechanism and applies it to multi-horizon time series prediction. While our proposed architecture adheres to the general encoder-decoder structure, it undergoes a significant renovation in the encoder component, wherein we incorporate a series of information contracting and expanding blocks inspired by the U-Net style architecture. The TSD approach facilitates the multiresolution analysis of saliency patterns by condensing multi-heads, thereby progressively enhancing the forecasting of complex time series data. Empirical evaluations illustrate the superiority of our proposed approach compared to other models across multiple standard benchmark datasets in diverse far-horizon forecasting settings. The initial TSD achieves substantial relative improvements of 31% and 46% over several models in the context of multivariate and univariate prediction. We believe the comprehensive investigations presented in this study will offer valuable insights and benefits to future research endeavors.
Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition}
Venue: arXiv cs.RO 2023
Year: 2023

We present a data-efficient algorithm for learning models for model-predictive control (MPC). Our approach, Jacobian-Regularized Dynamic-Mode Decomposition (JDMD), offers improved sample efficiency over traditional Koopman approaches based on Dynamic-Mode Decomposition (DMD) by leveraging Jacobian information from an approximate prior model of the system, and improved tracking performance over traditional model-based MPC. We demonstrate JDMD's ability to quickly learn bilinear Koopman dynamics representations across several realistic examples in simulation, including a perching maneuver for a fixed-wing aircraft with an empirically derived high-fidelity physics model. In all cases, we show that the models learned by JDMD provide superior tracking and generalization performance within a model-predictive control framework, even in the presence of significant model mismatch, when compared to approximate prior models and models learned by standard Extended DMD (EDMD).
Political advertisement on Facebook and Instagram in the run up to 2022 Italian general election
Venue: arXiv cs.CY 2023
Year: 2023

Targeted advertising on online social platforms has become increasingly relevant in the political marketing toolkit. Monitoring political advertising is crucial to ensure accountability and transparency of democratic processes. Leveraging Meta public library of sponsored content, we study the extent to which political ads were delivered on Facebook and Instagram in the run up to 2022 Italian general election. Analyzing over 23 k unique ads paid by 2.7 k unique sponsors, with an associated amount spent of 4 M EUR and over 1 billion views generated, we investigate temporal, geographical, and demographic patterns of the political campaigning activity of main coalitions. We find results that are in accordance with their political agenda and the electoral outcome, highlighting how the most active coalitions also obtained most of the votes and showing regional differences that are coherent with the (targeted) political base of each group. Our work raises attention to the need for further studies of digital advertising and its implications for individuals' opinions and choices.
Localization & Mapping Requirements for Level 2+ Autonomous Vehicles
Venue: arXiv cs.RO 2023
Year: 2023

Autonomous vehicles are being deployed with a spectrum of capability, extending from driver assistance features for the highway in personal vehicles (SAE Level 2+) to fully autonomous fleet ride sharing services operating in complex city environments (SAE Level 4+). This spectrum of autonomy often operates in different physical environments with different degrees of assumed driver in-the-loop oversight and hence have very different system and subsystem requirements. At the heart of SAE Level 2 to 5 systems is localization and mapping, which ranges from road determination for feature geofencing or high-level routing, through lane determination for advanced driver assistance, to where-in-lane positioning for full vehicle control. We assess localization and mapping requirements for different levels of autonomy and supported features. This work provides a framework for system decomposition, including the level of redundancy needed to achieve the target level of safety. We examine several representative autonomous and assistance features and make recommendations on positioning requirements as well map georeferencing and information integrity.
A Survey on Anonymous Communication Systems with a Focus on Dining Cryptographers Networks
Venue: arXiv cs.CR 2023
Year: 2023

Traffic analysis attacks can counteract end-to-end encryption and use leaked communication metadata to reveal information about communicating parties. With an ever-increasing amount of traffic by an ever-increasing number of networked devices, communication privacy is undermined. Therefore, Anonymous Communication Systems (ACSs) are proposed to hide the relationship between transmitted messages and their senders and receivers, providing privacy properties known as anonymity, unlinkability, and unobservability. This article aims to review research in the ACSs field, focusing on Dining Cryptographers Networks (DCNs). The DCN-based methods are information-theoretically secure and thus provide unconditional unobservability guarantees. Their adoption for anonymous communications was initially hindered because their computational and communication overhead was deemed significant at that time, and scalability problems occurred. However, more recent contributions, such as the possibility to transmit messages of arbitrary length, efficient disruption handling and overhead improvements, have made the integration of modern DCN-based methods more realistic. In addition, the literature does not follow a common definition for privacy properties, making it hard to compare the approaches' gains. Therefore, this survey contributes to introducing a harmonized terminology for ACS privacy properties, then presents an overview of the underlying principles of ACSs, in particular, DCN-based methods, and finally, investigates their alignment with the new harmonized privacy terminologies. Previous surveys did not cover the most recent research advances in the ACS area or focus on DCN-based methods. Our comprehensive investigation closes this gap by providing visual maps to highlight privacy properties and discussing the most promising ideas for making DCNs applicable in resource-constrained environments.