OpenRaiser/Intern-Atlas huggingface dataset:
On the Overconfidence Problem in Semantic 3D Mapping
Venue: arXiv cs.CV 2023
Year: 2023

Semantic 3D mapping, the process of fusing depth and image segmentation information between multiple views to build 3D maps annotated with object classes in real-time, is a recent topic of interest. This paper highlights the fusion overconfidence problem, in which conventional mapping methods assign high confidence to the entire map even when they are incorrect, leading to miscalibrated outputs. Several methods to improve uncertainty calibration at different stages in the fusion pipeline are presented and compared on the ScanNet dataset. We show that the most widely used Bayesian fusion strategy is among the worst calibrated, and propose a learned pipeline that combines fusion and calibration, GLFS, which achieves simultaneously higher accuracy and 3D map calibration while retaining real-time capability. We further illustrate the importance of map calibration on a downstream task by showing that incorporating proper semantic fusion on a modular ObjectNav agent improves its success rates. Our code will be provided on Github for reproducibility upon acceptance.
Depth Insight -- Contribution of Different Features to Indoor Single-image Depth Estimation
Venue: arXiv cs.CV 2023
Year: 2023

Depth estimation from a single image is a challenging problem in computer vision because binocular disparity or motion information is absent. Whereas impressive performances have been reported in this area recently using end-to-end trained deep neural architectures, as to what cues in the images that are being exploited by these black box systems is hard to know. To this end, in this work, we quantify the relative contributions of the known cues of depth in a monocular depth estimation setting using an indoor scene data set. Our work uses feature extraction techniques to relate the single features of shape, texture, colour and saturation, taken in isolation, to predict depth. We find that the shape of objects extracted by edge detection substantially contributes more than others in the indoor setting considered, while the other features also have contributions in varying degrees. These insights will help optimise depth estimation models, boosting their accuracy and robustness. They promise to broaden the practical applications of vision-based depth estimation. The project code is attached to the supplementary material and will be published on GitHub.
Tabular Few-Shot Generalization Across Heterogeneous Feature Spaces
Venue: arXiv cs.LG 2023
Year: 2023

Despite the prevalence of tabular datasets, few-shot learning remains under-explored within this domain. Existing few-shot methods are not directly applicable to tabular datasets due to varying column relationships, meanings, and permutational invariance. To address these challenges, we propose FLAT-a novel approach to tabular few-shot learning, encompassing knowledge sharing between datasets with heterogeneous feature spaces. Utilizing an encoder inspired by Dataset2Vec, FLAT learns low-dimensional embeddings of datasets and their individual columns, which facilitate knowledge transfer and generalization to previously unseen datasets. A decoder network parametrizes the predictive target network, implemented as a Graph Attention Network, to accommodate the heterogeneous nature of tabular datasets. Experiments on a diverse collection of 118 UCI datasets demonstrate FLAT's successful generalization to new tabular datasets and a considerable improvement over the baselines.
A Framework of Defining, Modeling, and Analyzing Cognition Mechanisms
Venue: arXiv cs.AI 2023
Year: 2023

Cognition is a core part of and a common topic among philosophy of mind, psychology, neuroscience, AI, and cognitive science. Through a mechanistic lens, I propose a framework of defining, modeling, and analyzing cognition mechanisms. Firstly, appropriate terms are introduced and used in explanations related to the framework and within the definition of a mechanism. I implicitly contend that this terminology essentially characterizes a conceptual world required for discussions in this paper. Secondly, a mathematical model of a mechanism based on directed graphs is proposed. Thirdly, the definition of a base necessary for a mechanism to be classified as a cognition mechanism is proposed. I argue that the cognition base has the features of the cognition self of humans. Fourthly, three ways to mechanistically look at mechanisms is defined and specific instances of them are suggested. Fifthly, standards for visualization and presentation of mechanisms, cognition mechanisms, and the instances to mechanistically look at them are suggested and used to analyze cognition mechanisms through appropriate examples. Finally, the features of this paper are discussed and prospects of further development of the proposed framework are briefly expressed.
UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized Multimodal Framework
Venue: arXiv cs.CV 2023
Year: 2023

In the current landscape of artificial intelligence, foundation models serve as the bedrock for advancements in both language and vision domains. OpenAI GPT-4 has emerged as the pinnacle in large language models (LLMs), while the computer vision (CV) domain boasts a plethora of state-of-the-art (SOTA) models such as Meta's SAM and DINO, and YOLOS. However, the financial and computational burdens of training new models from scratch remain a significant barrier to progress. In response to this challenge, we introduce UnifiedVisionGPT, a novel framework designed to consolidate and automate the integration of SOTA vision models, thereby facilitating the development of vision-oriented AI. UnifiedVisionGPT distinguishes itself through four key features: (1) provides a versatile multimodal framework adaptable to a wide range of applications, building upon the strengths of multimodal foundation models; (2) seamlessly integrates various SOTA vision models to create a comprehensive multimodal platform, capitalizing on the best components of each model; (3) prioritizes vision-oriented AI, ensuring a more rapid progression in the CV domain compared to the current trajectory of LLMs; and (4) introduces automation in the selection of SOTA vision models, generating optimal results based on diverse multimodal inputs such as text prompts and images. This paper outlines the architecture and capabilities of UnifiedVisionGPT, demonstrating its potential to revolutionize the field of computer vision through enhanced efficiency, versatility, generalization, and performance. Our implementation, along with the unified multimodal framework and comprehensive dataset, is made publicly available at https://github.com/LHBuilder/SA-Segment-Anything.
Learning interactions to boost human creativity with bandits and GPT-4
Venue: arXiv cs.AI 2023
Year: 2023

This paper considers how interactions with AI algorithms can boost human creative thought. We employ a psychological task that demonstrates limits on human creativity, namely semantic feature generation: given a concept name, respondents must list as many of its features as possible. Human participants typically produce only a fraction of the features they know before getting "stuck." In experiments with humans and with a language AI (GPT-4) we contrast behavior in the standard task versus a variant in which participants can ask for algorithmically-generated hints. Algorithm choice is administered by a multi-armed bandit whose reward indicates whether the hint helped generating more features. Humans and the AI show similar benefits from hints, and remarkably, bandits learning from AI responses prefer the same prompting strategy as those learning from human behavior. The results suggest that strategies for boosting human creativity via computer interactions can be learned by bandits run on groups of simulated participants.
Notes on the Sum-Rank Weight of a Matrix with Restricted Rank
Venue: arXiv cs.IT 2023
Year: 2023

These notes cover a few calculations regarding the sum-rank weight of a matrix in relation to its rank. In particular, a formula and lower bounds are given on the probability that a matrix of rank $t$ consisting of $\ell$ blocks has sum-rank weight $\ell t$.
Closed-Form Information-Theoretic Roughness Measures for Mixture Densities
Venue: arXiv cs.IT 2023
Year: 2023

We calculate the smoothest mixture density under a variety of prescribed specifications. This includes constraints on certain moments, specifications on density values and/or its derivatives, and prescribed probability masses in certain regions. As a roughness measure, we use Fisher Information (FI) in the space of mixtures $\cal M$. For mixtures, FI cannot be calculated in closed form. We define the space $\cal R$ of root mixtures (RMs) living on the Hilbert sphere. A transformation of FI to $\cal R$ admits a closed-form solution and yields the desired result in $\cal M$. This naturally leads to a tandem processing with two density representations maintained simultaneously in $\cal R$ and $\cal M$. FI is calculated in RM space $\cal R$ while the constraints are evaluated in mixture space $\cal M$.
Fused Breadth-First Probabilistic Traversals on Distributed GPU Systems
Venue: arXiv cs.DC 2023
Year: 2023

Probabilistic breadth-first traversals (BPTs) are used in many network science and graph machine learning applications. In this paper, we are motivated by the application of BPTs in stochastic diffusion-based graph problems such as influence maximization. These applications heavily rely on BPTs to implement a Monte-Carlo sampling step for their approximations. Given the large sampling complexity, stochasticity of the diffusion process, and the inherent irregularity in real-world graph topologies, efficiently parallelizing these BPTs remains significantly challenging. In this paper, we present a new algorithm to fuse massive number of concurrently executing BPTs with random starts on the input graph. Our algorithm is designed to fuse BPTs by combining separate traversals into a unified frontier on distributed multi-GPU systems. To show the general applicability of the fused BPT technique, we have incorporated it into two state-of-the-art influence maximization parallel implementations (gIM and Ripples). Our experiments on up to 4K nodes of the OLCF Frontier supercomputer ($32,768$ GPUs and $196$K CPU cores) show strong scaling behavior, and that fused BPTs can improve the performance of these implementations up to 34$\times$ (for gIM) and ~360$\times$ (for Ripples).
Bayes in the age of intelligent machines
Venue: arXiv cs.LG 2023
Year: 2023

The success of methods based on artificial neural networks in creating intelligent machines seems like it might pose a challenge to explanations of human cognition in terms of Bayesian inference. We argue that this is not the case, and that in fact these systems offer new opportunities for Bayesian modeling. Specifically, we argue that Bayesian models of cognition and artificial neural networks lie at different levels of analysis and are complementary modeling approaches, together offering a way to understand human cognition that spans these levels. We also argue that the same perspective can be applied to intelligent machines, where a Bayesian approach may be uniquely valuable in understanding the behavior of large, opaque artificial neural networks that are trained on proprietary data.
Predictive Minds: LLMs As Atypical Active Inference Agents
Venue: arXiv cs.CL 2023
Year: 2023

Large language models (LLMs) like GPT are often conceptualized as passive predictors, simulators, or even stochastic parrots. We instead conceptualize LLMs by drawing on the theory of active inference originating in cognitive science and neuroscience. We examine similarities and differences between traditional active inference systems and LLMs, leading to the conclusion that, currently, LLMs lack a tight feedback loop between acting in the world and perceiving the impacts of their actions, but otherwise fit in the active inference paradigm. We list reasons why this loop may soon be closed, and possible consequences of this including enhanced model self-awareness and the drive to minimize prediction error by changing the world.
Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study
Venue: arXiv cs.CL 2023
Year: 2023

With the ever-growing presence of social media platforms comes the increased spread of harmful content and the need for robust hate speech detection systems. Such systems easily overfit to specific targets and keywords, and evaluating them without considering distribution shifts that might occur between train and test data overestimates their benefit. We challenge hate speech models via new train-test splits of existing datasets that rely on the clustering of models' hidden representations. We present two split variants (Subset-Sum-Split and Closest-Split) that, when applied to two datasets using four pretrained models, reveal how models catastrophically fail on blind spots in the latent space. This result generalises when developing a split with one model and evaluating it on another. Our analysis suggests that there is no clear surface-level property of the data split that correlates with the decreased performance, which underscores that task difficulty is not always humanly interpretable. We recommend incorporating latent feature-based splits in model development and release two splits via the GenBench benchmark.
JediCode -- A Gamefied Approach to Competitive Coding
Venue: arXiv cs.CY 2023
Year: 2023

JediCode (name inspired from Star Wars) pioneers a transformative approach to competitive coding by infusing the challenge with gamified elements. This platform reimagines coding competitions, integrating real-time leaderboards, synchronized challenges, and random matchmaking, creating an engaging, dynamic, and friendly atmosphere. This paper explores JediCode's innovative features and architecture, shedding light on its user-centric design and powerful execution service. By embracing gamification, JediCode not only elevates the thrill of coding challenges but also fosters a sense of community, inspiring programmers to excel while enjoying the process.
igraph enables fast and robust network analysis across programming languages
Venue: arXiv cs.SI 2023
Year: 2023

Networks or graphs are widely used across the sciences to represent relationships of many kinds. igraph (https://igraph.org) is a general-purpose software library for graph construction, analysis, and visualisation, combining fast and robust performance with a low entry barrier. igraph pairs a fast core written in C with beginner-friendly interfaces in Python, R, and Mathematica. Over the last two decades, igraph has expanded substantially. It now scales to billions of edges, supports Mathematica and interactive plotting, integrates with Jupyter notebooks and other network libraries, includes new graph layouts and community detection algorithms, and has streamlined the documentation with examples and Spanish translations. Modern testing features such as continuous integration, address sanitizers, stricter typing, and memory-managed vectors have also increased robustness. Hundreds of bug reports have been fixed and a community forum has been opened to connect users and developers. Specific effort has been made to broaden use and community participation by women, non-binary people, and other demographic groups typically underrepresented in open source software.
Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving
Venue: arXiv cs.CV 2023
Year: 2023

Sensor fusion is crucial for an accurate and robust perception system on autonomous vehicles. Most existing datasets and perception solutions focus on fusing cameras and LiDAR. However, the collaboration between camera and radar is significantly under-exploited. The incorporation of rich semantic information from the camera, and reliable 3D information from the radar can potentially achieve an efficient, cheap, and portable solution for 3D object perception tasks. It can also be robust to different lighting or all-weather driving scenarios due to the capability of mmWave radars. In this paper, we introduce the CRUW3D dataset, including 66K synchronized and well-calibrated camera, radar, and LiDAR frames in various driving scenarios. Unlike other large-scale autonomous driving datasets, our radar data is in the format of radio frequency (RF) tensors that contain not only 3D location information but also spatio-temporal semantic information. This kind of radar format can enable machine learning models to generate more reliable object perception results after interacting and fusing the information or features between the camera and radar.
Interpretable pap smear cell representation for cervical cancer screening
Venue: arXiv cs.CV 2023
Year: 2023

Screening is critical for prevention and early detection of cervical cancer but it is time-consuming and laborious. Supervised deep convolutional neural networks have been developed to automate pap smear screening and the results are promising. However, the interest in using only normal samples to train deep neural networks has increased owing to class imbalance problems and high-labeling costs that are both prevalent in healthcare. In this study, we introduce a method to learn explainable deep cervical cell representations for pap smear cytology images based on one class classification using variational autoencoders. Findings demonstrate that a score can be calculated for cell abnormality without training models with abnormal samples and localize abnormality to interpret our results with a novel metric based on absolute difference in cross entropy in agglomerative clustering. The best model that discriminates squamous cell carcinoma (SCC) from normals gives 0.908 +- 0.003 area under operating characteristic curve (AUC) and one that discriminates high-grade epithelial lesion (HSIL) 0.920 +- 0.002 AUC. Compared to other clustering methods, our method enhances the V-measure and yields higher homogeneity scores, which more effectively isolate different abnormality regions, aiding in the interpretation of our results. Evaluation using in-house and additional open dataset show that our model can discriminate abnormality without the need of additional training of deep models.
Improving FSM State Enumeration Performance for Hardware Security with RECUT and REFSM-SAT
Venue: arXiv cs.AR 2023
Year: 2023

Finite state machines (FSM's) are implemented with sequential circuits and are used to orchestrate the operation of hardware designs. Sequential obfuscation schemes aimed at preventing IP theft often operate by augmenting a design's FSM post-synthesis. Many such schemes are based on the ability to recover the FSM's topology from the synthesized design. In this paper, we present two tools which can improve the performance of topology extraction: RECUT, which extracts the FSM implementation from a netlist, and REFSM-SAT, which solves topology enumeration as a series of SAT problems. In some cases, these tools can improve performance significantly over current methods, attaining up to a 99\% decrease in runtime.
Secure Instruction and Data-Level Information Flow Tracking Model for RISC-V
Venue: arXiv cs.CR 2023
Year: 2023

Rising device use and third-party IP integration in semiconductors raise security concerns. Unauthorized access, fault injection, and privacy invasion are potential threats from untrusted actors. Different security techniques have been proposed to provide resilience to secure devices from potential vulnerabilities; however, no one technique can be applied as an overarching solution. We propose an integrated Information Flow Tracking (IFT) technique to enable runtime security to protect system integrity by tracking the flow of data from untrusted communication channels. Existing hardware-based IFT schemes are either fine-, which are resource-intensive, or coarse-grained models, which have minimal precision logic, providing either control flow or data-flow integrity. No current security model provides multi-granularity due to the difficulty in balancing both the flexibility and hardware overheads at the same time. This study proposes a multi-level granularity IFT model that integrates a hardware-based IFT technique with a gate-level-based IFT (GLIFT) technique, along with flexibility, for better precision and assessments. Translation from the instruction level to the data level is based on module instantiation with security-critical data for accurate information flow behaviors without any false conservative flows. A simulation-based IFT model is demonstrated, which translates the architecture-specific extensions into a compiler-specific simulation model with toolchain extensions for Reduced Instruction Set Architecture (RISC-V) to verify the security extensions. This approach provides better precision logic by enhancing the tagged mechanism with 1-bit tags and implementing an optimized shadow logic that eliminates the area overhead by tracking the data for only security-critical modules.
Dimensionality Reduction of Dynamics on Lie Manifolds via Structure-Aware Canonical Correlation Analysis
Venue: arXiv cs.RO 2023
Year: 2023

Incorporating prior knowledge into a data-driven modeling problem can drastically improve performance, reliability, and generalization outside of the training sample. The stronger the structural properties, the more effective these improvements become. Manifolds are a powerful nonlinear generalization of Euclidean space for modeling finite dimensions. Structural impositions in constrained systems increase when applying group structure, converting them into Lie manifolds. The range of their applications is very wide and includes the important case of robotic tasks. Canonical Correlation Analysis (CCA) can construct a hierarchical sequence of maximal correlations of up to two paired data sets in these Euclidean spaces. We present a method to generalize this concept to Lie Manifolds and demonstrate its efficacy through the substantial improvements it achieves in making structure-consistent predictions about changes in the state of a robotic hand.
MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries
Venue: arXiv cs.CV 2023
Year: 2023

Medical image segmentation annotations exhibit variations among experts due to the ambiguous boundaries of segmented objects and backgrounds in medical images. Although using multiple annotations for each image in the fully-supervised has been extensively studied for training deep models, obtaining a large amount of multi-annotated data is challenging due to the substantial time and manpower costs required for segmentation annotations, resulting in most images lacking any annotations. To address this, we propose Multi-annotated Semi-supervised Ensemble Networks (MSE-Nets) for learning segmentation from limited multi-annotated and abundant unannotated data. Specifically, we introduce the Network Pairwise Consistency Enhancement (NPCE) module and Multi-Network Pseudo Supervised (MNPS) module to enhance MSE-Nets for the segmentation task by considering two major factors: (1) to optimize the utilization of all accessible multi-annotated data, the NPCE separates (dis)agreement annotations of multi-annotated data at the pixel level and handles agreement and disagreement annotations in different ways, (2) to mitigate the introduction of imprecise pseudo-labels, the MNPS extends the training data by leveraging consistent pseudo-labels from unannotated data. Finally, we improve confidence calibration by averaging the predictions of base networks. Experiments on the ISIC dataset show that we reduced the demand for multi-annotated data by 97.75\% and narrowed the gap with the best fully-supervised baseline to just a Jaccard index of 4\%. Furthermore, compared to other semi-supervised methods that rely only on a single annotation or a combined fusion approach, the comprehensive experimental results on ISIC and RIGA datasets demonstrate the superior performance of our proposed method in medical image segmentation with ambiguous boundaries.