OpenRaiser/Intern-Atlas huggingface dataset:
Open-world Text-specified Object Counting
Venue: arXiv cs.CV 2023
Year: 2023

Our objective is open-world object counting in images, where the target object class is specified by a text description. To this end, we propose CounTX, a class-agnostic, single-stage model using a transformer decoder counting head on top of pre-trained joint text-image representations. CounTX is able to count the number of instances of any class given only an image and a text description of the target object class, and can be trained end-to-end. In addition to this model, we make the following contributions: (i) we compare the performance of CounTX to prior work on open-world object counting, and show that our approach exceeds the state of the art on all measures on the FSC-147 benchmark for methods that use text to specify the task; (ii) we present and release FSC-147-D, an enhanced version of FSC-147 with text descriptions, so that object classes can be described with more detailed language than their simple class names. FSC-147-D and the code are available at https://www.robots.ox.ac.uk/~vgg/research/countx.
Probabilistic Adaptation of Text-to-Video Models
Venue: arXiv cs.AI 2023
Year: 2023

Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions. However, adapting these models to tasks with limited domain-specific data, such as animation or robotics videos, poses a significant computational challenge, since finetuning a pretrained large model can be prohibitively expensive. Inspired by how a small modifiable component (e.g., prompts, prefix-tuning) can adapt a large language model to perform new tasks without requiring access to the model weights, we investigate how to adapt a large pretrained text-to-video model to a variety of downstream domains and tasks without finetuning. In answering this question, we propose Video Adapter, which leverages the score function of a large pretrained video diffusion model as a probabilistic prior to guide the generation of a task-specific small video model. Our experiments show that Video Adapter is capable of incorporating the broad knowledge and preserving the high fidelity of a large pretrained video model in a task-specific small video model that is able to generate high-quality yet specialized videos on a variety of tasks such as animation, egocentric modeling, and modeling of simulated and real-world robotics data. More videos can be found on the website https://video-adapter.github.io/.
Auditable data structures: theory and applications
Venue: arXiv cs.DC 2023
Year: 2023

Every digital process needs to consume some data in order to work properly. It is very common for applications to use some external data in their processes, getting them by sources such as external APIs. Therefore, trusting the received data becomes crucial in such scenarios, considering that if the data are not self-produced by the consumer, the trust in the external data source, or in the data that the source produces, can not always be taken for granted. The most used approach to generate trust in the external source is based on authenticated data structures, that are able to authenticate the source when queried through the generation of proofs. Such proofs are useful to assess authenticity or integrity, however, an external user could also be interested in verifying the data history and its consistency. This problem seems to be unaddressed by current literature, which proposes some approaches aimed at executing audits by internal actors with prior knowledge about the data structures. In this paper, we address the scenario of an external auditor with no data knowledge that wants to verify the data history consistency. We analyze the terminology and the current state of the art of the auditable data structures, then we will propose a general framework to support external audits from both internal and external users.
Exploring the Boundaries: Thorough Software Testing for Safety-Critical Driving Scenarios Based on Kinematics in the Context of Autonomous Driving
Venue: arXiv cs.RO 2023
Year: 2023

This scientific publication focuses on the efficient application of boundary value analysis in the testing of corner cases for kinematic-based safety-critical driving scenarios within the domain of autonomous driving. Corner cases, which represent infrequent and crucial situations, present notable obstacles to the reliability and safety of autonomous driving systems. This paper emphasizes the significance of employing boundary value analysis, a systematic technique for identifying critical boundaries and values, to achieve comprehensive testing coverage. By identifying and testing extreme and boundary conditions, such as minimum distances, this publication aims to improve the performance and robustness of autonomous driving systems in safety-critical scenarios. The insights and methodologies presented in this paper can serve as a guide for researchers, developers, and regulators in effectively addressing the challenges posed by corner cases and ensuring the reliability and safety of autonomous driving systems under real-world driving conditions.
A phase-field chemo-mechanical model for corrosion-induced cracking in reinforced concrete
Venue: arXiv cs.CE 2023
Year: 2023

We present a new mechanistic framework for corrosion-induced cracking in reinforced concrete that resolves the underlying chemo-mechanical processes. The framework combines, for the first time, (i) a model for reactive transport and precipitation of dissolved Fe2+ and Fe3+ ions in the concrete pore space, (ii) a precipitation eigenstrain model for the pressure caused by the accumulation of precipitates (rusts) under pore confinement conditions, (iii) a phase-field model calibrated for the quasi-brittle fracture behaviour of concrete, and (iv) a damage-dependent diffusivity tensor. Finite element model predictions show good agreement with experimental data from impressed current tests under natural-like corrosion current densities.
A Simple yet Effective Self-Debiasing Framework for Transformer Models
Venue: arXiv cs.CL 2023
Year: 2023

Current Transformer-based natural language understanding (NLU) models heavily rely on dataset biases, while failing to handle real-world out-of-distribution (OOD) instances. Many methods have been proposed to deal with this issue, but they ignore the fact that the features learned in different layers of Transformer-based NLU models are different. In this paper, we first conduct preliminary studies to obtain two conclusions: 1) both low- and high-layer sentence representations encode common biased features during training; 2) the low-layer sentence representations encode fewer unbiased features than the highlayer ones. Based on these conclusions, we propose a simple yet effective self-debiasing framework for Transformer-based NLU models. Concretely, we first stack a classifier on a selected low layer. Then, we introduce a residual connection that feeds the low-layer sentence representation to the top-layer classifier. In this way, the top-layer sentence representation will be trained to ignore the common biased features encoded by the low-layer sentence representation and focus on task-relevant unbiased features. During inference, we remove the residual connection and directly use the top-layer sentence representation to make predictions. Extensive experiments and indepth analyses on NLU tasks show that our framework performs better than several competitive baselines, achieving a new SOTA on all OOD test sets.
Structural Similarities Between Language Models and Neural Response Measurements
Venue: arXiv cs.CL 2023
Year: 2023

Large language models (LLMs) have complicated internal dynamics, but induce representations of words and phrases whose geometry we can study. Human language processing is also opaque, but neural response measurements can provide (noisy) recordings of activation during listening or reading, from which we can extract similar representations of words and phrases. Here we study the extent to which the geometries induced by these representations, share similarities in the context of brain decoding. We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging. Code is available at \url{https://github.com/coastalcph/brainlm}.
Sampling binary sparse coding QUBO models using a spiking neuromorphic processor
Venue: arXiv cs.NE 2023
Year: 2023

We consider the problem of computing a sparse binary representation of an image. To be precise, given an image and an overcomplete, non-orthonormal basis, we aim to find a sparse binary vector indicating the minimal set of basis vectors that when added together best reconstruct the given input. We formulate this problem with an $L_2$ loss on the reconstruction error, and an $L_0$ (or, equivalently, an $L_1$) loss on the binary vector enforcing sparsity. This yields a so-called Quadratic Unconstrained Binary Optimization (QUBO) problem, whose solution is generally NP-hard to find. The contribution of this work is twofold. First, the method of unsupervised and unnormalized dictionary feature learning for a desired sparsity level to best match the data is presented. Second, the binary sparse coding problem is then solved on the Loihi 1 neuromorphic chip by the use of stochastic networks of neurons to traverse the non-convex energy landscape. The solutions are benchmarked against the classical heuristic simulated annealing. We demonstrate neuromorphic computing is suitable for sampling low energy solutions of binary sparse coding QUBO models, and although Loihi 1 is capable of sampling very sparse solutions of the QUBO models, there needs to be improvement in the implementation in order to be competitive with simulated annealing.
Mitigating Backdoor Attack Via Prerequisite Transformation
Venue: arXiv cs.CR 2023
Year: 2023

In recent years, with the successful application of DNN in fields such as NLP and CV, its security has also received widespread attention. (Author) proposed the method of backdoor attack in Badnet. Switch implanted backdoor into the model by poisoning the training samples. The model with backdoor did not exhibit any abnormalities on the normal validation sample set, but in the input with trigger, they were mistakenly classified as the attacker's designated category or randomly classified as a different category from the ground truth, This attack method seriously threatens the normal application of DNN in real life, such as autonomous driving, object detection, etc.This article proposes a new method to combat backdoor attacks. We refer to the features in the area covered by the trigger as trigger features, and the remaining areas as normal features. By introducing prerequisite calculation conditions during the training process, these conditions have little impact on normal features and trigger features, and can complete the training of a standard backdoor model. The model trained under these prerequisite calculation conditions can, In the verification set D'val with the same premise calculation conditions, the performance is consistent with that of the ordinary backdoor model. However, in the verification set Dval without the premise calculation conditions, the verification accuracy decreases very little (7%~12%), while the attack success rate (ASR) decreases from 90% to about 8%.Author call this method Prerequisite Transformation(PT).
Asymptotically Optimal Pure Exploration for Infinite-Armed Bandits
Venue: arXiv cs.LG 2023
Year: 2023

We study pure exploration with infinitely many bandit arms generated i.i.d. from an unknown distribution. Our goal is to efficiently select a single high quality arm whose average reward is, with probability $1-δ$, within $\varepsilon$ of being among the top $η$-fraction of arms; this is a natural adaptation of the classical PAC guarantee for infinite action sets. We consider both the fixed confidence and fixed budget settings, aiming respectively for minimal expected and fixed sample complexity. For fixed confidence, we give an algorithm with expected sample complexity $O\left(\frac{\log (1/η)\log (1/δ)}{η\varepsilon^2}\right)$. This is optimal except for the $\log (1/η)$ factor, and the $δ$-dependence closes a quadratic gap in the literature. For fixed budget, we show the asymptotically optimal sample complexity as $δ\to 0$ is $c^{-1}\log(1/δ)\big(\log\log(1/δ)\big)^2$ to leading order. Equivalently, the optimal failure probability given exactly $N$ samples decays as $\exp\big(-cN/\log^2 N\big)$, up to a factor $1\pm o_N(1)$ inside the exponent. The constant $c$ depends explicitly on the problem parameters (including the unknown arm distribution) through a certain Fisher information distance. Even the strictly super-linear dependence on $\log(1/δ)$ was not known and resolves a question of Grossman and Moshkovitz (FOCS 2016, SIAM Journal on Computing 2020).
Can Directed Graph Neural Networks be Adversarially Robust?
Venue: arXiv cs.LG 2023
Year: 2023

The existing research on robust Graph Neural Networks (GNNs) fails to acknowledge the significance of directed graphs in providing rich information about networks' inherent structure. This work presents the first investigation into the robustness of GNNs in the context of directed graphs, aiming to harness the profound trust implications offered by directed graphs to bolster the robustness and resilience of GNNs. Our study reveals that existing directed GNNs are not adversarially robust. In pursuit of our goal, we introduce a new and realistic directed graph attack setting and propose an innovative, universal, and efficient message-passing framework as a plug-in layer to significantly enhance the robustness of GNNs. Combined with existing defense strategies, this framework achieves outstanding clean accuracy and state-of-the-art robust performance, offering superior defense against both transfer and adaptive attacks. The findings in this study reveal a novel and promising direction for this crucial research area. The code will be made publicly available upon the acceptance of this work.
On Optimal Caching and Model Multiplexing for Large Model Inference
Venue: arXiv cs.LG 2023
Year: 2023

Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. In particular, the large-scale deployment of these models is hindered by the significant resource requirements during inference. In this paper, we study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing. Theoretically, we provide an optimal algorithm for jointly optimizing both approaches to reduce the inference cost in both offline and online tabular settings. By combining a caching algorithm, namely Greedy Dual Size with Frequency (GDSF) or Least Expected Cost (LEC), with a model multiplexer, we achieve optimal rates in both offline and online settings. Empirically, simulations show that the combination of our caching and model multiplexing algorithms greatly improves over the baselines, with up to $50\times$ improvement over the baseline when the ratio between the maximum cost and minimum cost is $100$. Experiments on real datasets show a $4.3\times$ improvement in FLOPs over the baseline when the ratio for FLOPs is $10$, and a $1.8\times$ improvement in latency when the ratio for average latency is $1.85$.
Evolving Knowledge Mining for Class Incremental Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Class Incremental Semantic Segmentation (CISS) has been a trend recently due to its great significance in real-world applications. Although the existing CISS methods demonstrate remarkable performance, they either leverage the high-level knowledge (feature) only while neglecting the rich and diverse knowledge in the low-level features, leading to poor old knowledge preservation and weak new knowledge exploration; or use multi-level features for knowledge distillation by retraining a heavy backbone, which is computationally intensive. In this paper, we for the first time investigate the efficient multi-grained knowledge reuse for CISS, and propose a novel method, Evolving kNowleDge minING (ENDING), employing a frozen backbone. ENDING incorporates two key modules: evolving fusion and semantic enhancement, for dynamic and comprehensive exploration of multi-grained knowledge. Evolving fusion is tailored to extract knowledge from individual low-level feature using a personalized lightweight network, which is generated from a meta-net, taking the high-level feature as input. This design enables the evolution of knowledge mining and fusing when applied to incremental new classes. In contrast, semantic enhancement is specifically crafted to aggregate prototype-based semantics from multi-level features, contributing to an enhanced representation. We evaluate our method on two widely used benchmarks and consistently demonstrate new state-of-the-art performance. The code is available at https://github.com/zhiheLu/ENDING_ISS.
Painsight: An Extendable Opinion Mining Framework for Detecting Pain Points Based on Online Customer Reviews
Venue: arXiv cs.AI 2023
Year: 2023

As the e-commerce market continues to expand and online transactions proliferate, customer reviews have emerged as a critical element in shaping the purchasing decisions of prospective buyers. Previous studies have endeavored to identify key aspects of customer reviews through the development of sentiment analysis models and topic models. However, extracting specific dissatisfaction factors remains a challenging task. In this study, we delineate the pain point detection problem and propose Painsight, an unsupervised framework for automatically extracting distinct dissatisfaction factors from customer reviews without relying on ground truth labels. Painsight employs pre-trained language models to construct sentiment analysis and topic models, leveraging attribution scores derived from model gradients to extract dissatisfaction factors. Upon application of the proposed methodology to customer review data spanning five product categories, we successfully identified and categorized dissatisfaction factors within each group, as well as isolated factors for each type. Notably, Painsight outperformed benchmark methods, achieving substantial performance enhancements and exceptional results in human evaluations.
Verifying C++ Dynamic Binding
Venue: arXiv cs.PL 2023
Year: 2023

We propose an approach for modular verification of programs written in an object-oriented language where, like in C++, the same virtual method call is bound to different methods at different points during the construction or destruction of an object. Our separation logic combines Parkinson and Bierman's abstract predicate families with essentially explicitly tracking each subobject's vtable pointer. Our logic supports polymorphic destruction. Virtual inheritance is not yet supported. We formalised our approach and implemented it in our VeriFast tool for semi-automated modular formal verification of C++ programs.
A Conditional Generative Chatbot using Transformer Model
Venue: arXiv cs.CL 2023
Year: 2023

A Chatbot serves as a communication tool between a human user and a machine to achieve an appropriate answer based on the human input. In more recent approaches, a combination of Natural Language Processing and sequential models are used to build a generative Chatbot. The main challenge of these models is their sequential nature, which leads to less accurate results. To tackle this challenge, in this paper, a novel architecture is proposed using conditional Wasserstein Generative Adversarial Networks and a transformer model for answer generation in Chatbots. While the generator of the proposed model consists of a full transformer model to generate an answer, the discriminator includes only the encoder part of a transformer model followed by a classifier. To the best of our knowledge, this is the first time that a generative Chatbot is proposed using the embedded transformer in both generator and discriminator models. Relying on the parallel computing of the transformer model, the results of the proposed model on the Cornell Movie-Dialog corpus and the Chit-Chat datasets confirm the superiority of the proposed model compared to state-of-the-art alternatives using different evaluation metrics.
Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distribution
Venue: arXiv cs.CV 2023
Year: 2023

Text-to-3D is an emerging task that allows users to create 3D content with infinite possibilities. Existing works tackle the problem by optimizing a 3D representation with guidance from pre-trained diffusion models. An apparent drawback is that they need to optimize from scratch for each prompt, which is computationally expensive and often yields poor visual fidelity. In this paper, we propose DreamPortrait, which aims to generate text-guided 3D-aware portraits in a single-forward pass for efficiency. To achieve this, we extend Score Distillation Sampling from datapoint to distribution formulation, which injects semantic prior into a 3D distribution. However, the direct extension will lead to the mode collapse problem since the objective only pursues semantic alignment. Hence, we propose to optimize a distribution with hierarchical condition adapters and GAN loss regularization. For better 3D modeling, we further design a 3D-aware gated cross-attention mechanism to explicitly let the model perceive the correspondence between the text and the 3D-aware space. These elaborated designs enable our model to generate portraits with robust multi-view semantic consistency, eliminating the need for optimization-based methods. Extensive experiments demonstrate our model's highly competitive performance and significant speed boost against existing methods.
Extending an Event-type Ontology: Adding Verbs and Classes Using Fine-tuned LLMs Suggestions
Venue: arXiv cs.CL 2023
Year: 2023

In this project, we have investigated the use of advanced machine learning methods, specifically fine-tuned large language models, for pre-annotating data for a lexical extension task, namely adding descriptive words (verbs) to an existing (but incomplete, as of yet) ontology of event types. Several research questions have been focused on, from the investigation of a possible heuristics to provide at least hints to annotators which verbs to include and which are outside the current version of the ontology, to the possible use of the automatic scores to help the annotators to be more efficient in finding a threshold for identifying verbs that cannot be assigned to any existing class and therefore they are to be used as seeds for a new class. We have also carefully examined the correlation of the automatic scores with the human annotation. While the correlation turned out to be strong, its influence on the annotation proper is modest due to its near linearity, even though the mere fact of such pre-annotation leads to relatively short annotation times.