OpenRaiser/Intern-Atlas huggingface dataset:
V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results
Venue: arXiv cs.CV 2024
Year: 2024

Detecting objects in real-world scenes is a complex task due to various challenges, including the vast range of object categories, and potential encounters with previously unknown or unseen objects. The challenges necessitate the development of public benchmarks and challenges to advance the field of object detection. Inspired by the success of previous COCO and LVIS Challenges, we organize the V3Det Challenge 2024 in conjunction with the 4th Open World Vision Workshop: Visual Perception via Learning in an Open World (VPLOW) at CVPR 2024, Seattle, US. This challenge aims to push the boundaries of object detection research and encourage innovation in this field. The V3Det Challenge 2024 consists of two tracks: 1) Vast Vocabulary Object Detection: This track focuses on detecting objects from a large set of 13204 categories, testing the detection algorithm's ability to recognize and locate diverse objects. 2) Open Vocabulary Object Detection: This track goes a step further, requiring algorithms to detect objects from an open set of categories, including unknown objects. In the following sections, we will provide a comprehensive summary and analysis of the solutions submitted by participants. By analyzing the methods and solutions presented, we aim to inspire future research directions in vast vocabulary and open-vocabulary object detection, driving progress in this field. Challenge homepage: https://v3det.openxlab.org.cn/challenge
Folk-ontological stances toward robots and psychological human likeness
Venue: arXiv cs.RO 2024
Year: 2024

It has often been argued that people can attribute mental states to robots without making any ontological commitments to the reality of those states. But what does it mean to 'attribute' a mental state to a robot, and what is an 'ontological commitment'? It will be argued that, on a plausible interpretation of these two notions, it is not clear how mental state attribution can occur without any ontological commitment. Taking inspiration from the philosophical debate on scientific realism, a provisional taxonomy of folk-ontological stances towards robots will also be identified, corresponding to different ways of understanding robotic minds. They include realism, non-realism, eliminativism, reductionism, fictionalism and agnosticism. Instrumentalism will also be discussed and presented as a folk-epistemological stance. In the last part of the article it will be argued that people's folk-ontological stances towards robots and humans can influence their perception of the human-likeness of robots. The analysis carried out here can be seen as encouraging a 'folk-ontological turn' in human-robot interaction research, aimed at explicitly determining what beliefs people have about the reality of robot minds.
A Brief Survey on Leveraging Large Scale Vision Models for Enhanced Robot Grasping
Venue: arXiv cs.RO 2024
Year: 2024

Robotic grasping presents a difficult motor task in real-world scenarios, constituting a major hurdle to the deployment of capable robots across various industries. Notably, the scarcity of data makes grasping particularly challenging for learned models. Recent advancements in computer vision have witnessed a growth of successful unsupervised training mechanisms predicated on massive amounts of data sourced from the Internet, and now nearly all prominent models leverage pretrained backbone networks. Against this backdrop, we begin to investigate the potential benefits of large-scale visual pretraining in enhancing robot grasping performance. This preliminary literature review sheds light on critical challenges and delineates prospective directions for future research in visual pretraining for robotic manipulation.
LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
Venue: arXiv cs.RO 2024
Year: 2024

In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize in different settings has often been less than desired. To address this, we introduce LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic learning tasks, scenarios, and environments. Additionally, we show that predicting intermediate 2-D representations, which we refer to as "visual traces", can help further align vision and action spaces for robot learning. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model, and we evaluate on 12 different tasks in the RLBench simulator as well as a physical Franka Emika Panda 7-DoF robot. Our experiments yield strong performance, demonstrating that LLARVA - using 2-D and language representations - performs well compared to several contemporary baselines, and can generalize across various robot environments and configurations.
Spectral Introspection Identifies Group Training Dynamics in Deep Neural Networks for Neuroimaging
Venue: arXiv cs.LG 2024
Year: 2024

Neural networks, whice have had a profound effect on how researchers study complex phenomena, do so through a complex, nonlinear mathematical structure which can be difficult for human researchers to interpret. This obstacle can be especially salient when researchers want to better understand the emergence of particular model behaviors such as bias, overfitting, overparametrization, and more. In Neuroimaging, the understanding of how such phenomena emerge is fundamental to preventing and informing users of the potential risks involved in practice. In this work, we present a novel introspection framework for Deep Learning on Neuroimaging data, which exploits the natural structure of gradient computations via the singular value decomposition of gradient components during reverse-mode auto-differentiation. Unlike post-hoc introspection techniques, which require fully-trained models for evaluation, our method allows for the study of training dynamics on the fly, and even more interestingly, allow for the decomposition of gradients based on which samples belong to particular groups of interest. We demonstrate how the gradient spectra for several common deep learning models differ between schizophrenia and control participants from the COBRE study, and illustrate how these trajectories may reveal specific training dynamics helpful for further analysis.
Language Modeling with Editable External Knowledge
Venue: arXiv cs.CL 2024
Year: 2024

When the world changes, so does the text that humans write about it. How do we build language models that can be easily updated to reflect these changes? One popular approach is retrieval-augmented generation, in which new documents are inserted into a knowledge base and retrieved during prediction for downstream tasks. Most prior work on these systems have focused on improving behavior during prediction through better retrieval or reasoning. This paper introduces ERASE, which instead improves model behavior when new documents are acquired, by incrementally deleting or rewriting other entries in the knowledge base each time a document is added. In two new benchmark datasets evaluating models' ability to answer questions about a stream of news articles or conversations, ERASE improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute. Code and data are available at https://github.com/belindal/ERASE
Closed-loop Teaching via Demonstrations to Improve Policy Transparency
Venue: arXiv cs.CY 2024
Year: 2024

Demonstrations are a powerful way of increasing the transparency of AI policies. Though informative demonstrations may be selected a priori through the machine teaching paradigm, student learning may deviate from the preselected curriculum in situ. This paper thus explores augmenting a curriculum with a closed-loop teaching framework inspired by principles from the education literature, such as the zone of proximal development and the testing effect. We utilize tests accordingly to close to the loop and maintain a novel particle filter model of human beliefs throughout the learning process, allowing us to provide demonstrations that are targeted to the human's current understanding in real time. A user study finds that our proposed closed-loop teaching framework reduces the regret in human test responses by 43% over a baseline.
Ethical Framework for Responsible Foundational Models in Medical Imaging
Venue: arXiv cs.CY 2024
Year: 2024

Foundational models (FMs) have tremendous potential to revolutionize medical imaging. However, their deployment in real-world clinical settings demands extensive ethical considerations. This paper aims to highlight the ethical concerns related to FMs and propose a framework to guide their responsible development and implementation within medicine. We meticulously examine ethical issues such as privacy of patient data, bias mitigation, algorithmic transparency, explainability and accountability. The proposed framework is designed to prioritize patient welfare, mitigate potential risks, and foster trust in AI-assisted healthcare.
The EarlyBird Gets the WORM: Heuristically Accelerating EarlyBird Convergence
Venue: arXiv cs.LG 2024
Year: 2024

The Lottery Ticket hypothesis proposes that ideal, sparse subnetworks, called lottery tickets, exist in untrained dense neural networks. The Early Bird hypothesis proposes an efficient algorithm to find these winning lottery tickets in convolutional neural networks, using the novel concept of distance between subnetworks to detect convergence in the subnetworks of a model. However, this approach overlooks unchanging groups of unimportant neurons near the search's end. We proposes WORM, a method that exploits these static groups by truncating their gradients, forcing the model to rely on other neurons. Experiments show WORM achieves faster ticket identification during training on convolutional neural networks, despite the additional computational overhead, when compared to EarlyBird search. Additionally, WORM-pruned models lose less accuracy during pruning and recover accuracy faster, improving the robustness of a given model. Furthermore, WORM is also able to generalize the Early Bird hypothesis reasonably well to larger models, such as transformers, displaying its flexibility to adapt to more complex architectures.
Data Petri Nets meet Probabilistic Programming (Extended version)
Venue: arXiv cs.PL 2024
Year: 2024

Probabilistic programming (PP) is a programming paradigm that allows for writing statistical models like ordinary programs, performing simulations by running those programs, and analyzing and refining their statistical behavior using powerful inference engines. This paper takes a step towards leveraging PP for reasoning about data-aware processes. To this end, we present a systematic translation of Data Petri Nets (DPNs) into a model written in a PP language whose features are supported by most PP systems. We show that our translation is sound and provides statistical guarantees for simulating DPNs. Furthermore, we discuss how PP can be used for process mining tasks and report on a prototype implementation of our translation. We also discuss further analysis scenarios that could be easily approached based on the proposed translation and available PP tools.
Understanding the Dynamics of the Stack Overflow Community through Social Network Analysis and Graph Algorithms
Venue: arXiv cs.SI 2024
Year: 2024

This thesis conducts a focused literature review on online communities, centering on Stack Overflow, employing social network analysis and graph algorithms. It examines the evolving landscape of health information quality within the digital ecosystem, emphasizing the challenges posed and the multifaceted nature of quality. The significance of online communities, notably Stack Overflow, as hubs for social interaction and knowledge sharing is underscored. Proposing advanced approaches, the thesis introduces an ensemble deep learning model for traffic flow forecasting, an efficient multi-objective optimization method for influence maximization, and a graph convolutional neural network-based approach for link prediction.
Decomposed evaluations of geographic disparities in text-to-image models
Venue: arXiv cs.CV 2024
Year: 2024

Recent work has identified substantial disparities in generated images of different geographic regions, including stereotypical depictions of everyday objects like houses and cars. However, existing measures for these disparities have been limited to either human evaluations, which are time-consuming and costly, or automatic metrics evaluating full images, which are unable to attribute these disparities to specific parts of the generated images. In this work, we introduce a new set of metrics, Decomposed Indicators of Disparities in Image Generation (Decomposed-DIG), that allows us to separately measure geographic disparities in the depiction of objects and backgrounds in generated images. Using Decomposed-DIG, we audit a widely used latent diffusion model and find that generated images depict objects with better realism than backgrounds and that backgrounds in generated images tend to contain larger regional disparities than objects. We use Decomposed-DIG to pinpoint specific examples of disparities, such as stereotypical background generation in Africa, struggling to generate modern vehicles in Africa, and unrealistically placing some objects in outdoor settings. Informed by our metric, we use a new prompting structure that enables a 52% worst-region improvement and a 20% average improvement in generated background diversity.
The Benefits and Risks of Transductive Approaches for AI Fairness
Venue: arXiv cs.LG 2024
Year: 2024

Recently, transductive learning methods, which leverage holdout sets during training, have gained popularity for their potential to improve speed, accuracy, and fairness in machine learning models. Despite this, the composition of the holdout set itself, particularly the balance of sensitive sub-groups, has been largely overlooked. Our experiments on CIFAR and CelebA datasets show that compositional changes in the holdout set can substantially influence fairness metrics. Imbalanced holdout sets exacerbate existing disparities, while balanced holdouts can mitigate issues introduced by imbalanced training data. These findings underline the necessity of constructing holdout sets that are both diverse and representative.
Outer Space Cyberattacks: Generating Novel Scenarios to Avoid Surprise
Venue: arXiv cs.CR 2024
Year: 2024

Though general awareness around it may be low, space cyberattacks are an increasingly urgent problem given the vital role that space systems play in the modern world. Open-source or public discussions about it typically revolve around only a couple generic scenarios, namely satellite hacking and signals jamming or spoofing. But there are so many more possibilities. The report offers a scenario-prompt generator -- a taxonomy of sorts, called the ICARUS matrix -- that can create more than 4 million unique scenario-prompts. We will offer a starting set of 42 scenarios, briefly describing each one, to begin priming the imagination-pump so that many more researchers can bring their diverse expertise and perspectives to bear on the problem. A failure to imagine novel scenarios is a major risk in being taken by surprise and severely harmed by threat actors who are constantly devising new ways, inventive and resourceful ways, to breach the digital systems that control our wired world. To stay vigilant, defenders likewise need to be imaginative to keep up in this adversarial dance between hunter and prey in cybersecurity. More than offering novel scenarios, we will also explore the drivers of the space cybersecurity problem, which include at least seven factors we have identified. For instance, the shared threat of space debris would seem to push rational states and actors to avoid kinetic conflicts in orbit, which weighs in favor of cyberoperations as the dominant form of space conflicts. Outer space is the next frontier for cybersecurity. To guard against space cyberattacks, we need to understand and anticipate them, and imagination is at the very heart of both cybersecurity and frontiers.
A Construction of Optimal Quasi-cyclic Locally Recoverable Codes using Constituent Codes
Venue: arXiv cs.IT 2024
Year: 2024

A locally recoverable code of locality $r$ over $\mathbb{F}_{q}$ is a code where every coordinate of a codeword can be recovered using the values of at most $r$ other coordinates of that codeword. Locally recoverable codes are efficient at restoring corrupted messages and data which make them highly applicable to distributed storage systems. Quasi-cyclic codes of length $n=m\ell$ and index $\ell$ are linear codes that are invariant under cyclic shifts by $\ell$ places. %Quasi-cyclic codes are generalizations of cyclic codes and are isomorphic to $\mathbb{F}_{q} [x]/ \langle x^m-1 \rangle$-submodules of $\mathbb{F}_{q^\ell} [x] / \langle x^m-1 \rangle$. In this paper, we decompose quasi-cyclic locally recoverable codes into a sum of constituent codes where each constituent code is a linear code over a field extension of $\mathbb{F}_q$. Using these constituent codes with set parameters, we propose conditions which ensure the existence of almost optimal and optimal quasi-cyclic locally recoverable codes with increased dimension and code length.
FAWN: Floor-And-Walls Normal Regularization for Direct Neural TSDF Reconstruction
Venue: arXiv cs.CV 2024
Year: 2024

Leveraging 3D semantics for direct 3D reconstruction has a great potential yet unleashed. For instance, by assuming that walls are vertical, and a floor is planar and horizontal, we can correct distorted room shapes and eliminate local artifacts such as holes, pits, and hills. In this paper, we propose FAWN, a modification of truncated signed distance function (TSDF) reconstruction methods, which considers scene structure by detecting walls and floor in a scene, and penalizing the corresponding surface normals for deviating from the horizontal and vertical directions. Implemented as a 3D sparse convolutional module, FAWN can be incorporated into any trainable pipeline that predicts TSDF. Since FAWN requires 3D semantics only for training, no additional limitations on further use are imposed. We demonstrate, that FAWN-modified methods use semantics more effectively, than existing semantic-based approaches. Besides, we apply our modification to state-of-the-art TSDF reconstruction methods, and demonstrate a quality gain in SCANNET, ICL-NUIM, TUM RGB-D, and 7SCENES benchmarks.
Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint
Venue: arXiv cs.CV 2024
Year: 2024

As we push the boundaries of performance in various vision tasks, the models grow in size correspondingly. To keep up with this growth, we need very aggressive pruning techniques for efficient inference and deployment on edge devices. Existing pruning approaches are limited to channel pruning and struggle with aggressive parameter reductions. In this paper, we propose a novel multi-dimensional pruning framework that jointly optimizes pruning across channels, layers, and blocks while adhering to latency constraints. We develop a latency modeling technique that accurately captures model-wide latency variations during pruning, which is crucial for achieving an optimal latency-accuracy trade-offs at high pruning ratio. We reformulate pruning as a Mixed-Integer Nonlinear Program (MINLP) to efficiently determine the optimal pruned structure with only a single pass. Our extensive results demonstrate substantial improvements over previous methods, particularly at large pruning ratios. In classification, our method significantly outperforms prior art HALP with a Top-1 accuracy of 70.0(v.s. 68.6) and an FPS of 5262 im/s(v.s. 4101 im/s). In 3D object detection, we establish a new state-of-the-art by pruning StreamPETR at a 45% pruning ratio, achieving higher FPS (37.3 vs. 31.7) and mAP (0.451 vs. 0.449) than the dense baseline.
Mutual Learning for Finetuning Click-Through Rate Prediction Models
Venue: arXiv cs.IR 2024
Year: 2024

Click-Through Rate (CTR) prediction has become an essential task in digital industries, such as digital advertising or online shopping. Many deep learning-based methods have been implemented and have become state-of-the-art models in the domain. To further improve the performance of CTR models, Knowledge Distillation based approaches have been widely used. However, most of the current CTR prediction models do not have much complex architectures, so it's hard to call one of them 'cumbersome' and the other one 'tiny'. On the other hand, the performance gap is also not very large between complex and simple models. So, distilling knowledge from one model to the other could not be worth the effort. Under these considerations, Mutual Learning could be a better approach, since all the models could be improved mutually. In this paper, we showed how useful the mutual learning algorithm could be when it is between equals. In our experiments on the Criteo and Avazu datasets, the mutual learning algorithm improved the performance of the model by up to 0.66% relative improvement.
Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Reddit
Venue: arXiv cs.CL 2024
Year: 2024

Online communities such as drug-related subreddits serve as safe spaces for people who use drugs (PWUD), fostering discussions on substance use experiences, harm reduction, and addiction recovery. Users' shared narratives on these forums provide insights into the likelihood of developing a substance use disorder (SUD) and recovery potential. Our study aims to develop a multi-level, multi-label classification model to analyze online user-generated texts about substance use experiences. For this purpose, we first introduce a novel taxonomy to assess the nature of posts, including their intended connections (Inquisition or Disclosure), subjects (e.g., Recovery, Dependency), and specific objectives (e.g., Relapse, Quality, Safety). Using various multi-label classification algorithms on a set of annotated data, we show that GPT-4, when prompted with instructions, definitions, and examples, outperformed all other models. We apply this model to label an additional 1,000 posts and analyze the categories of linguistic expression used within posts in each class. Our analysis shows that topics such as Safety, Combination of Substances, and Mental Health see more disclosure, while discussions about physiological Effects focus on harm reduction. Our work enriches the understanding of PWUD's experiences and informs the broader knowledge base on SUD and drug use.