OpenRaiser/Intern-Atlas huggingface dataset:
Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

This work addresses the challenge of quantifying originality in text-to-image (T2I) generative diffusion models, with a focus on copyright originality. We begin by evaluating T2I models' ability to innovate and generalize through controlled experiments, revealing that stable diffusion models can effectively recreate unseen elements with sufficiently diverse training data. Then, our key insight is that concepts and combinations of image elements the model is familiar with, and saw more during training, are more concisly represented in the model's latent space. We hence propose a method that leverages textual inversion to measure the originality of an image based on the number of tokens required for its reconstruction by the model. Our approach is inspired by legal definitions of originality and aims to assess whether a model can produce original content without relying on specific prompts or having the training data of the model. We demonstrate our method using both a pre-trained stable diffusion model and a synthetic dataset, showing a correlation between the number of tokens and image originality. This work contributes to the understanding of originality in generative models and has implications for copyright infringement cases.
Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation
Venue: arXiv cs.RO 2024
Year: 2024

To enable non-experts to specify long-horizon, multi-robot collaborative tasks, language models are increasingly used to translate natural language commands into formal specifications. However, because translation can occur in multiple ways, such translations may lack accuracy or lead to inefficient multi-robot planning. Our key insight is that concise hierarchical specifications can simplify planning while remaining straightforward to derive from human instructions. We propose Nl2Hltl2Plan, a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL) and solves the corresponding planning problem. The translation involves two steps leveraging Large Language Models (LLMs). First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations. Next, a fine-tuned LLM converts sub-tasks into flat LTL formulas, which are aggregated into hierarchical specifications, with the lowest level corresponding to ordered robot actions. These specifications are then used with off-the-shelf planners. Our Nl2Hltl2Plan demonstrates the potential of LLMs in hierarchical reasoning for multi-robot task planning. Evaluations in simulation and real-world experiments with human participants show that Nl2Hltl2Plan outperforms existing methods, handling more complex instructions while achieving higher success rates and lower costs in task allocation and planning. Additional details are available at https://nl2hltl2plan.github.io .
DaRec: A Disentangled Alignment Framework for Large Language Model and Recommender System
Venue: arXiv cs.IR 2024
Year: 2024

Benefiting from the strong reasoning capabilities, Large language models (LLMs) have demonstrated remarkable performance in recommender systems. Various efforts have been made to distill knowledge from LLMs to enhance collaborative models, employing techniques like contrastive learning for representation alignment. In this work, we prove that directly aligning the representations of LLMs and collaborative models is sub-optimal for enhancing downstream recommendation tasks performance, based on the information theorem. Consequently, the challenge of effectively aligning semantic representations between collaborative models and LLMs remains unresolved. Inspired by this viewpoint, we propose a novel plug-and-play alignment framework for LLMs and collaborative models. Specifically, we first disentangle the latent representations of both LLMs and collaborative models into specific and shared components via projection layers and representation regularization. Subsequently, we perform both global and local structure alignment on the shared representations to facilitate knowledge transfer. Additionally, we theoretically prove that the specific and shared representations contain more pertinent and less irrelevant information, which can enhance the effectiveness of downstream recommendation tasks. Extensive experimental results on benchmark datasets demonstrate that our method is superior to existing state-of-the-art algorithms.
Strong Data Processing Inequalities and their Applications to Reliable Computation
Venue: arXiv cs.IT 2024
Year: 2024

In 1952, von Neumann gave a series of groundbreaking lectures that proved it was possible for circuits consisting of 3-input majority gates that have a sufficiently small independent probability $δ> 0$ of malfunctioning to reliably compute Boolean functions. In 1999, Evans and Schulman used a strong data-processing inequality (SDPI) to establish the tightest known necessary condition $δ< \frac{1}{2} - \frac{1}{2\sqrt{k}}$ for reliable computation when the circuit consists of components that have at most $k$ inputs. In 2017, Polyanskiy and Wu distilled Evans and Schulman's SDPI argument to establish a general result on the contraction of mutual information in Bayesian networks. In this essay, we will first introduce the problem of reliable computation from unreliable components and establish the existence of noise thresholds. We will then provide an exposition of von Neumann's result with 3-input majority gates and extend it to minority gates. We will then provide an introduction to SDPIs, which have many applications, including in statistical mechanics, portfolio theory, and lower bounds on statistical estimation under privacy constraints. We will then use the introduced material to provide an exposition of Polyanskiy and Wu's 2017 result on Bayesian networks, from which the 1999 result of Evans-Schulman follows.
mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis
Venue: arXiv cs.CL 2024
Year: 2024

This paper introduces mhGPT, a lightweight generative pre-trained transformer trained on mental health-related social media and PubMed articles. Fine-tuned for specific mental health tasks, mhGPT was evaluated under limited hardware constraints and compared with state-of-the-art models like MentaLLaMA and Gemma. Despite having only 1.98 billion parameters and using just 5% of the dataset, mhGPT outperformed larger models and matched the performance of models trained on significantly more data. The key contributions include integrating diverse mental health data, creating a custom tokenizer, and optimizing a smaller architecture for low-resource settings. This research could advance AI-driven mental health care, especially in areas with limited computing power.
Segment Anything for Videos: A Systematic Survey
Venue: arXiv cs.CV 2024
Year: 2024

The recent wave of foundation models has witnessed tremendous success in computer vision (CV) and beyond, with the segment anything model (SAM) having sparked a passion for exploring task-agnostic visual foundation models. Empowered by its remarkable zero-shot generalization, SAM is currently challenging numerous traditional paradigms in CV, delivering extraordinary performance not only in various image segmentation and multi-modal segmentation (\eg, text-to-mask) tasks, but also in the video domain. Additionally, the latest released SAM 2 is once again sparking research enthusiasm in the realm of promptable visual segmentation for both images and videos. However, existing surveys mainly focus on SAM in various image processing tasks, a comprehensive and in-depth review in the video domain is notably absent. To address this gap, this work conducts a systematic review on SAM for videos in the era of foundation models. As the first to review the progress of SAM for videos, this work focuses on its applications to various tasks by discussing its recent advances, and innovation opportunities of developing foundation models on broad applications. We begin with a brief introduction to the background of SAM and video-related research domains. Subsequently, we present a systematic taxonomy that categorizes existing methods into three key areas: video understanding, video generation, and video editing, analyzing and summarizing their advantages and limitations. Furthermore, comparative results of SAM-based and current state-of-the-art methods on representative benchmarks, as well as insightful analysis are offered. Finally, we discuss the challenges faced by current research and envision several future research directions in the field of SAM for video and beyond.
Exploring Urban Comfort through Novel Wearables and Environmental Surveys
Venue: arXiv cs.HC 2024
Year: 2024

This study presents a comprehensive dataset capturing indoor environmental parameters, physiological responses, and subjective perceptions across three global cities. Utilizing wearable sensors, including smart eyeglasses, and a modified Cozie app, environmental and physiological data were collected, along with pre-screening, onboarding, and recurring surveys. Peripheral cues facilitated participant engagement with micro-EMA surveys, minimizing disruption over a 5-day collection period. The dataset offers insights into urban comfort dynamics, highlighting the interplay between environmental conditions, physiological responses, and subjective perceptions. Researchers can utilize this dataset to deepen their understanding of indoor environmental quality and inform the design of healthier built environments. Access to this dataset can advance indoor environmental research and contribute to the creation of more comfortable and sustainable indoor spaces.
Plan with Code: Comparing approaches for robust NL to DSL generation
Venue: arXiv cs.SE 2024
Year: 2024

Planning in code is considered a more reliable approach for many orchestration tasks. This is because code is more tractable than steps generated via Natural Language and make it easy to support more complex sequences by abstracting deterministic logic into functions. It also allows spotting issues with incorrect function names with the help of parsing checks that can be run on code. Progress in Code Generation methodologies, however, remains limited to general-purpose languages like C, C++, and Python. LLMs continue to face challenges with custom function names in Domain Specific Languages or DSLs, leading to higher hallucination rates and syntax errors. This is more common for custom function names, that are typically part of the plan. Moreover, keeping LLMs up-to-date with newer function names is an issue. This poses a challenge for scenarios like task planning over a large number of APIs, since the plan is represented as a DSL having custom API names. In this paper, we focus on workflow automation in RPA (Robotic Process Automation) domain as a special case of task planning. We present optimizations for using Retrieval Augmented Generation (or RAG) with LLMs for DSL generation along with an ablation study comparing these strategies with a fine-tuned model. Our results showed that the fine-tuned model scored the best on code similarity metric. However, with our optimizations, RAG approach is able to match the quality for in-domain API names in the test set. Additionally, it offers significant advantage for out-of-domain or unseen API names, outperforming Fine-Tuned model on similarity metric by 7 pts.
API-guided Dataset Synthesis to Finetune Large Code Models
Venue: arXiv cs.SE 2024
Year: 2024

Large code models (LCMs), pre-trained on vast code corpora, have demonstrated remarkable performance across a wide array of code-related tasks. Supervised fine-tuning (SFT) plays a vital role in aligning these models with specific requirements and enhancing their performance in particular domains. However, synthesizing high-quality SFT datasets poses a significant challenge due to the uneven quality of datasets and the scarcity of domain-specific datasets. Inspired by APIs as high-level abstractions of code that encapsulate rich semantic information in a concise structure, we propose DataScope, an API-guided dataset synthesis framework designed to enhance the SFT process for LCMs in both general and domain-specific scenarios. DataScope comprises two main components: Dsel and Dgen. On one hand, Dsel employs API coverage as a core metric, enabling efficient dataset synthesis in general scenarios by selecting subsets of existing (uneven-quality) datasets with higher API coverage. On the other hand, Dgen recasts domain dataset synthesis as a process of using API-specified high-level functionality and deliberately-constituted code skeletons to synthesize concrete code. Extensive experiments demonstrate DataScope's effectiveness, with models fine-tuned on its synthesized datasets outperforming those tuned on unoptimized datasets five times larger. Furthermore, a series of analyses on model internals, relevant hyperparameters, and case studies provide additional evidence for the efficacy of our proposed methods. These findings underscore the significance of dataset quality in SFT and advance the field of LCMs by providing an efficient, cost-effective framework for constructing high-quality datasets. This contribution enhances performance across both general and domain-specific scenarios, paving the way for more powerful and tailored LCMs.
GraphQL Adoption and Challenges: Community-Driven Insights from StackOverflow Discussions
Venue: arXiv cs.SE 2024
Year: 2024

GraphQL is a query language and web application programming interface (API) for client-server architecture. Its advantages include type-safe queries, which allow clients to retrieve the data they require precisely in a single request. As organizations adopt GraphQL for API implementations, it is imperative to understand its challenges and the software community's interests. To achieve this goal, we conducted a five-step mixed-method empirical analysis of 45K StackOverflow questions and answers on GraphQL. In the first step, we derive a reference architecture for the GraphQL ecosystem with five key layers. Second, we used topic modeling based on Latent Dirichlet Allocation (LDA) to automatically identify 14 topics and 47 subtopics. Third, we mapped discussion topics to architecture layers. Fourth, we manually investigate questions on each topic and subtopics to provide additional insight to the GraphQL stakeholders. Finally, we study topic difficulty, popularity, trends, and tradeoffs to provide insights into evolving community interests and challenges. Our results indicate that Client and Server are the top two architectural layers attracting discussion on SO. While earlier discussions on SO focused on building third-party applications consuming GraphQL APIs (i.e., API Integration) released by large organizations, recent trends suggest more organizations implementing APIs using GraphQL servers. Due to difficulty and lack of well-defined solutions, security remains a difficult and low-interest area. However, such a practice can lead to vulnerable APIs.
Kernelization for Orthogonality Dimension
Venue: arXiv cs.DS 2024
Year: 2024

The orthogonality dimension of a graph over $\mathbb{R}$ is the smallest integer $d$ for which one can assign to every vertex a nonzero vector in $\mathbb{R}^d$ such that every two adjacent vertices receive orthogonal vectors. For an integer $d$, the $d$-Ortho-Dim$_\mathbb{R}$ problem asks to decide whether the orthogonality dimension of a given graph over $\mathbb{R}$ is at most $d$. We prove that for every integer $d \geq 3$, the $d$-Ortho-Dim$_\mathbb{R}$ problem parameterized by the vertex cover number $k$ admits a kernel with $O(k^{d-1})$ vertices and bit-size $O(k^{d-1} \cdot \log k)$. We complement this result by a nearly matching lower bound, showing that for any $\varepsilon > 0$, the problem admits no kernel of bit-size $O(k^{d-1-\varepsilon})$ unless $\mathsf{NP} \subseteq \mathsf{coNP/poly}$. We further study the kernelizability of orthogonality dimension problems in additional settings, including over general fields and under various structural parameterizations.
Parameterized Algorithms for Optimal Refugee Resettlement
Venue: arXiv cs.GT 2024
Year: 2024

We study variants of the Optimal Refugee Resettlement problem where a set $F$ of refugee families need to be allocated to a set $L$ of possible places of resettlement in a feasible and optimal way. Feasibility issues emerge from the assumption that each family requires certain services (such as accommodation, school seats, or medical assistance), while there is an upper and, possibly, a lower quota on the number of service units provided at a given place. Besides studying the problem of finding a feasible assignment, we also investigate two natural optimization variants. In the first one, we allow families to express preferences over $P$, and we aim for a Pareto-optimal assignment. In a more general setting, families can attribute utilities to each place in $P$, and the task is to find a feasible assignment with maximum total utilities. We study the computational complexity of all three variants in a multivariate fashion using the framework of parameterized complexity. We provide fixed-parameter tractable algorithms for a handful of natural parameterizations, and complement these tractable cases with tight intractability results.
Level Up Your Tutorials: VLMs for Game Tutorials Quality Assessment
Venue: arXiv cs.CV 2024
Year: 2024

Designing effective game tutorials is crucial for a smooth learning curve for new players, especially in games with many rules and complex core mechanics. Evaluating the effectiveness of these tutorials usually requires multiple iterations with testers who have no prior knowledge of the game. Recent Vision-Language Models (VLMs) have demonstrated significant capabilities in understanding and interpreting visual content. VLMs can analyze images, provide detailed insights, and answer questions about their content. They can recognize objects, actions, and contexts in visual data, making them valuable tools for various applications, including automated game testing. In this work, we propose an automated game-testing solution to evaluate the quality of game tutorials. Our approach leverages VLMs to analyze frames from video game tutorials, answer relevant questions to simulate human perception, and provide feedback. This feedback is compared with expected results to identify confusing or problematic scenes and highlight potential errors for developers. In addition, we publish complete tutorial videos and annotated frames from different game versions used in our tests. This solution reduces the need for extensive manual testing, especially by speeding up and simplifying the initial development stages of the tutorial to improve the final game experience.
Efficient low rank model order reduction of vibroacoustic problems under stochastic loads
Venue: arXiv cs.CE 2024
Year: 2024

This contribution combines a low-rank matrix approximation through Singular Value Decomposition (SVD) with second-order Krylov subspace-based Model Order Reduction (MOR), in order to efficiently propagate input uncertainties through a given vibroacoustic model. The vibroacoustic model consists of a plate coupled to a fluid into which the plate radiates sound due to a turbulent boundary layer excitation. This excitation is subject to uncertainties due to the stochastic nature of the turbulence and the computational cost of simulating the coupled problem with stochastic forcing is very high. The proposed method approximates the output uncertainties in an efficient way, by reducing the evaluation cost of the model in terms of DOFs and samples by using the factors of the SVD low-rank approximation directly as input for the MOR algorithm. Here, the covariance matrix of the vector of unknowns can efficiently be approximated with only a fraction of the original number of evaluations. Therefore, the approach is a promising step to further reducing the computational effort of large-scale vibroacoustic evaluations.
Domination in Diameter-Two Graphs and the 2-Club Cluster Vertex Deletion Parameter
Venue: arXiv cs.CC 2024
Year: 2024

The s-club cluster vertex deletion number of a graph, or sccvd, is the minimum number of vertices whose deletion results in a disjoint union of s-clubs, or graphs whose diameter is bounded above by s. We launch a study of several domination problems on diameter-two graphs, or 2-clubs, and study their parameterized complexity with respect to the 2ccvd number as main parameter. We further propose to explore the class of problems that become solvable in sub-exponential time when the running time is independent of some input parameter. Hardness of problems for this class depends on the Exponential-Time Hypothesis. We give examples of problems that are in the proposed class and problems that are hard for it.
Multi-Modal Dialogue State Tracking for Playing GuessWhich Game
Venue: arXiv cs.AI 2024
Year: 2024

GuessWhich is an engaging visual dialogue game that involves interaction between a Questioner Bot (QBot) and an Answer Bot (ABot) in the context of image-guessing. In this game, QBot's objective is to locate a concealed image solely through a series of visually related questions posed to ABot. However, effectively modeling visually related reasoning in QBot's decision-making process poses a significant challenge. Current approaches either lack visual information or rely on a single real image sampled at each round as decoding context, both of which are inadequate for visual reasoning. To address this limitation, we propose a novel approach that focuses on visually related reasoning through the use of a mental model of the undisclosed image. Within this framework, QBot learns to represent mental imagery, enabling robust visual reasoning by tracking the dialogue state. The dialogue state comprises a collection of representations of mental imagery, as well as representations of the entities involved in the conversation. At each round, QBot engages in visually related reasoning using the dialogue state to construct an internal representation, generate relevant questions, and update both the dialogue state and internal representation upon receiving an answer. Our experimental results on the VisDial datasets (v0.5, 0.9, and 1.0) demonstrate the effectiveness of our proposed model, as it achieves new state-of-the-art performance across all metrics and datasets, surpassing previous state-of-the-art models. Codes and datasets from our experiments are freely available at \href{https://github.com/xubuvd/GuessWhich}.
Lifelong Reinforcement Learning via Neuromodulation
Venue: arXiv cs.LG 2024
Year: 2024

Navigating multiple tasks$\unicode{x2014}$for instance in succession as in continual or lifelong learning, or in distributions as in meta or multi-task learning$\unicode{x2014}$requires some notion of adaptation. Evolution over timescales of millennia has imbued humans and other animals with highly effective adaptive learning and decision-making strategies. Central to these functions are so-called neuromodulatory systems. In this work we introduce an abstract framework for integrating theories and evidence from neuroscience and the cognitive sciences into the design of adaptive artificial reinforcement learning algorithms. We give a concrete instance of this framework built on literature surrounding the neuromodulators Acetylcholine (ACh) and Noradrenaline (NA), and empirically validate the effectiveness of the resulting adaptive algorithm in a non-stationary multi-armed bandit problem. We conclude with a theory-based experiment proposal providing an avenue to link our framework back to efforts in experimental neuroscience.