OpenRaiser/Intern-Atlas huggingface dataset:
AIM 2024 Challenge on UHD Blind Photo Quality Assessment
Venue: arXiv cs.CV 2024
Year: 2024

We introduce the AIM 2024 UHD-IQA Challenge, a competition to advance the No-Reference Image Quality Assessment (NR-IQA) task for modern, high-resolution photos. The challenge is based on the recently released UHD-IQA Benchmark Database, which comprises 6,073 UHD-1 (4K) images annotated with perceptual quality ratings from expert raters. Unlike previous NR-IQA datasets, UHD-IQA focuses on highly aesthetic photos of superior technical quality, reflecting the ever-increasing standards of digital photography. This challenge aims to develop efficient and effective NR-IQA models. Participants are tasked with creating novel architectures and training strategies to achieve high predictive performance on UHD-1 images within a computational budget of 50G MACs. This enables model deployment on edge devices and scalable processing of extensive image collections. Winners are determined based on a combination of performance metrics, including correlation measures (SRCC, PLCC, KRCC), absolute error metrics (MAE, RMSE), and computational efficiency (G MACs). To excel in this challenge, participants leverage techniques like knowledge distillation, low-precision inference, and multi-scale training. By pushing the boundaries of NR-IQA for high-resolution photos, the UHD-IQA Challenge aims to stimulate the development of practical models that can keep pace with the rapidly evolving landscape of digital photography. The innovative solutions emerging from this competition will have implications for various applications, from photo curation and enhancement to image compression.
Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Despite extensive research, open-vocabulary segmentation methods still struggle to generalize across diverse domains. To reduce the computational cost of adapting Vision-Language Models (VLMs) while preserving their pre-trained knowledge, most methods freeze the VLMs for mask classification and train only the mask generator. However, our comprehensive analysis reveals a surprising insight: open-vocabulary segmentation is primarily bottlenecked by mask classification, not mask generation. This discovery prompts us to rethink the existing paradigm and explore an alternative approach. Instead of freezing the VLM, we propose to freeze the pre-trained mask generator and focus on optimizing the mask classifier. Building on the observation that VLMs pre-trained on global-pooled image-text features often fail to capture fine-grained semantics necessary for effective mask classification, we propose a novel Fine-grained Semantic Adaptation (FISA) method to address this limitation. FISA enhances the extracted visual features with fine-grained semantic awareness by explicitly integrating this crucial semantic information early in the visual encoding process. As our method strategically optimizes only a small portion of the VLM's parameters, it enjoys the efficiency of adapting to new data distributions while largely preserving the valuable VLM pre-trained knowledge. Extensive ablation studies confirm the superiority of our approach. Notably, FISA achieves new state-of-the-art results across multiple representative benchmarks, improving performance by up to +1.0 PQ and +3.0 mIoU and reduces training costs by nearly 5x compared to previous best methods. Our code and data will be made public.
On 1-Planar Graphs with Bounded Cop-Number
Venue: arXiv cs.DM 2024
Year: 2024

Cops and Robbers is a type of pursuit-evasion game played on a graph where a set of cops try to capture a single robber. The cops first choose their initial vertex positions, and later the robber chooses a vertex. The cops and robbers make their moves in alternate turns: in the cops' turn, every cop can either choose to move to an adjacent vertex or stay on the same vertex, and likewise the robber in his turn. If the cops can capture the robber in a finite number of rounds, the cops win, otherwise the robber wins. The cop-number of a graph is the minimum number of cops required to catch a robber in the graph. It has long been known that graphs embedded on surfaces (such as planar graphs and toroidal graphs) have a small cop-number. Recently, Durocher et al. [Graph Drawing, 2023] investigated the problem of cop-number for the class of $1$-planar graphs, which are graphs that can be embedded in the plane such that each edge is crossed at most once. They showed that unlike planar graphs which require just three cops, 1-planar graphs have an unbounded cop-number. On the positive side, they showed that maximal 1-planar graphs require only three cops by crucially using the fact that the endpoints of every crossing in an embedded maximal 1-planar graph induce a $K_4$. In this paper, we show that the cop-number remains bounded even under the relaxed condition that the endpoints induce at least three edges. More precisely, let an $\times$-crossing of an embedded 1-planar graph be a crossing whose endpoints induce a matching; i.e., there is no edge connecting the endpoints apart from the crossing edges themselves. We show that any 1-planar graph that can be embedded without $\times$-crossings has cop-number at most 21. Moreover, any 1-planar graph that can be embedded with at most $γ$ $\times$-crossings has cop-number at most $γ+ 21$.
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/
Age of Gossip in Networks with Multiple Views of a Source
Venue: arXiv cs.IT 2024
Year: 2024

We consider the version age of information (AoI) in a network where a subset of nodes act as sensing nodes, sampling a source that in general can follow a continuous distribution. Any sample of the source constitutes a new version of the information and the version age of the information is defined with respect to the most recent version of the information available for the whole network. We derive a recursive expression for the average version AoI between different subsets of the nodes which can be used to evaluate the average version AoI for any subset of the nodes including any single node. We derive asymptotic behavior of the average AoI on any single node of the network for various topologies including line, ring, and fully connected networks. The prior art result on version age of a network by Yates [ISIT'21] can be interpreted as in our derivation as a network with a single view of the source, e.g., through a Poisson process with rate $λ_{00}$. Our result indicates that there is no loss in the average version AoI performance by replacing a single view of the source with distributed sensing across multiple nodes by splitting the same rate $λ_{00}$. Particularly, we show that asymptotically, the average AoI scales with $O(\log(n))$ and $O(\sqrt{n})$ for fully connected and ring networks, respectively. More interestingly, we show that for the ring network the same $O(\sqrt{n})$ asymptotical performance on average AoI is still achieved with distributed sensing if the number of sensing nodes only scales with $O(\sqrt{n})$ instead of prior known result which requires $O(n)$. Our results indicate that the sensing nodes can be arbitrarily chosen as long as the maximum number of consecutive non-sensing nodes also scales as $O(\sqrt{n})$.
Beyond Following: Mixing Active Initiative into Computational Creativity
Venue: arXiv cs.HC 2024
Year: 2024

Generative Artificial Intelligence (AI) encounters limitations in efficiency and fairness within the realm of Procedural Content Generation (PCG) when human creators solely drive and bear responsibility for the generative process. Alternative setups, such as Mixed-Initiative Co-Creative (MI-CC) systems, exhibited their promise. Still, the potential of an active mixed initiative, where AI takes a role beyond following, is understudied. This work investigates the influence of the adaptive ability of an active and learning AI agent on creators' expectancy of creative responsibilities in an MI-CC setting. We built and studied a system that employs reinforcement learning (RL) methods to learn the creative responsibility preferences of a human user during online interactions. Situated in story co-creation, we develop a Multi-armed-bandit agent that learns from the human creator, updates its collaborative decision-making belief, and switches between its capabilities during an MI-CC experience. With 39 participants joining a human subject study, Our developed system's learning capabilities are well recognized compared to the non-learning ablation, corresponding to a significant increase in overall satisfaction with the MI-CC experience. These findings indicate a robust association between effective MI-CC collaborative interactions, particularly the implementation of proactive AI initiatives, and deepened understanding among all participants.
Probabilistic Spatiotemporal Modeling of Day-Ahead Wind Power Generation with Input-Warped Gaussian Processes
Venue: arXiv cs.LG 2024
Year: 2024

We design a Gaussian Process (GP) spatiotemporal model to capture features of day-ahead wind power forecasts. We work with hourly-scale day-ahead forecasts across hundreds of wind farm locations, with the main aim of constructing a fully probabilistic joint model across space and hours of the day. To this end, we design a separable space-time kernel, implementing both temporal and spatial input warping to capture the non-stationarity in the covariance of wind power. We conduct synthetic experiments to validate our choice of the spatial kernel and to demonstrate the effectiveness of warping in addressing nonstationarity. The second half of the paper is devoted to a detailed case study using a realistic, fully calibrated dataset representing wind farms in the ERCOT region of Texas.
Automated Spatio-Temporal Weather Modeling for Load Forecasting
Venue: arXiv cs.LG 2024
Year: 2024

Electricity is difficult to store, except at prohibitive cost, and therefore the balance between generation and load must be maintained at all times. Electricity is traditionally managed by anticipating demand and intermittent production (wind, solar) and matching flexible production (hydro, nuclear, coal and gas). Accurate forecasting of electricity load and renewable production is therefore essential to ensure grid performance and stability. Both are highly dependent on meteorological variables (temperature, wind, sunshine). These dependencies are complex and difficult to model. On the one hand, spatial variations do not have a uniform impact because population, industry, and wind and solar farms are not evenly distributed across the territory. On the other hand, temporal variations can have delayed effects on load (due to the thermal inertia of buildings). With access to observations from different weather stations and simulated data from meteorological models, we believe that both phenomena can be modeled together. In today's state-of-the-art load forecasting models, the spatio-temporal modeling of the weather is fixed. In this work, we aim to take advantage of the automated representation and spatio-temporal feature extraction capabilities of deep neural networks to improve spatio-temporal weather modeling for load forecasting. We compare our deep learning-based methodology with the state-of-the-art on French national load. This methodology could also be fully adapted to forecasting renewable energy production.
Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs
Venue: arXiv cs.CL 2024
Year: 2024

This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.
FastTalker: Jointly Generating Speech and Conversational Gestures from Text
Venue: arXiv cs.MM 2024
Year: 2024

Generating 3D human gestures and speech from a text script is critical for creating realistic talking avatars. One solution is to leverage separate pipelines for text-to-speech (TTS) and speech-to-gesture (STG), but this approach suffers from poor alignment of speech and gestures and slow inference times. In this paper, we introduce FastTalker, an efficient and effective framework that simultaneously generates high-quality speech audio and 3D human gestures at high inference speeds. Our key insight is reusing the intermediate features from speech synthesis for gesture generation, as these features contain more precise rhythmic information than features re-extracted from generated speech. Specifically, 1) we propose an end-to-end framework that concurrently generates speech waveforms and full-body gestures, using intermediate speech features such as pitch, onset, energy, and duration directly for gesture decoding; 2) we redesign the causal network architecture to eliminate dependencies on future inputs for real applications; 3) we employ Reinforcement Learning-based Neural Architecture Search (NAS) to enhance both performance and inference speed by optimizing our network architecture. Experimental results on the BEAT2 dataset demonstrate that FastTalker achieves state-of-the-art performance in both speech synthesis and gesture generation, processing speech and gestures in 0.17 seconds per second on an NVIDIA 3090.
Design of a Reformed Array Logic Binary Multiplier for High-Speed Computations
Venue: arXiv cs.AR 2024
Year: 2024

Binary multipliers have long been a staple component in digital circuitry, serving crucial roles in microprocessor design, digital signal processing units and many more applications. This work presents a unique design for a multiplier that utilizes a reformed-array-logic approach to compute the product of two unsigned binary numbers. We employed a multiplexer and a barrel shifter to multiply partial products in a single clock cycle to speed up the traditional array logic. In addition, we have employed a combination of Carry Save Adders (CSA) and Ripple Carry Adders (RCA) to accumulate the partial products instead of using standalone RCAs to speed up the multiplication process further. Finally, we have demonstrated our design to perform multiplication of two 16-bit unsigned binary numbers on Cadence Virtuoso. Our design is modular and can be scaled up or down to accommodate the multiplication of any n-bit unsigned numbers.
Frequency-based View Selection in Gaussian Splatting Reconstruction
Venue: arXiv cs.CV 2024
Year: 2024

Three-dimensional reconstruction is a fundamental problem in robotics perception. We examine the problem of active view selection to perform 3D Gaussian Splatting reconstructions with as few input images as possible. Although 3D Gaussian Splatting has made significant progress in image rendering and 3D reconstruction, the quality of the reconstruction is strongly impacted by the selection of 2D images and the estimation of camera poses through Structure-from-Motion (SfM) algorithms. Current methods to select views that rely on uncertainties from occlusions, depth ambiguities, or neural network predictions directly are insufficient to handle the issue and struggle to generalize to new scenes. By ranking the potential views in the frequency domain, we are able to effectively estimate the potential information gain of new viewpoints without ground truth data. By overcoming current constraints on model architecture and efficacy, our method achieves state-of-the-art results in view selection, demonstrating its potential for efficient image-based 3D reconstruction.
KinScene: Model-Based Mobile Manipulation of Articulated Scenes
Venue: arXiv cs.RO 2024
Year: 2024

Sequentially interacting with articulated objects is crucial for a mobile manipulator to operate effectively in everyday environments. To enable long-horizon tasks involving articulated objects, this study explores building scene-level articulation models for indoor scenes through autonomous exploration. While previous research has studied mobile manipulation with articulated objects by considering object kinematic constraints, it primarily focuses on individual-object scenarios and lacks extension to a scene-level context for task-level planning. To manipulate multiple object parts sequentially, the robot needs to reason about the resultant motion of each part and anticipate its impact on future actions. We introduce KinScene, a full-stack approach for long-horizon manipulation tasks with articulated objects. The robot maps the scene, detects and physically interacts with articulated objects, collects observations, and infers the articulation properties. For sequential tasks, the robot plans a feasible series of object interactions based on the inferred articulation model. We demonstrate that our approach repeatably constructs accurate scene-level kinematic and geometric models, enabling long-horizon mobile manipulation in a real-world scene. Code and additional results are available at https://chengchunhsu.github.io/KinScene/
SynChart: Synthesizing Charts from Language Models
Venue: arXiv cs.AI 2024
Year: 2024

With the release of GPT-4V(O), its use in generating pseudo labels for multi-modality tasks has gained significant popularity. However, it is still a secret how to build such advanced models from its base large language models (LLMs). This work explores the potential of using LLMs alone for data generation and develop competitive multi-modality models focusing on chart understanding. We construct a large-scale chart dataset, SynChart, which contains approximately 4 million diverse chart images with over 75 million dense annotations, including data tables, code, descriptions, and question-answer sets. We trained a 4.2B chart-expert model using this dataset and achieve near-GPT-4O performance on the ChartQA task, surpassing GPT-4V.
Time Constant: Actuator Fingerprinting using Transient Response of Device and Process in ICS
Venue: arXiv cs.CR 2024
Year: 2024

Command injection and replay attacks are key threats in Cyber Physical Systems (CPS). We develop a novel actuator fingerprinting technique named Time Constant. Time Constant captures the transient dynamics of an actuator and physical process. The transient behavior is device-specific. We combine process and device transient characteristics to develop a copy-resistant actuator fingerprint that resists command injection and replay attacks in the face of insider adversaries. We validated the proposed scheme on data from a real water treatment testbed, as well as through real-time attack detection in the live plant. Our results show that we can uniquely distinguish between process states and actuators based on their Time Constant.
Disentangling Questions from Query Generation for Task-Adaptive Retrieval
Venue: arXiv cs.CL 2024
Year: 2024

This paper studies the problem of information retrieval, to adapt to unseen tasks. Existing work generates synthetic queries from domain-specific documents to jointly train the retriever. However, the conventional query generator assumes the query as a question, thus failing to accommodate general search intents. A more lenient approach incorporates task-adaptive elements, such as few-shot learning with an 137B LLM. In this paper, we challenge a trend equating query and question, and instead conceptualize query generation task as a "compilation" of high-level intent into task-adaptive query. Specifically, we propose EGG, a query generator that better adapts to wide search intents expressed in the BeIR benchmark. Our method outperforms baselines and existing models on four tasks with underexplored intents, while utilizing a query generator 47 times smaller than the previous state-of-the-art. Our findings reveal that instructing the LM with explicit search intent is a key aspect of modeling an effective query generator.
Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications
Venue: arXiv cs.CL 2024
Year: 2024

Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.