OpenRaiser/Intern-Atlas huggingface dataset:
RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

Utilizing Vision-Language Models (VLMs) for robotic manipulation represents a novel paradigm, aiming to enhance the model's ability to generalize to new objects and instructions. However, due to variations in camera specifications and mounting positions, existing methods exhibit significant performance disparities across different robotic platforms. To address this challenge, we propose RoboUniView in this paper, an innovative approach that decouples visual feature extraction from action learning. We first learn a unified view representation from multi-perspective views by pre-training on readily accessible data, and then derive actions from this unified view representation to control robotic manipulation. This unified view representation more accurately mirrors the physical world and is not constrained by the robotic platform's camera parameters. Thanks to this methodology, we achieve state-of-the-art performance on the demanding CALVIN benchmark, enhancing the success rate in the $D \to D$ setting from 93.0% to 96.2%, and in the $ABC \to D$ setting from 92.2% to 94.2%. Moreover, our model exhibits outstanding adaptability and flexibility: it maintains high performance under unseen camera parameters, can utilize multiple datasets with varying camera parameters, and is capable of joint cross-task learning across datasets. Code is provided for re-implementation. https://github.com/liufanfanlff/RoboUniview
E-Mapper: Energy-Efficient Resource Allocation for Traditional Operating Systems on Heterogeneous Processors
Venue: arXiv cs.OS 2024
Year: 2024

Energy efficiency has become a key concern in modern computing. Major processor vendors now offer heterogeneous architectures that combine powerful cores with energy-efficient ones, such as Intel P/E systems, Apple M1 chips, and Samsungs Exyno's CPUs. However, apart from simple cost-based thread allocation strategies, today's OS schedulers do not fully exploit these systems' potential for adaptive energy-efficient computing. This is, in part, due to missing application-level interfaces to pass information about task-level energy consumption and application-level elasticity. This paper presents E-Mapper, a novel resource management approach integrated into Linux for improved execution on heterogeneous processors. In E-Mapper, we base resource allocation decisions on high-level application descriptions that user can attach to programs or that the system can learn automatically at runtime. Our approach supports various programming models including OpenMP, Intel TBB, and TensorFlow. Crucially, E-Mapper leverages this information to extend beyond existing thread-to-core allocation strategies by actively managing application configurations through a novel uniform application-resource manager interface. By doing so, E-Mapper achieves substantial enhancements in both performance and energy efficiency, particularly in multi-application scenarios. On an Intel Raptor Lake and an Arm big.LITTLE system, E-Mapper reduces the application execution on average by 20 % with an average reduction in energy consumption of 34 %. We argue that our solution marks a crucial step toward creating a generic approach for sustainable and efficient computing across different processor architectures.
Efficient course recommendations with T5-based ranking and summarization
Venue: arXiv cs.IR 2024
Year: 2024

In this paper, we implement and evaluate a two-stage retrieval pipeline for a course recommender system that ranks courses for skill-occupation pairs. The in-production recommender system BrightFit provides course recommendations from multiple sources. Some of the course descriptions are long and noisy, while retrieval and ranking in an online system have to be highly efficient. We developed a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We compare two summarizers for course descriptions: a LongT5 model that we finetuned for the task, and a generative LLM (Vicuna) with in-context learning. We experiment with quantization to reduce the size of the ranking model and increase inference speed. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. On the two labelled datasets, our proposed two-stage ranking with automatic summarization achieves a substantial improvement over the in-production (BM25) ranker: nDCG@10 scores improve from 0.482 to 0.684 and from 0.447 to 0.844 on the two datasets. We also achieve a 40% speed-up by using a quantized version of RankT5. The improved quality of the ranking was confirmed by the questionnaire completed by 29 respondents, but not by the A/B test. In the A/B test, a higher clickthrough rate was observed for the BM25-ranking than for the proposed two-stage retrieval. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5. In the online evaluation, however, other factors than relevance play a role (such as speed and interpretability of the retrieval results), as well as individual preferences.
DiffLoss: unleashing diffusion model as constraint for training image restoration network
Venue: arXiv cs.CV 2024
Year: 2024

Image restoration aims to enhance low quality images, producing high quality images that exhibit natural visual characteristics and fine semantic attributes. Recently, the diffusion model has emerged as a powerful technique for image generation, and it has been explicitly employed as a backbone in image restoration tasks, yielding excellent results. However, it suffers from the drawbacks of slow inference speed and large model parameters due to its intrinsic characteristics. In this paper, we introduce a new perspective that implicitly leverages the diffusion model to assist the training of image restoration network, called DiffLoss, which drives the restoration results to be optimized for naturalness and semantic-aware visual effect. To achieve this, we utilize the mode coverage capability of the diffusion model to approximate the distribution of natural images and explore its ability to capture image semantic attributes. On the one hand, we extract intermediate noise to leverage its modeling capability of the distribution of natural images, which serves as a naturalness-oriented optimization space. On the other hand, we utilize the bottleneck features of diffusion model to harness its semantic attributes serving as a constraint on semantic level. By combining these two designs, the overall loss function is able to improve the perceptual quality of image restoration, resulting in visually pleasing and semantically enhanced outcomes. To validate the effectiveness of our method, we conduct experiments on various common image restoration tasks and benchmarks. Extensive experimental results demonstrate that our approach enhances the visual quality and semantic perception of the restoration network.
Improving Weak-to-Strong Generalization with Reliability-Aware Alignment
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) are now rapidly advancing and surpassing human abilities on many natural language tasks. However, aligning these super-human LLMs with human knowledge remains challenging because the supervision signals from human annotators may be wrong. This issue, known as the "super-alignment" problem, requires enhancing weak-to-strong generalization, where a strong LLM must generalize from imperfect supervision provided by a weaker source. To address this issue, we propose an approach to improve weak-to-strong generalization by involving the reliability of weak supervision signals in the alignment process. In our method, we query the weak supervisor for multiple answers, estimate the answer reliability, and enhance the alignment process by filtering out uncertain data or re-weighting reliable data. Experiments on four datasets demonstrate that our methods effectively identify the quality of weak labels and significantly enhance weak-to-strong generalization. Our work presents effective techniques for error-robust model alignment, reducing error propagation from noisy supervision and enhancing the accuracy and reliability of LLMs. Codes are publicly available at http://github.com/Irenehere/ReliableAlignment.
Spatial Multiplexing in Near-Field Line-of-Sight MIMO Communications: Paraxial and Non-Paraxial Deployments
Venue: arXiv cs.IT 2024
Year: 2024

Sixth generation (6G) wireless networks are envisioned to include aspects of energy footprint reduction (sustainability), besides those of network capacity and connectivity, at the design stage. This paradigm change requires radically new physical layer technologies. Notably, the integration of large-aperture arrays and the transmission over high frequency bands, such as the sub-terahertz spectrum, are two promising options. In many communication scenarios of practical interest, the use of large antenna arrays in the sub-terahertz frequency range often results in short-range transmission distances that are characterized by line-of-sight channels, in which pairs of transmitters and receivers are located in the (radiating) near field of one another. These features make the traditional designs, based on the far-field approximation, for multiple-input multiple-output (MIMO) systems sub-optimal in terms of spatial multiplexing gains. To overcome these limitations, new designs for MIMO systems are required, which account for the spherical wavefront that characterizes the electromagnetic waves in the near field, in order to ensure the highest spatial multiplexing gain without increasing the power expenditure. In this paper, we introduce an analytical framework for optimizing the deployment of antenna arrays in line-of-sight channels, which can be applied to paraxial and non-paraxial network deployments. In the paraxial setting, we devise a simpler analytical framework, which, compared to those available in the literature, provides explicit information about the impact of key design parameters. In the non-paraxial setting, we introduce a novel analytical framework that allows us to identify a set of sufficient conditions to be fulfilled for achieving the highest spatial multiplexing gain. The proposed designs are validated with numerical simulations.
MegIS: High-Performance, Energy-Efficient, and Low-Cost Metagenomic Analysis with In-Storage Processing
Venue: arXiv cs.AR 2024
Year: 2024

Metagenomics has led to significant advances in many fields. Metagenomic analysis commonly involves the key tasks of determining the species present in a sample and their relative abundances. These tasks require searching large metagenomic databases. Metagenomic analysis suffers from significant data movement overhead due to moving large amounts of low-reuse data from the storage system. In-storage processing can be a fundamental solution for reducing this overhead. However, designing an in-storage processing system for metagenomics is challenging because existing approaches to metagenomic analysis cannot be directly implemented in storage effectively due to the hardware limitations of modern SSDs. We propose MegIS, the first in-storage processing system designed to significantly reduce the data movement overhead of the end-to-end metagenomic analysis pipeline. MegIS is enabled by our lightweight design that effectively leverages and orchestrates processing inside and outside the storage system. We address in-storage processing challenges for metagenomics via specialized and efficient 1) task partitioning, 2) data/computation flow coordination, 3) storage technology-aware algorithmic optimizations, 4) data mapping, and 5) lightweight in-storage accelerators. MegIS's design is flexible, capable of supporting different types of metagenomic input datasets, and can be integrated into various metagenomic analysis pipelines. Our evaluation shows that MegIS outperforms the state-of-the-art performance- and accuracy-optimized software metagenomic tools by 2.7$\times$-37.2$\times$ and 6.9$\times$-100.2$\times$, respectively, while matching the accuracy of the accuracy-optimized tool. MegIS achieves 1.5$\times$-5.1$\times$ speedup compared to the state-of-the-art metagenomic hardware-accelerated (using processing-in-memory) tool, while achieving significantly higher accuracy.
RAVEN: Multitask Retrieval Augmented Vision-Language Learning
Venue: arXiv cs.CV 2024
Year: 2024

The scaling of large language models to encode all the world's knowledge in model parameters is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. Existing methods focus on models designed for single tasks. Furthermore, they're limited by the need for resource intensive pre training, additional parameter requirements, unaddressed modality prioritization and lack of clear benefit over non-retrieval baselines. This paper introduces RAVEN, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning. By integrating retrieval augmented samples without the need for additional retrieval-specific parameters, we show that the model acquires retrieval properties that are effective across multiple tasks. Our results and extensive ablations across retrieved modalities for the image captioning and VQA tasks indicate significant performance improvements compared to non retrieved baselines +1 CIDEr on MSCOCO, +4 CIDEr on NoCaps and nearly a +3\% accuracy on specific VQA question types. This underscores the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.
Advancing operational PM2.5 forecasting with dual deep neural networks (D-DNet)
Venue: arXiv cs.LG 2024
Year: 2024

PM2.5 forecasting is crucial for public health, air quality management, and policy development. Traditional physics-based models are computationally demanding and slow to adapt to real-time conditions. Deep learning models show potential in efficiency but still suffer from accuracy loss over time due to error accumulation. To address these challenges, we propose a dual deep neural network (D-DNet) prediction and data assimilation system that efficiently integrates real-time observations, ensuring reliable operational forecasting. D-DNet excels in global operational forecasting for PM2.5 and AOD550, maintaining consistent accuracy throughout the entire year of 2019. It demonstrates notably higher efficiency than the Copernicus Atmosphere Monitoring Service (CAMS) 4D-Var operational forecasting system while maintaining comparable accuracy. This efficiency benefits ensemble forecasting, uncertainty analysis, and large-scale tasks.
Robust Classification of Dynamic Bichromatic point Sets in R2
Venue: arXiv cs.CG 2024
Year: 2024

Let $R \cup B$ be a set of $n$ points in $\mathbb{R}^2$, and let $k \in 1..n$. Our goal is to compute a line that "best" separates the "red" points $R$ from the "blue" points $B$ with at most $k$ outliers. We present an efficient semi-online dynamic data structure that can maintain whether such a separator exists. Furthermore, we present efficient exact and approximation algorithms that compute a linear separator that is guaranteed to misclassify at most $k$, points and minimizes the distance to the farthest outlier. Our exact algorithm runs in $O(nk + n \log n)$ time, and our $(1+\varepsilon)$-approximation algorithm runs in $O(\varepsilon^{-1/2}((n + k^2) \log n))$ time. Based on our $(1+\varepsilon)$-approximation algorithm we then also obtain a semi-online data structure to maintain such a separator efficiently.
Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos
Venue: arXiv cs.CV 2024
Year: 2024

Despite significant advancements in robotic systems and surgical data science, ensuring safe and optimal execution in robot-assisted minimally invasive surgery (RMIS) remains a complex challenge. Current surgical error detection methods involve two parts: identifying surgical gestures and then detecting errors within each gesture clip. These methods seldom consider the rich contextual and semantic information inherent in surgical videos, limiting their performance due to reliance on accurate gesture identification. Motivated by the chain-of-thought prompting in natural language processing, this letter presents a novel and real-time end-to-end error detection framework, Chain-of-Thought (COG) prompting, leveraging contextual information from surgical videos. This encompasses two reasoning modules designed to mimic the decision-making processes of expert surgeons. Concretely, we first design a Gestural-Visual Reasoning module, which utilizes transformer and attention architectures for gesture prompting, while the second, a Multi-Scale Temporal Reasoning module, employs a multi-stage temporal convolutional network with both slow and fast paths for temporal information extraction. We extensively validate our method on the public benchmark RMIS dataset JIGSAWS. Our method encapsulates the reasoning processes inherent to surgical activities enabling it to outperform the state-of-the-art by 4.6% in F1 score, 4.6% in Accuracy, and 5.9% in Jaccard index while processing each frame in 6.69 milliseconds on average, demonstrating the great potential of our approach in enhancing the safety and efficacy of RMIS procedures and surgical education. The code will be available.
VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation
Venue: arXiv cs.CL 2024
Year: 2024

Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false). We address this issue with VERISCORE, a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content. VERISCORE can be effectively implemented with either closed or fine-tuned open-weight language models, and human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks. We use VERISCORE to evaluate generations from 16 different models across multiple long-form tasks and find that while GPT-4o is the best-performing model overall, open-weight models such as Mixtral-8x22 are closing the gap. We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., long-form QA), highlighting the need for expanding factuality evaluation across tasks with varying fact density.
Understanding Routing-Induced Censorship Changes Globally
Venue: arXiv cs.NI 2024
Year: 2024

Internet censorship is pervasive, with significant effort dedicated to understanding what is censored, and where. Prior censorship work however have identified significant inconsistencies in their results; experiments show unexplained non-determinism thought to be caused by censor load, end-host geographic diversity, or incomplete censorship -- inconsistencies which impede reliable, repeatable and correct understanding of global censorship. In this work we investigate the extent to which Equal-cost Multi-path (ECMP) routing is the cause for these inconsistencies, developing methods to measure and compensate for them. We find ECMP routing significantly changes observed censorship across protocols, censor mechanisms, and in 17 countries. We identify that previously observed non-determinism or regional variations are attributable to measurements between fixed end-hosts taking different routes based on Flow-ID; i.e., choice of intra-subnet source IP or ephemeral source port leads to differences in observed censorship. To achieve this we develop new route-stable censorship measurement methods that allow consistent measurement of DNS, HTTP, and HTTPS censorship. We find ECMP routing yields censorship changes across 42% of IPs and 51% of ASes, but that impact is not uniform. We identify numerous causes of the behavior, ranging from likely failed infrastructure, to routes to the same end-host taking geographically diverse paths which experience differences in censorship en-route. Finally, we explore our results in the context of prior global measurement studies, exploring first the applicability of our findings to prior observed variations, and then demonstrating how specific experiments from two studies could be impacted by, and specific results are explainable by, ECMP routing. Our work points to methods for improving future studies, reducing inconsistencies and increasing repeatability.
Enhanced Data Transfer Cooperating with Artificial Triplets for Scene Graph Generation
Venue: arXiv cs.CV 2024
Year: 2024

This work focuses on training dataset enhancement of informative relational triplets for Scene Graph Generation (SGG). Due to the lack of effective supervision, the current SGG model predictions perform poorly for informative relational triplets with inadequate training samples. Therefore, we propose two novel training dataset enhancement modules: Feature Space Triplet Augmentation (FSTA) and Soft Transfer. FSTA leverages a feature generator trained to generate representations of an object in relational triplets. The biased prediction based sampling in FSTA efficiently augments artificial triplets focusing on the challenging ones. In addition, we introduce Soft Transfer, which assigns soft predicate labels to general relational triplets to make more supervisions for informative predicate classes effectively. Experimental results show that integrating FSTA and Soft Transfer achieve high levels of both Recall and mean Recall in Visual Genome dataset. The mean of Recall and mean Recall is the highest among all the existing model-agnostic methods.
CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement
Venue: arXiv cs.CV 2024
Year: 2024

Understanding how humans cooperatively rearrange household objects is critical for VR/AR and human-robot interaction. However, in-depth studies on modeling these behaviors are under-researched due to the lack of relevant datasets. We fill this gap by presenting CORE4D, a novel large-scale 4D human-object-human interaction dataset focusing on collaborative object rearrangement, which encompasses diverse compositions of various object geometries, collaboration modes, and 3D scenes. With 1K human-object-human motion sequences captured in the real world, we enrich CORE4D by contributing an iterative collaboration retargeting strategy to augment motions to a variety of novel objects. Leveraging this approach, CORE4D comprises a total of 11K collaboration sequences spanning 3K real and virtual object shapes. Benefiting from extensive motion patterns provided by CORE4D, we benchmark two tasks aiming at generating human-object interaction: human-object motion forecasting and interaction synthesis. Extensive experiments demonstrate the effectiveness of our collaboration retargeting strategy and indicate that CORE4D has posed new challenges to existing human-object interaction generation methodologies.