OpenRaiser/Intern-Atlas huggingface dataset:
A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns
Venue: arXiv cs.LG 2024
Year: 2024

As recommender systems become widely deployed in different domains, they increasingly influence their users' beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential issues like biases and ethical concerns. In this paper, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them -- notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. In our experiments, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.
Admittance Control-based Floating Base Reaction Mitigation for Limbed Climbing Robots
Venue: arXiv cs.RO 2024
Year: 2024

Reaction force-aware control is essential for legged climbing robots to ensure a safer and more stable operation. This becomes particularly crucial when navigating steep terrain or operating in microgravity environments, where excessive reaction forces may result in the loss of foot contact with the ground, leading to potential falls or floating over in microgravity. Furthermore, such robots are often tasked with manipulation activities, exposing them to external forces in addition to those generated during locomotion. To effectively handle such disturbances while maintaining precise motion trajectory tracking, we propose a novel control scheme based on position-based impedance control, also known as admittance control. We validated this control method through simulation-based case studies by intentionally introducing continuous and impact interference forces to simulate scenarios such as object manipulation or obstacle collisions. The results demonstrated a significant reduction in both the reaction force and joint torque when employing the proposed method.
Large Language Model Should Understand Pinyin for Chinese ASR Error Correction
Venue: arXiv cs.CL 2024
Year: 2024

Large language models can enhance automatic speech recognition systems through generative error correction. In this paper, we propose Pinyin-enhanced GEC, which leverages Pinyi, the phonetic representation of Mandarin Chinese, as supplementary information to improve Chinese ASR error correction. Our approach only utilizes synthetic errors for training and employs the one-best hypothesis during inference. Additionally, we introduce a multitask training approach involving conversion tasks between Pinyin and text to align their feature spaces. Experiments on the Aishell-1 and the Common Voice datasets demonstrate that our approach consistently outperforms GEC with text-only input. More importantly, we provide intuitive explanations for the effectiveness of PY-GEC and multitask training from two aspects: 1) increased attention weight on Pinyin features; and 2) aligned feature space between Pinyin and text hidden states.
JoyHallo: Digital human model for Mandarin
Venue: arXiv cs.CV 2024
Year: 2024

In audio-driven video generation, creating Mandarin videos presents significant challenges. Collecting comprehensive Mandarin datasets is difficult, and the complex lip movements in Mandarin further complicate model training compared to English. In this study, we collected 29 hours of Mandarin speech video from JD Health International Inc. employees, resulting in the jdh-Hallo dataset. This dataset includes a diverse range of ages and speaking styles, encompassing both conversational and specialized medical topics. To adapt the JoyHallo model for Mandarin, we employed the Chinese wav2vec2 model for audio feature embedding. A semi-decoupled structure is proposed to capture inter-feature relationships among lip, expression, and pose features. This integration not only improves information utilization efficiency but also accelerates inference speed by 14.3%. Notably, JoyHallo maintains its strong ability to generate English videos, demonstrating excellent cross-language generation capabilities. The code and models are available at https://jdh-algo.github.io/JoyHallo.
Generative Learning Powered Probing Beam Optimization for Cell-Free Hybrid Beamforming
Venue: arXiv cs.IT 2024
Year: 2024

Probing beam measurement (PBM)-based hybrid beamforming provides a feasible solution for cell-free MIMO. In this letter, we propose a novel probing beam optimization framework where three collaborative modules respectively realize PBM augmentation, sum-rate prediction and probing beam optimization. Specifically, the PBM augmentation model integrates the conditional variational auto-encoder (CVAE) and mixture density networks and adopts correlated PBM distribution with full-covariance, for which a Cholesky-decomposition based training is introduced to address the issues of covariance legality and numerical stability. Simulations verify the better performance of the proposed augmentation model compared to the traditional CVAE and the efficiency of proposed optimization framework.
A Novel Adaptive Fine-Tuning Algorithm for Multimodal Models: Self-Optimizing Classification and Selection of High-Quality Datasets in Remote Sensing
Venue: arXiv cs.CV 2024
Year: 2024

We propose an adaptive fine-tuning algorithm for multimodal large models. The core steps of this algorithm involve two stages of truncation. First, the vast amount of data is projected into a semantic vector space, and the MiniBatchKMeans algorithm is used for automated clustering. This classification ensures that the data within each cluster exhibit high semantic similarity. Next, we process the data in each cluster, calculating the translational difference between the original and perturbed data in the multimodal large model's vector space. This difference serves as a generalization metric for the data. Based on this metric, we select the data with high generalization potential for training. We applied this algorithm to train the InternLM-XComposer2-VL-7B model on two 3090 GPUs using one-third of the GeoChat multimodal remote sensing dataset. The results demonstrate that our algorithm outperforms the state-of-the-art baselines. various baselines. The model trained on our optimally chosen one-third dataset, based on experimental validation, exhibited only 1% reduction in performance across various remote sensing metrics compared to the model trained on the full dataset. This approach significantly preserved general-purpose capabilities while reducing training time by 68.2%. Furthermore, the model achieved scores of 89.86 and 77.19 on the UCMerced and AID evaluation datasets, respectively, surpassing the GeoChat dataset by 5.43 and 5.16 points. It only showed a 0.91-point average decrease on the LRBEN evaluation dataset.
Recent Advancement of Emotion Cognition in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Emotion cognition in large language models (LLMs) is crucial for enhancing performance across various applications, such as social media, human-computer interaction, and mental health assessment. We explore the current landscape of research, which primarily revolves around emotion classification, emotionally rich response generation, and Theory of Mind assessments, while acknowledge the challenges like dependency on annotated data and complexity in emotion processing. In this paper, we present a detailed survey of recent progress in LLMs for emotion cognition. We explore key research studies, methodologies, outcomes, and resources, aligning them with Ulric Neisser's cognitive stages. Additionally, we outline potential future directions for research in this evolving field, including unsupervised learning approaches and the development of more complex and interpretable emotion cognition LLMs. We also discuss advanced methods such as contrastive learning used to improve LLMs' emotion cognition capabilities.
ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practice
Venue: arXiv cs.LG 2024
Year: 2024

Detecting arousals in sleep is essential for diagnosing sleep disorders. However, using Machine Learning (ML) in clinical practice is impeded by fundamental issues, primarily due to mismatches between clinical protocols and ML methods. Clinicians typically annotate only the onset of arousals, while ML methods rely on annotations for both the beginning and end. Additionally, there is no standardized evaluation methodology tailored to clinical needs for arousal detection models. This work addresses these issues by introducing a novel post-processing and evaluation framework emphasizing approximate localization and precise event count (ALPEC) of arousals. We recommend that ML practitioners focus on detecting arousal onsets, aligning with clinical practice. We examine the impact of this shift on current training and evaluation schemes, addressing simplifications and challenges. We utilize a novel comprehensive polysomnographic dataset (CPS) that reflects the aforementioned clinical annotation constraints and includes modalities not present in existing polysomnographic datasets. We release the dataset alongside this paper, demonstrating the benefits of leveraging multimodal data for arousal onset detection. Our findings significantly contribute to integrating ML-based arousal detection in clinical settings, reducing the gap between technological advancements and clinical needs.
More Clustering Quality Metrics for ABCDE
Venue: arXiv cs.IR 2024
Year: 2024

ABCDE is a technique for evaluating clusterings of very large populations of items. Given two clusterings, namely a Baseline clustering and an Experiment clustering, ABCDE can characterize their differences with impact and quality metrics, and thus help to determine which clustering to prefer. We previously described the basic quality metrics of ABCDE, namely the GoodSplitRate, BadSplitRate, GoodMergeRate, BadMergeRate and DeltaPrecision, and how to estimate them on the basis of human judgements. This paper extends that treatment with more quality metrics. It describes a technique that aims to characterize the DeltaRecall of the clustering change. It introduces a new metric, called IQ, to characterize the degree to which the clustering diff translates into an improvement in the quality. Ideally, a large diff would improve the quality by a large amount. Finally, this paper mentions ways to characterize the absolute Precision and Recall of a single clustering with ABCDE.
Evaluating the plausibility of synthetic images for improving automated endoscopic stone recognition
Venue: arXiv cs.CV 2024
Year: 2024

Currently, the Morpho-Constitutional Analysis (MCA) is the de facto approach for the etiological diagnosis of kidney stone formation, and it is an important step for establishing personalized treatment to avoid relapses. More recently, research has focused on performing such tasks intra-operatively, an approach known as Endoscopic Stone Recognition (ESR). Both methods rely on features observed in the surface and the section of kidney stones to separate the analyzed samples into several sub-groups. However, given the high intra-observer variability and the complex operating conditions found in ESR, there is a lot of interest in using AI for computer-aided diagnosis. However, current AI models require large datasets to attain a good performance and for generalizing to unseen distributions. This is a major problem as large labeled datasets are very difficult to acquire, and some classes of kidney stones are very rare. Thus, in this paper, we present a method based on diffusion as a way of augmenting pre-existing ex-vivo kidney stone datasets. Our aim is to create plausible diverse kidney stone images that can be used for pre-training models using ex-vivo data. We show that by mixing natural and synthetic images of CCD images, it is possible to train models capable of performing very well on unseen intra-operative data. Our results show that is possible to attain an improvement of 10% in terms of accuracy compared to a baseline model pre-trained only on ImageNet. Moreover, our results show an improvement of 6% for surface images and 10% for section images compared to a model train on CCD images only, which demonstrates the effectiveness of using synthetic images.
Procedure Model for Building Knowledge Graphs for Industry Applications
Venue: arXiv cs.IR 2024
Year: 2024

Enterprise knowledge graphs combine business data and organizational knowledge by means of a semantic network of concepts, properties, individuals and relationships. The graph-based integration of previously unconnected information with domain knowledge provides new insights and enables intelligent business applications. However, knowledge graph construction is a large investment which requires a joint effort of domain and technical experts. This paper presents a practical step-by-step procedure model for building an RDF knowledge graph that interconnects heterogeneous data and expert knowledge for an industry use case. The self-contained process adapts the "Cross Industry Standard Process for Data Mining" and uses competency questions throughout the entire development cycle. The procedure model starts with business and data understanding, describes tasks for ontology modeling and the graph setup, and ends with process steps for evaluation and deployment.
Towards the Discovery of Down Syndrome Brain Biomarkers Using Generative Models
Venue: arXiv cs.CV 2024
Year: 2024

Brain imaging has allowed neuroscientists to analyze brain morphology in genetic and neurodevelopmental disorders, such as Down syndrome, pinpointing regions of interest to unravel the neuroanatomical underpinnings of cognitive impairment and memory deficits. However, the connections between brain anatomy, cognitive performance and comorbidities like Alzheimer's disease are still poorly understood in the Down syndrome population. The latest advances in artificial intelligence constitute an opportunity for developing automatic tools to analyze large volumes of brain magnetic resonance imaging scans, overcoming the bottleneck of manual analysis. In this study, we propose the use of generative models for detecting brain alterations in people with Down syndrome affected by various degrees of neurodegeneration caused by Alzheimer's disease. To that end, we evaluate state-of-the-art brain anomaly detection models based on Variational Autoencoders and Diffusion Models, leveraging a proprietary dataset of brain magnetic resonance imaging scans. Following a comprehensive evaluation process, our study includes several key analyses. First, we conducted a qualitative evaluation by expert neuroradiologists. Second, we performed both quantitative and qualitative reconstruction fidelity studies for the generative models. Third, we carried out an ablation study to examine how the incorporation of histogram post-processing can enhance model performance. Finally, we executed a quantitative volumetric analysis of subcortical structures. Our findings indicate that some models effectively detect the primary alterations characterizing Down syndrome's brain anatomy, including a smaller cerebellum, enlarged ventricles, and cerebral cortex reduction, as well as the parietal lobe alterations caused by Alzheimer's disease.
Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts
Venue: arXiv cs.CL 2024
Year: 2024

LLMs have demonstrated commendable performance across diverse domains. Nevertheless, formulating high-quality prompts to assist them in their work poses a challenge for non-AI experts. Existing research in prompt engineering suggests somewhat scattered optimization principles and designs empirically dependent prompt optimizers. Unfortunately, these endeavors lack a structural design, incurring high learning costs and it is not conducive to the iterative updating of prompts, especially for non-AI experts. Inspired by structured reusable programming languages, we propose LangGPT, a structural prompt design framework. Furthermore, we introduce Minstrel, a multi-generative agent system with reflection to automate the generation of structural prompts. Experiments and the case study illustrate that structural prompts generated by Minstrel or written manually significantly enhance the performance of LLMs. Furthermore, we analyze the ease of use of structural prompts through a user survey in our online community.
Toward a formalization of artifacts in GFO
Venue: arXiv cs.LO 2024
Year: 2024

The General Formal Ontology (GFO) is a top-level ontology that is designed to formally describe different domains of reality. Most recent advancements within GFO have been made in defining its modules of space and material objects, defining its functions, and a module for integrated data semantics. In this paper, I further develop the GFO towards the integration of artifacts, which are material objects that are intentionally made for a certain purpose. I discuss recent advancements in artifact ontology in philosophy and formal ontology alike, and provide basic categories and axioms for artifact description in GFO, while considering existing work within its space module, object-process integration and function module. Keywords. Artifact Ontology, GFO, Material Object, Formalization
Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?
Venue: arXiv cs.CV 2024
Year: 2024

Concept-based XAI (C-XAI) approaches to explaining neural vision models are a promising field of research, since explanations that refer to concepts (i.e., semantically meaningful parts in an image) are intuitive to understand and go beyond saliency-based techniques that only reveal relevant regions. Given the remarkable progress in this field in recent years, it is time for the community to take a critical look at the advances and trends. Consequently, this paper reviews C-XAI methods to identify interesting and underexplored areas and proposes future research directions. To this end, we consider three main directions: the choice of concepts to explain, the choice of concept representation, and how we can control concepts. For the latter, we propose techniques and draw inspiration from the field of knowledge representation and learning, showing how this could enrich future C-XAI research.
Quantum-Assisted Joint Virtual Network Function Deployment and Maximum Flow Routing for Space Information Networks
Venue: arXiv cs.NI 2024
Year: 2024

Network function virtualization (NFV)-enabled space information network (SIN) has emerged as a promising method to facilitate global coverage and seamless service. This paper proposes a novel NFV-enabled SIN to provide end-to-end communication and computation services for ground users. Based on the multi-functional time expanded graph (MF-TEG), we jointly optimize the user association, virtual network function (VNF) deployment, and flow routing strategy (U-VNF-R) to maximize the total processed data received by users. The original problem is a mixed-integer linear program (MILP) that is intractable for classical computers. Inspired by quantum computing techniques, we propose a hybrid quantum-classical Benders' decomposition (HQCBD) algorithm. Specifically, we convert the master problem of the Benders' decomposition into the quadratic unconstrained binary optimization (QUBO) model and solve it with quantum computers. To further accelerate the optimization, we also design a multi-cut strategy based on the quantum advantages in parallel computing. Numerical results demonstrate the effectiveness and efficiency of the proposed algorithm and U-VNF-R scheme.
A Centrality Approach to Select Offloading Data Aggregation Points in Vehicular Sensor Networks
Venue: arXiv cs.NI 2024
Year: 2024

This work proposes a centrality-based approach to identify data offloading points in a VSN. The solution presents a scheme to select vehicles used as aggregation points to collect and aggregate other vehicles' data before uploading it to processing stations. We evaluate the proposed solution in a realis tic simulation scenario derived from data traffic containing more than 700,000 individual car trips for 24 hours. We compare our approach with both a reservation-based algorithm and the optimal solution. Our results indicate an upload cost reduction of 30.92\% using the centrality-based algorithm and improving the aggregation rate by up to 10.45% when considering the centralized scenario.
First Place Solution to the Multiple-choice Video QA Track of The Second Perception Test Challenge
Venue: arXiv cs.CV 2024
Year: 2024

In this report, we present our first-place solution to the Multiple-choice Video Question Answering (QA) track of The Second Perception Test Challenge. This competition posed a complex video understanding task, requiring models to accurately comprehend and answer questions about video content. To address this challenge, we leveraged the powerful QwenVL2 (7B) model and fine-tune it on the provided training set. Additionally, we employed model ensemble strategies and Test Time Augmentation to boost performance. Through continuous optimization, our approach achieved a Top-1 Accuracy of 0.7647 on the leaderboard.
FullAnno: A Data Engine for Enhancing Image Comprehension of MLLMs
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal Large Language Models (MLLMs) have shown promise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they heavily depend on high-quality data in the Supervised Fine-Tuning (SFT) phase. The existing approaches aim to curate high-quality data via GPT-4V, but they are not scalable due to the commercial nature of GPT-4V and the simplicity of the prompts used to instruct the model. To this end, we devised the FullAnno system, which is a data engine that can generate large-scale, high-quality, and fine-grained image annotations consisting of the category and position of objects, region descriptions, text information, as well as image dense captions. This engine is characterized by its cascade annotation process, which involves multiple expert models and employs rich prompts to instruct LLMs in generating dense image captions. We re-annotated the COCO and Visual Genome datasets using our FullAnno system, tripling the number of object annotations and increasing the length of the original image captions by a factor of 15. Experiments show that the regenerated annotation can significantly enhance the capabilities of LLaVA-v1.5 on several benchmarks. The re-annotated data are available at: https://arcana-project-page.github.io