OpenRaiser/Intern-Atlas huggingface dataset:
Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity
Venue: arXiv cs.LG 2024
Year: 2024

Federated Learning (FL) stands to gain significant advantages from collaboratively training capacity-heterogeneous models, enabling the utilization of private data and computing power from low-capacity devices. However, the focus on personalizing capacity-heterogeneous models based on client-specific data has been limited, resulting in suboptimal local model utility, particularly for low-capacity clients. The heterogeneity in both data and device capacity poses two key challenges for model personalization: 1) accurately retaining necessary knowledge embedded within reduced submodels for each client, and 2) effectively sharing knowledge through aggregating size-varying parameters. To this end, we introduce Pa3dFL, a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. First, we decompose each layer of the model into general and personal parameters. Then, we maintain uniform sizes for the general parameters across clients and aggregate them through direct averaging. Subsequently, we employ a hyper-network to generate size-varying personal parameters for clients using learnable embeddings. Finally, we facilitate the implicit aggregation of personal parameters by aggregating client embeddings through a self-attention module. We conducted extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL. Our findings indicate that Pa3dFL consistently outperforms baseline methods across various heterogeneity settings. Moreover, Pa3dFL demonstrates competitive communication and computation efficiency compared to baseline approaches, highlighting its practicality and adaptability in adverse system conditions.
Multi-label Class Incremental Emotion Decoding with Augmented Emotional Semantics Learning
Venue: arXiv cs.AI 2024
Year: 2024

Emotion decoding plays an important role in affective human-computer interaction. However, previous studies ignored the dynamic real-world scenario, where human experience a blend of multiple emotions which are incrementally integrated into the model, leading to the multi-label class incremental learning (MLCIL) problem. Existing methods have difficulty in solving MLCIL issue due to notorious catastrophic forgetting caused by partial label problem and inadequate label semantics mining. In this paper, we propose an augmented emotional semantics learning framework for multi-label class incremental emotion decoding. Specifically, we design an augmented emotional relation graph module with label disambiguation to handle the past-missing partial label problem. Then, we leverage domain knowledge from affective dimension space to alleviate future-missing partial label problem by knowledge distillation. Besides, an emotional semantics learning module is constructed with a graph autoencoder to obtain emotion embeddings in order to guide the semantic-specific feature decoupling for better multi-label learning. Extensive experiments on three datasets show the superiority of our method for improving emotion decoding performance and mitigating forgetting on MLCIL problem.
Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning
Venue: arXiv cs.CV 2024
Year: 2024

Skeleton-based action representation learning aims to interpret and understand human behaviors by encoding the skeleton sequences, which can be categorized into two primary training paradigms: supervised learning and self-supervised learning. However, the former one-hot classification requires labor-intensive predefined action categories annotations, while the latter involves skeleton transformations (e.g., cropping) in the pretext tasks that may impair the skeleton structure. To address these challenges, we introduce a novel skeleton-based training framework (C$^2$VL) based on Cross-modal Contrastive learning that uses the progressive distillation to learn task-agnostic human skeleton action representation from the Vision-Language knowledge prompts. Specifically, we establish the vision-language action concept space through vision-language knowledge prompts generated by pre-trained large multimodal models (LMMs), which enrich the fine-grained details that the skeleton action space lacks. Moreover, we propose the intra-modal self-similarity and inter-modal cross-consistency softened targets in the cross-modal representation learning process to progressively control and guide the degree of pulling vision-language knowledge prompts and corresponding skeletons closer. These soft instance discrimination and self-knowledge distillation strategies contribute to the learning of better skeleton-based action representations from the noisy skeleton-vision-language pairs. During the inference phase, our method requires only the skeleton data as the input for action recognition and no longer for vision-language prompts. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that our method outperforms the previous methods and achieves state-of-the-art results. Code is available at: https://github.com/cseeyangchen/C2VL.
Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning
Venue: arXiv cs.AI 2024
Year: 2024

As the applicability of Large Language Models (LLMs) extends beyond traditional text processing tasks, there is a burgeoning interest in their potential to excel in planning and reasoning assignments, realms traditionally reserved for System 2 cognitive competencies. Despite their perceived versatility, the research community is still unraveling effective strategies to harness these models in such complex domains. The recent discourse introduced by the paper on LLM Modulo marks a significant stride, proposing a conceptual framework that enhances the integration of LLMs into diverse planning and reasoning activities. This workshop paper delves into the practical application of this framework within the domain of travel planning, presenting a specific instance of its implementation. We are using the Travel Planning benchmark by the OSU NLP group, a benchmark for evaluating the performance of LLMs in producing valid itineraries based on user queries presented in natural language. While popular methods of enhancing the reasoning abilities of LLMs such as Chain of Thought, ReAct, and Reflexion achieve a meager 0%, 0.6%, and 0% with GPT3.5-Turbo respectively, our operationalization of the LLM-Modulo framework for TravelPlanning domain provides a remarkable improvement, enhancing baseline performances by 4.6x for GPT4-Turbo and even more for older models like GPT3.5-Turbo from 0% to 5%. Furthermore, we highlight the other useful roles of LLMs in the planning pipeline, as suggested in LLM-Modulo, which can be reliably operationalized such as extraction of useful critics and reformulator for critics.
GenMix: Combining Generative and Mixture Data Augmentation for Medical Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we propose a novel data augmentation technique called GenMix, which combines generative and mixture approaches to leverage the strengths of both methods. While generative models excel at creating new data patterns, they face challenges such as mode collapse in GANs and difficulties in training diffusion models, especially with limited medical imaging data. On the other hand, mixture models enhance class boundary regions but tend to favor the major class in scenarios with class imbalance. To address these limitations, GenMix integrates both approaches to complement each other. GenMix operates in two stages: (1) training a generative model to produce synthetic images, and (2) performing mixup between synthetic and real data. This process improves the quality and diversity of synthetic data while simultaneously benefiting from the new pattern learning of generative models and the boundary enhancement of mixture models. We validate the effectiveness of our method on the task of classifying focal liver lesions (FLLs) in CT images. Our results demonstrate that GenMix enhances the performance of various generative models, including DCGAN, StyleGAN, Textual Inversion, and Diffusion Models. Notably, the proposed method with Textual Inversion outperforms other methods without fine-tuning diffusion model on the FLL dataset.
Automatic Counting and Classification of Mosquito Eggs in Field Traps
Venue: arXiv cs.AI 2024
Year: 2024

Insect pest control poses a global challenge, affecting public health, food safety, and the environment. Diseases transmitted by mosquitoes are expanding beyond tropical regions due to climate change. Agricultural pests further exacerbate economic losses by damaging crops. The Sterile Insect Technique (SIT) emerges as an eco-friendly alternative to chemical pesticides, involving the sterilization and release of male insects to curb population growth. This work focuses on the automation of the analysis of field ovitraps used to follow-up a SIT program for the Aedes albopictus mosquito in the Valencian Community, Spain, funded by the Conselleria de Agricultura, Agua, Ganaderia y Pesca. Previous research has leveraged deep learning algorithms to automate egg counting in ovitraps, yet faced challenges such as manual handling and limited analysis capacity. Innovations in our study include classifying eggs as hatched or unhatched and reconstructing ovitraps from partial images, mitigating issues of duplicity and cut eggs. Also, our device can analyze multiple ovitraps simultaneously without the need of manual replacement. This approach significantly enhances the accuracy of egg counting and classification, providing a valuable tool for large-scale field studies. This document describes part of the work of the project Application of Industry 4.0 techniques to the production of tiger mosquitoes for the Sterile Insect Technique (MoTIA2,IMDEEA/2022/70), financed by the Valencian Institute for Business Competitiveness (IVACE) and the FEDER funds. The participation of J.Naranjo-Alcazar, J.Grau-Haro and P.Zuccarello has been possible thanks to funding from IVACE and FEDER funds. The participation of D.Almenar has been financed by the Conselleria de Agricultura, Agua, Ganaderia y Pesca of the Generalitat Valenciana and the Subdireccion de Innovacion y Desarrollo de Servicios (TRAGSA group).
LightDE: A Lightweight Method for Eliminating Dangling Pointers
Venue: arXiv cs.CR 2024
Year: 2024

The widespread presence of Use-After-Free (UAF) vulnerabilities poses a serious threat to software security, with dangling pointers being considered the primary cause of these vulnerabilities. However, existing methods for defending against UAF vulnerabilities by eliminating dangling pointers need to interrupt the program's execution when encountering pointer assignment operations in order to store the memory addresses of the pointers in a specific data structure. This makes these methods not lightweight. To overcome this drawback, we propose a novel approach called LightDE. This method does not require storing the memory addresses of pointers during program execution. LightDE uses our proposed structure-sensitive pointer analysis method to determine which objects pointers point to and stores the pointing relationships in the program's data segment during program compilation. Since LightDE only needs to verify if pointers identified by the pointer analysis point to released objects when eliminating dangling pointers, it is very lightweight. Our experimental results show that LightDE can effectively defend against UAF vulnerabilities and the performance overhead it introduces is very low.
FinGen: A Dataset for Argument Generation in Finance
Venue: arXiv cs.CL 2024
Year: 2024

Thinking about the future is one of the important activities that people do in daily life. Futurists also pay a lot of effort into figuring out possible scenarios for the future. We argue that the exploration of this direction is still in an early stage in the NLP research. To this end, we propose three argument generation tasks in the financial application scenario. Our experimental results show these tasks are still big challenges for representative generation models. Based on our empirical results, we further point out several unresolved issues and challenges in this research direction.
Fast Evaluation of S-boxes with Garbled Circuits
Venue: arXiv cs.CR 2024
Year: 2024

Garbling schemes are vital primitives for privacy-preserving protocols and secure two-party computation. This paper presents a projective garbling scheme that assigns $2^n$ values to wires in a circuit comprising XOR and unary projection gates. A generalization of FreeXOR allows the XOR of wires with $2^n$ values to be very efficient. We then analyze the performance of our scheme by evaluating substitution-permutation ciphers. Using our proposal, we measure high-speed evaluation of the ciphers with a moderately increased cost in garbling and bandwidth. Theoretical analysis suggests that for evaluating the nine examined ciphers, one can expect a 4- to 70-fold improvement in evaluation performance with, at most, a 4-fold increase in garbling cost and, at most, an 8-fold increase in communication cost compared to the Half-Gates (Zahur, Rosulek and Evans; Eurocrypt'15) and ThreeHalves (Rosulek and Roy; Crypto'21) garbling schemes. In an offline/online setting, such as secure function evaluation as a service, the circuit garbling and communication to the evaluator can proceed in the offline phase. Thus, our scheme offers a fast online phase. Furthermore, we present efficient Boolean circuits for the S-boxes of TWINE and Midori64 ciphers. To our knowledge, our formulas give the smallest number of AND gates for the S-boxes of these two ciphers.
Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images
Venue: arXiv cs.CV 2024
Year: 2024

Vision-language models have emerged as a powerful tool for previously challenging multi-modal classification problem in the medical domain. This development has led to the exploration of automated image description generation for multi-modal clinical scans, particularly for radiology report generation. Existing research has focused on clinical descriptions for specific modalities or body regions, leaving a gap for a model providing entire-body multi-modal descriptions. In this paper, we address this gap by automating the generation of standardized body station(s) and list of organ(s) across the whole body in multi-modal MR and CT radiological images. Leveraging the versatility of the Contrastive Language-Image Pre-training (CLIP), we refine and augment the existing approach through multiple experiments, including baseline model fine-tuning, adding station(s) as a superset for better correlation between organs, along with image and language augmentations. Our proposed approach demonstrates 47.6% performance improvement over baseline PubMedCLIP.
Ovis: Structural Embedding Alignment for Multimodal Large Language Model
Venue: arXiv cs.CV 2024
Year: 2024

Current Multimodal Large Language Models (MLLMs) typically integrate a pre-trained LLM with another pre-trained vision transformer through a connector, such as an MLP, endowing the LLM with visual capabilities. However, the misalignment between two embedding strategies in MLLMs -- the structural textual embeddings based on an embedding look-up table and the continuous embeddings generated directly by the vision encoder -- makes challenges for a more seamless fusion of visual and textual information. We propose Ovis, a novel MLLM architecture designed to structurally align visual and textual embeddings. Ovis integrates an additional learnable visual embedding table into the visual encoder's process. To capture rich visual semantics, each image patch indexes the visual embedding table multiple times, resulting in a final visual embedding that is a probabilistic combination of the indexed embeddings. This structural approach mirrors the method used for generating textual embeddings. Empirical evaluations on various multimodal benchmarks show that Ovis outperforms open-source MLLMs of similar parameter scales and even surpasses the proprietary model Qwen-VL-Plus overall. These results highlight the potential of Ovis' structured visual representation for advancing MLLM architectural design and promoting more effective multimodal learning. Code, datasets, and models are available at https://github.com/AIDC-AI/Ovis.
Heuristic evaluations of back support, shoulder support, hand grip strength support, and sit-stand support exoskeletons using universal design principles
Venue: arXiv cs.HC 2024
Year: 2024

Occupational exoskeletons promise to reduce the incidence of musculoskeletal injuries; however, we do not know if their designs allow universal use by all workers. We also do not know how easy the tasks of assembling, donning, doffing, and disassembling exoskeletons are. The purpose of our study was to heuristically evaluate a back support, a shoulder support, a handgrip strength support, and a sit-stand exoskeleton for how well they are designed for universal use when assembling, donning, doffing, and disassembling the exoskeleton. Seven evaluators used universal design principles and associated criteria to independently evaluate and rate four exoskeletons when assembling, donning, doffing, and disassembling the devices. The rating scale was a Likert-type scale, where a rating of 1 represented not at all, and a rating of 5 represented an excellent design with respect to the universal design criteria for the task. The results indicate that providing perceptible information to the user, making the design equitable to use for a diverse set of users, making the design simple and intuitive to use with adequate feedback, and designing to prevent user errors, and when errors are made, allowing the user to recover quickly from the errors, were rated poorly. Assembling and donning tasks presented the most challenges.
Proportionally dense subgraphs of maximum size in degree-constrained graphs
Venue: arXiv cs.CC 2024
Year: 2024

A proportionally dense subgraph (PDS) of a graph is an induced subgraph of size at least two such that every vertex in the subgraph has proportionally as many neighbors inside as outside of the subgraph. Then, maxPDS is the problem of determining a PDS of maximum size in a given graph. If we further require that a PDS induces a connected subgraph, we refer to such problem as connected maxPDS. In this paper, we study the complexity of maxPDS with respect to parameters representing the density of a graph and its complement. We consider $Δ$, representing the maximum degree, $h$, representing the $h$-index, and degen, representing the degeneracy of a graph. We show that maxPDS is NP-hard parameterized by $Δ,h$ and degen. More specifically, we show that maxPDS is NP-hard on graphs with $Δ=4$, $h=4$ and degen=2. Then, we show that maxPDS is NP-hard when restricted to dense graphs, more specifically graphs $G$ such that $Δ(\overline{G})\leq 6$, and graphs $G$ such that $degen(\overline{G}) \leq 2$ and $\overline{G}$ is bipartite, where $\overline{G}$ represents the complement of $G$. On the other hand, we show that maxPDS is polynomial-time solvable on graphs with $h\le2$. Finally, we consider graphs $G$ such that $h(\overline{G})\le 2$ and show that there exists a polynomial-time algorithm for finding a PDS of maximum size in such graphs. This result implies polynomial-time complexity on graphs with $n$ vertices of minimum degree $n-3$, i.e. graphs $G$ such that $Δ(\overline{G})\le 2$. For each result presented in this paper, we consider connected maxPDS and explain how to extend it when we require connectivity.
SLIM: a Scalable Light-weight Root Cause Analysis for Imbalanced Data in Microservice
Venue: arXiv cs.SE 2024
Year: 2024

The newly deployed service -- one kind of change service, could lead to a new type of minority fault. Existing state-of-the-art methods for fault localization rarely consider the imbalanced fault classification in change service. This paper proposes a novel method that utilizes decision rule sets to deal with highly imbalanced data by optimizing the F1 score subject to cardinality constraints. The proposed method greedily generates the rule with maximal marginal gain and uses an efficient minorize-maximization (MM) approach to select rules iteratively, maximizing a non-monotone submodular lower bound. Compared with existing fault localization algorithms, our algorithm can adapt to the imbalanced fault scenario of change service, and provide interpretable fault causes which are easy to understand and verify. Our method can also be deployed in the online training setting, with only about 15% training overhead compared to the current SOTA methods. Empirical studies showcase that our algorithm outperforms existing fault localization algorithms in both accuracy and model interpretability.
MegActor: Harness the Power of Raw Video for Vivid Portrait Animation
Venue: arXiv cs.CV 2024
Year: 2024

Despite raw driving videos contain richer information on facial expressions than intermediate representations such as landmarks in the field of portrait animation, they are seldom the subject of research. This is due to two challenges inherent in portrait animation driven with raw videos: 1) significant identity leakage; 2) Irrelevant background and facial details such as wrinkles degrade performance. To harnesses the power of the raw videos for vivid portrait animation, we proposed a pioneering conditional diffusion model named as MegActor. First, we introduced a synthetic data generation framework for creating videos with consistent motion and expressions but inconsistent IDs to mitigate the issue of ID leakage. Second, we segmented the foreground and background of the reference image and employed CLIP to encode the background details. This encoded information is then integrated into the network via a text embedding module, thereby ensuring the stability of the background. Finally, we further style transfer the appearance of the reference image to the driving video to eliminate the influence of facial details in the driving videos. Our final model was trained solely on public datasets, achieving results comparable to commercial models. We hope this will help the open-source community.The code is available at https://github.com/megvii-research/MegFaceAnimate.
CSDO: Enhancing Efficiency and Success in Large-Scale Multi-Vehicle Trajectory Planning
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents an efficient algorithm, naming Centralized Searching and Decentralized Optimization (CSDO), to find feasible solution for large-scale Multi-Vehicle Trajectory Planning (MVTP) problem. Due to the intractable growth of non-convex constraints with the number of agents, exploring various homotopy classes that imply different convex domains, is crucial for finding a feasible solution. However, existing methods struggle to explore various homotopy classes efficiently due to combining it with time-consuming precise trajectory solution finding. CSDO, addresses this limitation by separating them into different levels and integrating an efficient Multi-Agent Path Finding (MAPF) algorithm to search homotopy classes. It first searches for a coarse initial guess using a large search step, identifying a specific homotopy class. Subsequent decentralized Quadratic Programming (QP) refinement processes this guess, resolving minor collisions efficiently. Experimental results demonstrate that CSDO outperforms existing MVTP algorithms in large-scale, high-density scenarios, achieving up to 95% success rate in 50m $\times$ 50m random scenarios around one second. Source codes are released in https://github.com/YangSVM/CSDOTrajectoryPlanning.