OpenRaiser/Intern-Atlas huggingface dataset:
SDRS: Shape-Differentiable Robot Simulator
Venue: arXiv cs.RO 2024
Year: 2024

Robot simulators are indispensable tools across many fields, and recent research has significantly improved their functionality by incorporating additional gradient information. However, existing differentiable robot simulators suffer from non-differentiable singularities, when robots undergo substantial shape changes. To address this, we present the Shape-Differentiable Robot Simulator (SDRS), designed to be differentiable under significant robot shape changes. The core innovation of SDRS lies in its representation of robot shapes using a set of convex polyhedrons. This approach allows us to generalize smooth, penalty-based contact mechanics for interactions between any pair of convex polyhedrons. Using the separating hyperplane theorem, SDRS introduces a separating plane for each pair of contacting convex polyhedrons. This separating plane functions as a zero-mass auxiliary entity, with its state determined by the principle of least action. This setup ensures global differentiability, even as robot shapes undergo significant geometric and topological changes. To demonstrate the practical value of SDRS, we provide examples of robot co-design scenarios, where both robot shapes and control movements are optimized simultaneously.
A Rhetorical Relations-Based Framework for Tailored Multimedia Document Summarization
Venue: arXiv cs.MM 2024
Year: 2024

In the rapidly evolving landscape of digital content, the task of summarizing multimedia documents, which encompass textual, visual, and auditory elements, presents intricate challenges. These challenges include extracting pertinent information from diverse formats, maintaining the structural integrity and semantic coherence of the original content, and generating concise yet informative summaries. This paper introduces a novel framework for multimedia document summarization that capitalizes on the inherent structure of the document to craft coherent and succinct summaries. Central to this framework is the incorporation of a rhetorical structure for structural analysis, augmented by a graph-based representation to facilitate the extraction of pivotal information. Weighting algorithms are employed to assign significance values to document units, thereby enabling effective ranking and selection of relevant content. Furthermore, the framework is designed to accommodate user preferences and time constraints, ensuring the production of personalized and contextually relevant summaries. The summarization process is elaborately delineated, encompassing document specification, graph construction, unit weighting, and summary extraction, supported by illustrative examples and algorithmic elucidation. This proposed framework represents a significant advancement in automatic summarization, with broad potential applications across multimedia document processing, promising transformative impacts in the field.
Multi-Attribute Constraint Satisfaction via Language Model Rewriting
Venue: arXiv cs.AI 2024
Year: 2024

Obeying precise constraints on top of multiple external attributes is a common computational problem underlying seemingly different domains, from controlled text generation to protein engineering. Existing language model (LM) controllability methods for multi-attribute constraint satisfaction often rely on specialized architectures or gradient-based classifiers, limiting their flexibility to work with arbitrary black-box evaluators and pretrained models. Current general-purpose large language models, while capable, cannot achieve fine-grained multi-attribute control over external attributes. Thus, we create Multi-Attribute Constraint Satisfaction (MACS), a generalized method capable of finetuning language models on any sequential domain to satisfy user-specified constraints on multiple external real-value attributes. Our method trains LMs as editors by sampling diverse multi-attribute edit pairs from an initial set of paraphrased outputs. During inference, LM iteratively improves upon its previous solution to satisfy constraints for all attributes by leveraging our designed constraint satisfaction reward. We additionally experiment with reward-weighted behavior cloning to further improve the constraint satisfaction rate of LMs. To evaluate our approach, we present a new Fine-grained Constraint Satisfaction (FineCS) benchmark, featuring two challenging tasks: (1) Text Style Transfer, where the goal is to simultaneously modify the sentiment and complexity of reviews, and (2) Protein Design, focusing on modulating fluorescence and stability of Green Fluorescent Proteins (GFP). Our empirical results show that MACS achieves the highest threshold satisfaction in both FineCS tasks, outperforming strong domain-specific baselines. Our work opens new avenues for generalized and real-value multi-attribute control, with implications for diverse applications spanning NLP and bioinformatics.
NADER: Neural Architecture Design via Multi-Agent Collaboration
Venue: arXiv cs.CV 2024
Year: 2024

Designing effective neural architectures poses a significant challenge in deep learning. While Neural Architecture Search (NAS) automates the search for optimal architectures, existing methods are often constrained by predetermined search spaces and may miss critical neural architectures. In this paper, we introduce NADER (Neural Architecture Design via multi-agEnt collaboRation), a novel framework that formulates neural architecture design (NAD) as a LLM-based multi-agent collaboration problem. NADER employs a team of specialized agents to enhance a base architecture through iterative modification. Current LLM-based NAD methods typically operate independently, lacking the ability to learn from past experiences, which results in repeated mistakes and inefficient exploration. To address this issue, we propose the Reflector, which effectively learns from immediate feedback and long-term experiences. Additionally, unlike previous LLM-based methods that use code to represent neural architectures, we utilize a graph-based representation. This approach allows agents to focus on design aspects without being distracted by coding. We demonstrate the effectiveness of NADER in discovering high-performing architectures beyond predetermined search spaces through extensive experiments on benchmark tasks, showcasing its advantages over state-of-the-art methods. The codes will be released soon.
Multi-view Fake News Detection Model Based on Dynamic Hypergraph
Venue: arXiv cs.LG 2024
Year: 2024

With the rapid development of online social networks and the inadequacies in content moderation mechanisms, the detection of fake news has emerged as a pressing concern for the public. Various methods have been proposed for fake news detection, including text-based approaches as well as a series of graph-based approaches. However, the deceptive nature of fake news renders text-based approaches less effective. Propagation tree-based methods focus on the propagation process of individual news, capturing pairwise relationships but lacking the capability to capture high-order complex relationships. Large heterogeneous graph-based approaches necessitate the incorporation of substantial additional information beyond news text and user data, while hypergraph-based approaches rely on predefined hypergraph structures. To tackle these issues, we propose a novel dynamic hypergraph-based multi-view fake news detection model (DHy-MFND) that learns news embeddings across three distinct views: text-level, propagation tree-level, and hypergraph-level. By employing hypergraph structures to model complex high-order relationships among multiple news pieces and introducing dynamic hypergraph structure learning, we optimize predefined hypergraph structures while learning news embeddings. Additionally, we introduce contrastive learning to capture authenticity-relevant embeddings across different views. Extensive experiments on two benchmark datasets demonstrate the effectiveness of our proposed DHy-MFND compared with a broad range of competing baselines.
Anvendelse av kunstig intelligens (KI) i Norge i norsk offentlig sektor 2024
Venue: arXiv cs.CY 2024
Year: 2024

There are great expectations for the use of AI in Norway. On the other hand, it is reported that the adoption of AI in Norway is slower than expected in both the private and public sectors. Using responses from NOKIOS Technology Radar 2017-2021, IT in Practice surveys conducted by Ramboll in 2021-2024, as well as another national survey as part of a five-year cycle, this article looks at reported and planned use of AI with a focus on local (municipalities) and national government agencies. IT in practice is distributed to a large number of Norwegian public agencies, with a response rate of over 5o percent. The most recent data (2024) presented in this article is based on responses from 335 public organizations, with 237 municipalities, and 98 public organizations at the national or regional level. The survey confirms that the use of AI is still at an early stage, although expectations are high for future use. -- Det er store forventninger til bruk av KI i Norge. På den annen side rapporteres det at adopsjonen av KI i Norge går tregere enn forventet både i privat og offentlig sektor. Ved hjelp av svar fra NOKIOS teknologiradar 2017-2021, IT i Praksis undersøkelser utført av Rambøll i 2021-2024, samt en annen nasjonal undersøkelse som en del av en femårig syklus, ser vi i denne artikkelen på rapportert og planlagt bruk av KI med fokus på lokale (kommuner) og nasjonale offentlige etater. IT i praksis distribueres til en lang rekke norske offentlige virksomheter, med en svarprosent på over 50 prosent. De nyeste dataene (2024) presentert i denne artikkelen er basert på svar fra 335 offentlige organisasjoner, med 237 kommuner, og 98 offentlige organisasjoner på nasjonalt eller regionalt nivå. Undersøkelsen bekrefter at bruken av KI fortsatt er på et tidlig stadium, selv om forventningene er høye til fremtidig bruk.
Manga Generation via Layout-controllable Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

Generating comics through text is widely studied. However, there are few studies on generating multi-panel Manga (Japanese comics) solely based on plain text. Japanese manga contains multiple panels on a single page, with characteristics such as coherence in storytelling, reasonable and diverse page layouts, consistency in characters, and semantic correspondence between panel drawings and panel scripts. Therefore, generating manga poses a significant challenge. This paper presents the manga generation task and constructs the Manga109Story dataset for studying manga generation solely from plain text. Additionally, we propose MangaDiffusion to facilitate the intra-panel and inter-panel information interaction during the manga generation process. The results show that our method particularly ensures the number of panels, reasonable and diverse page layouts. Based on our approach, there is potential to converting a large amount of textual stories into more engaging manga readings, leading to significant application prospects.
From Interests to Insights: An LLM Approach to Course Recommendations Using Natural Language Queries
Venue: arXiv cs.IR 2024
Year: 2024

Most universities in the United States encourage their students to explore academic areas before declaring a major and to acquire academic breadth by satisfying a variety of requirements. Each term, students must choose among many thousands of offerings, spanning dozens of subject areas, a handful of courses to take. The curricular environment is also dynamic, and poor communication and search functions on campus can limit a student's ability to discover new courses of interest. To support both students and their advisers in such a setting, we explore a novel Large Language Model (LLM) course recommendation system that applies a Retrieval Augmented Generation (RAG) method to the corpus of course descriptions. The system first generates an 'ideal' course description based on the user's query. This description is converted into a search vector using embeddings, which is then used to find actual courses with similar content by comparing embedding similarities. We describe the method and assess the quality and fairness of some example prompts. Steps to deploy a pilot system on campus are discussed.
The Rendezvous Between Extreme Value Theory and Next-generation Networks
Venue: arXiv cs.IT 2024
Year: 2024

Promising technologies such as massive multiple-input and multiple-output, reconfigurable intelligent reflecting surfaces, non-terrestrial networks, millimetre wave communication, ultra-reliable lowlatency communication are envisioned as the enablers for next-generation (NG) networks. In contrast to conventional communication systems meeting specific average performance requirements, NG systems are expected to meet quality-of-service requirements in extreme scenarios as well. In this regard, extreme value theory (EVT) provides a powerful framework for the design of communication systems. In this paper, we provide a comprehensive survey of advances in communication that utilize EVT to characterize the extreme order statistics of interest. We first give an overview of the history of EVT and then elaborate on the fundamental theorems. Subsequently, we discuss different problems of interest in NG communication systems and how EVT can be utilized for their analysis. We finally point out the open challenges and future directions of EVT in NG communication systems.
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
Venue: arXiv cs.LG 2024
Year: 2024

Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, adhoc tuning of learning rates poses a challenge, leading to inefficiencies in practice. To address this issue, recent research has focused on developing "learning-rate-free" or "parameter-free" algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of AdaGrad++ and Adam++.
Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

This study proposes a knowledge distillation algorithm based on large language models and feature alignment, aiming to effectively transfer the knowledge of large pre-trained models into lightweight student models, thereby reducing computational costs while maintaining high model performance. Different from the traditional soft label distillation method, this method introduces a multi-layer feature alignment strategy to deeply align the intermediate features and attention mechanisms of the teacher model and the student model, maximally retaining the semantic expression ability and context modeling ability of the teacher model. In terms of method design, a multi-task loss function is constructed, including feature matching loss, attention alignment loss, and output distribution matching loss, to ensure multi-level information transfer through joint optimization. The experiments were comprehensively evaluated on the GLUE data set and various natural language processing tasks. The results show that the proposed model performs very close to the state-of-the-art GPT-4 model in terms of evaluation indicators such as perplexity, BLEU, ROUGE, and CER. At the same time, it far exceeds baseline models such as DeBERTa, XLNet, and GPT-3, showing significant performance improvements and computing efficiency advantages. Research results show that the feature alignment distillation strategy is an effective model compression method that can significantly reduce computational overhead and storage requirements while maintaining model capabilities. Future research can be further expanded in the directions of self-supervised learning, cross-modal feature alignment, and multi-task transfer learning to provide more flexible and efficient solutions for the deployment and optimization of deep learning models.
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Venue: arXiv cs.CV 2024
Year: 2024

Vision-centric autonomous driving systems require diverse data for robust training and evaluation, which can be augmented by manipulating object positions and appearances within existing scene captures. While recent advancements in diffusion models have shown promise in video editing, their application to object manipulation in driving scenarios remains challenging due to imprecise positional control and difficulties in preserving high-fidelity object appearances. To address these challenges in position and appearance control, we introduce DriveEditor, a diffusion-based framework for object editing in driving videos. DriveEditor offers a unified framework for comprehensive object editing operations, including repositioning, replacement, deletion, and insertion. These diverse manipulations are all achieved through a shared set of varying inputs, processed by identical position control and appearance maintenance modules. The position control module projects the given 3D bounding box while preserving depth information and hierarchically injects it into the diffusion process, enabling precise control over object position and orientation. The appearance maintenance module preserves consistent attributes with a single reference image by employing a three-tiered approach: low-level detail preservation, high-level semantic maintenance, and the integration of 3D priors from a novel view synthesis model. Extensive qualitative and quantitative evaluations on the nuScenes dataset demonstrate DriveEditor's exceptional fidelity and controllability in generating diverse driving scene edits, as well as its remarkable ability to facilitate downstream tasks. Project page: https://yvanliang.github.io/DriveEditor.
MNet-SAt: A Multiscale Network with Spatial-enhanced Attention for Segmentation of Polyps in Colonoscopy
Venue: arXiv cs.CV 2024
Year: 2024

Objective: To develop a novel deep learning framework for the automated segmentation of colonic polyps in colonoscopy images, overcoming the limitations of current approaches in preserving precise polyp boundaries, incorporating multi-scale features, and modeling spatial dependencies that accurately reflect the intricate and diverse morphology of polyps. Methods: To address these limitations, we propose a novel Multiscale Network with Spatial-enhanced Attention (MNet-SAt) for polyp segmentation in colonoscopy images. This framework incorporates four key modules: Edge-Guided Feature Enrichment (EGFE) preserves edge information for improved boundary quality; Multi-Scale Feature Aggregator (MSFA) extracts and aggregates multi-scale features across channel spatial dimensions, focusing on salient regions; Spatial-Enhanced Attention (SEAt) captures spatial-aware global dependencies within the multi-scale aggregated features, emphasizing the region of interest; and Channel-Enhanced Atrous Spatial Pyramid Pooling (CE-ASPP) resamples and recalibrates attentive features across scales. Results: We evaluated MNet-SAt on the Kvasir-SEG and CVC-ClinicDB datasets, achieving Dice Similarity Coefficients of 96.61% and 98.60%, respectively. Conclusion: Both quantitative (DSC) and qualitative assessments highlight MNet-SAt's superior performance and generalization capabilities compared to existing methods. Significance: MNet-SAt's high accuracy in polyp segmentation holds promise for improving clinical workflows in early polyp detection and more effective treatment, contributing to reduced colorectal cancer mortality rates.
Knowledge Graph-Based Multi-Agent Path Planning in Dynamic Environments using WAITR
Venue: arXiv cs.MA 2024
Year: 2024

This paper addresses the challenge of multi-agent path planning for efficient data collection in dynamic, uncertain environments, exemplified by autonomous underwater vehicles (AUVs) navigating the Gulf of Mexico. Traditional greedy algorithms, though computationally efficient, often fall short in long-term planning due to their short-sighted nature, missing crucial data collection opportunities and increasing exposure to hazards. To address these limitations, we introduce WAITR (Weighted Aggregate Inter-Temporal Reward), a novel path-planning framework that integrates a knowledge graph with pathlet-based planning, segmenting the environment into dynamic, speed-adjusted sub-regions (pathlets). This structure enables coordinated, adaptive planning, as agents can operate within time-bound regions while dynamically responding to environmental changes. WAITR's cumulative scoring mechanism balances immediate data collection with long-term optimization of Points of Interest (POIs), ensuring safer navigation and comprehensive data coverage. Experimental results show that WAITR substantially improves POI coverage and reduces exposure to hazards, achieving up to 27.1\% greater event coverage than traditional greedy methods.
An Overview of Machine Learning-Driven Resource Allocation in IoT Networks
Venue: arXiv cs.NI 2024
Year: 2024

In the wake of disruptive IoT technologies generating massive amounts of diverse data, Machine Learning (ML) will play a crucial role in bringing intelligence to Internet of Things (IoT) networks. This paper provides a comprehensive analysis of the current state of resource allocation within IoT networks, focusing specifically on two key categories: Low-Power IoT Networks and Mobile IoT Networks. We delve into the resource allocation strategies that are crucial for optimizing network performance and energy efficiency in these environments. Furthermore, the paper explores the transformative role of Machine Learning (ML), Deep Learning (DL), and Reinforcement Learning (RL) in enhancing IoT functionalities. We highlight a range of applications and use cases where these advanced technologies can significantly improve decision-making and optimization processes. In addition to the opportunities presented by ML, DL, and RL, we also address the potential challenges that organizations may face when implementing these technologies in IoT settings. These challenges include crucial accuracy, low flexibility and adaptability, and high computational cost, etc. Finally, the paper identifies promising avenues for future research, emphasizing the need for innovative solutions to overcome existing hurdles and improve the integration of ML, DL, and RL into IoT networks. By providing this holistic perspective, we aim to contribute to the ongoing discourse on resource allocation strategies and the application of intelligent technologies in the IoT landscape.