Title: Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Respiratory sounds, multimodal learning, LLM-based generation, medical agents

Abstract: Deep learning-based respiratory auscultation provides critical acoustic cues for pulmonary assessment. However, conventional methods often compress signals into mel-spectrograms, discarding fine temporal dynamics, transient events, and semantic links to electronic health records (EHRs). The scarcity of high-quality multimodal datasets further constrains progress. To address these gaps, we present **_Resp-Agent_**, a multimodal agent system that tackles two key challenges: high-fidelity respiratory sound generation and accurate disease diagnosis, which can systematically identify and remedy model weaknesses in a closed loop. Specifically, we introduce a cascaded framework that transforms a text-only large language model (LLM) into a multimodal generator via modality injection. Conditioned on diagnostic text and style embeddings derived from real audio, the generator autoregressively produces discrete acoustic units (BEATs tokens), which are reconstructed into waveforms with a conditional flow-matching decoder. For classification, we design a Longformer-based model that fuses EHR text and respiratory features at the input level. 
The strategic global attention is proposed to enable the model to capture cross-modal long-range dependencies while detecting brief, low-energy events such as coughs and wheezes, yielding fine-grained disease predictions. To support this work, we construct **_Resp-229k_**, a multimodal corpus of 408 hours, 229k recordings, and 20 disease classes, each paired with expert-level natural language annotations distilled from structured EHRs. Extensive experiments show that our method consistently surpasses prior approaches, advancing robust and deployable respiratory intelligence. Data and code will be released upon acceptance.

TL;DR: We propose Resp-Agent, the first agent-based system for respiratory intelligence, integrating controllable sound synthesis and multimodal diagnosis to achieve state-of-the-art performance on ICBHI and our new large-scale Resp-229k benchmark.