TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization
Authors: Hugo Malard
Michel Olvera
Stéphane Lathuilière
Slim Essid
Venue: Accepted by TMLR ()

Abstract: Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.

Published in Transactions on Machine Learning Research (02/2026)
TACO: Training-free Sound Prompted Segmentation via
Semantically Constrained Audio-visual CO-factorization
Hugo Malard
LTCI, Télécom Paris, Institut Polytechnique de Paris, France
Michel Olvera
LTCI, Télécom Paris, Institut Polytecknique de Paris, France
Stéphane Lathuilière
Inria at Université Grenoble Alpes, CNRS, LJK, Fronce
Slim Essid
NVIDIA, Pronce
Reviewed on OpenReview: Attps: // openreveew. net/ forum? id- Xt9sd
Abstract
Large-scale pre-trained audio and image models demonstrate an unprecedented degroe of generalization, making them suitable for wide range of applications. Here, tackle the specific task of sound-prompte segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine- tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we int roduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interprotable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods. The code of our experiments is open source".
Introduction
Audio-visual perception has attracted considerable interest due to its practical applications in varions fields, especially robotics, video understanding (Chen et al. 2023 Shahabaz & Sarkar 2024 and recently, audio- visual language modeling (Malard et al. 2024 Chowdhury et al. 2024). A particularly compelling task this area is sound-prompted segmentation (Hamillon et al. 2024) also known as audio-visual segmentation (Zhou al. 2022) which involves identifying image regions associated with sounds in the accompanying audio. Whereas several prior approaches frame this task supervised segmentation problem (Liu al.
2023b Gao et al. 2024 Liu et al.) 2023a). some explore unsupervised learning, relying solely inherent cross-modal features rather than ground-truth audiovisual masks. Most of them proceed to aligning audio and visual representations through contrastive learning techniques (Sun et al. 2023 Senocak et al. 2023
Park et al. 2023) Such techniques exploit paired andio-visual data to learn a shared embedding space,
Work conducted while Telecom Paris.
* *Code available
https://githun hwtp:///gggaaaaaaaa//oo---- Traini/n free-Sound- Sound-Prompted Prompted- git
Segmentation Prompt.d--Semmettttio
factorizatioo,

---

Published in Transactions on Machine Learning Research (02/2026)
Joint audio-visual deep feature decomposition
Image factors & activations
CLIP
Sem Co-NMF
Sounding activation
Sounding factor
Image
CLAP
X
Word bank
Audio
Audio factors & activations
Figure 1: Our method takes a representation of an image and its associated audio as input, decomposing them into a product of 'semantic' factors and (spatial or temporal) activations. This decomposition enables locating parts of the original input corresponding to the concept present in both the image and the audio.
where correspouding audio and visual features are brought closer together, while non-corresponding pairs are pushed apart. The main challenge lics in learning local features that can be used for segmenting audio-related concepts, as videos lypically provide global-level alignment (as opposed to local alignment, where each part of the image has a match with a defined part of the audio). Recently, a few works have made significant progress in this direction through the use of pre-trained models. Park et al. (2024) use a pro-trained CLIP encoder (Radford al.) 2021) and audio tokenizer within relatively complex framework that translates audio signals into tokens compatible with CL.IP's text encoder. This is combined with a grounding mechanism, enabling audio-driven segmentation of the image. Similarly, DenseAV (Hamilton et al. 2024) identifies image regions linked to sounds without supervision. Using DINO image backbone {Caron et al. 2021 and the
HUBERT audio transformer (Hsu el al. 2021 it employs multi-head feature aggregation for contrastive learning on vidco-audio pairs.
These approaches tackle sound-prompted segmentation through carefully designed architectures and dedicated training procedures, which often compromise the model's generalization capabilities and abilities in other tasks. Here, we propose radically different. strategy. Specifically, we ask: Can sound-prompted visual segmentation be achieved using only pre-trained audio and image models, in unsupervised manner, by analyzing Uheir frozen feature representations?
To answer this question, we introduce TACO: a Training-free Audio-visual CO-fuctriiation approach, which identifies audio and visual tokens that are co-activated in the input signal. Leveraging pre-trained CLIP and CLAP (Elizalde et al.] 2023) backbones for image and audio, respectively, we introduce a semantically constrained Non-negativv Matrix co-Factorization (Sem co-NMF) framework. This co-NMF formulation enables audio-visual corrospondence analysis without the need for training. Specifically, using sct of semantic anchor words projected into the CLIP and CLAP spaces, are able to match audio features with visual ones in an interpretable fashion. Through simple inference-time optimization, our framework establishes local correspondence between visual and audio signals, and further performs sound-prompted segmentation in a training-free manner, as illustrated in Figure [ Furthermore, the interpretable factors identified by our decomposition framework allow us to directly prompt pre-trained open-vocabulary segmenter, such as FC-CLIP YYu al.) 2023b), enhancing segmentation quality while remaining zero-shot.
Our unsupervised approach stands out for its training- free paradigm which preserves the generalization capabilities of the pretrained models, well as the interpretability offered by the NMF framework. Here, training-free mcans that the method does not rely on any datasct, opting instead for sample-spccific oplimization. This training-free setup holds great potential for multi-Lask scenarios, enabling a single model to support multiple downstream applications without additional fine-tuning. Additionally, leveraging the robustness of the frozen CLIP and CLAP models enables us achieve significantly superior performance compared to existing unsupervised training-basse methods on established benchmarks. To summarize, our contributions are:
2

---

Published in Transactions on Machine Learning Research (02/2026)
We introduce TACO, a training-free approach for sound-prompte segmentation based on pre-trained deep representations. Our framework leverages the inherent interpretability of matrix factorization, enabling clear, semantically grounded visualization of interactions between segmentation outputs and the concepts identified within the signal.
We revisit soft. co-NMF for audiovisual sound source localization by constraining the semantics of the decomposition and incorporating novel penalty function. This new decomposition (called Sem co- NMF) enables finding audio-visual correspondences across unaligned sound and image representation backbones.
Our Sem co-NMF framework is meant to extract both an audiovisual segmentation and encode audiovisual concepts that can be incorporated into a pre-trained open-vocabulary model (namely, FC-CLIP (Yu et al. 2023b)) ), prompting its decoder with the most activated factors identified by our decomposition system, allowing significant leap in accuracy.
We rigorously validate our approach through extensive experiments, providing both quantitative and qualitative analyses. We demonstrate the superior performance of our unsupervised (training-free) approach across four datasets and multiple variants of the sound-prompted segmentation task. The code will be released as open-source upon acceptance.
2 Related Work
Audio visual lcarning Audio-visual learning has been widely explored to model correspondences between visual content and Lemporally evolving audio signals. Early works on audio-visual event localization explicitly study temporal alignment between sound events and visual dynamics in unconstrained videos (Tian et a 2018a), while subsoquent approaches learn shared audio-visual representations through weak or self-supervision for cross-modal grounding and retrieval (Arandjelovic & Zisserman 2017 Mo & Morgado 2022a).. More recent. methods leverage large-scale pre-training to learn unified spatio-tempora multimodal representations enabling flexible transfer across audio-visual tasks (Gong et al. 20233 Girdhar et al., 2023). Rather than learning audio-visual aligmment through training, we leverage frozen pro-trained representations and training-five free co- co-factorization to directly extract local audio-visual correspondences.
Unsupervised Sound-Prompted Segmentation The sound- cound-poompted segmentation task aims to localize the object in an image that emits the sound present in the associated audio. While often referred to as audio-visual segmentation (Zhou et al., 2022) adopt the term used by HHamilton et al.
2024) throughout.
this paper, as it more clearly indicates that only the image is segmented, not the audio. Recent methods
primarily utilize cross-modal attention for sound-prompled segmentation (Tian et al. 20188b Senocak el al.]
2019: 2018). often combined with a contrastive loss that aligns audio and image global representations, hypothesizing that this process would implicitly foster the emergence of local alignment. Extonding this contrastive learning framework, numerous enhancements have been introduced. These include integrating challenging negatives from background areas (Chen et al. 2021). adopting iterative contrastive learning with pseudo-labels from earlier model epochs (Lin et al. 2023), maintaining transformation invariance and equivariance through geometric consistency (Liu et al. 2022), leveraging semantically similar hard positives (Senocak et al. 2022). and implementing false negative-aware contrastive learning via intra-modal similarities (Sun et al. 2023).
More recently, large pre-trained models have started to be used in order to benefit from their rich data representation. ACL-SSL (Park et al. 2024) relies on the CLIP image encoder as well as a segmentation model CLIPP-Sg (Lüddecke & Eekkr 2022 b, and learns an additional module that projects the audio the CLIP space. Relying on DINO and HUBERT (further fine-tuned), [aamltto al. (2024) trained a multi-head module that identifies the localization of the sound in the image features. Unlike these methods, our framework uses training-free free setup with pre-trained models and matrix decomposition, requiring no training data and preserving the model's generalization abilitics. Finally, few recent methods attempt to perform sound-prompted segmentation in a training- free manner by chain-prompting multiple single- modality models. In particular, Box- Proopp (Yu et al. 2023a combines CLAP (Elizalde et al.) 2023), GroNndiggDINO (Liu et al., 2024), and SAM (Kirillov et al. 2023) to perform the task. Extending this paradigm to very large models, OpenAVS (Chen et al.] (2255) integrates the reasoning capabilities of Large Language Models, such as GPT-4o-mini, with specialized frameworks including Pengi (Deshmukh et al.
3

---

Published in Transactions on Machine Learning Research (02/2026)
2023 GroundigDINO (Liu et al. 2024 and SAM2 (Ravi et al. 2024 While these approaches achieve good segmentation performance, they sulfer from prohibitive computational cost. In contrast, our method focuses on directly decomposing features from the audio and image encoders to discover audiovisual correspondences, limiting the computational overhead.
NMF and Deep Feature Factorization Non-negative matrix factorization (NMF) has been applied in diverse fields, including audio souree separation (Grass Erdogan 2011), document clustering (Xu al. 2003), and face recognition (Guillamet & Vitria 2002). Previous research has expanded NMF to multiple
layers (Cichocki et al. 2006), implemented NMF using neural networks (Dziugaite & Roy 2015), and utilized NMF approximations as inputs to neural networks (Vu et al.) 2016). More recently, it has been applied to decompose deep neural features, for concept discovery (Collins et al.
2018) interpretability (Parekh et al.,
2022 2024) or audio-visual source separation (Parekh et al. 2017]. 7) While it has already been applied to co- co-factorize audio and visual (handcrafted) features (Seichepine et al.
2014), to the best of our knowledge, this
is the first work applying to deep features for cound-rrompted segmentation, and the first co-factorize representations from different spaces while enforcing semantic matching.
3
TACO: Training-freet semantically constrained Audio-visual CO-factorization
Our methodology leverages the capacity of recent large pre-trained models to effectively localise patterns within their input signals YYu et al., 2023b). Localization here refers to identifying spatial regions in images and spectro-tempont segments in audio. From this observation, we hypothesize that separately trained audio and visual encoders contain all the necessary information for sound-prompted segmentation without requiring additional training. To achieve this, we propose method, TACO, which identifies concepts that co-activate across audio and image representations exploiting the Non-negative Matrix Factorization (NMF) framework (Lee & Seung 2000p. NMF provides a training-free approach that jointly decomposes each signal into an activation matrix and concept matrix. The activation matrix, in particular, can be interpreted as a segmentation matrix, highlighting regions of concept alignment.
Before detailing our approach, it is important to note that NMF is tool that is meant to interpret input matrices composed of only non-negative values. The non-ncgativity constraints are key to the interpretability offered by the NMF framework. However, deep representations often include negative coefficients. A simple solution is to clip these negative values to zero, ensuring matrix non non-negativity. While this may lead to some information loss, our preliminary experiments show that with backbones like CLIP (Radford et al.) 2021) and CLAP (Elizalde et al. 2023), representation power remains unaffected for our tasks (further details are in Sec. Section 3.1 provides the necessary background NMF, and section 3.2 introduces our framework for extracting audio-visus concepts using NMF, while section 3.3 describes complete solution for audio-source segmentation, which integrates our factorization approach into pre-trained open- opca-vocabulary image segmenter, enabling fine segmentation in training-fres free manner.
3.1 Background
[Lee & Seung 2000) involves decomposing non-negative matrix X E R. N.C into two non-negative matrices U E R x and V E R KxC such that X x UV. In typical machine learning applications of NMF Giilis 2014), N represents the number of observations and C donotes the number of (non-negative) features per observation, while K is the rank the factorization, controlling the number of factors or components into which the original matrix is decomposed. Given a distance D the matrix space, NMF can be formulated as the following constrained optimization problem:
NMF
Non-negative Matrix Factorization (NMF)
min D(X(UV).
(1)
U.V20
The matrix V is called factor matrix, and the U matrix corresponds to the activations those factors.
Co-NMF extends the NMF framework multiple data views analyzed in parallel. Assuming two data matrices X1 NNC and X2 R Rxx containing different modalities (such as audio and visual streams of video), NO-NMP exploits the mutual information shared between these modalities. It assumes that.
4

---

Published in Transactions on Machine Learning Research (02/2026)
the different modalities share a common activation matrix U. In particular, Yokoya al. 2011 Yoo & Choi (2011) decompase these matrices into a shared activation matrix U E R with view-specific factors Vi E + XxC and 1/2 1 Rxxx This formulation forces the activation matrix U to be consistent across both modalities, reflecting the intuition that events occurring in the audio also happen at the same time in the image stream. Although the factors are encoded in different spaces, they should remain the same. Formally, given distances D2 and D2, co-NMF solves the following problem:
min D ( + UVV)) V.Vi.V22
(2)
subject to non-negativity constraints on U, V1, and V2.
Note that, as there might be some small temporal shift between the audio and image events, enforcing the same U matrix for both decompositions can be overly restrictive. To mitigate this issue, soft non-negative Matrix Co-Faclorization (soft co-NMF) (Seichepine al., 2014) introduces two separate activation matrices and includes a penalty term to minimize their dissimilarity. They are obtained by solving the following optimization problem, with a penalty function P. and hyperparameter 3p > 0
(((X)) + D2((2)U2VV) + P(U,,,, V1, U2, V2,V,
min
(3)
subject to non-negativity constraints on U1, U2, Vi and V/2
3.2 Sem Co-NMF:
Revisiting soft NM-NMF
This section details our redefinition of soft co-NMF to semantically constrained co-NME (Sem Co-NMF) and its application in TACO.
Soft Co-NMF for token decomposition We assume access to an input video with corresponding audio signal. For simplicity, let us initially consider the central frame of the video; we will later exlend our approach to handle the entire video, incorporating the temporal dimension. Additionally, we assume the availability of trained audio and image backbones. Feeding the audio signal into the audio backbone produces matrix XA E R RNrxCa, where Nr is the number of tokens, and C' is the token dimension. Similarly, the image encoder maps an image to fcature representation X, E IR dimensions, and C₁ is the number of channels. Note that the spatial dimensions are flattened, resulting in a 2D matrix compatible with NMF inputs.
HW WXx,,, where H and W represent the spatial
In the original formulation, soft co-NMF factorizes visual and audio matrices containing temporal features (i.c. the first dimension corresponds to time). The penalty term is applied to the activations to ensure that concepts align temporally in both audio and image. However, in our case, the first dimension of the image feature corresponds to space (as aim for spatial decompasition for segmentation), hence the revision of the factorization model.
In our Sem Co-NMF scheme, we aim at factorizing XA into matrices UA E Nr and VA E R along with X, into U2 E IR. HW*K and V1 RRKK, where K denotes the predefined number of factors. Restricting the decomposition of the U, and UA matrices to the interval [0, 1] instead of R4 provides an interpretable, well-calibrated scale, where each concept is either activated (near one) or inactive (near zero) in each token. Thus, we re-parameterize the problem using the sigmoid function and optimize matrices Ü E N and Ú1 E "W such that UA- = (Ü,)) and U, = o(ū)) with U1 E [0, "/WK and UA E [0, N Specifically, the k--th column of U, and UA can be interpreted as the activation of the hth factor for each token. In this way, the Hadamard product between the original tokens and a column of Uy UA can be viewed soft masking, highlighting the portion of the original input that activates the tth factor. To jointly decompose the image and audio features, we cannot constrain the activations to be identical, images encode spatial information and audio captures spectro-tempore patterns. Additionally, constraining the factors VA and V, is challenging, as they reside in different spaces. To address this, we introduce a penalty function that ensures semantic proximity between the decompositions.
Penalty function Applying soft co-NMF to a specific problem requires careful definition of the penalty term P in Eq. equation 3. as it impacts both the quality of the factorization and its interpretability. To
5

---

Published in Transactions on Machine Learning Research (02/2026)
Co Co-factorization
U1
K
FC-CLIP
C₁
CLIP
II
Ci
HaW
X
K
Image
U A
K
h-
A
Ca
1111
CLAP
N1
X
Nr
Audio
K
Complete pipeline: both the audio and the image are encoded using their respective encoder and their representations ions are used to perform the co-NMF. FC-CLIP is prompted using the image factors (V,) and the segmentation corresponding to the sounding image factor (V/+*) is kept as the final segmentation.
Figure 2:
achieve a semantically consistent factorization of audio-visual observations, P must effectively capture the semantic similarity between the image and audio features. However, since the backbones (e.g., CLIP and CLAP) are not aligned, audio and image factors are encoded in two different subspaces, making 1ι Le norms unsuitable as penalty functions. Moreover, audio and image representations are not necessarily commensurable: they often differ in dimensionality, precluding direct use of standard distances. To address this, we propose projecting the audio and visual feature representations into a unified feature space to enable direct comparison. Specifically, we estimate aligned semantic descriptors Di= and DA* for the 1th column of Uj and UA, respectively. These descriptors are computed using J semantic anchors, {b/}/=1 and (bA))=1 which lie in the same spaces X, and XA- Fach anchor pair (b). b'A) represents shared semantic concept, enabling the estimation of aligned semantic descriptors. Specifically, to estimate aligned semantic anchors, we utilize CLIP (Radford et al.) 2021) and CLAP (Elizalde et al. 2023) as image and audio backbones. Since these encoders are aligned with text encoders, we can exploit. their respective text embeddings to align identical words across audio and image spaces. Thus, we define b/- = (t,) and b/ = =t((t,) where ε and E are the text encoders of CLIP and CLAP, respectively, and {1/)/=1 represents predefined word bank.
CLIP Space
CLIP
D l
Image
avgl
Uf)
Penalty
avgl
6 UA)
D At
CLAP
Audio visual semantic space
Audio
CLAP Space
Figure 3: As CLIP and CLAP encode audio and images in different spaces, our method employs semantic anchors project semantic components (representations soft-masked by U and U+) in an audio-visual semantic space where standard distances can be considered to compute the penalty function.
We now describe the process of estimating the semantic descriptors from the matrices UA and U, using the aligned semantic anchors. For simplicity, we detail only the estimation of semantic image descriptors; analogous steps apply the audio descriptors. First, we introduce semantic components C associated each column k of Uy. Since Uf represents the spatial activation of the tth factor (corresponding to V/),,

---

Published in Transactions on Machine Learning Research (02/2026)
it can be interpreted as soft segmentation mask for the imput X₁. Applying these soft-masks to X, and averaging the resulting representations extracts the concept associated with the hth factor from the image features. The hth image semantic component is defined as:
C₁ = avg(X, OUf),
(4)
where avg(.) donotes spatial average pooling (i.e. f: "Wwc, - Rt) and @ is the Hadamard product. We define the semantic image descriptor D by computing the cosine similarity between Cf and each semantic anchor:
D/ = (Ctos(C,,,) bf), "H, bf)1
55)
where cos(.) denotes cosine similarity. Intuitively, a descriptor quantifies the similarity of the semantic component with each semantic anchor (illustrated in Figure Note that Vf could have been used to play the role of semantic components instead of Cý. However, V7 is less resilient to variations of the semantic anchors. Indeed, during the NMF optimization, V can take any range of values, potentially degenerating to values very similar to those of br, which would be equivalent to performing a simple classification. In contrast, each C+ inherently constrained to be a weighted average of elements from the original representations, thus discouraging it from merely replicating values from by- We ablate this choice in Scction [ Audio semantic descriptors A* are computed similarly to their image counterparts. The penalty function is then defined as the cross-entropy between the image and audio semantic descriptors (alternative distances in Appendix B). However, enforcing similarity across all semantic descriptors may be detrimental, as certain semantic components may be visually present but inaudible, or vice versa. To address this, alignment is applied only to the closest audio and image semantic components. This choice is ablated in Section i Using the L2 norm for reconstruction, we obtain VA, Vi, UA, and U, by solving the following optimization problem:
also explored
min U.
I UAVA 2 + |x, V + Bp min CE(D)e,Da=]]
where CE denotes the cross-cntropy function. The pscudo-code of TACO is given in Appendix
Interpretation After optimization, the ttth row of Uy represents the segmentation of the hth factor in V/ (i.e. its hthh column) within the original image. The index & that realizes min; CE(D),, Da²)) corresponds to the dominant semantic component shared between the audio and image modalities, denoted as k* Consequently, row K-t of U,, referred as Ut", can be interpreted as the segmentation of the image region associated with the sound in the audio signal. Likewise, the segmentation of the spectrogram for this same semantic component is represented by Uk.
Temporal consistency To incorporate the temporal dimension in both the audio and visual inputs, if dealing with a video sequence, we introduce a temporal consistency regularization term in the decomposition process. Given a video, we extract T frames at a constant interval, and construct T audio-image pairs, denoted by [x and { by extracting the consecutive non-overlapped audio frames centered around an image frame position. For each frame pair, we perform a decomposition into matrices VA, V1, UAi U1, introducing regularization term R. to encourage consisteney between the primary shared factors, Vy." and VA," across consccutive frames. Formally, we add the following R, rogularization term to the optimization objective of Eq. equation D
T
R₂ = - Bremp Σ
cos(V Vk' Vk' + cossVk V A. V)
(7)
where Stemp 2 0 controls the temporal regularization.
3.3 Improved Segmentation with an Open Vocabulary Segmenter
Our soft co-NME approach generates a segmentation mask for objects producing sound in the audio input through the t- row of U,. However, the segmentation accuracy is limited, as the CLIP features in the
7

---

Published in Transactions on Machine Learning Research (02/2026)
visual imput Xy were trained only for image-level semantic alignment, not for precise segmentation. To enhance segmentation quality, we propose to integrate a pre-trained open-vocabulary segmentation model. To maintain compatibility with our NMF framework and keep the approach zero-shot, we select a model based on CLIP that proserves alignment with the text encoder without additional inc-tuning of the CLIP encoder. Here, we use the FC-CLIP model (Yu el al., 2023b), which segments images using CLIP's image encoder via text prompts encoded using the CLIP text encoder. A valuable feature of FC-CLIP is its shared CLIP embedding space for both image and text inputs, which allows it to be prompted by vectors from the image space. Leveraging this, we prompt the FC-CLIP decoder with factors identified by our NMF framework. More precisely, in our soft co-NMF decompasition, the rows of matrix V1 represent the factors in the image space. Therefore, we can use them to prompt FC-CLIP, treating each factor in V1 as segmentation class. Specifically, v/8 describes the shared "sounding" factor wetween audio and visual inputs. Since V resides in the CLIP space, it can also be compared to class name embeddings, allowing us to infer the segmented object's class names and achieve sound prompted image semantic segmentation. The complete pipcline is depicted in Figure [ Note that, incorporating FC-CLIP is computationally efficient, as the CLIP representation is computed only once before the soft Co-NMF decomposition, with the lightweight FC-CLIP decoder adding minimal overhead (there is 200M parameters for the CLIP encoder and ~20 for the FC-CLIP decoder).
4 Results and Discussion
Implementation details. To extract the modality-specifie representations we utilize the MSCLAP 2023 Elizalde et. al. 2023) (~80M parameters) audio backbone and the OpenCIIP (Cherti et al. 2023) ConxNeXt- Large CLIP trained on LAIÓN 2B (Schuhmann et al.) 2022] (~200M parameters) as they exhibit higher performance and generalization than the aligned audio-visual models audio and image only tasks (Elizalde et al. 2023). We fix the decomposition coefficients to B, = 125, and K = 8, and we ablate those choices in Appendix B. All the matrices UA,VA,Ui and V, are initialized randomly from Gaussians. We set the image and audio frame rate to 1 and use Bremp = 1 for the temporal consistency constraint. Since it targets single-source segmentation, we set Stemp = 0 for the multi-source task. Those hyperparameters were selected based on preliminary experiments conducted on the AVSBench validation set. The optimization problem is solved using gradient descent in 1800 steps with a learning rate of 0.25. Note that, this optimization induces minimal computational overhead as detailed in the Appendix 4.5). For semantic segmentation, we use the class names provided with the dataset as the word bank. For non-semantic segmentation, no predefined class set is needed, we therefore used general class list: the set of 527 AudioSet (Gemmeke et al.) 2017) tags. We ablate this choice in Appendix [ Since the U, and UA matrices are randomly initialized, we ran each experiment three times and report the average score as well as the variance.
AndioSet/Gemmkke
segmentation and two for sound-prompted scmantic sogmentation. For round-prompted segmentation, we use: AVS-Bench ZZhou et al. 2022), with binary segmentation maps for audia-visually related pixels, divided into single-source (S4, 5K samples) and multi-source (MS3, 400 samples) datasets. We also use ADE20k sound- sund-prompted (Hamilton et al.) 2024), ; which pairs ADE20k scgmentation masks and Images(Zhon et al.) 2019) with audio from VGGSound (Chen et al. 2020). For sound-prompted semantic segmentation, we created ADE Sound Prompted Semantic: a semanntc version of the ADF20k sound-prompted dataset, by assigning class labels to segmentation masks. We also conduct experiments on AVSS (Zhou et al. 2025), ; (~~1K samples) comprising single and multi-source sounding objects. Regarding metrics, we adopt standard segmentation metrics for evaluation: mean Intersection over Union (mIoU) and F-score for the semantic segmentation tasks, and mcan Average Precision (mAP) as well as mask Intersection over Union (mask-IoU) and F-score for the binary segmentation lasks. Each experiment involving NMF decompasition was conducted three times, with the results reported as the average and standard deviation of the scores. Baseline. In the absence of prior work on unsupervised sound-prompted semantic segmentation, and to have a fair competitor that relies on the same models, we consider a straightforward yet effective baseline approach: Audio-prompteed-FCCC.. (A-FC-CLIP). This method takes an audio input and its corresponding image, utilizing CLAP to determine the most likely class from the dataset's class pool. FC-CLIP is then prompted to segment this class.
Datasets and Metrics. We evaluate our method on five datasets: three for
To investigate the resilience of CLIP representations to clamping negative values, using the baseline setup, we prompted FC-CLIP with only the non-negative part of the CL.IP text encoding and evaluated its performance.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Results in Appendix A confirm the validity of clamping, supporting the use of NMF-based methods to leverage their interpretability.
In the following subsection, we compare our approach with other unsupervised methods tackling the task of sound-prompted segmentation. Most of the methods rely on fine-tuning (even if done unsupervisodly) which significantly modifies the original representations of pre-trained models. In contrast, our method is training-free and fully preserves the generalization ability of the models builds upon.
4.1
Segmentation
We now discuss experiments on (non-semantic) Sound-Prompled Segmentation. Table 1 compares our approach with other unsupervised methods on the AVS-Bench, in both single and multi-source setups. For completeness, we also include in the comparison Box-Prompt (Yu et al. 2023a), method which benefits from SAM, much richer segmentatiin model than FC-CLIP, making the comparison not fair. Importantly, this approach also assumes that the number of sources known a priori (one in S4 and two in MS3 information not available in real world scenario), while TACO does not rely on this additional information. In contrast, our method is able to extract audiovisual factors and encodes a variable number of sources within the sounding factor. TACO demonstrates strong performance relative to unsupervised training- based alternatives, outperforming the best competitor method by ~- and ~2 points of mloU in S4 and MS3 tasks. Our method also largely outperforms the A-FC-CLIP baseline that uses the same pre-trained models, underlining the effectiveness of our decomposition approach. Interestingly, using UF directly as segmentation map yields results competitive with state-of-tho-art methods, despite our model never being exposed to any audiovisual data (only out perfformde by MarginNCE (Park et al. 2023) in terms of F-score). TACO even outperforms Box- nox-Prompt, despite the latter's reliance on SAM and its access to prior information regarding the number of sources.
In the multi-source sctup, while baseline performance drops substantially due to its limitation in handling only single concept, our method shows robust performance. We hypothesize that robustness arises because Vi" encodes information about all the sounding objects rather than just one, within the sounding factor. The hypothesis is explored qualitatively in Section 4.3.1 Table 2 presents performance results on the recently proposed ADE Sound Prompted (ADE SP) dataset, along with variations of our approach on the same dataset to understand its gains. We compare existing methods with TACO variants and our decomposition without FC-CLIP. As in AVSBench, using only Uf" from our decomposition yields strong segmentation, outperforming prior audio visual models. Notably, unlike methods such as DenseAV (Hamilton et al. 2024). CAVMAF (Gong et al. 2023), DAVENet (Hsu et al.] 2019), and ImageBind (Girdhar et al.) 2023) which are trained on audio-image pains, our approach uses only frozen features. By decomposing these fcatures to identify matching factors, we show that they inherently capture the spatial information needed for localization. Predictably, the use of FC-CLIP segmenter substantially refines the segmentation and the scores. Designing a decomposition specifically tailored to be a compatible prompt for FC-CLIP is a core component of our method and a key direction for achieving significantly better results.
Remarkably, TACO matches the results of Box-Prompt without requiring SAM or prior knowledge of
$4
MS3
Method
Training-free
Segmenter
Sonroees
mask-
F-Score
mask-IoU t
F-Srore
SLAVC [Mo & Morgado
2022b
X
28.10
34.60
24.37
25.56
MargiuNCE Park al.
X
33.27
45.33
27.31
31.56
FNAC Senorak al.
23
X
27.15
31.40
21.98
22.50
Alignment Sun
1123
X
29.60
35.90
TACO w/o segmenter
X
29.68-t0.10
41.91=0.099
25.
25.88=0.91
30.72++0.95
Box-Prompt Yu
2023a
SAM
51.2
61.5
41.8
47.8
ACL-SSI [Park 222
CLIPSeg
59.76
69.03
41.08
46.67
A-FC-CLIP
FC-CLIP
51.52
57.25
33.67
35.87
TACO
FC-CLIP
64.04+ 0.25*
71.50=+0.20
43.15 +00.91
47.5+
Table 1: Quantitative results of sound-prompted segmentation on the AVS-Bench test sets. Best is in bold and second best underlined. The column #Sources indicates whether the number of sources is known. The indicates a valuuu lower than 0.01 when comparing to the best performing model.
9

---

Published in Transactions on Machine Learning Research (02/2026)
the source count. It even outperforms Box-Prompt in terms of mask-loU. This demonstrates the inherent robustness of our method when handling multiple sources. To gain further insight, we now examine variants of TACO. In the TACO w/o component variant, using V and V instead of the soft mask representations as semantic components during the optimization process (i.., using = and D -==v) leads to degraded performance. In TACO No-min, we replace the min operator in equation G with an average, penalizing all concept distances instead of just the closest. This results a noticeable drop in performance. The last variant, TACO w/o penalty, shows the results using 3, =,=0 (i.e., without penalization), which equivalent to performing two independent decompositions and choosing k arg ming CE(D), D) ;resulting in big performance drop.
4.2 Sound-Prompter Semantic Segmentation
Sound- Prompted Semantic Segmentation differs from Gound-Prompiln Segmentation seen in the previous section as it requires the estimation of class labels corresponding to the mask. In TACO, classifying the audio prompt v/" allows us to obtain the label of the segment. We run experiments on the AVSS and ADE20k SP Semantic datasets. As shown in Table 3 our method greatly outperforms the baseline on the two datasets as it can take advantage both the image and the audio decompositions (+8 and 15 points of mloU AVSS and ADE SP Semantic, respectively).
On the ADE SP Semantic, which contains a single class per image, the method performs well. However, on the AVSS datasct that contains samples with multiple classes, the performances drop. We attribute this to the class decoding process: since estimate the class label as the closest one to the value of V we consider the full segmentation as single class, Nonetheless, the class-agnostic segmentation masks are accurate as the decomposition encodes all the concepts in a single vector (shown in Appendix
4.3 Qualitative Analysis
This qualitative analysis is structured in two parts: we first assess TACO's performance across diverse scenarios, and then deepen our understanding through a set of ablation studies.
4.3.1 Qualitative Analysis of TACO
We begin this qualitative analysis by presenting in Figure input images along with their associated V reshaped as matrix, the segmentation from TACO, and an illustration of the associated audio. These examples highlight both the segmentation quality of TACO and the method's interprelability. Specifically, U3" reveals the tokens attended by the decomposition, from which V serves as the prompt for FC-CLIP. The first row shows challenging example from the S4 datasct, where the sound source is nearly invisible, yet. both TACO and V successfully segment it. While recent works (Juanola al. 2025 underline how andio-visual segmenter models fail to deal with negative audio (i.e. audio depicting an event not present in the image), the second row illustrates the robustness of our method that particular problem. As there is no common concept between the audio and image (given that the sound track of this example features
Segmenter
Training-rree
m-1OUt
mAPt
DAVENet Hsu et al.
2019
17.0
16.8
CAVMAE
Gong
21P3
20.5
26.0
ImageBina Bind
Girdbar et al
2123
18.3
18.1
DenseAV
[amiiton ct
2024
25.5
32.4
TACO w/o segmenter (U7
27.74+0.48
35.75++0.33
A-FC-CLIP
FC-CLIP
45.31
50.36
No-min
29.93+11.0 1.95
34.59++3.17
w w/o component
22.70+0.74
23.68-4-.008
TACO
PC-CLIP
w/ penalty
06.15
30.17:4:0.12
Proposed
51.57.10.86
60.01=1.38
Table 2: Quantitative results of sound-prompted segmentation the ADE SP dataset. The * indicates p-value lower than 0.01 when comparing to the best performing model.
10

---

Published in Transactions on Machine Learning Research (02/2026)
AVSS
ADE SP Semantic
Method
mloU
F-Score 1
mloU f
F-Seore f
A-FC-CLIP
11.95
13.56
20.91
24.95
TACO
20.50 #0.10"
23.02 =0.00"
37.70=1.611
41.39=11.2"
Table 3: Quantitative results of sound-prompled semantic segmentation on the AVSS and ADE SP Semantic datasets. The * indicates a p-value lower than 0.01 when comparing to the best performing model.
Audio visual imput
UH
TACO segmentation
Audio-visual Input
Ul
TACO segmentation
Similarities
0,022
0,110
up
-0, 0,0327
0,1291
0,0411
t +
0, 1236
0,0467
0,0883
Figure 4: Qualitative segmentation examples from ADE SP and S4 datasets: TACO performs well even in challenging cases.
Figure 5: Multiple source segmentation examples. As the sources change during the video, TACO's segmen- tation changes well.
music instead of the original audio recording of the scene), the estimated V contains only noise, therefore FC-CLIP does not find it in the image, hence the final segmentation empty, showing the resilience of TACO those negative audios. Finally, the last row illustrates the case of multiple potential sources in the image. Even though the image features Guitar, Djembe, and singers, the model correctly segments only the people as only singing voice is heard on this excerpt.
Figure 5 shows a segmentation example from the multi-source MS3 dataset. To gain insight into the model's ' behavior, we manually selected words describing sounding entities in the image and examined the similarity V/ with them. In the first row, the sound heard a baby babbling ( in the first column), while the image contains both the baby, woman, and sofa. As expected, the closest word to V is "Baby babbling" (see in the last column). In the second row, both the women and the baby are making sounds (6 + the first column). While one might expect these two sounds to be encoded in two different factors, they are both encoded in the same sounding factor V as its similarity is maximal with "A baby babbling and a woman talking'
in the last column), which is coherent with the segmentation. This observation reinforces our hypothesis made in section 4.1 the decomposition encodes not just a common concept between audio and image features, but all the sounding objects, explaining its performance in the multi-source task.
4.3.2 Qualitative Ablation studies
In this section, we evaluate the impact of two components of the method: the penalty loss and the temporal consistency loss. First, figure shows qualitative examples obtained by TACO on the $4 dataset, with and without the penalty loss. Not penalizing the decomposition to be performed jointly is essentially equivalent to rumning two independent NMFs (one on the image and one the audio features), and selecting the sounding factor the one that minimizes the distance between semantic descriptors (CE(D)s, DAv)), )). However, this strategy does not take into account the mutual dependence between the visual and auditory sourees, and therefore cannot ensure that. the selected components represent the same underlying object, and base the decomposition on is clearly present in both modalitics (sce the bus example in the figure), struggles scenes where either the image or the audio contains multiple salient elements, resulting in incorrect or fragmented localization. Second, figure 7 illustrates the impact of temporal consistency the S4 dataset. Even when the visual
sounding factors. While it may perform well when a single dominant concept
11

---

Published in Transactions on Machine Learning Research (02/2026)
Input
Input
TACO Penalty
TACO Consistency
TACO w/o Penalty
TACO w/o Consistency
Figure 6: Qualitative visualization of the impact of the penalty loss the segmentation in the S4 dataset.
Figure 7: Qualitative comparison of TACO segmenta- lion with and without temporal consistency the S4 dataset
content does not change significantly over time, the audio may vary, making segmentation more challenging at different timesteps. Temporal consistency helps such cases by encouraging the model maintain the same sounding factor across timestamps.
4.4 Word bank Analysis
Figure shows the mIoU of TACO on the S4 dataset when varying the proportion of AudioSet tags used in the word bank.
8
8
meou
Interestingly, performance remains stable until half of the tags are removed. It then drops when using only 20% of the Lags, and even more significantly when fewer lags are used. This behavior underlines that AudioSet tags diverse enough to express wide range of concepts, and that removing a large portion of them (half) does not significantly impact the model.
3.0
0.4
Figure mloU of TACO on S4, with vary- ing number of tags the word bank.
In order to vary the content of the word bank, and not only its size, we prompted GPT-5.2 to generate a large list of sounding objects (407), which we used as the word bank in TACO. Table compares the performance of TACO using the AudioSet word bank and the generated one. The difference in performance is negligible (less than 0.2 in mask- IoU and 0.35 in F-score). This not only demonstrates the robustness of our method to variations in the word bank, but also shows that constructing this list not a bottleneck in TACO.
Word Bank
mask-IoU
F-Score ?
AudioSet tags GPT5.2 generated
64.05 10.25
71.50 -0.20
63.83 ++0.29
71.18 +0.29
Table 4: Ablation on S4 dataset: choice of general word bank.
4.5 Computational overhead
The proposed NMF-based decomposition introduces additional computational costs compared to the bascline (consisting in prompting FC-CLIP with the predicted class from CLAP) as it requires optimization for each Figure g shows the performance (mAP) of TACO depending on the number of iterations in the optimization process as well as the associated relative computing time for a complete inference on the ADE SP test set. The performance plateaus after 1000 iterations, which takes 20% additional time compared to the baseline on single Nvidia A100 80G. Additionally, the amount of memory required to
12

---

Published in Transactions on Machine Learning Research (02/2026)
Proportional computing time
1.01
1.10
1.20
1.31
1.42
65
TACO
Baseline
60
mAP
55
50
45
40
0
250
500
750
1000 1250 1500 1750 2000
Iterations
Figure 9: Proportion of additional computing time induced by the decomposition. The decomposition allows improvement of ~ 12 points with only 20% of additional compute time compared to the bascline (A-FC-CLIP).
decompose an audiovisual imput is 425, MB (576 image tokens for an image with resolution 764 x 764, and audio tokens for 10-sccond audio clip).
5 Conclusion and limitations
We introduced TACO, a method for training-Iree sound-prompted visual segmentation, effectively incor- porating an open vocabulary image segmentation model, namely FC-CLIP, by prompting with factors extracted through a semantically motivated co-factorization of audio-visual deep embedding representations. Our method provides framework for unsupervised audio-visual segmentation, enabling the extraction and visualization of sounding concepts shared across audio rograms and image regions, making it inherently interpretable. TACO outperforms concurrent approaches, achieving state-of-the-art results. However, al- though the Sem CO-NMF incurs lower costs compared to pretrained models, it still introduces additional computational overhead during inference, which may pose challenges in resource-consee constrained settings. Finally, while TACO can support semantic segmentation, the decoding of multiple labels in the same image remains challenging and should be improved in further work.
6 Acknowledgement
This work was supported by the Audible project, funded by French BPI. This work was performed using AI resources from GENCIIDDISS (Grant 2025-AD01 101 488511)
13

---

Published in Transactions on Machine Learning Research (02/2026)
References
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sell-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In 2020 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2020.
Honglie Chen, Weidi Xie, Triantafyllos
Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2021.
Shengkai Chen, Yifang Yin, Jinming Cao, Shili Xiang, Zhenguang Liu, and Roger Zimmermann. Ope- navs: Training-free open-vocabulary audio visual segmentation with foundational models. arXiv preprint arXiv:2505.01445 2025.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio--ubbiitleeer omni-modality foundation model and dataset. Advonces in Neural Information Processing Systems, 36:72842 72866, 2023.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision, pp. 52-70 20-70. Springer, 2024.
Andrzej Cichocki, Rafal Zdunek, et al. Multilayer nonnegative matrix factorisation. ELECTRONICS LETTERS-IEE, 42(16), 2006.
Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. In Proceedings of the European Conference on Computer Vision (ECCV), 2018
Soham Deshmukh,, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems,
2023.
Gintare Karolina Dziugaite and Daniel M Roy. Neural network matrix factorization. arXiv preprint arXis:1511.064443 2015.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 12155 12163, 2024.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017.
Nicolas Gillis. The why and how of nonnegative matrix factorization. Regularization, optimization, kernels, and support vector machines, 12(257), 2014.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the TEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition, 2023.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuchne, and James R Glass. Contrastive audio-visual masked auloencoder. In The Eleventh International Conference on Learning Representations, 2023.
Emad M Grais and Hakan Erdogan. Single channel speech music separation using nonnegative matrix factorization and spectral masks. In 2011 17th International Conference on Digital Signal Processing (DSP), IEEE, 2011.
David Guillamet and Jordi Vitria. Non-negative matrix factorization for face recognition. In Catalonion Conference on Artificial Intelligence. Springer, 2002.
Mark Hamillon, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the' chirp' from the' chat': Selfsupervised visual grounding of sound and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2024.
Wei-Ning Hsu, David Harwath, and James Glass. Transfer learning from audio-visual grounding to speech recognition. Interspeech 2019, 2019.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29, 2021.
Xavier Juanola, Gloria Haro, and Magdalena Fuentes. A critical assessment of visual sound source localization models including negative audio. In ICASSP 0025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5. IEEE, 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustalson, Tele Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015 4026, 2023.
Daniel Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000.
Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Unsupervised sound localization via iterative contrastive learning. Computer Vision and Image Understanding, 2023.
Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu. Audio-visual segmentation by exploring cross-modal mutual semantics. In Proceedings of the 31sl ACM International Conference on Multimedia, pp. 7590-7598, 2023a.
Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya. Zhang. Audio-aware query-enhanced transformer for audio-visual segmentation. aixiv preprint ar Xi0: 13236, 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38 55. Springer, 2024.
Timo Lüddeeke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEFE/CVF conference on computer vision and paltern recognition, 2022.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Hugo Malard, Michel Olvera, Stéphane Lathuilière, and Slim Essid. An eye for an car: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching. In The Thirly- eighth Annual Conference on Neural Information Processing Systems, 2024.
Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In European Conference Computer Vision, pp. 218-234 Springer, 2022a.
Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Newral Information Processing Systems, 2022b.
Jayneel Parekh, Sanjeel Parekh, Pavlo Mozharovskyi, Florence d' d'Alché Buc, and Gaël Richard. Listen to interpret: Post-hoc interpretability for audio networks with nmf. Advances in Neural Information Processing Systems, 35, 2022.
Jayneel Parckh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept- basod explainability framework for large multimodal models. Aduances in Newral Informalion Processing Systems, 37, 2024.
Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc QK Duong, Patrick Pérez, and Gaël Richard. Guiding audio source separation by video object information. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2017.
Sooyoung Park, Arda Senocak, and Joon Son Chung, Marginnce: Robust sound localization with a negative margin. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Sooyoung Park, Arda Senocak, and Joon Son Chung. Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprini N44::448800714 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text. models. Advances in Neural Information Processing Systems, 35, 2022.
Nicolas Seichepine, Slim Essid, Cédric Févotte, and Olivier Cappé. Soft nonnegative matrix co-factorization. IEFE Transactions on Signal Processing, 62()2), 2014.
Arda Senocak, Tac-Hyun Oh, Junsik Kim, Ming- Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and So Kweon. Learning to localize sound sources in visual scenes: Analysis and applications. IEEE transactions on pattern analysis and machine intelligence, (5), 2019.
Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
Arda Senocak, Hyconggon Ryu, Junsik Kim, Tac-Hyun Oh, Hanspeter Pfister, and Joon Son Ching. Sound source localization is all about cross-modal alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Ahmed Shahabaz and Sudeep Sarkar. Increasing importance of joint analysis of audio and video in computer vision: A survey. IEFE Access, 2024.
Weixuan Sun, Jiayi Zhang, Jianyuan Wang. Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning misiol source localization via false negative aware contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained vidcos. In Proceedings of the European conference on computer vision (ECCV), pp. 247-263, 2018a.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), 2018b.
Thanh T Vu, Benjamin Bigot, and Eng Siong Chng. Combining non-negative matrix lactorization and deep neural networks for speech enhancement and automatic speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016.
Wei Xu, Xin Liu, and Yihong Gong. Document clustering based non-negative matrix factorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrievol, 2003.
Naoto Yokoya, Takehisa Yairi, and Akira Iwasaki. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 2011.
Jiho Yoo and Seungjin Choi. Matrix co-factorization on compressed sensing. In Proceedings of the Twenty- Second international joint conference on Artificial Intelligence- Volume Volume Two, 2011.
Jiarui Yu, Haoran Li, Yanbin Hao, Jinmeng Wu, Tong Xu, Shuo Wang, and Xiangnan He. How can contrastive pre-training benefit audio-visual segmentation? a study from supervised and zero-shot perspectives. BMVC, pp. 367-374, 2023a.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances in Newral Information Processing Systems, 36, 2023b.
Bolci Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Compuler Vision, 127, 2019.
Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visuse segmentation. In Earopean Conference Computer Vision. Springer, 2022.
Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, al. Andio-visual segmentation with semantics. International Journal of Compuler Vision, 2025.
17

---

Published in Transactions on Machine Learning Research (02/2026)
The appendix is organized as follows: the first part presents analysis of the clamping resilience well as additional ablations and hyperparameter variations. The second part presents a detailed pseudo-code of the method, followed by an analysis of the encoding costs induced by the pretrained models. The last parts discuss the metrics used and show additional qualitative visualizations and analysis of segmentation from TACO.
A Analysis of clamping resilience
To assess the resilience of CLIP representations to clamping negative values, we evaluate FC-CLIP using only the nonnegative portion (obtained via ReLU) of the CLIP text encoding, variant referred to as A+-FC-CLIP Table 5 shows the performance degradation across all datasets. Except for the ADE SP dataset (~3.5 points drop), the decrease never exceeds points. This experiment validates the use of clamping, enabling the application of NMF-based methods so as to benefit from their interpretability.
ADE SP
S4
MS3
AVSS
ADE SP S.
A-FC-CLIP A+-FC-CLIP
45.31
51.52
33.67
11.95
24.95
41.89
51.63
31.67
11.82
23.73
Table 5: Impact of clamping on the representation before feeding FC-CLIP. We report mloU on five datasets. ADE SP S. corresponds the ADE SP Semantic segmentation dataset. mask-IoU reported for S4 and S3 and mloU for all the other datasots.
Analysis of design choices
Hyperparameter analysis. sshows the performance of our method the S4 dataset using different numbers of components in the decomposition (values of K). Results are stable across multiple values, showing the resilience of the method with respect to the number of factors.
Table 7 shows the performance of TACO (on the S4 dataset) with respect to the hyperparameter 3p. The performance remaining stable across multiple values of the hyperparameter indicates the robustness of the method. TACO-KL shows the performance of TACO when using the KL-divergence instcad of the cross entropy to measure dissimilarity between the semantic descriptors. Using KI-divergence instead of cross entropy affects significantly the performance of TACO.
Additionally, ablate the temporal consistency component by setting Btemp 0. This significantly degrades performance, highlighting the effectiveness of that regularization. Table S presents the mask- scores on
S4
S4
Method
B
Atemp
mask-ToUl
F-Score
K
mask-Iol IoU t
F-Score f
100
63.45
70.57
TACO
125
64.04
71.50
61.74
68.87
150
63.90
71.52
8
64.04
71.50
TACO
125
533.47
39.93
10
63.92
71.54
TACO-KL.
125
58.74
65.69
12
63.78
71.28
TACO Words/2
125
62.91
70.16
TACO Words/1
125
55.30
62.10
Table 6: Variation of the number of factors
Table 7: Variation of the design choices
the three used sound-prompted segmentation datasets using either a general class list or one specific to the dataset (i.e the list of all the classes of the dataset). Using a specific class list overall improves both TACO results (+1 point on $4, 3.5 points on ADE SP and 2 points in MS3). Interestingly, the segmentation using U.! does not seems be improved lot by the specific word bank (+1 point on S4 and MS3, and -5 points ADE SP).
18

---

Published in Transactions on Machine Learning Research (02/2026)
Formulation choices. Because multiple-soumd-sourre (MS3) scenarios more complex and challenging, they provide complementary test of the contribution of each component of our method. As such, Table g reports the results of TACO with and without our penalty function. We obtained results in line with what we observe in the single-sound-source case (see Table 2 without penalty, the performance is almost halved, underlining the effectiveness of our proposed mechanism. encodes all sounding sources, even in the multi-source sctting, we modified the formulation of the penalty loss and penalized the bottom-2 values instead of taking only the minimum, i.e,, we forced two factors to be shared between the image and the audio. Line bottom-2 of Table g shows the performance of TACO using both the bottom-2 formulations of the penalty function. Using the original min formulation yields better results, indicating that the decomposition more effectively encodes all sounding factors into a single concept. than when two are forced.
order verily whether the sounding factor
Method
mask-ToU f
F-Score f
TACO
43.15 =0.91
47.5 +0.95
TACO w/o penalty
23.52 10.54
24.23 10.65
TACO bottom-2
41.12 +0.90
45.17 +1.14
Table 9: Ablation MS3 dataset: impact of penalty function.
C TACO pseudo-code
The following pseudo-code details the exact computations performed by TACO. Note that this is the pseudo code for the case of a static frame, in the case of multiple frames (like video) Eq equation 7 should be added to C.
The visualization of the dccomposition given in Figure 10 UA and VA and Un and V,. The kth column of UA and U, that minimizes the cross entropy between the semantic descriptors is used to force the unify the 2 decompositions.
The input matrices are decomposed in 2 matrices:
D Encoder MACS analysis
The table 10 shows the cost (measured in MACs) of the image and audio encoder used in our method and concurrent ones. While our image encoder is larger, the efficiency of the CLAP audio encoder (HTSAT- Swin Transformer) significantly reduccs the overall computation. As a result, our total encoding cost is lower than that of Hamilton el al. (2024) and Park el al. (2024).
E Metrics discussion
Throughout this work, we reported mask-loU as one of the primary metrics for sound- segmentation the AVSBench dataset, instead of mean-IoU. While mean-IoU is commonly used in segmentation tasks, it may not be the most appropriate metric in the context of binary segmentation. This is because mean- IoU averages the IoU of both the foreground and the background (foU(foreground) + (oU(background)) potentially overemphasizing background accuracy, which is often less critical in binary tasks. In contrast,
Word Bank
S4
MS3
ADE SP
General
27.78+0.06
25.8810.91
32.42+0.74
U.
Specific
28.76:40.06
26.5740.17
27.74410.8
Specific
64.73::0.16
45.1540.80
54.05+1140 51.5740.86
TACO
General
64.0
43.1540.91
Table Comparison of the two word banks on sound-prompted segmentation datasets. mask-loU is reported for S4 and MS3, while mloU is reported for ADE SP.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm TACO algorithm
Input: Audio and image features (XA; XT), number of iterations Titts semantic anchors (1/)/11 1allz1 learning rate 7,, parameter Sp. FCCLIP model s(.) Initialize: UA U, Va, V, NO,
3: for iteration = 1 to Nite do
4: Compute semantic components: C = ( ④ Uf,, C₁ = = U cos(CC b2))
cos(CC),b) bb)
5:
Compute semantic descriptors: Djc
AD
cos(C), b/)/
cos(C22, b/2)/
Compute total cost:
L= |X
UAVA 11 + |X - U1Vill
7:
+ 3p ming CE(D,t, D
8:
Update paramelers:
9:
UA t UA- A
10:
U, Ur 1UT
DC
11:
VA + VA 1] AVA
BC
12:
V1 t V1 1881
13: end for
14: k*
arg ming CE(D),,DAt AA)
15: segmentations t =Vi, X,)
16: Return: segmentation [k-]
X1
U1
T
W
Toten
8
O
)
HW
HW
reshape
CE(D(X,
) D
X A
U
reshape
NT
VA
O
x
CA
Nr.
NT
Ca
Figure 10: Illustration of the decomposition
20

---

Published in Transactions on Machine Learning Research (02/2026)
Method
Image Encoder
Audio Encoder
Total
Hamilton el al. (2024)
1.69 x 10th
6.72 x 10
8.41 x 10tH
Park et al. [2024
1.69 x 10th
4.48 x 10th
6.17 x 10SH
Ours
3.44 X 10
6.31 x 10
4.07 10
Table 10: Comparison of model MACs for image and audio encoders.
laput image
TACO prediction
Ground Truth segmentation
Figure 11: Semantic segmentation examples from AVSS.
mask-IoU focuses solely on the IoU of the foreground, making it a more relevant and precise measure of segmentation quality in scenarios where the foreground object is the primary area of interest. However, we noticod that several works report mcan-IoU in their papers, while in practice they mcasure mask-IoU, This discrepancy appears to stem from unintentionally replicating error in the original repository of dataset. AVSBench, which leads to potentially unfair comparisons between works. This discrepancy poses challenges when comparing results, particularly for works without available code the exact metric used cannot always be confirmed. Consequently, we were unable to compare our results with works for which the code was not available (like Bhosale et al.), as there was no assurance regarding the metrics used.
F Semantic segmentation analysis
Figure presents qualitalive evidence supporting the hypothesis mentioned in Section 4.2 the semantic segmentation performance is hindered by the class inference process. Specifically, in a scenario with two sound sources, while both sources may be well segmented, each being encoded within the sounding concept through decomposition, the use of single factor infer the class leads to both sources being assigned the same class. This significantly penalizes the segmentation score.
21

---

Published in Transactions on Machine Learning Research (02/2026)
G Qualitative results of single and multiple sources, sound-prompted segmentation
Figure 12 presents randomly sampled segmentation examples from the ADE Sound prompted datasct, as well as their associated activation matrices U which enlightens the regions of the image that is used to extract the prompt to the segmenter model.
Figure 13 presents the equivalent analysis applied to the multi-sources dataset MS3.
Typical failure cases
Figure 14 shows typical failure cases of TACO. The first row depicts a case where the decomposition correctly identifies that the concept to segment is a piano (as shown by U/ ) but the segmenter model fails clearly segment it, resulting in inaccurate segmentation. The second row showcases an example where the decomposition fails to extract clear concept, as it is unable to encode both audio concepts effectively, yielding partial and inaccurate segmentation. Finally, the third row highlights special failure case where the decomposition successfully encodes one concept but neglects the second. We hypothesize that this issue arises from limitations in the performance of the audio model (CLAP audio encoder).
22

---

Published in Transactions on Machine Learning Research (02/2026)
Imput image
Ut
TACO prediction
Ground Truth
Park
Figure 12: Sound-promptes segmentation examples from ADE SP.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Imput image
U1
TACO prediction
Ground Truth
Bsknimals.com
Bstonasttra Era
Figure 13: Sound promptod-segmmentttion examples from MS3.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Inputt image
TACO prediction
Ground Truth
Figure 14: Examples of typical failure cases on MS3.
25


Reviews:
[{"review_id": "yTVil8INVY", "reviewer": "Reviewer_MjHF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TACO, a training-free approach for audio-visual segmentation that operates entirely on frozen representations from pretrained models (CLIP for images and CLAP for audio). The method performs a semantically constrained audio–visual co-factorization at inference time to discover latent factors that are jointly activated by both modalities. To bridge the mismatch between CLIP and CLAP embedding spaces, TACO uses a semantic anchor word bank to align audio and visual factors via a dedicated penalty. The most strongly aligned “sounding factor” is then used to produce a segmentation, either directly from the factor activations or by prompting an open-vocabulary segmenter (FC-CLIP) with the discovered image factor. Experiments across multiple benchmarks show that TACO achieves promising performance.\n\nThe underlying problem formulation is not new, and there exists prior zero-shot audio-visual segmentation approach that leverages CLAP together with pretrained grounding and segmentation models (e.g., Grounding-DINO and SAM). In this context, the primary contribution of this work lies in its use of a soft co-NMF framework, with textual semantic anchors serving as a bridge, to associate audio and visual representations in a training-free manner for audio-visual segmentation.", "claims_and_evidence_yn": "No", "claims_explanation": "- The central motivation of this paper is to answer the question: Can sound-prompted visual segmentation be achieved using only pretrained audio and image models, in an unsupervised manner? The authors argue that prior work largely relies on carefully designed architectures and task-specific training procedures. However, to the best of my knowledge, there already exist works that address this problem without additional training. For example, Yu et al. [1] explored a zero-shot formulation of audio-visual segmentation as early as 2023, using CLAP to capture semantic concepts from audio and then prompting a segmentation model. The insight is conceptually close to the motivation of this paper. More recently, Chen et al. [2] proposed a fully training-free, open-vocabulary audio-visual segmentation framework using foundation models. These prior works are not discussed in the paper, making the claim of being “the first training-free approach for sound-prompted segmentation” inaccurate.\n\n- Regarding the proposed approach, it is unclear how the method handles multiple instances of the same object category. For example, if multiple guitars appear in the video but only one is producing sound, the discovered image factor—used as a global prompt—may lead the model to segment all guitars, including non-sounding instances. This raises concerns about correct instance-level localization.\n\n- The method’s performance and efficiency appear to be sensitive to the choice and size of the vocabulary used as semantic anchors. A large vocabulary may significantly increase optimization time during inference, potentially limiting scalability. A small one will definitely drop the model performance.\n\n- The approach primarily selects a single best-matching factor to represent the sounding concept.  The authors provided results on the MS3 set, which contains multi-source videos.  However, it remains unclear how the method explicitly handles multiple simultaneous sound sources, beyond implicitly encoding them within a single factor, which may limit its applicability in more complex auditory scenes. \n\n[1] Yu, Jiarui, Haoran Li, Yanbin Hao, Jinmeng Wu, Tong Xu, Shuo Wang, and Xiangnan He. \"How Can Contrastive Pre-training Benefit Audio-Visual Segmentation? A Study from Supervised and Zero-shot Perspectives.\" In BMVC, pp. 367-374. 2023.\n\n[2]Chen, Shengkai, Yifang Yin, Jinming Cao, Shili Xiang, Zhenguang Liu, and Roger Zimmermann. \"OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models.\" arXiv preprint arXiv:2505.01448 (2025).", "audience_yn": "Yes", "audience_explanation": "Yes. The paper would interest part of the TMLR audience, particularly researchers in multimodal learning,  training-free methods, and foundation model reuse. It demonstrates that frozen pretrained audio–visual representations can be aligned via semantic factorization to achieve strong segmentation performance offers a novel and relevant perspective for the community.", "requested_changes": "I hope the authors can adequately address the concerns raised above and make the necessary revisions to strengthen the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hlgMk2cKkE", "reviewer": "Reviewer_wRnq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes TACO, a training-free approach to sound-prompted image segmentation that operates entirely on frozen pretrained audio and visual encoders. The core idea is to jointly decompose audio and visual token representations using a semantically constrained audio–visual co-NMF objective, which reveals shared and interpretable latent factors across modalities. A semantic penalty based on text-anchored descriptors aligns the decompositions, enabling the method to identify the dominant “sounding” factor. This factor is then used both as a coarse segmentation mask and as a prompt to an open-vocabulary segmentation model for refined and semantic segmentation.\n\nKey strengths include the genuinely training-free formulation, the interpretability of the factorization-based approach, and strong empirical performance across several sound-prompted segmentation benchmarks. Notable weaknesses include inference-time optimization cost and limitations in multi-class semantic decoding, which are acknowledged but not fully resolved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides clear and generally convincing evidence in support of its claims. The method is evaluated across multiple benchmarks covering both binary and semantic sound-prompted segmentation, and results are consistently competitive or superior to prior unsupervised or training-based baselines. The experimental section includes meaningful ablations on semantic constraints, number of factors, word-bank size, and descriptor design, which help isolate the contributions of different components. Qualitative visualizations further support the interpretability claims by showing that the selected factors correspond well to sounding objects.\n\nWhile inference-time optimization introduces additional complexity, runtime behavior is analyzed and performance saturation is empirically demonstrated. Overall, the empirical evidence is sufficient to support the central claims of the paper.", "audience_yn": "Yes", "audience_explanation": "The paper will likely be of interest to researchers working on multimodal learning, audio–visual understanding, unsupervised representation learning, and methods that leverage frozen foundation models. The training-free perspective and factorization-based formulation provide a useful alternative to data-hungry training pipelines, and the approach may inspire related work in other cross-modal or weakly supervised settings. As such, the findings are relevant to a meaningful subset of TMLR’s readership.", "requested_changes": "1. **Clarify inference-time optimization behavior (non-critical).**\n\n   Provide additional discussion or statistics on convergence stability, sensitivity to initialization, or adaptive stopping criteria, given the reliance on per-sample optimization.\n\n2. **Expose semantic-anchor sensitivity more clearly (non-critical).**\n\n   The appendix shows performance degradation when the word bank is aggressively reduced. Briefly summarizing this sensitivity in the main paper would clarify underlying assumptions.\n\n3. **Address multi-class semantic decoding limitations (non-critical).**\n\n   Consider a simple extension or baseline that decodes multiple factors in multi-source scenes to assess whether the current limitation is fundamental or largely a design choice.\n\n4. **Sharpen the “training-free” framing (editorial).**\n\n   Explicitly distinguish between the absence of dataset training and the use of inference-time optimization to avoid potential misinterpretation.", "broader_impact_concerns": "I do not see significant broader impact or ethical concerns specific to this work. The method relies on standard pretrained models and is intended for general computer vision and audio–visual understanding tasks. As with related segmentation techniques, potential misuse would fall under broader concerns already associated with visual recognition systems, rather than arising uniquely from this contribution.", "additional_comments": "Overall, this is a solid and well-executed paper with a clear conceptual contribution and strong empirical support. While there are practical limitations related to inference-time optimization and semantic decoding, these do not undermine the core idea. With minor clarification and framing improvements, the paper would make a valuable contribution to the literature on training-free and unsupervised multimodal learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZDDgsTKiyM", "reviewer": "Reviewer_UpGK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\nThis paper introduces TACO, a training-free framework for sound-prompted segmentation. The method addresses the challenge of identifying image regions corresponding to sounds using only frozen, pre-trained audio (CLAP) and image (CLIP) models. The core technical contribution is a Semantically Constrained Non-negative Matrix Co-Factorization (Sem co-NMF), which identifies shared interpretable concepts across modalities by projecting deep features into a unified semantic space using a word bank as anchors. These identified factors are then used to prompt an open-vocabulary segmenter (FC-CLIP) for high-quality mask generation. Experimental results across five datasets demonstrate that TACO achieves state-of-the-art performance in unsupervised settings, significantly outperforming previous training-based unsupervised methods. Overall, I have a positive view of this paper.\n\n\n## Strengths\n- The paper is well-structured and clearly written, with helpful visualizations and mathematical derivations.\n- Most research work on sound-prompted segmentation focus on the supervised setting, which the pixel-level masks are required. This paper explores a training-free approach, which is valuable and interesting.\n\n## Weaknesses\n-  The method relies heavily on a \"word bank\" to act as semantic anchors. While the authors ablate this using AudioSet tags, there is a risk that the performance is sensitive to the vocabulary size and relevance. If a sound occurs that is not well-represented in the word bank, the \"semantic descriptors\" might become noisy, leading to a breakdown in the co-factorization.\n- class inference in Semantic Segmentation. As admitted by the authors, the semantic segmentation performance on AVSS is limited because the model collapses multiple objects into a single class label during decoding. If an image has two distinct sounding objects (e.g., a dog and a person), TACO segments both but likely assigns them both the same label, which is a significant architectural limitation for true semantic segmentation.\n- While the authors state the optimization overhead is minimal, requiring 1000-1800 iterations of gradient descent per sample at inference time is a disadvantage compared to a single-pass feed-forward network.\n- Failure in multi-source scenarios: qualitative analysis (Figure 13) shows that the model still struggles to encode multiple distinct audio concepts effectively, often neglecting one or providing partial results. This suggests the \"min\" operator in the penalty function might be too aggressive, focusing only on the single strongest shared concept.\n- The related work in current manuscript primarily focuses on the sound-prompted segmentation, which focuses on spatial-level audiovisual understanding. Introducing more relevant contexts on a broader audio-visual learning, such as temporal learning, would make the manuscript more comprehensive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The proposed method is clearly elaborated and it makes sound. The authors verify the method on standard datasets, which demonstrate the its effectiveness and superiority.", "audience_yn": "Yes", "audience_explanation": "This paper focuses on the sound-prompted segmentation, which may be attracted to audiovisual learning and segmentation community.", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "Not required. But it would be better to add a paragraph to claim the Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jF46oDU8oN", "reviewer": "Reviewer_MPy6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes TACO, a training-free framework for sound-prompted image segmentation, where the goal is to segment objects in a static image that correspond to a given audio signal. The method leverages frozen, large-scale pre-trained models—CLIP for vision and CLAP for audio—and introduces a semantically constrained co-nonnegative matrix factorization (co-NMF) to align audio and visual representations without any additional model training.\nThe key idea is to jointly decompose audio and image features into a shared set of low-rank semantic factors, which are further regularized by text-based semantic anchors derived from CLIP’s language space. These factors provide coarse spatial localization, which is then refined using an open-vocabulary segmentation model (FC-CLIP) to obtain high-quality masks. The approach is fully unsupervised, doesn't rely on paired video supervision, and operates entirely at inference time.\nKey strengths:\n1. Introduces a novel, principled training-free formulation for sound-prompted segmentation.\n2. Effectively reuses powerful pre-trained multimodal models without fine-tuning.\n3. The co-NMF formulation yields interpretable semantic factors and spatial activations.\n4. Demonstrates strong empirical performance on multiple benchmarks, outperforming prior unsupervised methods by a significant margin.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides a clear methodological description, including the motivation for using co-NMF, the semantic anchoring mechanism, and the integration with an open-vocabulary segmentation model. The experimental evaluation is extensive, covering several widely used benchmarks for sound-prompted or audio-visual segmentation (e.g., AVS-Bench S4/MS3, AVSS, ADE-based settings), and comparisons are made against relevant unsupervised and training-free/no-free baselines.\nQuantitative results consistently show that TACO significantly outperforms prior unsupervised methods and even approaches the performance of some trained alternatives, which supports the main empirical claims. Qualitative visualizations further illustrate that the learned factors correspond to meaningful semantic regions and align well with the provided audio prompts.\n\nHowever, there are minor concerns regarding clarity and reporting:\n1. Some tables report unusually large standard deviations that seem inconsistent with the deterministic nature of the inference procedure, suggesting adding statistical significance analyses to strengthen the empirical claims. \n2. Runtime and computational overhead are discussed but could be quantified more rigorously. Although the method is training-free, Figure 8 indicates that the iterative co-NMF optimization leads to increased inference time. An analysis of how inference cost scales with problem size (e.g., number of image patches, feature dimensionality, or factor rank) would help clarify its practical applicability.\nOverall, the evidence is convincing and largely supports the claims.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a timely and relevant problem at the intersection of multimodal learning, weakly supervised segmentation, and interpretability. Satisfying the audience of TMLR for machine learning.\nThe training-free nature of TACO is especially appealing in scenarios where labeled data or large-scale retraining is impractical. The use of co-NMF as a semantic alignment tool provides methodological insights, and contributes both a practical system and a conceptual framework for inference-time multimodal alignment.", "requested_changes": "1. Clarify and correct numerical reporting in experimental tables: Some reported standard deviations appear unusually large and may result from formatting or alignment issues. The authors should carefully verify all numerical entries, ensure consistent reporting of mean ± standard deviation, and clarify the number of runs used to compute these statistics. Suggesting adding statistical significance analyses to strengthen the empirical claims. \n2. Provide a more detailed and broader analysis of inference-time efficiency: Since the method relies on iterative co-NMF optimization at inference time, the paper should report more setup information (eg. memory usage), and iteration counts on representative hardware, along with comparisons to baseline methods.\n3. Improve reproducibility scaling details: The authors should clearly specify all hyperparameters (e.g., NMF rank, regularization weights, iteration limits, initialization strategy) and provide a concrete plan to facilitate replication.", "broader_impact_concerns": "1. Inference efficiency and scaling behavior:\nAlthough the method is training-free, its practical feasibility for real-world industrial deployment remains to be further evaluated. In particular, additional analysis is needed for large-scale and distributed settings, such as inference-time performance under billion-scale data regimes or high-QPS (queries per second) scenarios.\n2. Bias and representational limitations:\nThe method relies on large pre-trained models and benchmark datasets, with evaluations conducted on relatively homogeneous data types. It would be valuable to extend the experiments to a more diverse set of datasets, as well as to compare performance across different types of pre-trained models (e.g., Whisper), to better assess generalization and representational biases.\n3. Robustness to noisy or adversarial audio:\nThe system may be sensitive to noisy, misleading, or adversarial audio signals, which could lead to incorrect segmentation outcomes in real-world applications. Evaluating robustness under such conditions would strengthen the practical relevance of the method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)