Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion
Authors: Dexuan Ding
Lei Wang
Liyun Zhu
Tom Gedeon
Piotr Koniusz
Venue: ICLR 2025 Poster (2025)
Keywords: feature fusion
video anomaly detection
graph learning
multi-modal learning
Primary area: learning on graphs and other geometries & topologies

Abstract: In computer vision tasks, features often come from diverse representations, domains (e.g., indoor and outdoor), and modalities (e.g., text, images, and videos). Effectively fusing these features is essential for robust performance, especially with the availability of powerful pre-trained models like vision-language models. However, common fusion methods, such as concatenation, element-wise operations, and non-linear techniques, often fail to capture structural relationships, deep feature interactions, and suffer from inefficiency or misalignment of features across domains or modalities. In this paper, we shift from high-dimensional feature space to a lower-dimensional, interpretable graph space by constructing relationship graphs that encode feature relationships at different levels, e.g., clip, frame, patch, token, etc. To capture deeper interactions, we expand graphs through iterative graph relationship updates and introduce a learnable graph fusion operator to integrate these expanded relationships for more effective fusion. Our approach is relationship-centric, operates in a homogeneous space, and is mathematically principled, resembling element-wise relationship score aggregation via multilinear polynomials. We demonstrate the effectiveness of our graph-based fusion method on video anomaly detection, showing strong performance across multi-representational, multi-modal, and multi-domain feature fusion tasks.

Published as a conference paper at ICLR 2025
LEARNABLE EXPANSION OF GRAPH OPERATORS FOR MODAL FEATURE FUSION
Dexuan Ding'
Lei Wang* 1.2
Liyun Zhu'
Tom Gedeon
Piotr Koniusz?
Australian National University, "Data6//CSIRC 'Curtin University
ABSTRACT
In computer vision tasks, features often come from diverse representations, do- mains (e.g., indoor and outdoor). and modalities (e.g., text, images, and videos). Effectively fusing these features is essential for robust performance, especially with the availability of powerful pre-trained models like vision-language models. However, common fusion methods, such as concatenation, element-wise oper- ations, and non-linear techniques, often fail to capture structural relationships, deep feature interactions, and suffer from inefficiency tures across domains or modalities. In this paper, we shift from high-dimensional feature space
misalignment of fea-
oower-dimensional. interpretable graph space by constructing relationship graphs that encode feature relationships at different levels, e.g.. clip, frame, patch, token, etc. To capture deeper interactions, we expand graphs through iterative graph relationship updates and introduce learnable graph fusion op- erator to integrate these expanded relationships for more effective fusion. Our approach is relationship-centre operates in a homogeneous space, and is mathe- matically principled, resembling element-wise relationship score aggregation via multilinear polynomials. We demonstrate the effectiveness of our graph-based based fusion method on video anomaly detection, showing strong performance across multi-represennationall, multi-modal, and multi-domain feature fusion tasks.
INTRODUCTION
Whole fruits
Sliced fruit
Fruit salad
Mixed juice
Figure 1: Can squeeze more? This figure shows feature fusion in computer vision, from whole fruits (raw features) to sliced fruit (early fusion) and fruit salad (late fusion). The juice represents our graph-based fusion approach, which mixes multi-modal data for richer insights.
Imagine preparing fruit salad (see Figure T}. Initially, we slice fruits like apples, bananas, and oranges into distinct pieces, each retaining its unique flavor. This is analogous to features in multi- modal data, sourced from different modalities such as text, images, or videos. Combining these fruit slices resembles traditional early fusion methods in computer vision, where features are con- catenated but remain largely independent of each other (Snoek et al., 20055 Gadzicki et al.) 2020] Baruum et al. 2020). Next, might cut the fruit into smaller pieces and mix them further, but the distinct flavors persist. This reflects late fusion methods, which combine outputs from separately trained models on different modalities (Snoek et al., 2005| Bodla et al. 2017 Wang al.) 2019a). While some integration occurs, the deeper interactions between the features are still missing, just as the flavors in the salad remain separate. Finally, we use fruit mixer. This tool thoroughly blends
Conresponning author (lei.v @ adu..cd..au)

---

Published as a conference paper at ICLR 2025
the fruits, creating smooth, unified mixture where each flavor enhances the whole. This blending captures the essence of feature fusion. Our proposed graph-based fusion method parallels the fruit mixer, it doesn't 't just combine features but captures their complex, multi-level relationships. By focusing on interactions between feature relationships, we aim for a richer, more integrated fusion, revealing insights that traditional methods miss.
Traditional fusion techniques like concatenation, element-wise operations, or attention mecha- nisms (Dai et al. 202T) often capture shallow superficial interactions. These approaches typi- cally overlook deeper, structural relationships between feature elements (Atrey et al. 2010 Feng al.al,22100 2019)). limiting their ability to align features across different modalities or domains. Further- more, they often suffer from inefficiencies in computation and alignment. Our motivation for this work arises from the limitations of current methods, which struggle to blend and enhance feature relationships meaningfully. We propose paradigm shift from high-dimensional feature spaces to lower-dimensional, interpretable graph spaces. Instead of relying on raw features, our approach emphasizes the fusion of relationships between features, similar to how a fruit mixer blends dis- tinct flavors into a cohesive whole. Specifically, we introduce relationship graphs, such as similarity graphs, intermediary representations that encode the relationships between entities like frame-, patch-, or token-level features from videos. These graphs provide a more compact and interpretable representation of the data (Mai et al. 2020). In a similarity graph, nodes correspond to entities (e.g.. video clips, frames, or patches), while edges represent their relationships. To capture more complex interactions. we use iterative graph relationship updates to refine existing connections. This process reveals deeper structural insights that are often overlooked by traditional fusion methods.
al.,
We also introduce a learnable weight matrix, the graph fusion operator, which combines different graph relationship updates. Unlike simple concatenation or addition that treats all feature compo- nents equally, our learnable mechanism dynamically weights the contributions of different graph relationship updates, resulting in better fusion performance across various modalities, domains, and representations. Our graph-based fusion operates lower-dimensional, homogeneous space, of- fering several advantages. First, by representing relationships instead of individual features, it re- duces dimensionality and computational costs. Second, the homogeneous space allows consistent fusion across domains and modalities, aligning features into common structure. Third, it provides better interpretability by focusing on relationships rather than abstract features. The use of iterative graph relationship updates reveals refined feature interactions. Lastly, the learnable fusion mecha- nism adapts to specific tasks via learning objectives, improving both performance and efficiency.
Furthermore, our approach can be interpreted as multilinear polynomial with learnable coefficients, where relationship scores are aggregated across iterative graph relationship update sequences. This mathematically grounded framework generalizes simple linear operations, capturing more complex interactions between graph relationship updates, Our main contributions are as follows:
We propose a novel graph-based fusion framework, termed EGO fusion, which effectively cap- tures multtireppesentattinnal, multi-modal, and multi-domain relationships through relationship graphs, thereby enriching feature representations.
ii We introduce a learnable graph fusion operator that dynamically integrates different graph rela- tionship updates, facilitating deeper interactions among features and balancing self relfrooaaionssspss with inter-feature relationships.
iii We establish a theoretical connection between our graph fusion approach and multilinear poly- nomials, providing insights into feature interactions. We empirically validate our method in video anomaly detection, demonstrating improvements in both performance and interpretability over traditional feature-level fusion techniques.
RELATED WORK
Traditional feature fusion. Traditional fusion methods in multi-modal learning (D mello & Kory) 2015) typically rely on simple operations such as concatenation, element-wise addition, or multi- plication (Chen et al. 2023a). Early Fusion techniques (Snoek et al., 2005). for example, combine features from different modalities, such as text, images, and videos, into single high digh-dimensiooal feature vector. While straightforward, this approach often leads to overfitting and increased com- putational complexity due to the high dimensionality of the concatenated features. Moreover, early
2

---

Published as a conference paper at ICLR 2025
N
N
Oraph Pouzer Expurssion
Awi
Σ
Σ pm0
F
Nx
Apo
Apg
N
Gruph Plucenn Aspaniston
RP
R
(
F2
R2
NxN
Kandom Pairuése Sumpie
dn
Grüph
Graph Ruslon Operaator
Rlynonston
Fu
Figure 2: EGO fusion, our traph-based fusion framework, comprises three key components: (i) relationship graph reconstruction, (ii) graph expansion via element-wise multiplication, and (iii) graph fusion operator (learnable A) that aggregates representations within unified graph space.
fusion tends amplify noise from heterogeneous data sources, negatively impacting performance in complex tasks (Liu al. 2016). Late Fusion, the other hand, merges the outputs of inde- pendently trained models from different modalities (Snock et al.) 2005). This approach alleviates some dimensionality issues, allowing each model to focus on learning -opelity-specific features before integration. However, it fails to capture deep interactions between modalities, limiting its effectiveness. More advanced techniques, such as attention mechanisms (Vaswani] (2017) Dai al. 2021], have been introduced to dynamically weigh features based on importance. Nonetheless, these methods often overlook structural relationships between features, restricting their ability to model complex interactions. Neural network- non non-linear fusion introduces learnable layers to the process (Wang et al., (2019a) Wang & Koniusz 2021), but these models often lack transparency, making it difficult to interpret how individual features contribute to predictions. Aligning features from disparate modalities remains a challenge, further complicating fusion of non-comparable data.
Graph-based fusion. To overcome the limitations of traditional methods, recent work has turned to graph-based approaches (Liao et al., 2013) (Feg et al., 22099) Iyer et al., 2020; Chen Zhang [0200] Mai et al.) 2020 /Zhang et al. 2024), which emphasize the relationships between features rather than these methods, data is represented as graph, with nodes corresponding to entities such frames, patches, or tokens, and edges capturing similarities or interactions (Iyer) cct al. 2020). This structured representation allows for more interpretable and -aaee feature fusion. Graph Convolutional Networks (GCNs) have been widely used for modeling relationships across modalities (Zhang et al.) a 2020), particularly in video understanding tasks (Huang et al.) 2020)] Gkalelis et al. 2021). where capturing temporal and spatial relationships is critical. GCNs aggregate information from neighboring nodes, enabling the modeling of context-dependent interactions. De- spite their effectiveness, most graph-based methods capture only first-order relationships, limiting their ability to model complex, multi-step dependencies between features (Chen & Zhang 2020).
the features themselves.
Our work uses iterative graph relationship updates, which enable flexible and localized information fusion while preserving the graph structure. This enables our method to capture refined and more nuanced structural interactions, revealing insights that conventional fusion techniques often miss.
Interpretable and efficient fusion. As machine learning models grow in complexity, there is in- creasing demand for fusion methods that balance interpretability and computational efficiency. Tra- ditional approaches like concatenation and neural-based fusion often operate in high-dimensional spaces, which can lead inefficiencies and hinder transparency. Recent advancements have fo- cused designing more interpretable fusion strategies (Ma et al. 2016). For example, Capsule Networks (Sabour et al., 20177) and attention mechanisms (Vaswani, 2017) aim to provide greater insight into the fusion process by highlighting important features. However, these methods still suffer from the computational burdens associated with high-dimensional data, particularly in large- scale, multi-modal tasks.
Our approach offers a different solution by shifting from feature-level fusion to relationship fusion, operating in lower-dimensiooal graph space. This transition improves both interpretability
3

---

Published as a conference paper at ICLR 2025
and efficiency, it focuses capturing feature relationships rather than raw features. Using iter- ative graph relationship updates, our method models deeper interactions among features, enabling more effective fusion while avoiding the nigh-dimensional computations of traditional methods. Ad- ditionally, our learnable graph fusion operator dynamically weights feature interactions, leading to more adaptive and task-specifie fusion process.
3 APPROACH
This section introduces our proposed method, Expansion of Graph Operators (EGO) fusion. We be- gin by defining key notations, followed by the construction of our relationship graph, the formulation of iterative graph relationship updates as graph expansions, and our fusion strategy.
Notations. Let Ir 1.2...., ; vectors by lowercase boldface, e.g., 2; matrices by uppercase boldface, egg., X; and tensors by calligraphic letters, e.g..
represent the index set. Scalars are denoted by regular fonts, egg.
3.1 EGO: EXPANSION OF GRAPH OPERATORS
Relationship graph of unit-level features. Text, images, and videos be used extract vari- ous unit-level features (see definition in Appendix A),, ranging from word- and paragraph-level to patch-, clip-, frame-, cube-, token-level, using pre-trained models. These heterogeneous fea- tures are then transformed into a homogencous graph space by modeling pairwise relationships among unit-level features, such as similarities, distances, or other relevant metries. Since distances and similarities are inversely related, meaning high similarity corresponds to low feature distance (see proof in Appendix B). similarity scores are particularly effective for encoding local relation- ships among units, helping to identify which feature points are close or similar. In the resulting relationship graph, e.g., similarity graph, each unit feature point represented as a node, and the graph structure captures the local neighborhood of these unit-level features.
To show this process, we consider extracting unit-level feature representations from multiple pre- trained models multi-modal sources. We denote these feature representations as F(1) E R F(2) Nxd2
F(T) E Nxdr, Here, F(m) E Nxd... (m E Ir) denotes the feature set from the m-th model modality, where N corresponds to the number of unit-level features, and each feature exists in dn dimensions. We begin by computing pairwise relationships between unit-level features within each model modality as follows:
$t = rrffff
where r(--) relationship function, e.g., cosine similarity, Gaussian kernel, or another distance metric. The term sij represents the relationship score between unit-level features f; and f, from the feature set F (the model or modality index omitted for simplicity). Using these pairwise relationships, we build a relationship graph represented by the matrix:
R [s.ple..)eIN
(2)
where R E xN captures the pairwise relationships between unit-level features. A value of the matrix indicates strong relationship, e.g., two unit-level features are identical in visual or textural concepts, while value of 0 means no relationship. This matrix serves as the adjacency matrix for the graph.
However, while similarity- or distance-based graphs are useful, they often fail to capture the global structure of the data, as they predominantly rely on local information. To address this limitation, propose novel graph expansion approach based on iterative graph relationship updates. This method enhances the graph's 's representation by expanding graph relationships in a more controlled and localized manner. Through iterative refinement, nodes incorporate richer information.
Graph powers vs. Iterative graph relationship updates. Graph powers refer to the repeated mul- tiplication of graph's 's adjacency matrix, which shows multi-step connections between nodes. For a graph represented by the adjacency matrix R, the k-th power of the graph, denoted as R² un- covers relationships between nodes that are lo steps apart. Specifically, each element R indicates
4

---

Published as a conference paper at ICLR 2025
the cumulative influence of all paths of length & between nodes i and This mechanism is graph. However, our approach focuses on iterative
ful for modeling long long-range dependencies graph relationship updates using element-wise multiplication rather than traditional graph powers with matrix multiplication. Instead of matrix exponentiation, we iteratively refine the relationship graph through series of adaptive updates, providing more direct control over feature propagation. This process enables the model to dynamically capture both local and global dependencies without the constraints imposed by power-based adjacency transformations.
To operationalize this, consider two distinct relationship graphs R(a) and R(b)- We construct sequence of iterative relationship updates for each model or modality:
G (()) = Rla) R
R.P.
E (RNxx((P+1
(a)
(3)
(b) RBy R
2 R
N (b)
where R. (a) and R/ represent the relationship graphs obtained after p and q iterative updates for (a) and (()) respectively. The initial relationship graph R serves as baseline, typically preserving self-connections via an identity matrix, i.e., R =
These sequences reflect how information propagates between nodes over multiple refinement steps, offering insights into both local and global relationships within the graph. Unlike static graph power expansions, our iterative approach dynamically adjusts node interactions each step, allowing for more flexible and expressive feature integration process.
Graph fusion operator. We introduce novel graph fusion operator, denoted @, designed to integrate information from different modalities by merging their relationship graphs through iterative updates. This operator enhances the model's capacity to learn complex representations by incorporating both direct and higher-order relationships within the data. Mathematically, the graph fusion is expressed as follows:
G (a)
A G
Q
RS
(a) ap R
P=U
Q apbg
ΣΣ
R
R R%b)
(4)
p=
where a = [apleez and b= [byleZ are the modality graph update selectors, and A a&be (PHDX(HH) with representing the outer product. We also propose advanced variant:
Q P
G
G=EAAp p=0p=0
Ap.
RRa @ R%,),
(5)
where A E R(P+1)x(Q+I) is a learnable weight matrix that modulates the fusion process. The operator @ denotes element-wise multiplication. In Appendix C we provide a detailed expla- nation of our graph fusion process, including the derivation of equation [] and the relationship and between equation and equation The fused relationship graph G E 1 is the result of this integration, enabling the model to optimize the combination graph relationship updates through backpropagation, thus improving the fusion of features across different levels.
We observe that Raa R in equation 4 and equation 5 captures all possible combinations of it- erative graph relationship updates, while A provides the appropriate weights for the fusion process. Incorporating the initial relationship graph, e.g.. Rº, in the fusion process is critical; it preserves self- connections and maintains original feature information. This mechanism enables adaptive weight- ing, balancing the influence of direct relationships and progressively refined interactions, ensuring the model emphasizes the most relevant connections tailored the specifie task.
5

---

Published as a conference paper at ICLR 2025
65
11
13
15
13
14
6
16
16
17
18
30
18
30
18
30
1
19
19
29
20
20
20
28
21
22
23
24
25
26
13D visual features
(b) SimCSE text embeddings
(c) Fused relationship graph
Figure 3: Comparison of relationship graphs on ShanghaiTech. The graphs are constructed using cosine similarity to represent relationships among features: (a) visual features, (b) text embeddings, and (c) the fused graph that integrates both modalities. In each graph, nodes represent clip-level (or unit-level; see Appendix A features, with numbers indicating the sequence order of the video clips. Edges, shown in green, represent cosine similarity between features, with darker shades indi- cating stronger connections. Anomaly nodes and their connections are highlighted in purple (e.g., connection from node 10). The fused relationship graph, generated using our EGO fusion method, effectively integrates visual and textual information into unified structure, resulting in fewer connections among abnormal nodes. This effect is achieved through our regularization term in equation 7 which encourages anomaly nodes have fewer connections than normal nodes, gardless of connection strength. Appendix includes additional visualizations.
To enhance the fusion diverse features across varying representations, domains, and modalities, implement random sampling strategy during each training iteration. By randomly sampling two relationship graphs for the fusion process, we ensure that the integration occurs within a homoge- neous graph space while still facilitating the amalgamation of features from disparate domains. This random sampling introduces variability and robustness into the fusion process, significantly boost- ing the model's capacity learn meaningful representations. Furthermore, this strategy enriches the feature set and cultivates comprehensive understanding of inter-relationships across different data modalities, ultimately leading improved performance. [equreeccmparee the original relationship graphs for visual and textual features with our EGO-fused relationship graph.
3.2 CONNECTING TO MULTILINEAR POLYNOMIALS
Our graph fusion process intrinsically connected to multilinear polynomials, which provide a robust mathematical framework for aggregating multiple powers of relationship scores between two relationship graphs. Specifically, each entry in the fused graph G is a function of the element-wise relationship scores between the two input graphs. These scores are then combined in manner that parallels how terms in a multilinear polynomial are constructed. This allows us to capture both linear and nonlinear interactions between modalities.
We can express each entry in the fused relationship graph, Gt,, as multilinear combination of the relationship scores from the two graphs, G(a) and G(b). at different powers. Rewriting equation or cquation based on equation I we get:
Gij = Ao,o + A0,18(0)a.5 + A0,2070)as + ... + AR. (
A p.q (.9. (D)s
(6)
p=0== 9=0
The term Ap.g serves as learnable coefficient that controls the relative importance of the interaction between the p-th power of the relationship score from modality and the g-th power from modality b. This general form is closely related to multilinear polynomials, where the powers the relationship scores (i..,, s ( and og ( represent the different degrees of interaction between the modalities.

---

Published as a conference paper at ICLR 2025
Interpretation of multilinear polynomial terms. The structure of this multilinear polynomial offers valuable insights into the fusion process. Each individual term, such as o (a),, repre- sents interactions between higher-order relationships in the graphs ((a) and G These higher- order terms capture increasingly complex dependencies between the two graphs, allowing the model to learn not only from direct pairwise relationships (as seen in the linear terms) but also from more subtle. nonlinear relationships that emerge from specific combinations of scores.
Linear terms like A or A (a)i represent simple linear combinations of relationship scores from the two graphs. These terms capture first-order interactions, essentially weighting how much each modality's s direct relationship contributes to the fused graph. Quadratic and higher-order terms like
or (2) (()),, capture cross-modality interactions that go beyond simple weighting. These terms allow the model to learn relationships in which one modality's 's re- lationship score influences the contribution of another modality, enabling more sophisticated fusion strategies. For instance, if two modalities appear only weakly related initially, higher-order terms can help reveal deeper latent connections. This formulation shows that our graph fusion is, in fact, more general and flexible version of graph-based fusion methods, capable of modeling complex interactions between different modalities. Further insights can be found in Appendices D and
3.3 MULTI-MODAL VIDEO ANOMALY DETECTION
We present our approach video anomaly detection for several key reasons:
i. Multi-modality fusion. Robust video anomaly detection requires integrating multiple modali- ties, such as video, audio, and text (Wu et al. 2020 Chen et al. 2023a, Wu et al. 2024). These modalities can be easily obtained, e.g., by using pre-trained video captioning models to gener- ate accompanying text data. For human-relatee anomaly detection, poses can be extracted using OpenPose (Cao et al. 2017) then embedded into pose features via ST-GCN et al.) 2018). This enables us to explore and evaluate the efficacy of multi-modal feature fusion, where com- bining complementary information across modalities enhance anomaly detection performance.
ii Multirrepresentationnal fusion. Current state-of- sfate-of-the-art video anomaly detection methods typ- ically rely on pre-trained action recognition or motion-based models for feature extraction (Zhul et al. 2024). These models, such as 13D (Carreira & Zisserman 2017). C3D (Tran et al.) 2015) and SwinTransforme
(Swin')) et al. 2022), offer distinct perspectives on the same modal- ity, e.g., videos, extracting different features that represent various aspects of motion, appear- ance, or temporal dynamics. Our EGO fusion approach is well-suited to this setting, allowing to combine features from different representations within the same modality, thus enriching the representational capacity of the model and potentially boosting detection accuracy.
iii. Multi-domain fusion. Existing video anomaly detection datasets often represent a single do- main scenario, such as videos captured from specific locations like streets university cam- puses (Zhu et al. 2024). This limitation offers an opportunity for us to explore multi-domain feature fusion, where features from different environments or scenarios can be integrated to cre- ate a more generalized and robust anomaly detection framework. This cross-domain learning can enhance the model's 's adaptability and performance across diverse settings.
iv. Binary classification as foundational task. Video anomaly detection is commonly framed as a binary classification problem, where the objective is to distinguish between normal and anomalous events. This straightforward approach facilitates intuitive visualization and analysis of the model's 's performance when implementing our fusion technique, enabling us to gain deeper insights into the impact of feature fusion on detection accuracy. By starting with binary task, we establish a robust foundational benchmark that allows for iterative refinement and testing of our fusion strategies. Furthermore, once our framework is validated in this context, extending to more complex multi-class classification tasks becomes a natural next step.
Degree variance regularization. Regularization plays a crucial role in enhancing our fused graph representation, which captures the intricate relationships between nodes. In this context, each entry in the fused graph represents the weight of the edge connecting two nodes, providing a quantitative measure of their interconnections. To assess the connectivity of each node, calculate the sum of the relationship scores for each row in the fused graph. This results in a single value for each node, representing the total weight of all edges linked to it; this value is commonly referred to the
7

---

Published as a conference paper at ICLR 2025
0.95
0.95
0.95
0.85
0.85
0.85
0.75
0.75
0.75
0.65
0.85
0.65
Avenue
Avenue
Avenue
Pad2
Pad2
ShT
ShT
ShT
0.35
0.35
Street
0.35
Streat
0.0
0.2
0.4
0.6
0.8
9
13
17
21
25
29
0.0 0.001 0.1 0.5 1.0
3.0 10.0
(a) a
(b)
(c) À
Figure 4: Hyperparameter evaluations for (a) cut-off threshold a, (b) top maximum degrees, and
(c) À in the regularization term across all four video anomaly detection datasets, using I3D visual features and text features our EGO fusion framework.
weighted degree. A weighted degree of zero indicates that node is isolated with no connections, while a higher weighted degree signifies greater interconnectivity, either through larger number of edges or edges with higher weights. To facilitate an effective fusion process, we introduce a degree variance regularization term that operates on the fused graph representation for anomaly detection:
N Σ
N Σ
t=
Var
GTاl 2 )
- Var TopMax
Ciyls 2 a
where À is penalty parameter that controls the strength of this regularization, G+ and G' denote the graphs corresponding to abnormal and normal behaviors, respectively. The parameter a acts as a cut-off threshold, filtering connections such that, for example, a 0.5 excludes relationship scores below this value. Additionally, TopMax, (-) selects the top k maximum degree values from the abnormal graph. The purposes of this regularization term to ensure that unit-level normal features forming nodes have similar degrees in both normal and abnormal graphs.
Our regularization term can be integrated into the original anomaly detection classification loss, such Binary Cross Cross-Entropy (BCE) loss. By emphasizing the variance in node connectivity, enhance the model's sensitivity to anomalies while promoting a balanced representation across the graph.
EXPERIMENTS
4.1 SETUP
Datasets. We select the following datasets for our evaluation: (i UCSD Ped2 (Ped2) features 16 training and 21 testing videos pedestrians, with anomalies like eyclists, skateboarders, and cars on paths. (ii) ShanghaiTech (ShT) has 330 training and 107 testing videos across 13 campus scenes, with 130 abnormal events such as cyclists and fights. (iii) CUHK Avenue (Avenue) includes 16 training and 21 testing videos, with 47 anomalies like running, walking in the wrong direction, and object throwing. (iv) Street Scene (Street) comprises 46 training and 35 testing videos of two-lane street, capturing 205 anomalies like jaywalking. U-turns, and car ticketing.
Features. We use popular models pretrained on Kinetics-400 (Kay et al.) 2017) as feature ex- tractors, including I3D (Carreira & Zisserman 2017) for 2048-dim. features from each 16-frame segment. We also extract 4096- and 1024 features using pretrained C3D (Tran et al. 2015) and SwinT (Liu al.) (022), respectively. For text feature extraction, we apply SwinBERT (Lin ee al. 2022), pretrained on VATEX (Wang et al. (019b), to generate dense video captions for every 64-frame segment. We then use SimCSE (Gao
2021) obtain 768-dim. text embeddings.
Metrics. Following common practice (Chen et al. 2023a), we consider Area Under the ROC curve (AUC) which is widely used for evaluation Video anomaly detection. Similar to existing methods like (Tian et al. 2021 Chen et al. 2023b a). which evaluate frame-level performance by repeating snippet-level predictions (e.g., 16 times) to frame-level labels, we adopt evaluation method
haappet-by-snippee do not have access frame-level features. To obtain snippet-level labels, we derive them from the frame-level labels: if any anomaly occurs within a 16 frame snippet, we label the snippet abnormal; otherwise, it labeled normal.
8

---

Published as a conference paper at ICLR 2025
100
75
50
25
0
1.00
0.50
0.00
0.50
1.00
Figure 5: (Top row): The effects of P (for visual feature) and Q (for text feature) the learnable graph operator. (Bottom row): The learned optimal A for (from left to right) UCSD Ped2, Shang- haiTech, CUHK Avenue, Street Scene, and joint training on both UCSD Ped2 and ShanghaiTech.
Table 1:
Experimental results feature-level and graph-level fusion across four video anomaly detection datasets, including single-modality comparisons. Graph-level single-modality and tradi- tional methods similarity graph representations for anomaly detection.
UCSD Ped2 ShanghaiTech CUHK Avenue Street Scene
13D visual 78.90
95.87
37.25
74.53
Text only
80.02
83.39
65.19
69.34
Feature-level
Concatenation
86.72
96.07
43.22
75.42
Addition
86.20
95.77
57.44
75.05
Product
62.72
94.15
32.04
75.59
MTN fusion
92.80
96.37
62.06
71.50
13D visual
68.89
69.88
58.72
49.12
Text only
43.03
85.59
42.36
55.27
Graph-level
Concatenation
63.45
88.68
50.09
48.97
Addition
57.88
44.07
40.24
57.18
Product
43.07
86.49
44.34
66.52
EGO (ours) 93.23
97.26
83.10
77.61
Baselines. We compare EGO fusion with both feature- and graph-based approaches, including traditional techniques like simple concatenation and element-wise fusion. We also report single- modality performance for comparison. Additionally,
reproduce the results of recent multi- modal fusion method (Chen et al.) (023a) to show the effectiveness of our approach. Our classifica- tion layer, following the EGO fusion, consists of two fully connected (FC) layers (N N-1) with a ReLUU in between, followed by a Sigmoid. For simplicity. we set to 32. We use cosine sim- ilarity create relationship graphs in experiments. We set training epochs to 30-50, depending on the datasets; for example, we use 50 for multi-domain experiments with Ped2 and ShT.
4.2 EVALUATION
Discussion on hyperparameter evaluations. Figure 4 shows our results. First, the cut-off threshold for filtering weak relationships varies by dataset: higher values (e.g., 0.8 and 0.9) work better for Ped2 and Street, while lower a is needed for Avenue, likely due to more background motion in the latter. Second, the optimal & for selecting maximum degree values of normal nodes in the abnormal relationship graph is similar across ShT (11), Ped2 (17), and Avenue (15). This similarity may arise from the datasets being captured on campus and featuring comparable anomalies, like cyclists. Interestingly, the optimal & for Street is just 1, which may reflect its complexity due to diverse anomalies two-lane street setting. Additionally, the optimal regularization penalty parameter for Street is low (le-4), suggesting minor effect of regularization its feature fusion. In contrast, a larger À (e.g., I) yields the best performance on Ped2.
A closer look at the learnable graph operator. We set both P (for I3D visual features) and Q (for SimCSE text embeddings) in equation 5 to range from 1 to 10 and conduct grid search to evaluate their impact our EGO fusion. We evaluate the framework on all four individual anomaly detection datasets, as well as on a combination of ShT and Ped2, as shown in Figure [ The results indicate that P and Q significantly influence EGO fusion performance, suggesting that the fusion

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of MTN fusion (feature-level) and EGO fusion (graph-level). ShanghaiTTcc (ShT) is used for multi-represenaational and multi-modality fusion, while UCSD Ped2 (Ped2) and ShT are used for multi-domain fusion. Unlike MTN, which fuses two features at time, EGO fusion enables simultaneous fusion of multiple features for greater flexibility. Training times for one epoch (in seconds) with a batch size of 32 on an Nvidia RTX 4070 GPU are also reported, with model sizes indicated in blue next to their respective models.
MTN (29.0M) EGO[0.09
Train
Test
AUC
Time
AUC
Time
I3D + C3D
I3D + C3D
89.25
13.6
87.17
7.8
Multi-represent.
I3D + SwinT
I3D + SwinT
88.80
9.7
89.85
4.9
C3D + SwinT
C3D + SwinT
84.45
12.0
85.52
5.7
I3D + C3D + Swin'T I3D) + C3D Swin'T'
N/A
95.38
9.0
Visual + Text
Visual + Text
96.37
97.26
Multi-modality
Visual + Pose
Visual + Pose
95.48
96.04
Text Pose
Text + Pose
94,49
95.77
Visual + Text + Pose
Visual + Text + Pose
N/A
97.79
Ped2 only
56.21
58.30
Multi-domain
Ped2 + ShT
ShT only
96.04
95,10
Pcd2 ShT
94.60
92.11
process is (i) dependent on dataset complexity, (ii) affected by the quality of visual and text features, and (iii) guided by the learning objective, e.g., anomaly detection. By adjusting these parameters, the framework can prune irrelevant connections and strengthen useful ones across different feature modalities. The learned A shows that a higher-order relationship graph is required for visual features in Ped2 and ShT, as well for their combined dataset. Conversely, text features necessitate higher- order graph for ShT and Street. This indicates that in more complex scenes, text features can enhance anomaly detection performance.
Graph-level fusion vs. feature-level fusion. As shown in Table I for individual modalities such as visual-only or text-only inputs, raw features typically outperform their corresponding graph rep- resentations. This is because raw features are high-dimensional (e.g.. I3D visual features are 2048- dimensional, and text features are 768-dimensional), containing rich semantic information directly, whereas graph representations primarily capture relationships and are much lower-dimensional We also observe that feature-level fusion techniques, such as concatenation, addition, and product, tend to enhance performance compared to using individual feature modalities. However, these same methods do not perform as well when applied to graph-level fusion, indicating the need for more sophisticated fusion strategies at the graph level. Additionally, we compare approach to the popular Multi-scale Temporal Network (MTN) fusion (Chen et al. 2023a). which fuses visual and text features in end-to-end learnable manner the feature level. Our EGO fusion consistently outperforms MTN fusion on all benchmarks while being more lightweight and interpretable.
Unified fusion across representations, modalities, and domains. Table 2 summarizes our ex- periments on multi-repreeennttiionall multi-modal, and multi-domain feature fusion. As shown, our EGO fusion outperforms the MTN fusion (Chen et al., 2023a), despite its simplicity. A key ad- vantage of EGO fusion is its ability to integrate multiple modalities. representations, and domains simultaneously. whereas MTN limited by its network design to fusing only two features time. Moreover, our framework is more lightweight and significantly faster to train.
5 CONCLUSION
In this paper, we introduce a novel graph-based fusion framework, termed EGO fusion, which en- hances feature integration. By using relationship graphs on raw features, our approach captures deeper interactions through iterative graph relationship updates and a learnable fusion operator. Our framework not only reduces dimensionality and computational costs but also improves interpretabil- ity by emphasizing feature relationships. Experiments in video anomaly detection show that our method outperforms traditional fusion techniques, highlighting the potential of relationship-drie fusion approaches in using multi- re-rresnnaationall lieve our contributions will inspire further research into understanding complex feature interactions, ultimately leading to more robust model performance across diverse applications.
and
features. We be-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTI
Dexuan Ding conducted this research under the supervision of Lei Wang as part of his final year honors project at ANU. We extend our gratitude to Qixiang Chen for reviewing the model code and producing excellent plots. This work was also supported by the NCI National AI Flagship Merit Al- location Scheme, and the National Computational Merit Allocation Scheme 2024 (NCMAS 2024), with computational resources provided by NCI Australia, an NCRIS-enabled capability supported by the Australian Government.
REFERENCES
Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb EI Saddik, and Mohan S Kankanhalli. Multi- modal fusion for multimedia analysis: survey, Multimedia systems, 16:345-379, 2010.
George Barnum, Sabera Talukder, and Yisong Yue. On the benefits of early fusion in multimodal representation learning. arXiv preprint 2xXi::2011.0719,, 2020.
Navaneeth Bodla, Jingxiao Zheng, Hongyu Xu, Jun-Cheng Chen, Carlos Castillo, and Rama Chel- lappa. Deep heterogeneous feature fusion for template-based face recognition. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 586-595. IEEE, 2017.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi multi-person 2d pose estimation using part affinity fields. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Jiayi Chen and Aidong Zhang. Hgmf: heterogeneous graph- fusion for multimodal data with incompleteness. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp. 1295-1305 2020.
Weiling Chen, Keng Teck Ma, Zi Jian Yew, Minhoe Hur, and David Aik-Aun Khoo. Tevad: Im- proved video anomaly detection with captions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5549-5559, 2023a.
Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-Chung Wu. Mgfn: Magnitude-contrrssinn glance-anddfocus network for sepely-s video anomaly detection. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 387- 395, 2023b.
Marco Cuturi. Fast global alignment kernels. In Proceedings of the International Conference Machine Learning (ICML), 2011.
Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In Proceedings of the International Conference Machine Learning (ICML), 2017.
Yimian Dai, Fabian Gieseke, Stefan Oehmcke, Yiquan Wu, and Kobus Barnard. Attentional feature fusion. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3560-3569, 2021.
Sidney K D' D'mello and Jacqueline Kory. A review and meta- anely-sn of multimodal affect detection systems. ACM computing surveys (CSUR), 47(3):1-36, 2015.
Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training frame- work for video anomaly detection, 2021. RtRLhhtps::// /ttp/aaxiivi.////a//eee www.aiiww....aaaaoooooooa wttp:///.rrrgaaa///////0// wbsss//14
01633
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks.
In Proceedings the AAAI conference on artificial intelligence, volume 33, pp. 3558-3565, 2019.
Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early late fusion in multi- modal convolutional neural networks. In 2020 IEEE 23rd international conference on information fusion (FUSION), pp. 1-6. IEEE, 2020.

---

Published as a conference paper at ICLR 2025
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. Empirical Methods Natural Language Processing (EMNLP), 2021.
Nikolaos Gkalelis, Andreas Goulas, Damianos Galanopoulos, and Vasileios Mezaris. Objectgraphs: Using objects and graph convolutional network for the bottom-up recognition and explanation of events in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3375-3383, 2021.
Yi He, Xin Liu, Yiu-Ming Cheung, Shu-Juan Peng, Jinhan Yi, and Wentao Fan. Cross-graph atten- tion enhanced multi-modal correlation learning for fine- grained image-tex: retrieval. In SIGIR, pp. 1865-1869, 2021. httpp://///ooo
org/ 110.001111//34445111
Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. Location- aware graph convolutional networks for video question answering. In Proceedings of the AAAI Conference Arlificial Intelligence, volume 34, pp. 11021-11028, 2020.
Geoffrey Iyer, Jocelyn Chanussot, and Andrea L Bertozzi. A graph-based approach for data fusion and segmentation of multimodal images. IEEE Transactions on Geoscience and Remote Sensing, 59(5):449-4442, 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint wwXiv:1705.06955 2017.
François Le Gall. Algebraic complexity theory and matrix multiplication. Proceedings of the 39th International Symposium on Symbolic and Algebraic Computation, ISSAC '14, pp. 23, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450325011, 110.....11112222211122
doi:
10.1145/2608628.26222222 2627499. /ttp::tss///oi
Wenzhi Liao, Rik Bellens, Aleksandra Pizurica, Sidharta Gautama, and Wilfriid Philips. Graph- based feature fusion of hyperspectral and lidar remote sensing data using morphological features. In Proc. IGARSS, pp.
2013.
Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 17949 17958, 2022.
Jingjing Liu, Shaoting Zhang, Shu Wang, and Dimitris N Metaxas. Multispectral deep neural net- works for pedestrian detection. arXiv preprint 1:X1v:1611.02644 2016.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3202-3211, 2022.
Gang Ma, Xi Yang, Bo Zhang, and Zhongzhi Shi. Multi-fcature fusion deep networks. Neurocom- puting, 118:164-171. 2016.
Sijie Mai, Haifeng Hu, and Songlong Xing. Modality to modality translation: An adversarial repre- sentation learning and graph fusion network for multimodal fusion. In proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 164-172, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Cees GM Snoek, Marcel Worring, and Arnold WM Smeulders. Early versus late fusion in semantic video analysis. In Proceedings of the 13th annual ACM international conference on Multimedia, pp. 399 402, 2005.
Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Cameiro. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 4975 4986, 2021.
12

---

Published as a conference paper at ICLR 2025
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 448994497, 2015.
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Lei Wang and Piotr Koniusz. Self-supervisee action recognition by statistical moment and subspace descriptors. In Proceedings of the 29th ACM international conference on multimedia, pp. 4324- 4333, 2021.
Lei Wang and Piotr Koniusz. Temporal-viewpoint transportation plan for skeletal few-shot action recognition. In Proceedings of the Asian Conference on Computer Vision, pp. 4176-4193, 2022a.
Lei Wang and Piotr Koniusz. Uncertainty-dtw for time series and sequences. In European Confer- ence on Computer Vision, pp. 176-195, Springer, 2022b.
Lei Wang, Du Q Huynh, and Moussa Reda Mansour. Loss switching fusion with similarity search for video classification. In 2019 IEEE international conference on image processing (ICIP), pp- 974-978. IEBE, 2019a.
Lei Wang, Jun Liu, and Piotr Koniusz. 3d skeleton-based few-shot action recognition with jeanie is not so naïve. arXiv preprint 1rXiv:2112.12666 2021.
Lei Wang, Jun Liu, Liang Zheng, Tom Gedeon, and Piotr Koniusz. Meet jeanie: similarity measure for 3d skeleton sequences via temporal-viewpoin alignment. International Journal of Computer Vision, pp. 1-32, 2024.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A acage,scale, high-quality multilingual dataset for iideo-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019b.
Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 322-339. Springer, 2020.
Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang. Vadelip: Adapting vision-languag models for weakly supervised video anomaly detec- tion. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6074-6082, 2024.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial Temporal Graph Convolutional Networks for Skeleton- Based Action Recognition. In AAAI, 2018
Hongfeng Yu, Fanglong Yao, Wanxuan Lu, Nayu Liu, Peiguang Li, Hongjian You, and Xian Sun. Text-image matching for cross-modal remote sensing image retrieval via graph neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Sensing, PP:1-13, 01 2022. doi: 10.1 10.1109/SSTARS... 2022..333...22.2311551
Junbin Zhang, Pei-Hsuan Tsai, and Meng-Hsun Tsai. Semantic2graph: graph-based multi-modal feature fusion for action segmentation in videos. Applied Intelligence, 54(2):2084-2099, 2024.
Qi Zhang, Jianhang Zhou, and Bob Zhang, Graph based multichannel feature fusion for wrist pulse diagnosis. IEEE Journal of Biomedical and Health Informatics, 25(10):3732-3743 2020.
Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection, 2023. URL/ittpsss//rrxii.oor////////
05160
Liyun Zhu, Lei Wang, Arjun Raj. Tom Gedeon, and Chen Chen. Advancing video anomaly detec- tion: A concise review and new dataset. In The Thirly-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
13

---

Published as a conference paper at ICLR 2025
A UNIT-LEVEL REPRESENTATION
Unit-level refers to the fundamental, granular components of data that can be segmented and analyzed independently. Depending on data modality, unit-level entities include words or paragraphs in text, patches in images, frames, clips, cubes in videos, and tokens in sequential data. These units represent the smallest meaningful segments of the data, serving as essential building blocks for deeper analysis and processing.
Unit-level features are the extracted representations of these granular data components. Derived from individual units, such as words or paragraphs in text, patches in images, frames, cubes, or clips in videos, or tokens in sequences, unit-level features capture the distinctive characteristics of each unit. These features are widely used in recent advances like large language models (LLMs) and vision-language models (VLMs), where the ability to capture fine-grained details at the unit level has significantly improved tasks such text generation, image classification, object recog- nition, and multimodal understanding. By using unit-level features, modern models achieve supe- rior performance in higher- level tasks, including classification. recognition, prediction, and even cross cross-modal modal tasks, where detailed analysis is crucial for accurate and contextually rich outcomes.
B RELATIONSHIP BETWEEN FEATURE DISTANCE AND SIMILARITY
widely used feature distance measure the Euclidean distance. When we use Euclidean distance to measure the similarity between two network-encoded, L'-nomaalieed features from two images, we obtain the following expression:
[6x(X) 6(Y) (6(X), - 2(0( (
= 2 2(0(X),6(Y)
= 2 - (
(8)
In this equation, o(X) and <(Y) represent the feature maps of images X and Y. respectively,
while k(.,-) denotes various types of similarity measures. These measures can include dynamic time warping (DTW) (DTW)CCuturi 2011) and its variants such as Soft-DDTW (Cuttuu & Blondel 2017),
uncertainty-Dee -TW (Wang & Koniusz 2022b), and JEANIE (Wang et al. 2021 Wang & Koniusz 2022a Wang et al. 2024), or kernels such as intersection and radial basis function (RBF))ernn) as well as simpler metrics like cosine similarity. Since the features ((X) and o(Y) are Iz-nomaalzedd, both ((x(X), ((X) and k()(Y), ((Y)) equal 1.
This shows that the Euclidean distance between two I, z.2-normalzzed features is directly related to the similarity measure between them. Specifically, the Euclidean distance can be expressed terms of the similarity function k(.,), where higher similarity, measured by t(o(X), o Y)), leads to a smaller Euclidean distance. the case where the features are identical, the similarity reaches its maximum value of 1, and the Euclidean distance is zero. Conversely, ity decreases, the distance increases. Thus, this formulation highlights the inverse relationship between distance and similarity for normalized features, emphasizing that both metrics are funda- mentally tied to how well the features align.
the similar-
C DERIVATION OF GRAPH FUSION OPERATOR IN EGO FUSION
We begin by applying learnable weights, denoted as a and b to two iterative graph relationship updates ((a) and Goy) These vectors learn select graph relationship updates for the fusion of two
14

---

Published as a conference paper at ICLR 2025
graphs. The process can be expressed as follows:
Q Σ9
G
(()
A
R
b
Σ
apby
[
q=0
P
(
R
0g
@
P
R
@ 1bo, b1. ... .bo
RBb
R
R
Rio
0=0 p=0
ap
R
Tagbu agdi andy aιdo aid abb
ando
R
(a)
R
R
(a)
R
R
R
R
3 R
Q
P
abbo
R
R
R
R
R
R
Σ
(a)
(b)
(b)
(a)
(b)
0= p=0
Lapto apbi apbs
apbo_
R
R (6)
R
O
R
R
R
R
R R(y.
(a)
(a)
(a)
R
R
R
R (b)
R
R
R
R
(a)
(a)
aa)
P
R
R
R
R
R
R
R
A
(9)
400
RS
R
R
R
R
O
R
R
(b)
In this formulation, X represents the outer product, @ denotes element-wise multiplication, and is an operation we define that functions similarly the outer product but uses element-wise multiplication for the fusion process.
As shown in equation g instead using two independent vectors a and b that result in A & b, we can make A fully learnable matrix. This allows the model to explore more efficient and flexible fusion process by learning the weights directly, enabling more effective integration of information from both graphs.
D NONLINEAR RELATIONSHIPS AND EXPRESSIVITY
The ability to model these nonlinear relationships through multilinear polynomials provides signifi- cant expressivity for graph fusion. By including powers of the similarity scores, the fused graph effectively capture subtle nuances in the relationships between the modalities that a purely linear model might miss. This is especially important in cases where different modalities carry comple- mentary information. For example:
Complementary modality information. Consider video data with accompanying textual descrip- tions. The visual modality might capture general scene, while the text might provide specific context (e.g., objects or actions). Linear combinations of similarities might only highlight directly overlapping information, but higher-order polynomial terms can reveal deeper, context-driven rela- tionships, such as when visual cues and descríptive language align only in specific scenarios.
Disentangling complex correlations. Higher-order terms also help disentangle complex correla- tions between modalities. For instance, a quadratic term might reveal situations where both modali- ties strongly correlate with particular feature (e.g., object presence in an image and its mention in the text) only when viewed together, even individually their similarities are weak. The polynomial structure allows the fusion operator to amplify such interactions.
E LEARNABLE WEIGHTS AND OPTIMIZATION
The learnable coefficients Ap.g (p E Ip, E IQ) play a crucial role in determining how the contri- butions from different terms are weighted. During training, the model optimizes these coefficients
15

---

Published as a conference paper at ICLR 2025
through backpropagation balance the linear and nonlinear terms in the multilinear polynomial. This enables the model to learn the most relevant interactions for the task at hand, whether they be direct similarities (captured by lower-order terms) more complex, indirect relationships (captured by higher-order terms).
Multilinear fusion in practice. In practical applications, the multilinear polynomial fusion ap- proach offers several benefits: (i) Adaptability: The fusion process can adapt to different levels of interaction between modalities, adjusting the contribution of each term based on the complexity of the task. (ii) Robustness: By incorporating both linear and nonlinear terms, the fused graph is more robust to noisy or missing data. Even if one modality's similarity is weak, higher-order terms involving the other modality can still provide meaningful information. (iii) Improved task perfor- mance: Tasks such as classification, retrieval, or multi-mode modal feature learning can benefit from this fusion strategy, as it enables the model to use both direct and subtle modality interactions for better decision-making
F OPTIMAL HYPERPARAMETERS FOR EACH DATASET
Table 3: Optimal hyperparameters for each dataset.
Dataset
Operator
À
k
Best AUC
CUHK Avenue
& b
2
7
1
0.5
10
83.10
Shanghai Tech (ShT)
A
2
6
1
0.5
10
97.26
UCSD Ped2 (Ped2)
A
4
1
0.5
10
93.23
Street Scene
A
77.61
Combined (ShT + Ped2)
A
4
0.001
0.5
92.88
Selecting the appropriate hyperparameters is critical for achieving optimal performance across dif- ferent datasets. As shown in Table 3 we determine the best hyperparameters by conducting grid search across five datasets, including one combined dataset (ShanghaiTech and UCSD Ped2). The fusion operator corresponds two distinct learnable weight matrix representations: it can either be modeled as matrix A (see equation 5) the outer product of and b (see equation 4.. The remaining hyperparameters are fine-tuned through grid search to ensure the best performance.
G RELATIONSHIP BETWEEN EGO AND ITS WORLD APPLICATIONS
Iterative graph relationship updates play a crucial role in modeling complex interactions within graphs, enabling more refined and adaptive fusion of information. Unlike traditional methods that rely on direct connections, our approach incrementally enhances relationships through element- operations, allowing for more controlled and localized expansion of graph structure.
In video anomaly detection, iterative updates uncover subtle dependencies between frames by con- tinuously refining relationships over multiple iterations. This process improves the detection of anomalous behaviors that single-frame analysis might overlook by capturing evolving contextual cues.
For multi-modal data fusion, iterative graph relationship updates facilitate seamless alignment be- tween different modalities. For example, image-captioning tasks, textual descriptions may ref- erence background elements indirectly. Our approach adaptively strengthens such implicit connec- tions, improving the integration of visual and textual representations. Similarly, in social network analysis, iterative updates refine the understanding of information diffusion and influence chains. In- stead of relying on predefined connectivity patterns, this approach dynamically adjusts relationships, better capturing the evolving nature of interactions across network.
By expanding graph relationships in an iterative and structured manner, our method provides a more flexible and effective framework for capturing nuanced dependencies across various domains.
16

---

Published as a conference paper at ICLR 2025
Table 4: Experimental results XD- and UCF-Crime.
XD-Violence (AP)
UCF-Crime (AUC)
I3D visual
60.96
76.02
Text only
51.31
69.23
Feature-level
Concat.
59.93
67.84
Addition
58.79
68.93
Product
24.10
50.23
MIN fusion
77.17
85.14
13D visual
34.14
61.57
Text only
24.42
58.72
Graph-level
Concat.
27.89
63.16
Addition
27.66
65.30
Product
24.10
50.23
EGO (ours)
65.77
81.71
Table 5: Experimental results on the Multi-Scenario Anomaly Detection (MSAD) Dataset.
Venue
MSAD
MIST (I3D) Feng al. 2021
ICCV 2021
86.65
MIST (Swin (Feng et al. 2021
ICCV 2021
85.67
UR-DMU (Zhou
20231
AAAI 2023
85.02
UR-DMU (Swin hhou et al.
2023
AAAI 2023
72.36
MGFN (I3D) el al. 2023b
AAAI 2023
84.96
MGFN (S (Chen et 2023b
AAAI 2023
78.94
MTN (Ihe) et al. 2023a
CVPRW2023
86.82
MTN (Swin'T) (Chen et 2023a
CVPRW2023
83.60
EGO (ours)
87.36
H EGO WITH EXISTING FRAMEWORKS AND PIPELINES
EGO Fusion is designed with modularity at its core, ensuring compatibility with both (i) widely used machine learning frameworks, such as Py PyTorch and TensorFlow, and (ii) diverse model architectures. By using standard adjacency matrices and similarity metrics, EGO Fusion easily integrates with pre- trained models and raw feature pipelines. It can scamlessly integrated with vidco backbones such C3D, I3D, Swin'Tranfformer. as well GCN-based models for human skeleton sequences in pose modality (as demonstrated Table (). Additionally, EGO Fusion supports text embedding models, including SiMCSE.
I SCALABILITY OF EGO FOR LARGE-SCALE DATASETS
Computational complexity. EGO graphs are inherently small, such 32 adjacency matrices when a video is divided into N = 32 video clips (a.k.a. temporal blocks), Computing matrix-matrix multiplications with non-parallelized algorithm has complexity of 22.37 (Le Gall 2014). On GPUs (parallel hardware), this complexity is significantly reduced to Olog(N)) Consequently. the dominant computational complexity arises from equation 3 which is {( Q) log( Element- wise multiplications in equation are highly parallelizable, making their complexity negligible for typical values of P = Q 8.
EGO Fusion is thus highly scalable, using lower-dimensional relationship graph spaces to reduce computational overhead compared to feature-level (high-dimensional) fusion methods. Further- more, the learnable graph operator effectively balances self-connections with iterative graph rela- tionship updates. As shown in Table 2 EGO reduces training times by up to 50% compared to MTN on the ShanghaiTech dataset. For real-time applications, the learnable graph operator selec- tively refines key relationships to ensure computational efficiency. Additionally, sparse matrix (fast) representations can be used for further performance optimization.
17

---

Published as a conference paper at ICLR 2025
Table 6: Experimental results by anomaly type (11 main anomaly types) the MSAD dataset.
Assault
Explosion
Fighting
Fire
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM (Tian 2021
68.1
67.3
46.8
60.4
89.6
93.0
61.3
81.2
MGFN Chen et al. 2023b
59.7
59.0
64.5
71.9
89.4
93.5
86.0
93.0
UR-DMU (hou al. 2023
56.9
64.5
67.9
74.5
83.9
90.4
61.2
82.9
EGO (Ours)
52.2
57.5
57.6
74.4
66.5
72.8
62.9
86.7
Object Falling
People Falling
Robbery
Shooting
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM
Tian ct al. 2021
94.7
96.7
56.5
50.4
65.7
81.2
78.2
84.7
MGFN
Chen al.
2023b
90.9
94.8
52.7
47.8
86.7
86.8
88.5
UR-DMU Zhou el al.
2023
92.1
95.8
42.5
43.7
63.5
79.3
81.4
87.8
EGO (Ours)
92.3
94.8
35.4
43.8
64.8
87.5
68.6
78,4
Traffic Accident
Vandalism
Water Incident
Overall
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM (Tia el al. 2021
62.2
$1.8
85.2
76.1
96.3
99.1
66.3
MGFN (Chen et al.
20236
68.6
54.5
82.4
80.1
85.5
97.0
85.0
63.5
UR-DMU (Zhou et al.
2023
62.0
55.6
84.7
77.0
98.5
99.5
85.0
68.3
EGO (Ours)
69.9
64.3
88.1
81.4
81.9
95.4
87.3
64,4
Datasets. We evaluate EGO on large-scale anomaly detection datasets and have selected the fol- lowing: (i) XD-Violence: This dataset contains 3,954 training videos and 800 testing videos across various scenes, primarily from games or movies. It features anomalies such as abuse, explosions, fighting. and riots. (ii) (CF-Crime: This dataset includes 1,610 training videos and 290 testing videos of real-world multi-scene surveillance footage. It encompasses 13 types of anomalies, in- cluding fires, fights, and robberies. (iii) Multi-Scenario Anomaly Detection (MSAD): Introduced by Zhu et al.] (2024), this dataset consists of 360 training videos and 360 testing videos (Protocol íi) from multi-scenario surveillance footage. contains 55 types of anomalies, including assaults, explosions, and people falling.
Setups. For visual feature extraction, we use I3D pretrained on Kinetics-400 to obtain 2048- dimensional features. For text feature extraction, use SimCSE (as described in Sec. 4.1 to generate 768-dimensional text embeddings. On the MSAD dataset, we additionally extract 1024-
dimensional features using pretrained I3D and SwinTransformer (SwinT), respectively. Below, show evaluations on three large-scale anomaly detection datasets. We also present an evaluation by scenario (14 scenarios) MSAD, and the results (AUC) follow the Protocol ii of (Zhu et 2024
and 7), we draw the follow- ing insights: (i) EGO maintains high efficiency large and complex datasets, (ii) Compared to recent deep learning models, such as the latest MTN fusion (Chen et al. (023a), our EGO fusion demonstrates robustness in multi-scenario anomaly detection for static surveillance data. This is tributed to the ability of our relationship graph to effectively capture feature relationships, enabling the modeling of richer and more nuanced structural interactions that conventional fusion methods often overlook. (iii) EGO performs well on UCF-Crime videos, even in cases where the footage in- cludes frequently changing camera viewpoints. (iv) EGO is designed with computational efficiency in mind, using only 0.091M I parameters, significantly fewer than MTN (which uses 29.0M parame- ters). This reduction in parameters results in lower memory usage and faster inference times. Note that the weighted sums in EGO Fusion dynamically learned through task-specific optimization process, allowing for flexible, data-driven adaptation to the unique complexities of each dataset.
Evaluations. As shown in the large-scale evaluations (Table 4
Existing cross-modal graph methods (Yu et al. 2022 He et al. 2021) primarily rely on static inter-graph convolutions. In contrast, EGO captures higher-order relationships through iterative graph relationship updates: (i) We integrate both direct and iteratively refined connections. (ii) Equation 6 that the adjacency relationships between modalities are aligned via attention, i.e., G Ep E, Apo ( P R_ (iii) Our polynomial expansion scheme accounts for the fact that latent correlations within each modality may vary dynamically. Additionally,
Discussions.
18

---

Published as a conference paper at ICLR 2025
Table 7: Experimental results by scenario (14 total scenarios) the MSAD dataset.
Frontdoor
Highway
Mall
Office
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM (Tian 2021
84.1
81.1
63.7
4.1
87.2
72.2
78.1
68.8
MGFN CCenn et al. 2023b
86.4
85.1
79.7
4.1
65.3
56.6
75.1
62.4
UR-DMU (hou al. 2023
84.8
82.8
31.5
1.3
91.0
83.8
77.8
67.3
EGO (Ours)
85.2
81.6
80.2
30.8
82.3
73.4
80.0
71.7
Park
Parkinglot
Pedestrian st.
Restaurant
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM
Tian et al. 2021
69.0
25.6
74.4
35.9
97.4
50.6
96.1
91.9
MGFN
Chen et al. 2025b
77.9
38.3
68.1
14.5
88.0
20.4
95.8
91.8
UR-DMU Zhou 2023
87.8
36.2
91.4
53.9
81.9
11.5
93.1
87.4
EGO (Ours)
93.5
44.3
96.8
75.2
97.5
52.0
94.3
73.9
Road
Shop
Sidewalk
Street highview
Method
AUC
AP
AUC
AP
AUC
AP
AUC
AP
RTFM (Tian et al. 2021
54.0
77.3
52.5
17.1
43.3
12.3
MGFN (Chen ct al.
2023b
77.9
49.7
84.9
77.2
85.5
62.3
87.6
40.7
UR-DMU (Zoou et al.
2023
83.0
64.4
81.3
64.5
86.5
64.1
85.0
37.7
EGO (Ours)
89.8
64.6
83.4
72.2
87.1
45.0
28.2
10.1
Train
Warchouse
Overall
Method
AUC
AP
AUC
AP
AUC
AP
RTFM (Tian et al. 2021
66.9
3.9
69.5
37.4
86.7
66.3
MGFN (Chen ct al. 2023b
53.0
3.1
72.3
30.9
85.0
63.5
UR-DMU iZhou et al. 2023
59.0
3.1
81.2
59.1
85.0
68.3
EGO (Ours)
80.8
7.8
84.7
46.6
87.3
64.4
variance regularization loss in equation compares the oolynomial-aggreaatee variances of regular and anomalous G, novel aspect of our approach. In contrast, methods such (Yu et al. 2022] He et al. 202I) use fixed graphs and cross-attention graphs, respectively. The approach in (He et al.) 2021) also relies feature reconstructor losses per modality and triplet loss. Thus, these architectures differ fundamentally from ours. Note that (Yu et al. 2022] He et al.] (022]) are designed for remote sensing retrieval and image-text retrieval, whereas EGO is tailored for anomaly detection.
This richer interaction is reflected in our experimental results, where EGO consistently outperforms baseline methods, including state-of-the-aa cross-modal approaches, in anomaly detection tasks. Specifically, compared with other anomaly detection models: (i) Our EGO fusion consists of only two fully connected (FC) layers with ReLU activation in between, followed by Sigmoid activa- tion. (ii) In contrast, attention- based fusion operates at the feature level, which can be noisy. (iii) By operating the graph level, our approach is more robust, as demonstrated in the results above.
Additionally, uttention-basee fusion (He et al. 2021) typically requires three projection layers (for query, key, and value), leading fusion mechanisms are often constrained to single modality. such as the self-attention mechanisms used in models like MGFN (Chen et al., 2023b) and UR-DMU Zhou et al., 2023). In contrast, our EGO fusion demonstrates strong performance across mulli-representationaal multi-modal, and multi-domain feature fusion tasks.
significantly higher number of learnable parameters. Their
COMPARISON OF TRAINING AND TESTING TIMES
We compare the training and testing times per video across six datasets using both the EGO fusion and the recent MTN fusion approaches. All experiments conducted single Nvidia V100 GPU with a batch size 32. The training and testing times are measured at the sample level, representing the time required to process single video.
19

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of training and testing times for EGO and MTN fusion across six datasets.
EGO: Training (s)
EGO: Testing (s)
MTN: Training (s)
MTN: Testing (s)
UCSD Ped2
0.0878 + 0.0043
0,0033 # 0.0030
0,3656 # 0.0137
0,1256 # 0.0226
ShanghaiTech
0.1537 = 0.0160
0.0077 0.0041
0.4481 0.0232
0.1479 # 0.0127
CUHK Avenue
0.1472 + 0.0211
0.0078 = 0.0058
0.4338 + 0.0518
0.1384 + 0.0352
Street Scene
0.2064 + 0.0270
0.0173 = 0.0067
0.6179 # 0.0654
0.1596 = 0.0199
XD-Violence
0,2151 = 0.0440
0.0196 = 0.0089
0.5135 # 0.0361
0.1523 + 0.0122
UCF-Crime
0.2427 = 0.0361
0.0195 = 0.0071
0.6484 # 0.0492
0.1699 + 0.0257
Table 9: Performance of EGO in visual and text fusion under varying noise conditions on text features using the ShanghaiTech dataset. We report the Area Under the Curve (AUC).
Condition
Original
10% Noise
30% Noise 50% Noise
Train on Noisy, Test on Clean
97.26
96.01
95.98
95.58
Train Clean, Test Noisy
97.26
95.96
95.86
95.62
Train on Noisy, Test on Noisy
97.26
95.92
95.76
94.86
As shown in Table 8 EGO fusion achieves significantly lower training and testing times compared to MTN fusion, while delivering comparable or even superior performance in terms of results.
K ROBUSTNESS AND CROSS- DANSSET GENERALIZATION OF EGO FUSION
Handling noisy features. EGO effectively addresses noisy irrelevant features in the input data through its degree variance regularization mechanism (equation[)) 7 This approach promotes sparsity by diminishing the influence of weak or irrelevant connections in the fused graph, thereby helping to isolate and preserve meaningful relationships.
To evaluate EGO's performance under noisy conditions, we add Gaussian noise to the text features in the Shanghai dataset. The results are presented in Table 9 with the noise ratio indicated as percentage.
Our findings show that, in the presence of noise, it becomes more challenging to construct mean- ingful relationship graph, as the connections between features weaken with increasing noise. Nev- ertheless, EGO maintains high performance, with accuracy remaining within 2% of the baseline (Original) across all noise levels. This demonstrates EGO's ability to effectively isolate relevant fea- tures and minimize the impact of noise, even when the relationship graph is less reliable. The degree variance regularization enables EGO to prioritize stronger, more meaningful connections while de- emphasizing weaker irrelevant ones. This mechanism enhances the model's 's robustness in noisy environments, ensuring that noise has minimal impact on overall performance.
Non-robust feature extraction models. While high-quality features typically improve perfor- mance, EGO Fusion effectively mitigates the impact of less robust inputs by focusing on the relative relationships between features rather than their direct quality. Even when suboptimal features from lower-capace models are used, EGO's regularization mechanisms and learnable fusion operator help maintain stable performance.
Table 10: EGO performance different feature combinations. We report the Area Under the Curve (AUC).
Feature Combination
EGO
I3D + SwinT
89.85
I3D + C3D
87.17
SwinT C3D
85.52
I3D SwinT C3D
95.38
20

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of MTN fusion and EGO fusion performance in cross-dataset evaluation. Both models are trained on the ShanghaiTech dataset and evaluated on the UCSD Ped2, CUHK Avenue, Street Scene, XD-Violence, and UCF-Crime datasets. We report the Area Under the Curve (AUC).
Dataset
UCSD Ped2
CUHK Avenue
Street Scene
XD-Violence UCF-Crime 29.65
MTN fusion 50.49
46.99
28.94
35.08
EGO (ours) 48.03
49.35
36.76
30.52
57.84
As shown (Zhu et al. 2024), 13D and Swin'T features generally outperform C3D features, indi- cating that C3D features are less robust. Below, we evaluate EGO's performance when fusing C3D features on the ShanghaiTech dataset, with results provided Table TO These results suggest that fusing two features, one of which is less robust, leads to a slight drop in performance. For instance, I3D + Swin'T outperforms IDD C3D by more than 2%, and SwinT + I3D outperforms SwinT C3D by more than 4%. However, when fusing all three features (I3D + SwinT C3D), performance improves compared to any pairwise combination. These findings highlight EGO's robustness to variability in feature extraction quality.
Cross-dataset evaluation. In this section, we evaluate the performance of EGO fusion and MTN fusion using I3D visual features and text features, as described in Section 4.1 The models trained on the ShanghaiTech dataset and tested across several other datasets, including UCSD Ped2 (Ped2), CUHK Avenue (Avenue), Street Scene (Street), LD-Violence, and UCF-Crime.
As shown in Table I1 EGO fusion demonstrates strong performance in cross-dataset evaluations. Specifically: (i) Cross-dataset generalization: Despite being trained on the ShanghaiTech dataset, EGO fusion achieves competitive results on other datasets, such as 48.03 on Ped2 and 49.35 on Av- enue. (ii) Diverse scenarios: The results on datasets with varying motion patterns and complexities, such as XD- ED-Violence (30.52) and UCF-Crime (57.84), highlight EGO fusion's 's ability to adapt to datasets with diverse domain characteristics.
These results highlight EGO fusion's strong ability to generalize beyond the training dataset, con- sistently performing well across datasets with varying characteristies and challenges.
L ADDITIONAL VISUALISATIONS
Below, we provide additional visualizations normal and abnormal relationship graphs for visual features, text embeddings, and our EGO-fused representation selected video samples.
21

---

Published as a conference paper at ICLR 2025
10
9
7
16
11
5
12
13
13
13
3
14
15
16
16
16
17
31
18
30
18
30
18
30
19
19
19
29
20
20
20
28
21
27
22
23
24
25
28
13D visual features
(b) SimCSE text embeddings
(c) Fused relationship graph
10
13
13
13
14
14
14
6
16
0
16
16
30
18
30
16
30
19
19
1
20
20
28
(d) visual features
(e) SimCSE text embeddings
(f) Fused relationship graph
12
13
13
13
14
1
14
16
16
0
16
17
18
30
18
3
18
30
19
19
20
20
28
25
26
I3D visual features
(h) SimCSE embeddings
(i) Fused relationship graph
Figure 6: Comparison of relationship graphs on ShanghaiTech. The graphs are constructed using cosine similarity to represent relationships among features: (first column) visual features, (second column) text embeddings, and (third column) the fused graph, which integrates both modalities. In each graph, nodes represent clip-level (or unit-level) features, with numbers indicating the sequence order the video clips. Edges, shown in green, represent cosine similarity between features, with darker shades indicating stronger connections. Anomaly nodes and their connections are highlighted in purple. The fused relationship graph, generated using our EGO fusion method, effectively inte- grates visual and textual information into unified structure (see Figures (c), (f), and (i) for more details).
22

---

Published as a conference paper at ICLR 2025
13
13
14
6
16
16
18
30
18
30
18
19
19
19
20
20
20
24 26
23 24
13D visual features
(b) SimCSE embeddings
(c) Fused relationship graph
10
12
13
13
13
14
14
15
15
15
16
16
16
17
18
30
18
30
18
30
1
19
19
29
20
20
20
28
21
22
23
24
25
28
(d) visual features
(e) SimCSE text embeddings
(f) Fused relationship graph
10
10
7
6
12
13
13
14
14
14
15
15
15
16
16
0
16
18
30
18
30
18
30
19
19
19
29
20
20
28
21
27
24
22
25
22
23
24
25
26
I3D visual features
(h) SimCSE embeddings
(i) Fused relationship graph
Figure 7: Comparison of relationship graphs: the first shows UCSD Ped2, the second row shows CUHK Avenue, and the third row shows Street Scene. The graphs are constructed using cosine similarity to represent relationships among features: (first column) visual features, (second column) text embeddings, and (third column) the fused graph, which integrates both modalities. In each graph, nodes represent clip-level (or unit-level) features, with numbers indicating the sequence order of the video clips. Edges, shown in green, represent cosine similarity between features, with darker shades indicating stronger connections. Anomaly nodes and their connections are highlighted in purple. The fused relationship graph, generated using our EGO fusion method, effectively integrates visual and textual information into unified structure (see Figures (c), (f), and (i) for more details).
23


Reviews:
[{"review_id": "gW8wku8gpg", "reviewer": "Reviewer_jZ3p", "summary": "The paper introduces a graph-based method for multi-modal feature fusion in computer vision tasks. It shifts from high-dimensional feature spaces to interpretable graph spaces by constructing relationship graphs. This method leverages graph power expansions to capture multi-level interactions and introduces a learnable graph fusion operator for dynamic integration of graph powers. The approach is validated through experiments on video anomaly detection, demonstrating improved performance and deeper insights compared to traditional fusion techniques.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The graph-based fusion approach is innovative as it focuses on relationship-centric fusion, potentially capturing deeper interactions and structural relationships.\n2. The use of graph power expansions to model multi-hop connections is a strong point.\n3. Introducing a learnable weight matrix to dynamically integrate different graph powers is a key advancement.\n4. This paper is easy to follow.", "weaknesses": "1. While the method is sophisticated, its complexity might hinder interpretability. The relationship between graph power expansions and their real-world applications could be elaborated further for clarity.\n2. The scalability of the graph-based approach for large-scale datasets or real-time applications is not fully addressed. More discussion on computational efficiency would strengthen the paper.\n3. The integration of the LEGO method with existing machine learning frameworks or pipelines is not clearly addressed. If the possible combination of LEGO and the existing method is given, it will help the method to be widely used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NgI0MpKV07", "reviewer": "Reviewer_Fqv1", "summary": "This article focuses on multimodal feature fusion. The paper establishes separate graphs for each individual modality and then performs intra-graph convolution on each modality's graph, finally fusing the node features of different convolution depths. The authors believe that this approach can effectively achieve deeper fusion of multimodal features and avoid feature redundancy.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The illustration of the method is clear, and this method is easy to follow. The performance is credible.", "weaknesses": "Currently, there are many methods based on cross-modal graph neural networks that create their own graph networks for features from different modalities, and then use inter-graph convolution to obtain cross-modal embeddings, facilitating feature propagation and fusion in the process. \n\nCompared to these methods, the improvements presented in this paper seem to consist only of weighted sums at different convolution depths. The authors need to explain the advantages of this approach over previous methods.\n\n[1] Yu H, Yao F, Lu W, et al. Text-image matching for cross-modal remote sensing image retrieval via graph neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 16: 812-824.\n[2] He Y, Liu X, Cheung Y M, et al. Cross-graph attention enhanced multi-modal correlation learning for fine-grained image-text retrieval[C]//Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021: 1865-1869.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kU8EsgRDJJ", "reviewer": "Reviewer_Po6J", "summary": "LEGO is a novel graph-based feature fusion framework integrating multi-representational, multi-modal, and multi-domain relationships through relationship graphs, enhancing feature representations. It introduces a learnable graph fusion operator that dynamically combines different graph powers, facilitating deeper feature interactions and balancing self- and inter-feature relationships. LEGO theoretically connects graph fusion with multilinear polynomials, providing valuable insights into feature interactions. Empirical evaluations in video anomaly detection demonstrate that LEGO outperforms traditional feature-level fusion methods.", "questions": "1. Given that graph power expansion and tensor operations can be computationally intensive, how does LEGO Fusion scale with larger datasets? \n2. Since LEGO Fusion relies heavily on high-quality feature extraction, how does the framework perform when using features from models that may not be as robust or are trained on different datasets?\n3. How does LEGO Fusion handle noisy or irrelevant features within the input data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework seamlessly combines visual and text features, leveraging their complementary information to enhance anomaly detection. It utilizes graph power expansion and dynamically learns optimal weights for merging different graph powers, allowing the model to prioritize relevant relationships.\n3. LEGO requires significantly fewer parameters than traditional methods like MTN fusion, resulting in faster training times.\n4. LEGO consistently outperforms baseline and state-of-the-art methods across multiple datasets.", "weaknesses": "(See the questions section below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)