Towards context and domain-aware algorithms for scene analysis
Authors: Ibrahim Serouis
Florence Sèdes
Venue: Accepted by TMLR ()

Abstract: Interpersonal interactions and social situations in multimedia content encompass a rich blend of visual, textual, audio and contextual cues as well. However, contextual data integration in multimodal scene analysis research has often been overlooked, leading to incomplete interpretations. For instance, recognizing that two combatants in a video are positioned within a designated ring with a dedicated referee drastically alters the perception from a simple scuffle to a structured martial arts contest.

This paper presents an innovative approach to scene analysis in video content, which not only incorporates contextual data but also emphasizes the most significant features during training. Additionally, we introduce a methodology for integrating domain knowledge into our framework. We evaluate our proposed methodology using two comprehensive datasets, demonstrating promising results compared to a baseline study using one of the datasets. These findings underscore the importance of integrating contextual data into multimodal video analysis, while also recognizing the challenges associated with their utilization.

Published in Transactions on Machine Learning Research (01/2025)
Towards context and domain-aware algorithms for scene analysis
Ibrahim MOHAMED SEROUIS
sbrohim tbrshim.mohameddeee www.mmmmdddedeeddsiiiiitttte
CNRS, IRIT Université de Toulouse, Toulouse, France
Florence SÈDES
Soresri..edes@hiri...
CNRS, IRIT Université de Toulouse, Toulouse, France
Reviewed on OpenReview: oppereviee. net/ net/forum? id= JQGmb VK4
Abstract
Interpersonal interactions and social situations in multimedia content encompass rich blend of visual, textual, audio and contextual cues well. However, contextual data integration in multimodal scene analysis research has oflen been overlooked, leading to incomplete interpretations. For instance, recognizing that two combatants in a video are positioned within a designated ring with & dedicated referee drastically alters the perception from a simple scuffle to a structured martial arts contest.
This paper presents an innovative approach to scene analysis in video content, which not only incorporates contextual data but also emphasizes the most significant features during training. Additionally, we introduce a methodology for integrating domain knowledge into our framework. We evaluate our proposed methodology using two comprehensive datasets, demonstrating promising results compared to baseline study using of the datasets. These findings underscore the importance of integrating contextual data into multimodal video analysis, while also recognizing the challenges associated with their utilization.
1 Introduction
Within social situations, the human brain engages in a multifaceted analysis of situations and interac- tions, drawing upon a diverse array of sensory inputs. This cognitive process exlends beyond the realm of visual cues, encompassing auditory stimuli, environmental context, cultural influences, and educational backgrounds, thereby contributing to comprehensive understanding of social dynamics. In the growing landscape of multimodal learning rescarch, multitude of avenues has emerged, particularly in the realm of human-centric situation understanding and interaction analysis (Qodseya (2020) Kukleva et al. 2020) Liu
et al. 2019)).
Nevertheless, the literature has focused mainly on short actions sequences of frames KKndratyuk et al. (2021) Piergiovanni et al. (2022) and, most recently, as combination of textual and visual cues (Jiang al. (2018). Zhang al. (2022a)) or RGBD data (Xu al. (2023), Dai et al. (2017)) for studying situations consider truthful that more complete understanding of interpersonal interactions and social situations in broader sense only be achieved by incorporating contextual cues such as the context of the scene (for instance, courtroom drama, wedding, rehearsal, or humorous exchange between characters), its exact location, the relationships between the characters (e.g., friends, lovers, siblings, etc.), or some attributes of the characters such as their occupation, which can influence the power dynamics in the interactions. Also, reasoning on interconnected knowlodge for such problems would potentially allow for better interpretability than class activation methods (Selvaraju et al. (2017)), which be crucial in some scenarios.
in video content. However,

---

Published in Transactions on Machine Learning Research (01/2025)
Contributions. In this research, provide an interpretable methodology employing interconnected knowl- edge to capture high- level concepts like scene context and character relationships, while automatically high- lighting relevant features at training time. Our methodology is assessed on two scene analysis tasks, yielding promising results. We also provide a framework for integrating domain-knowledge in our methodology, which lew of the studies in scene analysis lackle, generalizable to many graph-learning situations.
The goal of our work is to put forward a comprehensive approach to guide systems towards deeper, higher- level and human-like understanding of interactions between humans. Our findings underscore the relevance of incorporating contextual data in scene analysis and illuminate the challenges that may arise from their usage. To the best of our knowledge, this study one of the rare works attempting to interpret scenes using context ual data, particularly through a knowledge-oriented approach, while also providing methods to integrate cultural factors or domain-specific insights.
Paper structure: Scction offers an overview of prior rescarch, detailing their contributions, limitations, and our study's positioning. In Section we formally outline the research question and evaluation tasks. Section delves into our approach to processing input data. Our methodology for learning various levels of information is elucidated in Section 5 followed by experimental findings in Section G Section 7 elucidates our method for integrating domain knowledge within our framework, followed by a discussion in Section g on the limitations of our approach, possible enhancements for our methodology, and the use and eaction of 4open.
contextual data. Code samples are available at :
2 Related work
Multimodal scene analysis. Previous studies in multimodal scene analysis has predominantly focused on the fusion of visual, textual, and audio cues to understand interactions within multimedia content (Stergion & Poppe (2018), Kukleva et al.] (2020), PPiergivaanni et al.] (2022) However, the integration of contextual data, such as the context of the scene or the relationships between the characters involved, has often been neglected in these studies.
Graph Neural Networks for scene analysis. Graph neural networks (GNNs) have emerged as powerful
tool for analyzing structured data, including multimedia content (Wu et al. (220 ). They have been suc- cessfully applied to lasks such as image classification, object detection, action recognition (Li et al. (019b)). Shi et al. (2019). Zhang et al. (2020), Zhang et al. (2022b)) ) and video understanding. However, few of these methods included contextual information, and there is a scant exploration of this direction in the literature.
Context integration in video analysis. Some recent efforts have begun to explore the importance of contextual information in video analysis. For instance, |Wang et al. (2016) proposed framework for action recognition in vidcos that incorporates spatial and temporal context. Similarly, Zhang (2023) introduced a method to recognize skeleton-based human activities in videos by considering the scene context as defined by a combination of the action and the location where it occurs, highlighting the potential benefits of integrating contextual data for more accurate analysis of video content. However, these methods did not inchude additional contextual information such as the relationships between the characters on-screen, or the potential speech during the interactions. They and are limited to action recognition, few provide insights on the importance of each feature, and do not provide framework how to include domain-specifte knowledge into their methodology.
By building upon and addressing the limitations of existing research, our proposed approach aims to advance the field of multimodal scene analysis by effectivcly integrating contextual data and possibly domain-spociftc knowledge insights into the learning process.
3 Objective of the framework and evaluation tasks
Consider a graph G = (V,E) consisting of various sets of vertices V and edges E that connect the ver- tices, such as E C (x,,)) (a, y) E 2 and æ + y) where æ and y are neighboring nodes. In various applications, such as scene analysis, graphs may exhibit heterogeneous structures, with vertices and edges of diverse types. For instance, nodes representing characters within scene may convey distinct information
2

---

Published in Transactions on Machine Learning Research (01/2025)
compared to nodes representing the context of the scene. However, in the absence of domain-specifie knowl- edge, discerning the significance of different types of connections or nodes poses a challenge. In the context of a training deep learning model on such graphs, how can we leverage the varying levels of information encoded within these node sets, and assess the importance of crucial connections or features during training? Moreover, how could we adapt our learning methodology to integrale domain-specific knowledge?
The global objective of our framework is to propose a methodology that integrates various contextual data while addressing this challenge. Our proposed approach will be assessed through two distinct tasks, objec- tification detection and interaction classification.
3.1 Objectification detection
Objectification detection a high-level interpretative task that consists to predict whether a scene was objectifying or not, meaning that a character was portrayed as a mere object in a scene, either through dialogues that reduce the character to its physical attributes, narratives that treat the character as a prize be won or a tool to be used, camera angles that focus on specific body parts. We evaluate our approach the ObyGaze12 12 dataset (Tores et al. (2024)), which comprises over 1600 scenes of varying lengths annotated with respect to 4 objectification tags: Easy Negative (453 samples) for scenes in which the assessors held a high degree of confidence that objectification was absent, Hard Negative (711 samples) for scenes in which the asscssors exhibited a reasonable level of confidence regarding the absence of objectification, Not Sure (397 samples) for scenes presenting ambiguous characteristics, thereby leading to uncertainly among assessors regarding the presence or absence of objectification, and Sure (353 samples) for scenes where the assessors had strong conviction that objoctification was unequivocally present. The scene graphs encompass scenes, places, contexts, characters along with their attributes, and the objectification lags. Domain-specific knowledge and intepretability insights can be crucial for this kind of task, especially juxtaposed to the opinion of expert annotators on the reasons for objectification furnished in the dataset. Classification is performed on the Scone node, which contains the objectification tag.
3.2 Interaction classification
Given input graph containing scenes, places, contexts, characters along with their emotional states, and interactions between characters including their dialogues, this task aims to classify the interactions accurately. We employ the MovieGraphs human-centric situations datasct (Vicol et al. (2018)) for evaluation, which offers graphs with 101 annolated interaction classes (e.g asks, fights (wilh), hugs, Uhrealens, explains (to)...) spanning more than 17,000 instances. It is a wide collection comprising 7637 scenes of varying lengths from 51 movies (with release dates ranging from 1945 to 2014) represented in a graph format. These graphs encapsulate comprehensive scene-specific information, encompassing character identities, physical and personal/filmm attributes (such as sex, occupation, age, ethnicity, appearance), character relationships, interpersonal interactions, and chronological markers. Additionally, the dataset includes separated subtitle files for each movie. For exhaustive details on the dataset, refer to the original study. Classification is performed on the Interaction node, which contains the interaction class.
4 Dataset processing
4.1 Speech features
Although the identity of the speaker is not mentioned in both datasets, we extract the dialogs and map each interaction with the corresponding dialogs if there is overlap lasting more than one second or equivalent to the duration of the interaction. Subsequently, we generate the representation of dialogs using a BERT
DDvlin et al. )219)) embedding model.
More specifically, we run an inference on each line for each dialog on the tokenized version of the speech file. Subsequently, leveraging Transformers (Wolf et al. (2020)) BERT embedding model for sequence classification, we extract the hidden states of all the lines. Finally, we perform a Max Pooling operation Gholamalinezhad & Khosravi (2020)) over one dimension of the text embedding here obtained, and pass the
3

---

Published in Transactions on Machine Learning Research (01/2025)
output to two fully-connected (dense) layers with relu and tanh activations, to obtain the final representation. While more recent. models exist for extracting textual features, models using the simpler BERT-base uncased produced better results for both tasks.
4.2 Interaction graphs generation
MovieGraphs dataset (Vicol al. (2018)) contains scene graphs with at least one interaction per graph. Mcanwhile, one of our rescarch objectives entails the classification of individual interactions.
To address this challenge, we have devised an approach that involves the creation of a dedicated graph for each interaction, guided by two fundamental rules: (1) characters who are not involved in the current interaction are not included in the interaction graph, and (2) any clement present in the original scene graphs that pertains to the entirety of the scene or lack a chronological indicator (timestamps), will be linked to each corresponding derived interaction.
Subsequently, the interactions and relationship classes are grouped respectively in 101 and 15 similar cat- egories, identically from baseline study [Kukleva al (2020)). This process results in an interaction graph as shown in Fig 1 which is generated from the complete scene graph and retains exclusively pertinent or interconnected information. The here obtained graphs are then serialized and stored in a .Ifrecord files, which will serve as the data source for training the Tensorflow (Abadi et al. (2015) models.
description: string
name: string
Emotion
description: string
value: string
Context
Attribute
dircmstance
summary: string
name: string
name: string stringattribute
value: string
Attribute
Character
involves role: string
Interaction
has
Scene
linked linked 19
Tolc: string
id: string
where D
hat speech
has speech
why
Place
involves role: string
Relat onshirpppp string
description: string
Reason
Speech
Speech
suns Tole-
linked
transcript: [string]
description: string transcript: [string]
string sttrrbut value: string
Character
name string
Figure 1:
Typical interaction graph. The grey node represents the root node (Interaction Entity). All the nodes represent elements that pertain to the entire scene or, at least, the singular interaction.
4.3 Data split
For the interaction classification lask, we reproduced the same data split as in the baseline study, LiREC (Kukleva et al. (2020)), leading to roughly 12000 training samples and 4000 validation samples. For the ob- joctification detection task, 60% percent of the dataset was allocated for model training, while the remaining 40% was reserved for the validation of results. It is worth noting that alternative partitioning strategies could have been employed had larger dataset been available.
4

---

Published in Transactions on Machine Learning Research (01/2025)
5 Proposed framework: learning process
In this section, we introduce our methodology, a three-step approach grounded in Graph Neural Networks (GNNs). It offers versatile pipeline for training on various levels of information and exploring various graph architectures, without substantial deviations from the foundational structure. It establishes a cohesive framework for the models, founded on the principles of feature mapping, as expounded 5.1 followed by graph updates, elaborated in 5.2 and culminating the classification and interpretability processes, detailed and 5.4
The model's architecture is shown in Figure illustrating the approach for objectification detection. The sole distinction between the two methodologies lies in the penultimate dense layer: it features kernel size of 512 for objectification detection and 1024 for interaction classification. These parameter values were selected based on achieving optimal experimental results.
Nodes their attributes
embeddings
Input Contexts, Relationships, Speech...
Node 1
If flag=true
Altibutes
Adjacency matrix
Node
embedding
Feature mapping
Node 3
If flag=false
Emply
Atbilates
map
GCN update : for each node in the sets nodes of interest
Message
passing iterations
Graph update
Updated weights Dense(convolution (messages from neighboring nodes) pooling))
Readout Dense layer (root node): kernel <64x512>
Inference Dense (Readout): kernel <64x512>
Classification
P(root_node=XX
...
P(root_node=Y)
Probability the root node to belong each class
Figure 2: Proposed framework. Each phase of the proposed framework is depicted by at least one layer (rectangular boxes), and its shape is explicitly defined within the '<>" symbols.
5.1 Feature mapping
The feature mapping phase is what separates our framework from other graph neural networks learning cycles (i.e Feature Mapping, then Graph Updates, then Classification).

---

Published in Transactions on Machine Learning Research (01/2025)
In this preliminary stage, we establish set of transformations designed for the individual nodes edges within our graph. These transformations are contingent upon the particular type of node edge, enabling us to craft the feature representation through tailored adjustments.
The selection of node and edge types: the models have been trained on the 2 potential combinations of input node states (with 7 corresponding to the types of nodes in both datasets), utilizing an automated script and training flags to specify the usage of particular node information during the training process. We loop through the list of nodes sets, check the value of the flag for each node type, then assign its representation based on the flag. The training flags exhibit binary values, with false denoting absence of information pertaining to the current node (we then generate an empty or masked tensor), denoting the presence of an emply information that shouldn't be treated as proper weights, or true, meaning that a proper representation will be computed using a <50000x32> embedding layer. This process is detailed in Algorithm a Algorithm
The initialization of the initial state of each node type: Node sets other than the subtitles are embedded through an embedding model with vocabulary size of 50000 and dense embedding size of 32, yielding their initial state.
This phase culminates in the generation of the graph representation matrix which is derived from the com- bination of the adjacency matrix and the nodes features set vector.
5.2 Graph update
Herein, we make decisive architectural choices that tailor the model to our algorithm. The four primary ar- chitectural designs for Graph Neural Networks were explored for our specific problem: and Graph Attention
Networks (Dwivedi & Bresson (2021)), Graph Convolutional Networks (GCN) (Kipf & Welling (2017)
GraphSAGE works (Hamilton et al. (2017) and Vanilla. Message Passing Neural Networks (annilla
MPNN) (Gimmer et al. (2017)), with layer support provided by Sensorflow-GNi (Ferludin et al. 2022
Further insights into each architecture are claborated in Appendix A
The Graph Update phase iteratively refines the node and edge features, enabling the GNN to learn more abstract and informative representations from the graph structure and the feature data. During our graph update phase, masked or emply nodes are updated based on the aggregated information (embeddings) from their neighboring non-masked nodes.
This phase ultimately yiclds collection of hidden states corresponding to cach pair of nodes and edges within the graph representation. This collection is represented by layer whose configuration varies in accordance with the chosen model architecture. The number of the parameters on this phase vary depending on the specified training flags.
5.3 Classification
Final stop, based on the previously acquired representation, it consists in the determination of the class assigned to the root node (Interaction node for interaction classification, Scene node, for objectification detection).
Comprising two sub-steps, this phase maintains uniform structure, irrespective of the selected graph update technique:
Retrieving the hidden state of the root node produced through the graph update phase, with a readout function.
2. Strictly speaking, classification of this state via fully connected layer classification head: pre- diction of class of the root node. Its formal definition is as follows:
i = R(H GG)
(1)
Where & is the prediction, R the readout function, H. the hidden states for the node sets in v, and the sets of nodes used for the readout.

---

Published in Transactions on Machine Learning Research (01/2025)
This phase yields the probability of the graph belonging to each class as its output.
5.4 Interpretability insights
In addressing the inherent black-box nature of Deep Learning models, known for producing results without clear interpretability (Hussain (2199)), we introduce an interpretability sub-module. This sub-module aims to elucidate the influence of input features on the final output of the model.
To derive these insights, we calculate the sensitivity or gradients the last hidden states, with respect to individual elements the weight matrices the previous layers. Tt. provides global view of the impact of each input element on the output. This operation can be articulated as follows:
(h++)
Sensitivity = aw
(2)
Where (h++1) are the hidden states of the layer 1+ (last layer), P and q the indexes for respectively the rows and the columns of the weight matrix, and wiy the element the p-th row and the q-th column of the weight matrix W
This computation results in sensitivity matrix for each weight matrix, where the elements aggregated by the mean of the elements, akin to some implementations of GRAD-CAM (Selaaraju et al. (2017)) ) yielding
score. Given that the messages during the graph update phase may possess diverse dimensions, each
sensitivity score is normalized by the reciprocal of the number of rows in its corresponding message:
Score Sensilivily
(3)
6 Experimentation and results
6.1 Implementation details
GNN implementation. In all considered architoctures, key parameters remain consistent, including the dimensionality of message exchanges between nodes and their neighbors across graph update layers, set at 64. Similarly, the number of iterations for message passing remains uniform at 1, does the message pooling method, employing sunmation across neighboring nodes. For GraphSAGE, GCN and GAT models, the common choices are related to the nodes where the convolution (for GCN and GAT) or neighborhood sampling and aggregation (for GraphSAGF) are computed. For our study, we convoluted or sampled over the nodes of interest: Interaction, Character, and Scene, chosen due to their extensive connectivity with other graph components.
Loss function. Our models are trained using sparse categorical cross entropy loss function (Mao et al. )) which is variation of the (categorical) cross entropy that uses integers instead of onc-hot vectors for the classes oulputs. Its formal definition is follows:
N
[ww) = N
1
Σ
Ie log(ū:) + - 1)) log(1 9:)
(4)
Where N is the number of samples, Yt represents the true class, and Sr the predicted class.
Training parameters. The batch size for interaction classification was adapted to our bascline study, LIREC (Kukleva et al. (2020)). Both models were trained using an Adam oplimizer for 200 epochs; however,
training was automatically halted using early stopping (Yao et al. (2007) once the monitored value (accuracy or binary accuracy) on the validation set exhibited no further improvement for a consccutive period of 40 epochs. For a comprehensive overview of the model implementations, please refer Table [ which summarizes the pertinent details.

---

Published in Transactions on Machine Learning Research (01/2025)
Table Implementation details for both tasks
(a) Interaction classification
(b) Objectification detection
Parameter
Value
Parameter
Value
Optimizer
Adam * 10e
Optimizer
Adam
Learning rate
Learning rate
cosine decay(10 4800 steps)
Batch size
64
Batch size
32
Early stopping
top-5 accuracy
Early stopping
Validation accuracy
Epochs
200
Epochs
200
Dropout rate
0.1
Dropout rate
0.2
6.2 Evaluation metrics
Accuracy. Used for both tasks, it quantifies the frequency at which the model achieves correct prediction, meaning that the class predicted by the model corresponds to the actual class. The mathematical expression for computing this metric is detailed as follows.
Top-5 accuracy. Variation of the top-n accuracy metric, this approach assesses how often the class predicted by the model falls within the top 5 probabilities when prediction is indeed accurate. We use it match the metrics given by our baseline study. In the context of problem where classes are loosely annotated, such as asks, answers, lalks, and explains, the relevance of top-5 accuracy becomes evident. This metric proves valuable because distinguishing between the latter two classes can be particularly challenging even for human.
Used for objectification detection, it measures the difference between two probability distributions: the true distribution and the predicted distribution.
Loss (cross-entropy loss), as defined in Section 6.1
Area Under the Curve (AUC). We use it evaluate the objectification detection task. When presented with randomly chosen positive and randomly negative instance, the Area Under the Curve (AUC) rep- resents the probability that the classifier possesses the capability to discern their respective classes (Fawcett (2006)). It is the area under the Receiver Operating Characteristic (ROC) curve, which is graphical rep- resentation that illustrates the performance of the True Positive Rate (TPR) in comparison to the False Positive Rate (FPR) (Monaghan al. (202T)) at various threshold values The AUC can be formally defined as:
AUC (PR(FPR-'(2)) (z))dz ==
(5)
When determining the best models, a comprehensive evaluation considers all of these metrics. For instance, models exhibiting elevated accuracy rates in both training and validation stages, yet possessing an AUC score < 0.5 are deemed ineffective, due their inability to provide discrimination capacity beyond that of random chance.
F1 score. It is measure of test's accuracy that combines both precision and recall. It ranges from 0 to 1, with higher score indicating a more accurate test. It's calculated as:
F1 = (Precision * Recall)
(6)
(Precision + Recall)
6.3 Results
6.3.1
Objectification detection
Based on the results presented in Table 2a when considering all pertinent metrics, our best models have demonstrated the capability to attain accuracy levels substantially exceeding those of random chance (i.e, 50% accuracy score), indicative of their efficacy the binary classification task. However, due to the
8

---

Published in Transactions on Machine Learning Research (01/2025)
class imbalance with 353 instances categorized Sure and 453 Easy Negative, cannot solely rely accuracy. Therefore, the utilization of the AUC metric is warranted in addressing this issue. Leveraging the ROC curve, as advocated in previous empirical investigations (Fawcett (2006)) for its capacity to greatly but not completely decouple classifier performance from class skew, they also exhibit a great level of performance, exceeding those of random classifier (í.e, a 0.5 AUC score). is noleworthy that GCN (graph convolut ional networks) models, using combination of contextual (f.e, scene graphs) data and speech data, exhibited superior intrinsic performance compared to our baseline Tores et al. (2024), and MPNNs showed Fl-score levels similar to our baseline.
In order to mitigate class imbalance, we oversampled the minority class (casy negative examples). We managed to obtain better results for all the models, except for GraphSAGE models. As we can observe in Table 2b] GCN and GraphSAGE models did not need the introduction of speech features to achieve better results when using oversampling. However, all the models shown a slight rise their validation loss, but they managed to outperformod our baseline study in terms of Fi-score.
Table 2: Best results on the validation set for the objectification task, for each architecture. SSG=Seenc Graphs, S=Speoch, I=Image, NC=Not Communicated)
(a) Results without oversampling
Model
Data
Accuracy (%)
AUC
Fi-score
Loss
GCN
SG S
65.62
0.64
0.81
0.70
Tores ct. al
N.C
N.C
0.79
N.C
MPNN
SG
60.93
0.63
0.79
0.76
SAGE
SG
63.28
0.62
0.77
0.81
(b) Results with oversampling
Model
Data
Accuracy (%)
AUC
F1-Score
Loss
GCN
SG
66.40
0.69
0.81
0.76
MMPNN
SG +
62.10
0.66
0.8
0.90
SAGE
SG I
61.32
0.6258
0.8
0.75
Tores et. al
N.C
N.C
0.79
N.C
Utilizing the sensitivity scores delineated in Section 5.4 we can offer insights into the docision-makine process of models. Based the scores provided in Table 3a we infer that optimal model (GCN) assigns greater significance to the weights associated with the activity context of the scene and the altributes of the featured characters, while altributing minimal importannce to biases related to the location of the scene in the decision-makinn making process. It is important to note that the model's 's heightened emphasis on the context is promising, aligning with its prominence in the decision- makinn process observed by expert annotators (Toees et al. 2024)), ) as outlined in Table 3b
(a) Sensitivity scores for the best model.
(b) Top reasons (concepts) for objectification high-
(W- Weightss B-Biases.)
lighted by expert annotators
Unit
Score
Reason
Appearances
(W) Activity/ Contextt of the scene
3.27
Speech
410
(W) Attributes (Body, Clothes, Occupation)
2.12
Activities
156
(B) Location of the scene
0.015
Clothes
112
(W) Relationship between characters
0.004
Body
86
Table 3: Comparative analysis of sensitivity scores and top reasons for objectification
9

---

Published in Transactions on Machine Learning Research (01/2025)
6.3.2 Interaction classification
The findings presented in Table demonstrate that, by exclusively leveraging contextual data, we can achieve top-1 and top-5 accuracy levels that outshine the performance achieved by utilizing both visual and dialog data. This observation underscores the significance of contextual information our classification task. Moreover, the incorporation of dialog data resulted in notable enhancements in training and validation accuracy. The rise in raw accuracy can be attributed to the enhanced performance achieved for certain prevalent verbal interactions within the dataset. An illustrative example is the asks class, which accounts for approximately 14% of the training and validation set.
Table 4: Results on the validation set for the best models, sorted by top-1 accuracy (V= (V= D=Dialogs, F=Face tracks, SG=Scene Graphs). To ensure fair comparison, adopted the same batch size, training, validation, and test data splits our baseline study, LIReC Kukleva et al. (2020)):
Model
Data
Accuracy (%)
Top-5 accuracy (%)
LIReO (baseline) LIRcC
18.7
45.8
22.4
50.6
LIReC
V D
25
54.8
LIReC
V + D
26.1
57.3
GCN
SG +S
33.4
68.12
Ablation study
Our ablation study focuses on objectification detection. In this section, we will compare the results on the validation set for the models without each contextual cue, and juxtapose them the results of the sensitivity scores outlined in Table 3
The results presented in Table are largely consistent with our sensitivity scores calculations. Models without the strongest cue, the context of the scene, yielded the poorest overall performance (marginally lowest AUC and accuracy). The recond-worst results were observed when the characters' attributes were climinated. A slight improvement in results was achieved by removing only the scene location, while the best results were obtained by excluding the scene location, and the relationship between the characters. These findings could potentially corroborate insights derived from the sensitivity scores for each contextual cue.
Table 5: Mean results on the validation set for the ablation study.
Removed cuc
Accuracy (%)
AUC
Loss
Activity/ CilvvityCContex Context of the scene
55.98
0.50
0.74
Attributes (Body, Clothes, Occupation)
56.09
0.54
0.76
Location of the scene
56.39
0.54
0.75
Relationship between the characters
57.22
0.54
0.75
7 Integrating domain-knowledge
In various real-world scenarios such healthcare diagnostics or financial risk assessment, domain know]- edge plays a pivotal role. Similarly, in our objectification delection lask, understanding the nuances of objectification and its contextual implications crucial. However, integrating domain knowledge into com- plex architectures poses a significant challenge, and few studics have addressod this aspect their proposod methodologics. Aside from traditionally assigning edge weights, few studies approach this problem for Graph Neural Networks. To address this gap, we propose two practical approaches for seamlessly integrating do-
10

---

Published in Transactions on Machine Learning Research (01/2025)
main knowledge into framework: (1) Node Ranking, detailed in Section 7.1 and (2) The probabilistic
approach, explained in Section 7.2
7.1 Node ranking
Collaborating with domain experts enables us to discern the significance of specific nodes within the dataset. By assigning ranks to nodes based on the importance of the information they represent, we can then impose constraints on the weights of certain layers. For instance, we may stipulate that the mean absolute value of weights associated with igger-ranked nodes (ma) should exceed those linked to lower-ranked nodes (m)). If the condition is not satisfied, we penalize the weights W; of lower-ranked nodes by a values, such as:
W; = W; m
my
Consoquently, the contributions of highly ranked nodes to the overall inference process would be prioritizod over those of less ranked nodes. These constraints can be seamlessly integrated into popular graph neural network (GNN) frameworks found in the literature, such DGL (Wang et al. (2019)), Torch-Geometric (Fey & Lenssen 22019)) ) or Tensorflow-GNN (Ferludin et al. (2022)). Howver, early iterations of this process did not manage to increase the results, although performed without domain experst.
7.2 Probabilistic neuro-symbolic approach
Given some specific conditions, we can create probabilistic problem in the form of Bayesian Network exploiting the outputs of the models, with tools such as Problog (De Raedt et al. (2007)).
For instance, in the context of objectification detection, the model output (probability of the scene being objectifying) can serve as the initial probabilistic fact (prior knowledge). Subsequently, we can update this knowledge by incorporating domain-knowlede insights regarding the likelihood of a scene being objectifying given certain factors, such as the presence of naked woman or the depiction of domestic activity.
The core concept. of this idea involves establishing a set of probabilistic facts Py lo refine prior knowledge (in our case, specifically the likelihood of a scene being objectifying). For each probabilistic fact Pfi, we define a confidence interval [Pf:- Pf;+ Pf+ +f++PA Pft Pf** * a]. The process then involves performing inferences on the values within this interval, beginning at the lower bound (or start and the higher bound) and incrementally increasing/ decrensinnneennnngggggg each probabilistic fact Pft by k. The process can be halted by early stopping if the results do not increase according the selected metrics.
An examination of the distributions of the results pre and post-update using KL divergence Kullback & Leibler (1951)) for instance would provide insights on the updated results. We can also infer uncertainty on the updated distribution post-application using uncertainty estimation (Shannon (1448))
Early iterations of this process, applied to the objectification detection task, managed to increase both accuracy (from 66.2 to 70.2) and AUC score (from 0.69 to 0.81) metrics. However, it was done without a team spocialized in the matter, which can be crucial for the task. Nonetheless, we are currently collaborating with experts in social sciences and humanities integrate more expert domain-speeific knowledge. The architecture of the proposed framework is available in Appendix E
8 Discussion
8.1
Limitations
For objectification detection, top-performing models (GCNs) consistently struggled in scenes involving spe- cific combinations such as [Type of Plan+Soundlrack) [Soundtrack+ as reasons for objectification (less than 10% accuracy), possibly due to insufficient data. Also, certain infor- mation such as the type of plan and narratology cannot be effectively used by the models. For interaction classification, top-performing models consistently failed on detecting interactions heavily influenced by visual factors, such as walks, grabs, or holds.
or solely Soundtrack or Voice
11

---

Published in Transactions on Machine Learning Research (01/2025)
Also, our interpretability framework at the moment fails to capture the importance of speech nodes for Graph Convolutional Networks. This might be due to some instances with unconnected gradients ween the embedding of the speech and the final output, or NaN losses in convolution layer. We aim to mitigate this effect on future iterations.
8.2 Integrating image features
Initially, attempted integrating frame features extracted using X-Clip (Ni et al. (2022)) for computing the representation of the clip directly into model training, by introducing a "frame" node. However, this method resulted in a significant increase in loss for the objectification detection task, from 0.8 to an average of 10, with only a marginal 0.1% improvement in accuracy. Moreover, it substantially elevated memory consumption during graph embedding and update phases, nearing the limits of our computing unit.
For the second approach, we trained the models on frame features for each movie and combined outputs with dense layer. While this approach led to reduced errors, it resulted in slight decline in binary accuracy. Additionally, we encountered challenges post-training, particularly in computing gradients for models, and sometimes, training failures due to kernel crashes caused by memory leak. Similar trends were also observed in the interaction classification task.
The absence of improvement with the incorporation of image features may be attributed to the elements of the graphs being annotated predominantly by annotators after thorough video analysis, as stated in the original study by Vicol al. (2018), lcading to potential rodundancy in information.
8.3 Contextual data extraction for scene analysis
Acquiring contextual data, particularly regarding character relationships, is more feasible within film data compared to typical social scenarios. However, the associated costs may hinder extensive research in this direction. For instance, the MovieGraphs dataset offers comprehensive human-annotated classifications, curated through labor-intensive processes. Yet, this approach may not be accessible to all researchers due to its time and resource intensity. Alternative methods, such as employing visual place recognition algorithms (Zhang et al. (2021) or leveraging contemporary computer vision techniques for age and gender extraction (Huang et al. [2017 Dong al. (2016); i Jia et al. (2016)), could streamline and automate data collection. Additionally, emotion and relationship recognition algorithms [oo (2018): Kukleva et al. (2020)) provide further insights. However, the context of the scene remains hard to extract in the cases where metadata are not provided or human input is not present.
While AI models may accelerate data annotation, human intervention remains crucial for specifying character relationships. AI may struggle with nuanced contextual interpretations, such as distinguishing between genuine and faked emotions. Thus, human input is indispensable for refining extraction outputs. Also, it is important to pay attention to the potential for cumulative errors with multiple extraction algorithms. While contextual data can yield a deeper level of interpretation, it is imperative to emphasize the crucial role played by the establishment of precise and unambiguous guidelines for annotating and collecting data, in their successful integration into the realm of multimodal scone analysis and deep learning. Notwithstanding, a comprehensive pipeline for extracting contextual data will be addressed in a forthcoming publication.
9 Conclusion
This research has been dedicated to the integration of contextual cues for scene analysis, driven by the imper- ative to augment the comprehension of social scenarios by computational systems beyond the traditional use of image and text. To address this, we introduced an innovative approach that integrates contextual infor- mation into our analysis pipeline, yielding better results than a state-of-the-art algorithm on an interaction classification task.
Starting from the promising results here obtained, significant improvements would be achievable with more data-centric approaches. In particular, we could reduce the human-annotated target classes within the
12

---

Published in Transactions on Machine Learning Research (01/2025)
interaction classes to enhance the generalization capabilities of the model, iterate more balanced dataset.
Moreover, an intriguing avenue for future investigation involves proposing a pipeline to automate the extrac- tion of contextual data using cutting-edge techniques. Additionally, we aspire to collaborate with experts in objectification and various social science domains to leverage their domain-specific insights in refining our methodology. We aim to explore these aspects in our future research endeavors, and apply our framework to diverse datasets and human-centric situations tasks.
Broader Impact Statement
As our methodology continues to evolve, we anticipate expanding its scope to tackle pertinent societal issues such as the depiction of sexism and racism in films or scenes. By employing artificial intelligence techniques for high-level interpretative lasks in scene analysis, we aim to contribute to the advancement of media landscape characterized by inclusivity and respect for all genders and races. This integration into production, evaluation, and moderation practices holds promise for reshaping the media environment positively.
Failure to address these aspects meticulously may result in perpetuating biases present in the training data, lcading to misinterpretations or the reinforcement of existing stcreotypes. This, in turn, could inadvertently exacerbate social issues rather than ameliorating them. Possible solutions that we may envisage for mitigating this issue are:
Diverse and representative data: ensuring that training datasets are diverse and representative of various demographics, cultures, and perspectives. This helps mitigate bias by providing a more comprehensive understanding of the data distribution.
Regular audits and evaluations: conducting regular audits and evaluations of Al systems to assess their performance, identify any biases or unintended consequences, and take corrective actions as necessary.
Education and awareness: providing education and training programs bias, diversity, and inclusion for Al developers, users, and stakeholders. This helps raise awareness of potential biases and encourages proactive efforts to address them.
Transparency and accountability: maintaining transparency in the development and deployment of Al systems, including disclosing the data sources, algorithms used, and decision-making processes. Establish mechanisms for accountability to address any unintended consequences or biases that may arisc.
Privacy protection: As we deal with human content, data should be anonymized in order safeguard individual's entities and ensure ethical data handling practices throughout the rescarch process.
Acknowledgments
This work has been supported by the French National Research Agency through the ANR TRACTIVE project ANR-21-CE38 00012-01.
References
Marti Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Πya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distríbuted Systems. 2015.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Compuler Vision and Pattern Recognition (CVPR), IEEE, 2017.
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: a probabilistic prolog and its application in link discovery. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAT07, pp. 2468 2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. event-place: Hyderabad, India.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheyust. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding, 2019. URL arXiv:1810.04805 [cs].
wttp://wwww....aaa/////./11 04805.
Yuan Dong, Yinan Liu, and Shiguo Lian. Aulomatic age estimation based on deep learning algorithm. Newrocomputing, 187:4-10, 2016. ISSN 0925-2312. doi: htw.:////..eecom.1155..... 7/www. [tw.aaaccciiiiic....oooaaaaaaaaa
URL https:
Vijay Prakash Dwivodi and Xavier Bresson. A Generalization of Transformer Networks to Graphs, January 2021. URL http:///arii..og//a///221......3
arXiv:2012.09699 [es].
Tom Fawcett. An introduction to ROC analysis. Pattern Recognation Letters, 27(8) 2006. ISSN 01678655 doi: (ttp///.attee.2000000 50167865505003081
URL http://www..aneeeeeeeeeeeerrrree
Oleksandr Ferludin, Arno Eigenwillig, Martin Blais, Dustin Zelle, Jan Pfeifer, Alvaro Gonzaez- Wai Lok Sibon Li, Sami A Abu-Fl-Haija, Peter Battaglia, Neslihan Bulut, Jonathan Halcrow, Filipe Miguel Gonçalves de Almeida, Pedro Gonnet, Liangze Jiang, Parth Kothari, Silvio Lattanzi, André Lin- hares, Brandou Mayer, Vahab Mirrokui, John Palowitch, Mihir Paradkar, Jennifer She, Anton Tsitsulin, Kevin Villela, Lisa Wang, David Wong, and Bryan Perozzi. TF-GNN: Graph Neural Networks in Tensor- Flow, 2022. URL
httpaaawww...rras/aaaaaaaaaaa/ 035222 arXiv:2207.035: 222.222222 [physics, stat].
Matthias Fey and Jan E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Represenlation Learning on Gruphs and Munifolds, 2019.
Hossein Gholamalinezhad and Hossein Khosravi. Pooling Methods in Deep Neural Networks, Review. CoRR, abs/2009.07485, 2020. URL https://arxiv. http::///axxxv.org/bbbbb.e wttg///////..////////0/0 arXiv: 2009.07485.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning, pp. 1263 1272. PMLR, 2017. URUiattpp://prcceednggg.u mlr. prress//70/gilmerr/rr. http:///eee./eooooeeeeee////rr/r 3498.
ISSN: 2640-
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS'17, pp. 1025-1035, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978-1-5108-60966 event-place: Long Beach, California, USA.
Jin Huang, Bin Li, Jia Zhu, and Jian Chen. Age classification with deep learning face representa- tion. Mullimedia Tools and Applications, 76(19):20231 20247, 2017. ISSN 1573-7721 doi: 10.1007/ s11042-017-4644- URL httpp:://oi...oo http:///oooorr//////// htt.:///ww......//100000000----.
Jabbar Hussain. Deep Learning Black Box Problem. Master's thesis, Uppsala University, Department of Informatics and Media, 2019. Backup Publisher: Uppsala University, Department of Informatics and Medía.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Sen Jia, Thomas Lenfae,l-eelfree, and Nello Cristianini. Gender Classification by Deep Learning Millions of Weakly Labelled Images. In 2016 IEEE 16th International Conference on Dala Mining Workshops (ICDMW), pp. 462 467, 2016. doi: 10.1109/10 /009///IMW.... abstractt/docmeet//888877 ISSN: 2375-9259.
/ttp///D........0. 20166.072. URL https::/ieeexpplore ieee. org/
Yu-Gang Jiang, Zuxuan Wu, Jinhui Tang, Zechao Li, Xiangyang Xue, and Shih-Fu Chang. Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification. TEEE Transactions on Multimedia, 20(11):3137 3147, 2018. ISSN 1941-0077. doi: 10.1109/TMM..21 201..........22.0000.0 URL
https://ieeexpporee ieee.iieeecoom httg:///aattattttcttaaaa//////// on Multimedia..
Conference Name: IEEE Transactions
Thomas N. Kipf and Max Welling. Semi-Superrised Classification with Graph Convolutional Networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview Reene 2017. URL SJ
Byoung Chul Ko. A Brief Review of Facial Emotion Recognition Based on Visual Information. Sensors, 18 (2), 2018. ISSN 1424-8220. doi: 1..3300/s18200A/ URL httppp//wwww..mppp.oooo22222222/
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong, Mobile Video Networks for Efficient. Video Recognition. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16015 -6025,, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-665444509-2 doi: 10.1109 10.1109/CVPR /C/////PR24447..7 wwwp:4447.2222....1 2021.0107210. 01576.. URL org//documett/857826000
MoViNets:
1eee.
Anna Kukleva, Makarand Tapaswi, and Ivan Laptev. Learning interactions and relationships between movie characters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9849-9858, 2020.
Solomon Kullback and Richard A Leibler. On information and sulficiency. The annals of mathemalical statistics, 22(():79-86 86, 1951. Publisher: JSTOR.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. Publisher: leee.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as enns? In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9267-9276, 2019a.
Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-Structuraal Graph Con- volutional Networks for Skeleton- Based Action Recognition. In 2019 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3590-3598, 2019b. doi: 10.1109 Ctt00//CCP.22000.0
Xinchen Liu, Wu Liu, Meng Zhang, Jingwen Chen, Lianli Gao, Chenggang Yan, and Tao Mei. Social Relation Reasoning. Tn 2019 IEFE/CVF Conference on Computer Vision and Pattern Recogrition (CVPR), pp. 3561 3569, Long Beach, CA, USA, 2019, IEEE. ISBN 978-1-72813-291 doi: 10. 10.1109/CVII CCC00//CVRR2209.0066666 URL https://heeepploree ieee. org/ 2ocument/8954352/
Recognition From Videos via Multi-Scale
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: theoretical analysis and appli- cations. In Proceedings of the 40th International Conference on Machine Learning, 1CML'23 JMLR.org, 2023. Place: Honolulu, Hawaii, USA.
Thomas F. Monaghan, Syed N. Rahman, Christina W. Agudelo, Alan J. Wein, Jason M. Lazar, Karel Ev- eraert, and Roger R. Dmochowski. Foundational Statistical Principles in Medical Research: Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Value. Medicina, 57(5):503, 2021. ISSN 1010-660X. doi: 00.3390/medicina57700 PMC8156820
URL https://ww/ ncbi, ,nlm. nih, gww//pccaarticccss//
15

---

Published in Transactions on Machine Learning Research (01/2025)
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807- 807-814, 2010.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding Language-Image Pretrained Models for General Video Recognition. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022. pp. 1-18, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19772-7
AJ Piergiovanni, Anelia Angelova, and Michael Ryoo. Tiny video networks. Applied AI Letters, 3(1):c38, 2022. Publisher: Wiley Online Library.
Mahmoud Qodseya. Managing heterogeneous cues in social conterts. A holistic approach for social interac- tions analysis. PhD thesis, Universite Toulouse Paul Sabatier, 2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Bass -aaeentt-Bsddd Localization. In 2017 IEEE International Conference on Compuler Vision (ICCV), pp. 618 626, 2017. doi: 10.1109/ ICCV.2017.74.
C. E. Shannon. A mathematical theory communication. 379-423, 1948. doi: 10.1002/j-15588-7900 1wt.:..19...ttt.....
The Bell System Technicol Journal, 27(3):
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-Stream Adaptive Graph Convolutional Networks for -kased Action Recognition. In 2019 IEEE/CVP Conference on Computer Vision and Paltern Recognition (CVPR), pp. 12018 12027, 2019. doi:
Alexandros Stergiou and Ronald Poppe. Understanding human-human interactions: survey, Xrviv preprint arXiv:1808.0000 2, 2018.
Julie Tores, Lucile Sassatelli, Hui- Yin Wu, Clement Bergman, Lea Andolfi, Victor Ecrement, Frederic Pre- cioso, Thierry Devars, Magali Guaresi, Virginie Julliard, and Sarah Lecossais. Visual Objectification in Films: Towards a New AI Task for Video Interpretation, 2024. eprint: 2401.13296.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler. MovieGraphs: Towards Understanding Hurnan-Centri Situations from Videos. In IEEE Conference on Compuler Vision and Pallern Recognation (CVPR), 2018.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong Hc, George Karypis, Jinyang Li, and Zheng Zhang. Deep Graph Library: A Graph-Centric, Highly-Pertee Performant Package for Graph Neural Networks. orXiv preprint XXw:11999.01315 2019.
Yilin Wang, Suhang Wang, Jiliang Tang, Ncil O'Hare, Yi Chang, and Baoxin Li. Hicrarchical attention network for action recognition in videos. Xi preprint 1rXic:1607.064 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ArL Natural Language Processing. In Qun Liu and David Schlangen (eds.), ) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Lin- guistics. doi: 10.18653/ v1 22000.emmlp-demme URL https://aclaanthology orgg/000000. emn.--demos demos. 6
16

---

Published in Transactions on Machine Learning Research (01/2025)
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey graph neural networks. TEEE transactions on neural networks and learning systems, 32(1 (2:1):4 24, 2020. Publisher: IEEE.
Yiteng Xu, Peishan Cong, Yichen Yao, Runnan Chen, Yuenan Hou, Xinge Zhu, Xuming He, Jingyi Yu, and Yuexin Ma. Human-centric Scone Understanding for 3D Large-scale Scenarios. In Proceedings of the IEEE/CVP International Conference Computer Vision (ICCV), pp. 20349 20359. 2023.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On Early Stopping in Gradient Descent Learning, Construclive Approzimmation, 26(2):289 -15, 315, 2007. ISSN 1432-0940. doi: 00.10077800355--000003322 URL http:s///abb.org//..0000///8885
Zhang, Beibei, Fang, Yaqun, Ren, Tongwei, Wu, and Gangshan. Multimodal Analysis for Deep Video Understanding with Video Language Transformer Proceedings of the 30th ACM International Conference on Multimedia, 2022a. URL https: acm. http://../aaaa/111111//333333630
Jiaxu Zhang, Gaoxiang Ye, Zhigang Tu, Yongtao Qin, Qianqing Qin, Jinlu Zhang, and Jun Liu. A spatial attentive and temporal dilated (SATD) GCN for skeleton-based action recognition. CAAI Transactions on Intelligence Technology, 7(1):46-55 55, 2022b. Publisher: Wiley Online Library.
Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, and Nanning Zheng. Semantics- Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition. In 2020 IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1109-1118, 2020. doi: 10.1109/ STTP4420002020.0000
Wenxian Zhang. Scene context-aware graph convolutional network for skeleton-based action recognition. IET Computer Vision, 2023. Publisher: Wiley Online Library.
Xiwu Zhang, Lei Wang, and Yan Su. Visual place recognition: A survey from deep learning perspective. Pattern Recognition, 113:107760, 2021. ISSN 0031-3203. doi:
URL https:
17

---

Published in Transactions on Machine Learning Research (01/2025)
A Graph Neural Networks architectures explored
As stated in Section 5.2 four primary architectural designs for Graph Neural Networks were explored for our specific problem: and Graph Attention Networks (Dwivedi Bresson (2021)). ). Graph Convolutional Networks (GCN) [Kipl & Welling (2017)), GraphSAGE. networks (Hamilton et al. (2017) and Vanilla
Message Passing Neural Networks (Vanilla MPNN) (Gilmer al. (2017)). In the subsequent paragraphs, we will elaborate on the principles of each architecture, and provide few additional implementation details.
GCN models. These architectures apply the principles of convolution (LeCun et al. 1998 in conventional Neural Networks, to graphs. They enable learning representations of nodes in a graph that incorporate information from their local neighborhood and can capture complex relationships and structures within the graph. There are many variations of the original GCN architectures in the literature, such as DeepGCN (Li al. (2019a)) Fast-localized spectral GCN (Defferrard et al. (2016)), but our graph convolutional layer corresponds (Kipf & Welling (2017)
observed
that computes:
E
(WT xxft-11) +b
(8)
JEN(A)(() V deg(i) dde(j))
(k-1) the node features node j in layer (k-1). Where W is weight matrix applied to the node features, 2j b the bias vector applied the aggregated output, and i and j the neighboring nodes.
Vanilla MPNN models. They exchange information between nodes through message passing steps, up- dating node representations based on information from neighbors, without convolution. The Vanilla MPNN update layer can be formally defined as (Gilmer et al. (2017)):
h Uk m
(9)
Where k++ are the hidden states at layer &e + 1, U the node update function, h the hidden states at layer k, and m the message at layer l + 1.
GraphSAGE models. They work by sampling and aggregating information from the neighborhood of each node iteratively. In each iteration, GraphSAGE samples a fixed number of neighbors for each node, aggregates their features, and updates the node's representation. The GraphSAGE update layer can be formally defined as (Hamilton et al. (017))
h (W wwt.ccccattt.h-11 (h)
(10)
Where a represents the nonlinear activation function, W the weight matrix at layer k, concat the function that concatenates the representation the node and its aggregated representation, 1g-1 "tee representation of the node layer k-,, and h()() the aggregated representation of the node and its immediate neighborhood.
GAT models. These architectures apply the principles of Attention (Vaswani et al. 2017)) in traditional neural networks to graphs. They use attention mechanisms weigh the importance of neighboring nodes' features when updating node's representation. The GAT update layer can be formally defined as (Dwivedi
& Bresson ))
h 11-1) =a
Σ ai W0 (1) JEN((
(11)
Where h (1+1) is the updated representation of node i at layer + 1, or is the activation function, ReLU (aar & Hinton (2010)) in our case, (()) represents the neighbors node i, Oij represents the attention weight between nodes and j, (1)) the weight matrix for layer I, and h2 at layer I.
(1) is the representation of node
18

---

Published in Transactions on Machine Learning Research (01/2025)
B
Algorithm: generation of the training flags
Algorithm Training flags generation
Require: node sels & null, size(node sets) = 7 b O. Set the possible values for the training flags values <- [True, False]
b 1. Generate the combinations (with repetition) of the values for a size = size(node_scta
Irain flugs t carlesion product(values, repeal size(nodes list))
2. Loop through the training flags and compute the representation for flag in training_flags do repr &- feature f _atieeemapppnnniiiiee (node_ sets, flag) end for b Description: The training flags exhibit binary values, with false denoting absence of information
pertaining to the current node, or true, meaning that a proper representation will be computed using the <50000<32> embedding layer. Given the existence of seven distinct node types in our representation, we generate combinations of lists with size of seven, comprising elements "true" and 'false.' Subsequently, we iterate over the training flags list to compute the representation for each node, contingent upon the flag value corresponding to its respective type.
C Algorithm: Computing the representation of the nodes
Algorithm Computing the representations of nodes
Require: node sets F null, flag F null
b Loop through the list of nodes sets and check the value of the flags for each node type and assign its representation based on the flag
for node in node sets do & Check the value of the flags for each node type and assign its representation based on the flag
if flaglnode("Mng [yppe"] True then
representation [nneee t- embedding(node) clse
represenlationnnodee + empty tensor() end if end for return (representation)
Description: While looping through the list of nodes, based the value of the training flag for the current node, we compute an empty feature map or send the information through an embedding layer. The empty feature as empty tensor with nil weights when the training flag is False. To make parallel with human logic, would the equivalent of saying: we know about the existence X node, but we do not know anything about it.
19

---

Published in Transactions on Machine Learning Research (01/2025)
D
Training flags for the best models
Table 6: Training flags for the best models
(a) Objectification detection
(b) Interaction classification
Flag
Value
Flag
Value
Attribute
True
Attribute
False
Place
True
Place
False
Character
False
Character
False
Context
True
Context
True
Relationship
True
Relationship
True
Emotion
False
Emotion
True
Speech
True
Speech
True
20

---

Published in Transactions on Machine Learning Research (01/2025)
E
Proposed framework for the neuro-symbolic approach
embeddinge
Input graph Emotions, Piaces, Contexts,, Reiationships, Speech..
Node
RZIDARN
Aurtbuee
Node
Node
Note
Note
Embedding
Node 3
GCN update: for each node in the sets of nodes of interest
Message passing iterations
Experte insights
Updated weights Dense(convolution (messages from neighboring nodes) + pooling))
Confidence interval
GCN
Fully connected layer (root states)
P(root_node=x)
Probabilistic facts/rules
Probability of
-.11 P(root_node=Y)
Symbolic component
Probabilities
Combining GNN prediction and symbolic component
Final probabilities
Prior knowledge P(root_ node=X)
Probabilistic facts/rules
Observations
Probablistic facts and rules update prior knowledge
Input of the Bayesian problem: prior knowledge
Figure 3: Integrating domain knowledge: protolype
21


Reviews:
[{"review_id": "qrbyNGmGPc", "reviewer": "Reviewer_TMWG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. Intuitive and well-motivated approach for utilizing graph machine learning for the task of scene understanding\n2. The paper is easy to follow and contains a good amount of technical details that is often missing in other papers, providing a good overview over the method and data preprocessing etc.\n\t\n## Weaknesses\n1. The lack of visual features is somewhat concerning to claim human-like understanding\n2. The empirical evaluation compares only with a few (old) baselines\n3. The paper introduces some interpretability features but lacks their empirical evaluation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a graph-centric deep learning approach for scene analysis. For this, the authors construct an interaction graph containing the context and interactions. Based on this graph, the authors devise a deep learning architecture with graph neural networks at its core to tackle two tasks: (a) objectification detection and (b) link classification. The core aim of this work is to provide a human-like understanding of interactions between humans. It is one of the few works in the field that attempts to incorporate contextual data for the interpretation of scenes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The paper could be more explicit about the targets in the tasks. I assume objectification detection is a graph classification task while interaction classification is an edge-level classification task. Perhaps one example per task would also be helpful (e.g., added to the appendix).\n2. The empirical evaluation could be more comprehensive in comparing to other approaches or the authors should elaborate on why there are not alternative approaches.\n3. The related work section only contains 9 references. Is this really comprehensive? If so, the authors should be explicit about why there are only so few relevant works.\n\n## Minor\n- Why past tense in 5.4 \"we introduce**d** an interpretability\"\n- It is confusing that Figure 2 follows page 6, given that the reader expects an equation next. Perhaps move Eq 4 to the top of page 7, there is plenty of space\n- It is uncommon to use * for multiplication. Consider \\cdot instead. Also, in the equations, the authors should escape the math mode for terms like \"TPR,\" \"Precision,\" etc.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ucdz5URNqz", "reviewer": "Reviewer_3bJ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a novel method for scene analysis by explicitly incorporating contextual information.\n\nThe experimental evaluation is comprehensive, with ablation studies and detailed analysis.\n\nExperimental results show improvements over baseline approaches.\n\nWeaknesses:\n\nIt is unclear how scene elements are mapped to interaction graphs. I was unable to understand how characters are identified in videos, what attributes are tracked, and how they're extracted. It is also unclear how scene and context information are extracted. The embedding process is also unclear, as there are no details on whether or how different node types are embedded differently.\n\n The details provided about the datasets are insufficient. For the ObjectifyGaze Dataset, the authors need to briefly introduce the source of the videos and the lengths of the scenes. For the MovieGraphs Dataset, the authors need to introduce the characteristics of the 51 movies, the lengths of the scenes, and the interaction classes involved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach to scene analysis that incorporates contextual data into graph neural networks. The authors evaluate their methodology on two tasks: objectification detection and interaction classification in video content. The work introduces methods for integrating domain expertise and provides interpretability insights through sensitivity analysis of the model's decision-making process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My major concern with this paper is the insufficient detail provided in the methodology description. This lack of detail makes it difficult to understand the model architecture and reproduce the results. The authors need to provide more comprehensive information in the manuscript to ensure readers can fully understand and implement the approach.", "broader_impact_concerns": "The authors have list broader impact in the Broader Impact Statement Section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YA6oDeeRs1", "reviewer": "Reviewer_4U97", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper provides a comprehensive and detailed description of the method, from data preprocessing to hyperparameter selection, which ensures reproducibility along with the supplementary code provided.\n\n- Although I am not familiar with the tasks this paper addresses, particularly objectification detection, the authors have conducted a thorough review of previous literature to give the audience more context.\n\n- The experiments not only achieve better performance than the selected baselines but also prove the major claims regarding the integration of contextual information and domain knowledge that were made at the beginning of the paper.\n\nWeakness:\n- First, the authors emphasize the importance of contextual information, yet they do not clearly define or provide references about what are contextual information. In the introduction, the statement \"can only be achieved by incorporating contextual cues such as the context of the scene\" offers very limited information. Specifically, the issue is twofold: firstly, besides the context of the scene, what other entities contain necessary context? Secondly, what specifically are these contexts, such as for a scene?\nIn Section 2, the authors provide examples like \"relationships between the characters on-screen, or the potential speech during the interactions.\" In Section 8.3, further examples are given, such as visual place recognition, age and gender, as well as emotion and relationship. These examples do not clearly define contextual information, suggesting that any information could be considered as contextual information. This lack of clarity undermines the contribution of integrating contextual information since it cannot be distinctly identified. Moreover, the examples are inconsistent with the pipeline diagram in Figure 1, where the context node is connected to the scene, while character relations are linked to two characters, clearly indicating different concepts. \nIn summary, the authors should provide a clear definition of contextual information—what it includes and what it does not. Without this, the first contribution of the paper does not hold.\n\n- Second, assuming there is a clear definition of contextual information, how it is integrated into the classification should also be further justified. The authors add the contextual information as a node and use existing graph neural networks. Since there is no contribution to the model itself, the authors should explain why graph neural networks are the best choice, rather than other structures such as transformers, or even off-the-shelf vision-language models (https://multimodal-vid.github.io/).\n\n- Finally, the experiments should be improved to better support the contributions. Firstly, the discussion of different choices of graph neural networks, which could be added to the appendix, is unnecessary. Secondly, after having a clear definition of contextual information, the authors should not only provide sensitivity scores for each type of contextual feature but also conduct ablation studies excluding each type of contextual information. Lastly, the claim of limited contribution from image features is unconvincing. Instead, visual input is important as some of the contextual information and the information for the naive Bayesian post-processing are derived from visual features. Therefore, rather than simply using existing feature extractors in parallel, I would suggest the authors extract different types of information from visual features using the same backbone feature maps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on multimodal video classification tasks and proposes incorporating \"contextual information\" such as scene context, along with integrating \"domain knowledge\" including feature importance and simple Bayesian rules into the classification process. To achieve this, the authors utilize existing graph neural networks, integrating the contextual information as one of the graph nodes. The feature importance is emphasized by constraining the weight magnitude associated with less significant nodes, and the Bayesian rules are applied as a post-processing step following the classification by the graph neural network. The effectiveness of the proposed method is demonstrated through experiments on two video classification tasks: objectification detection and the more general interaction classification. In both tasks, the proposed method outperforms existing baselines, showcasing its effectiveness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness for the required changes.", "broader_impact_concerns": "The authors have discussed the Broader Impact sufficiently.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o68o1SZ9bV", "reviewer": "Reviewer_dXtd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a strength, the paper paper underscores the importance of contextual data for scene analysis, providing a promising framework for building more human-like AI systems capable of understanding the subtleties of social interactions. \n\nThey validate their method on two tasks. In the first, objectification detection aiming to identify scenes containing objectification, they show promising results  when combining scene graph (SG) and speech (S) and highlight the importance of contextual data for this task, as this baseline surpass these presented with SG data only. \n\nThe second task. Interaction classification, aimes to categorize interpersonal interactions, and they demonstrate that incorporating contextual information given by the SG (and also combined with dialogs) leads to better performance compared to using visual, dialogue data and face tracks in isolation or combined. \n\nWeaknesses:\nThe validity of the proposal with no comparison with existing approaches is limited. The weakest aspect of the paper is comparison with existing methods and references to existent literature in the topic. The results of alternative approaches under the same tasks are minimal and/or missing. A quick search on results for the MovieGraphs dataset resulted in the following leaderboard:    \nhttps://paperswithcode.com/dataset/moviegraphs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research paper proposes a framework for analyzing scenes in video content by integrating contextual data, such as character relationships and scene context, into the analysis.\nThey proposed adopt graph neural networks (GNNs) to process the relationships and interconnected knowledge within a scene into a three-step process: (i) Feature mapping:  Each node and edge in the graph is assigned a representation based on its type (character, scene, relationship, etc.); (ii) Graph update: The model learns the importance of different nodes and connections by iteratively updating the graph representations using different GNN architectures; (iii) Classification: Based on the learned representations, the model classifies the overall scene or individual interactions within the scene.\n\nThe framework is evaluated on two tasks: objectification detection (Tores et al 2024) and interaction classification (Vicol 2018), and claim to achieve promising results compared to baseline methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Suggestions for improving the paper:\n\n- Definition of Objectification Detection Task: Provide a clearer definition of the \"objectification detection\" task in the end of Section 3. The term \"objectification\" is often associated with negative connotations, and the task itself may not be widely known.\nDataset Description: Enhance the description of the dataset from Tores. Specify the number of objectification tags and provide some examples to aid in understanding the task.\n\n- Method Description: For easier comprehension, start with an abstract overview of the full method. Isolate this description from hyperparameter and implementation details. Currently, Section 5 partially serves this purpose, but some components are described in Section 4. Reorganize the paper by moving Section 5 upwards and adding missing elements. Then, introduce an experimental section for implementation details.\n\n- Figure 2 Placement: Move Figure 2 to the beginning of the method section. Include a brief description of its components (Feature map, Graph Update, etc.) to help readers understand the overall composition before delving into details.\n\n- Clarification of the claims: are the authors presenting a novel methodology or investigating various Graph Neural Network (GNN) architectures (GCN, MPNN, Sage) for the given tasks? Table 2 seems to support the latter, while Table 4 supports the former.\n\n- External results comparison: The comparison of the objectification detection results with external sources appears to be missing. Table 1 only includes variant comparisons. Additionally, the argument presented in Table 3 is challenging to comprehend without prior knowledge of Tores 2024. How can expert annotations be utilized to interpret the presented results? Speech is not mentioned in other tables, and the scene is mentioned in Table 3a but not in Table 3b. What is the performance of Tores et al in this task?\n\n- Clear distinction of Tores and Vicol baseline results on the tables: in its current version is hard for the reader not familiar with these two references, where their corresponding baseline results appear on the presented tables.", "broader_impact_concerns": "Authors express Concerns on diverse and representative datasets, regular audits and evaluations, education and awareness and transparency and accountability. As deals with human content and interaction I suggest adding privacy protection such as to  safeguard individuals' identities and ensure ethical data handling practices throughout the research process.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a graph learning approach for scene analysis. The method integrates contextual information using GNNs for two tasks: Objectification detection and Interaction classification.

The paper was reviewed by expert reviewers and the summary of their findings are as follows:

Strengths
- Incorporating contextual cues and graph-based modeling is a promising direction for scene understanding.
- The paper provides detailed technical descriptions that facilitate reproducibility.
- There are extensive ablation studies demonstrating the significance of contextual data for the tasks.

Weaknesses
- The lack of a clear and consistent definition of "contextual information" undermines the core claim of the paper.
- Comparisons with existing state-of-the-art methods on the datasets are partially insufficient.
- The dataset descriptions, particularly ObjectifyGaze, lack sufficient detail for readers to assess the validity of the evaluation.

Requested changes in the revision:
- Authors should provide a clear and consistent definition of "contextual information" and its role in the model. Clearly outline how scene elements are mapped to interaction graphs, including character identification and node embeddings. Add a high-level overview of the method before delving into implementation details.
- Authors should enhance Dataset Descriptions. For ObjectifyGaze, describe the source of videos, the objectification tags, and examples. For MovieGraphs, provide details on the dataset’s composition and interaction classes.
- Authors should address all stated minor issues

This paper proposes an interesting method for scene understanding but requires revisions to ensure clarity and rigor. Once these changes are made, the paper will make a valuable contribution to the field.

Note: some of these changes are already made in the revision during the discussion period and kept here for the completeness of the summary.
Decision: Accept (poster)