Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 10:50:06

Abstract: Benefiting from strong generalization ability, pre-trained vision-language models (VLMs), e.g., CLIP, have been widely utilized in zero-shot scene understanding. Unlike simple recognition tasks, grounded situation recognition (GSR) requires the model not only to classify salient activity (verb) in the image, but also to detect all semantic roles that participate in the action. This complex task usually involves three steps: verb recognition, semantic role grounding, and noun recognition. Directly employing class-based prompts with VLMs and grounding models for this task suffers from several limitations, e.g., it struggles to distinguish ambiguous verb concepts, accurately localize roles with fixed verb-centric template input, and achieve context-aware noun predictions. In this paper, we argue that these limitations stem from the model’s poor understanding of verb/noun classes. To this end, we introduce a new approach for zero-shot GSR via Language EXplainer(LEX), which significantly boosts the model’s comprehensive capabilities through three explainers: 1) verb explainer, which generates general verb-centric descriptions to enhance the discriminability of different verb classes; 2) grounding explainer, which rephrases verb-centric templates for clearer understanding, thereby enhancing precise semantic role localization; and 3) noun explainer,  which creates scene-specific noun descriptions to ensure context-aware noun recognition. By equipping each step of the GSR process with an auxiliary explainer, LEX facilitates complex scene understanding in real-world scenarios. Our extensive validations on the SWiG dataset demonstrate LEX’s effectiveness and interoperability in zero-shot GSR.

Main paper: 
 introduction: 
 Traditional recognition methods heavily rely on the quality of vast dataset annotations, often limited in generalizing across diverse or unseen scenarios [35] . Zero-shot learning emerges to enable models to identify classes they have never been directly trained on, thus greatly enhancing their universality in real-world settings [38, 43, 44] . Recently, advancements in pre-trained visionlanguage models (VLMs) [19, 20, 27] , e.g., CLIP [37] , have shown excellent generalization capabilities across various recognition tasks, setting new benchmarks in zero-shot learning. Particularly, CLIP incorporates dual encoders: an image encoder and a text encoder.The former processes visual input into visual features, and the latter translates texts into semantic features. This architecture facilitates the alignment of visual and text data within a unified semantic space. Leveraging class-based prompts like "A photo of [NOUN CLASS]" or "A photo of [VERB CLASS]", CLIP effectively compares images against prompts in the learned semantic space, enabling the zero-shot recognition of both unseen objects [37] and actions [29] . Nevertheless, training-free zero-shot grounded situation recognition (ZS-GSR) presents a more intricate challenge than basic object or action recognition [36] . It requires not only identifying actions (verbs) depicted in images but also discerning and localizing the semantic roles involved [36, 42] . Such a comprehensive task contributes to effectively understanding what is happening (e.g., buying in Figure 1 ), who is involved (e.g., AGENT and SELLER), where it is taking place (e.g., PLACE), etc. Consequently, ZS-GSR extends beyond simple class recognition to a structured understanding of scenes, requiring accurate modeling of the relationships among objects within an event. As illustrated in Figure 1 , a typical GSR approach typically consists of three steps: • Verb Recognition. This step aims to identify the verb category of the entire image. The most common zero-shot method for this sub-task uses a set of class-based prompts "a photo of [VERB CLASS]" for each verb category [37] . These prompts are then fed into the text encoder to obtain semantic embedding, while the whole image is passed through an image encoder to get visual embedding. By comparing these two embeddings in the same semantic space, CLIP can achieve zero-shot verb recognition. • Semantic Role Grounding. Given the predicted verb class and its predefined template 1 , this step aims to precisely localize each semantic role within the image. Existing methods for this subtask often adopt an open-world grounding model, e.g., Grounding DINO [30] , with an object-contained grounding language as input [17, 49, 54] . As for GSR, the input can be the provided templates in the dataset, e.g., the template of buys: "the AGENT buys GOODS with PAYMENT from the SELLER in a PLACE". • Noun Recognition. This step aims to identify the noun category for each semantic role. Similar to the verb, a most direct zeroshot method can use noun class-based prompts like "A photo of {NOUN CLASS}" to compare with visual features of grounded semantic role for classification [37] . For example in Figure 1 , the grounded semantic role AGENT is classified as woman.However, these simple methods that rely entirely on class-based prompts at every step are limited by: 1) Ambiguous Action Concepts: Foundational VLMs primarily concentrate on understanding images or bag of entities, often overlooking the semantics and structures inherent in actions [10, 26, 28] . Directly classifying verbs with class-based prompts may not fully capture the nuanced meanings of actions, leading to CLIP misunderstanding the verb's embedding and generating inaccurate predictions. For instance in Figure 2 (a), the model projects the two embeddings into adjacent locations in semantic space and recognizes an activity as "coloring" due to the colorful background. It overlooks the actual activity of "studying" that entails a complex interaction with stationery and books. 2) Constrained Role Grounding: The effectiveness of semantic role grounding within visual grounding models critically depends on the quality of the text prompt [39, 46] . The employment of fixed and simplex templates 1 for semantic role grounding is inherently constrained by the simplified sentence structure and ambiguous verb. Once the model encounters unfamiliar or complex verb classes, it easily leads to misalignment and localization errors for the involved roles. As depicted in Figure 2 (b), with templates centered on the unfamiliar "climb" class, grounding models have difficulty in locating the "TOOL". 3) Context-agnostic Noun Prediction: Noun prediction is often conditioned on the specific verb and semantic role present within a scene [36, 42] . However, when employing class-based prompts for noun classification, CLIP tends to concentrate on the category with the highest prediction confidence. This method overlooks the crucial consideration of whether the noun accurately fits the specific role defined in the scene's template, leading to contextually inappropriate predictions. For example, in Figure 2 (c), VLMs directly recognize the "hand" that occupies the main area of the image, ignoring the characteristics of the AGENT.In this paper, we argue that these limitations stem from the model's insufficient understanding of individual classes. When humans encounter an unfamiliar word, such as "soliloquy", we often turn to a dictionary to serve as an explainer, providing a clear and accessible definition that illuminates the term's meaning from a common understanding perspective. For instance, the term "soliloquy" in a theatrical script refers to "a character speaking their thoughts aloud when alone or regardless of any hearers". Fortunately, recent advancements in large language models (LLMs), e.g., GPT [4] , have benefited from extensive training across diverse datasets, endowing them with broad world knowledge.Inspired by human reliance on external sources for deeper understanding, we propose the method for zero-shot grounded situation recognition via Language EXplainer (LEX). It employs LLMs to serve as explainers at crucial steps of the GSR process. Specially, for verb recognition, we devise a verb explainer to prompt LLMs to generate general verb-centric descriptions. This approach provides CLIP with multiple enriched "explanations" of each verb, e.g., "books or notes displayed" in Figure 2 (a), leading to more accurate verb classification. Additionally, we implement an offline description weighting strategy that takes into account the discriminability of the verb-centric descriptions without any training data. As for semantic role localization, we design a grounding explainer to prompt the LLMs to rephrase the original verb-centric template from multiple perspectives. These reformulated sentences, as seen in Figure 2 (b), serving as text inputs for the grounding model, are capable of enhancing the model's understanding of the intricate relationships between semantic roles. Lastly, for noun recognition, we propose a noun explainer to prompt LLMs to generate noun descriptions conditioned on both verb and semantic roles. As is illustrated in Figure 2 (c), noun explainer prevents the misclassification of a noun such as "hand" when the contextual role and action pertain to "woman" applying lotion, thus mitigating the risk of contextually inappropriate classifications.To verify the effectiveness of our LEX, we conduct extensive experiments and ablation studies on SWiG dataset. Each of our explainers can be utilized as a plug-and-play module to improve the overall performance of zero-shot GSR.In summary, we make the following contributions in this paper:(1) We propose a novel training-free zero-shot GSR framework LEX that incorporates LLMs as explainers within each process, enhancing understanding of complex visual scenes. (2) We introduce three explainers: verb explainer, grounding explainer, and noun explainer. Each serves as a plug-andplay module to enhance the performance of zero-shot GSR.  
related work: 
 Grounded Situation Recognition (GSR). Despite deep learning's notable achievements in image classification [7, 31, 34] , object detection [55] , and image segmentation [6, 22] , its comprehension of complex scenes remains limited. Techniques, e.g., scene graph generation [25, 45, 52] and human-object interaction detection [1, 5, 16] , aim to parse scene contents via relation graphs. Admittedly, these tasks can provide structured visual scene representations to assist downstream tasks [23] . However, they only focus on modeling with dyadic relationships between a subject and an object, ignoring the diversity of roles in visual events [8] . GSR emerges as a solution for holistic scene understanding by predicting actions (verbs) and detecting noun entities of each semantic role [36] . Pratt et.al. [36] first proposed an RNN-based two-stage framework, which detects verbs in the first stage and predicts the nouns with bounding boxes in the second stage. Subsequent methods [8, 10, 42] improved this two-stage pipeline by employing a transformer-based model and considering semantic relations among verbs and semantic roles. However, previous efforts that depend on annotated training samples may face challenges like sample noises [23, 24] and long-tailed distribution [9, 40, 48] , potentially limiting real-world applicability.Our approach predicts verbs, nouns, and their associated bounding boxes without training data, showcasing robust generalization and interpretability in real-world scenarios.Foundation Models. Foundation models are typically pre-trained on extensive training data. Owing to their strong generalization capabilities, foundation models are often applied to a variety of downstream tasks [3] . Particularly in the natural language processing (NLP) area, large language models (LLMs) [11, 12, 18] , such as GPT-3 [14] , PaLM [2] , OPT [53] , and LLaMA [41] , trained on extensive web-scale text datasets, exhibit formidable capabilities ranging from text prediction to contextually relevant text generation. In the field of cross-modal research, vision-language models (VLMs), e.g., CLIP [37] and ALBEF [21] , have been trained on extensive image-text pairs via contrastive learning, facilitating the cohesive integration of information between these modalities. Subsequent models, such as BLIP [20] and BLIP-2 [19] , leverage diverse datasets and Q-Former's trainable query vectors to further refine this alignment. Despite the remarkable transferability of VLMs, prevailing approaches often rely on class-based hard prompts as input to the text encoder, leading to a narrow focus on specific visual features and disregarding contextual information in the surroundings. In this paper, we employ various prompts to guide LLMs as different explainers to generating "class explanations", significantly enhancing VLM's understanding of intricate scenes. 
zs-gsr via language explainer: 
 Formulation. Given an image 𝐼 , zero-shot GSR aims to identify a structured frame To address the constraints of the baseline, we propose a new framework LEX for ZS-GSR. As illustrated in Figure 3 , it comprises three components: verb recognition via verb explainer, role localization via grounding explainer, and noun recognition via noun explainer. LEX enhances the model's understanding of different classes by adding an auxiliary explainer at each step.F 𝑣 = {𝑦 𝑣 , F R 𝑣 }, where 𝑦 𝑣 ∈ V denotes the salient verb category, F R 𝑣 = {(𝑟, 𝑦 𝑛 , b) | 𝑟 ∈ R 𝑣 }This component aims to recognize the salient verb category within an image. It comprises three steps: verb-centric description generation, description weighting, and verb classification. EQUATIONwhere 𝐿𝐿𝑀 (•) is the decoder of the LLM. The verb-centric description generation prompt input consists of three parts 2 : 1) In-context examples, the description instruction for the verb, along with some examples of the generated results. This part adopts in-context learning [13, 33] to make LLM generate analogous results. 2) Verb class 𝑦 𝑣 , the target category that needs to generate the description. 3) Instruction, the sentence used to command the LLM to generate the description, e.g., "what are the useful visual features for the event of 'rehabilitating': AGENT rehabilitates ITEM at a PLACE". These generated descriptions highlight various unique visual features associated with specific action categories, for example, "looking around for searching goods", "pushing a shopping cart or carrying a basket", contribute to enhancing the distinguishability between "shopping" and other similar actions like "buying".Weighting. Considering the variances in description quality and its different impact on verb classification, we devise a description weighting strategy. Recent methods try to select descriptions from the perspective of the discrimination of image 2 The detailed description generation prompts are in the Appendix. features [47] and the coverage among concepts [50] . However, these methods require a large number of annotated training images and are prone to overfitting to seen scenarios, which are not suitable for training-free zero-shot GSR. Thanks to the property of VLMs to align visual and text modes in a shared space, we can replace an annotated image with a complex scene description that contains the verb category [15] . Specifically, we first utilize the LLMs to generate a set of complex scene descriptions D 𝑠 |𝑣 = {𝑑 𝑠 } for each verb 𝑦 𝑣 ∈ V. These scene descriptions are fed into the text encoder 𝑇 (•) to generate scene text embeddings {t 𝑠 }, used as "image" for subsequent description weighting 2 . For instance in Figure 4 , the verb biting's scene description is used instead of an annotated image.Intuitively, the greater the distinctiveness of a verb description in differentiating between its own class of scenes and other classes, the more important it is. Following [47] , we apply the discriminability score 𝐷𝑖𝑠 (•) to measure the distinctiveness of a verb description. The correlation 𝜌 (•, •) between a verb description embedding and a set of scene text embeddings with class 𝑦 𝑣 is denoted as:EQUATIONwhere 𝜙 (•, •) denotes the cosine similarity, t 𝑑 𝑣 is the text embedding of a verb description 𝑑 𝑣 . The conditional likelihood of aligning scenes of a class given 𝑑 𝑣 can be written as:EQUATIONThe discriminability score 𝐷𝑖𝑠 (𝑑 𝑣 ) can be measured by the entropy over all verb classes, written as:EQUATIONThe smaller the discriminability score, the greater the distinction 𝑑 𝑣 offers for its category. Hence, the weight of a general verb description can be denoted as:EQUATIONwhere 𝑤 𝑣 (𝑑 𝑣 ) represents the weight of verb description 𝑑 𝑣 .In this step, we compute the similarity score between visual and textual features to obtain the probability distribution of verbs. We calculate the final distribution as follows:EQUATIONwhere 𝜆 is utilized as a balance factor between classed-based prompts and description-based prompts.This module aims to locate each semantic role more accurately via an auxiliary grounding explainer. Specifically, we first employ a grounding explainer to rephrase the verb-centric templates in a more comprehensible manner. Subsequently, we use the Grounding DINO to generate candidate bounding boxes for the semantic roles.Due to the ambiguity of the original fixed verb-centric template, using it as grounding text input for the Grounding DINO model may result in limited or inaccurate semantic role localizations. To enhance the comprehensibility of grounding language, we introduce a grounding explainer to "explain" these fixed templates. To be specific, we employ a rephrased template generation prompt, utilizing LLM as grounding explainers to produce rephrased grounding templates D 𝑔 = {𝑑 𝑔 } for each verb class 𝑦 𝑣 ∈ V, expressed as:EQUATIONSimilarly, the rephrased template generation prompt input also comprises three components: in-context examples, template 𝑇 𝑃 𝑣 , and instruction 2 . The first two components are analogous to verb-centric description generation. As for the last component, the instruction sentences are designed to enable LLM to generate role-contained templates that are easy to understand, i.e., "Please generate new sentences that detailedly rephrase this sentence to make it easier to understand {TEMPLATE}. Note that you must keep the original words: {SEMANTIC ROLES}".Given an unlabeled input image 𝐼 and the grounding text input 𝑑 𝑔 , the Grounding DINO model is adept at associating visual features with corresponding object labels (i.e., semantic roles). More concretely, the Grounding DINO model outputs a sequence of bounding boxes B 𝑔 = {b}, associated object labels R 𝑔 = {𝑦 𝑟 }, and confidence scores C 𝑔 = {𝑐} as follows:EQUATIONwhere 𝐷𝐼 𝑁𝑂 (•) is the decoder of Grounding DINO. In this module, we utilize the noun explainer to achieve contextaware noun classification. To be specific, it consists of four steps: noun filtering, scene-specific noun description generation, noun pre-prediction, and noun refinement, as displayed in Figure 5 .Filtering. This step filters out noun categories that are unlikely to appear in the specific semantic roles from class-level.Leveraging the common sense contained in LLM, we can inquire about reasonable noun categories through a simple prompt that includes a specified semantic role 2 . For instance, we can query "Given entity list {Noun CLASS}, which entities are most likely to be the result of a predicted semantic role PLACE" to filter unreasonable noun categories, e.g., man. This filtering step not only enhances the computational efficiency of subsequent noun classification by reducing the number of traversed categories but also improves the accuracy of classification results.3.3.2 Scene-Specific Noun Description Generation. In particular, the cropped candidate box inevitably contains more than one object, and the object normally has more than one possible category. Reliance on class-based prompts often results in CLIP focusing on the most salient noun category directly, thereby ignoring the specific scene context. To address this, we incorporate a noun explainer to "explain" class-based prompts associated with each noun category within diverse scenes 2 . This noun explainer adopts a scene-specific noun description generation prompt, guiding LLM to produce descriptions D 𝑟 𝑛 = {𝑑 𝑛 } that reflect the intricate context of each scene for each noun class 𝑦 𝑛 ∈ N :EQUATIONDifferent from verb-centric description generation, the input prompt for scene-specific noun description generation is conditioned on extra scene information (verb-centric template and semantic role) to provide contextual information. The instruction is like this: "Please describe the visual features that can distinguish the noun entity {NOUN CLASS} corresponding to {SEMANTIC ROLE} in the scene: AGENT writes on TARGET using a TOOL at a PLACE". The generated descriptions (e.g., "holding a pen and other tools in hand") reflect the characteristic of corresponding semantic roles (e.g., AGENT ), thereby ensuring more accurate and contextually relevant noun categorization.In this step, we first compute the similarity between visual embeddings of all cropped boxes and text embedding to obtain the probability distribution of nouns. Similar to the verb classification process, this entails the incorporation of both class-based and description-based approaches to yield the pre-predict results for nouns:EQUATIONwhere t 𝑑 𝑛 is the text features of the noun description 𝑑 𝑛 . Due to the requirement of invoking a large number 3 of LLMs for generating scene descriptions akin to description weighting in verb recognition, the weights 𝑤 𝑛 (𝑑 𝑛 ) are uniformly assigned as 1/|D 𝑟 𝑛 |. 3.3.4 Noun Refinement. After localization by Grounding DINO, a semantic role may correspond to multiple candidate boxes due to the integration of different 𝑑 𝑔 ∈ D 𝑔 . Directly selecting the box with the highest confidence often relies solely on the localization model's understanding of the semantic role, overlooking the contextual relevance within the scene. To mitigate this, we propose to refine noun classification and bounding box selection via a global comparison. This process contrasts the visual features of the entire image v 𝑣 against the text features {t 𝑡𝑝 } of templates { 𝑇 𝑃 𝑣 } whose semantic roles are filled iteratively with predicted nouns {𝑦 𝑛 } in the previous step, e.g., "woman is studying in library" in Figure 5 . The noun filled in the template with the highest similarity along with its corresponding bounding box is the final prediction, written as:𝑇 𝑃 * 𝑣 = arg max 𝜙 (v 𝑣 , {t 𝑡𝑝 }) F R 𝑣 = {(𝑟, 𝑦 𝑛 , 𝑏) | 𝑦 𝑛 ∈ 𝑇 𝑃 * 𝑣 }.(11) 3 Since noun description is conditioned on both verb and semantic role, the call number of LLMs at least is| V | × | R 𝑣 | × | N |As seen in Figure 5 , the noun "woman" for the semantic role "AGENT " and its bounding box is selected as the final prediction result. 
experiment 4.1 experimental setup: 
 4.1.1 Dataset. We evaluated our method on the SWiG dataset [36] , which extends the imSitu dataset [51] by incorporating additional bounding box annotations for all visible semantic roles (69.3% of semantic roles have bounding boxes). In the SWiG dataset, each image is annotated with a verb, followed by a set of semantic roles ranging from 1 to 6 (3.55 on average), each verb is annotated with three verb frames by three separate annotators. SWiG contains 25200 testing images with 504 verb categories, 190 semantic role categories, and 9929 noun entity categories.4.1.2 Evaluation Metrics. We employed the same five evaluation metrics as [36] , which include: 1) verb: the accuracy of verb prediction. 2) value: the noun prediction accuracy for each semantic role.3) value-all (val-all): the noun prediction accuracy across the entire set of semantic roles. 4) grounded-value (grnd): the accuracy of the bounding box for each semantic role, where the predicted bounding box must achieve an IoU value of at least 0.5 with the ground-truth bounding box. 5) grounded-value-all (grnd-all): the accuracy of bounding box across the entire set of semantic roles. Besides, these metrics were presented under three evaluation settings: 1) Top-1-verb, 2) Top-5-Verb and 3) Ground-Truth-Verb, verbs are selected based on the top-1 prediction, top-5 predictions, and corresponding ground-truth, respectively. If the predicted verbs are incorrect in the Top-1/5-verb settings, the other four metrics (value, val-all, grnd, and grnd-all) are considered incorrect.4.1.3 Implementation Details. We used the vision transformer with a base configuration (ViT-B/32) as the default backbone for CLIP. We adopted GPT-3.5-turbo for LLM, recognized as a widely utilized LLM in existing works. More details are left in the Appendix.4.1.4 Baselines. We compared our proposed method LEX with four strong baselines: 1) CLS, which uses class-based prompts to calculate the text embedding for both verb and noun classification. Leverage verb-centric template as the text input of Grounding DINO for role grounding. 2) TEM, which uses the verb-centric template to enhance the original class-based prompts for verb classification, compared with CLS. 3) CLSDE, which generates general descriptions of each noun category without considering scene and role by LLMs to enhance the original class-based prompts for noun classification, different from CLS. 4) RECODE, which generates semantic role descriptions as visual cues to enhance the original class-based prompts for verb classification, compared to CLS.We evaluated the performance of our proposed LEX and four strong baselines on the test set of SWiG dataset. From Table 1 , we have the following observations: 1) The CLS baseline relying solely on class-based prompts, demonstrates inferior performance, particularly in the prediction of bounding boxes and nouns. 2) In the Top-1 and Top-5 verb settings, TEM exhibits a slight performance advantage over CLS, attributed to the enhanced scene information As aforementioned, each component of our proposed LEX framework can serve as a plug-and-play module for zero-shot GSR. This section 4 ablated all the proposed components on the test split of SWiG dataset [36] .Components in Verb Recognition. We first investigated the two major elements utilized in verb recognition: 1) Verb Explainer, which denotes using verb-centric descriptions for verb 4 More ablation studies are left in the Appendix. classification (Sec. 3.1.3); and 2) Weighting, which denotes the employment of description weighting for the text embeddings of all generated verb-centric descriptions (Sec. 3.1.2). Results are shown in Table 2 . The first row corresponds to the performance of the CLS baseline. As seen, CLS achieves 30.18% top-1 and 55.49% top-5 verb accuracy. Upon applying the proposed verb explainer (the second row), we observe consistent and substantial improvements for both top-1 accuracy (30.18% → 32.13%) and top-5 accuracy (55.49% → 57.84%). This highlights the importance of our "recognition with explainer" strategy and validates the viability of our motivation. Moreover, LEX achieves better performance across all metrics with the weighting strategy. This indicates that the proposed verb explainer and weighting strategy can work synergistically.Component in Semantic Role Grounding. We next studied the impact of our Grounding Explainer, which adopts rephrased verb-centric templates as grounding input (Sec. 3.2.1). As illustrated in Table 3 , our grounding explainer proves to be effective, e.g., 0.81% accuracy improvement in terms of the grounding (grnd) metrics.Components in Noun Recognition. We further analyzed the influence of three major components proposed for noun recognition: 1) Filter, which denotes filtering those unreasonable nouns by using LLMs (Sec. 3.3.1); 2) Noun Explainer, which denotes using scene-specific noun descriptions for noun prediction (Sec. 3.3.3); and 3) Refine, which denotes using contextual information of the scene to refine final results (Sec. 3.3.4) . From the results in Table 4 , we have the following findings: 1) Filtering those unreasonable noun candidates leads to significant performance gains across all noun metrics(e.g., 15% in value metrics and 3.64% in val-all metrics). 2) Substantial improvements can be made by incorporating the noun explainer which aims to provide more contextual information (e.g., 28.51% → 29.16% in value metrics). 3) Furthermore, after incorporating the refine module, we achieve considerable gains of 0.88% in value metrics and 1.51% in grnd metrics. 4) Finally, by integrating three core components, LEX delivers the best performance across all metrics. This validates the effectiveness of our comprehensive modular designs.4.3.4 Different Architectures of CLIP. Last, we examined the impact of the utilized CLIP's architectures. As outlined in Table 5 , regardless of the visual encoder employed, our LEX demonstrates consistent and substantial improvements across all metrics. Such impressive results further verify the robustness of our approach.We visualized CLIP's attention maps for various images and query prompts in Figure 6 . As is displayed in Figure 6 (a), we can observeThe class-contained/verb-centric template contains both verb class and its associated semantic roles, which is provided in the dataset. 
 
 Appendix:  


Decision: All reviewers think the motivation and idea is novel and inspiring. Particularly, reviewer tvvz think the performance gap between zero-shot GSR and the supervised counterpart is large, thus does not think the zero-shot direction is not convincing. After checking the comments and author’s rebuttal, the AC think there are some research and practical values for the proposed zero-shot GSR task, thus encouraging the future research on this direction. Acceptance is recommended.

Reviews: review: paper_summary: paper_summary: The paper propose a new method for zero-shot grounded situation recognition in images. They introduce a novel training-free zero-shot GSR framework LEX that incorporates LLMs as explainers, enhancing understanding of complex visual scenes. The proposed method not only understands the coarse level classes, but also the holistic scene through verbs, semantic roles, their nouns, and their grounding in a zero shot fashion. They evaluate the method on the SWIG dataset.
strength_weakness: strength_weakness: A novel zero-shot method for a complex task which generally requires complex and structured supervision in terms of - verbs, semantic roles, nouns, and groundings. 

Solving a complex structured task with LLMs as planners without requiring expensive annotations is inspiring. This presents a new direction for future research in holistic scene level understanding with limited supervision.
limitations: limitations: Performance of the proposed zero-shot approach is quite low when compared with the supervised counterpart. The gap seems too big to be closed. Specially the value and value all metric are way below the SOTA. 

The performance on value and value-all is very low, what is the random performance? 

The method seems to work well for verbs but not for semantic role labels and noun. Which is the main proposal on the paper- going beyond simple scene level classes. All the complexity of the approach for noun prediction and grounding doesn't seem worth it.

What is the reason behind such low performance? With the current performance, zero-shot SRL doesn't seem like a convincing direction.
paper_summary: paper_summary: The paper proposes a training-free zero-shot GSR framework LEX to introduce LLM, enhancing the understanding of visual scenes. LEX introduces three explainers. Extensive experiments on the SWiG dataset demonstrate the effectiveness and interpretability of LEX.
strength_weakness: strength_weakness: 1. Easy understanding.
2. The paper proposes three explainers to prompt zero-shot GSR. 
3. The good experimental results demonstrate the performance surpasses that of previous state-of-the-art methods.
limitations: limitations: 1. Introducing the LLM mat increases inference time. Please provide more effectiveness analysis, e.g., inference time.
2. The text generated by large language models may introduce additional noise. The author how to ensure that it is consistently better than the original text.
3. The analysis of some hyperparameters is missing.
paper_summary: paper_summary: The authors proposed a novel approach LEX for zero-shot GSR utilizing large language models to provide rich, contextual explanations at each critical stage of the GSR process. By innovating beyond the constraints of conventional class-based prompt inputs, our approach leverages LLMs to clarify ambiguous actions, accurately ground semantic roles, and ensure context-aware noun identification.
strength_weakness: strength_weakness: The motivation of this paper is very clear. The framework of this paper is very intuitive.
limitations: limitations: (1) In Table 1, TEM lacks literature citations.
(2) It is suggested to add some more methods for comparison to demonstrate the cutting edge of the proposed method.