VideoPhy: Evaluating Physical Commonsense for Video Generation

## Abstract

Abstract Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts and styles. Due to their ability to synthesize realistic motions and render complex objects, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy , a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate a list of 688 688 688 captions that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., VideoCrafter2) and closed models (e.g., Lumiere from Google, Pika). Further, our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, Pika, generates videos that adhere to the caption and physical laws for only 19.7 % percent 19.7 19.7\% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we also supplement the dataset with an auto-evaluator, VideoCon-Physics , to assess semantic adherence and physical commonsense at scale.

## 1 Introduction

The ability to synthesize high-quality videos for a broad range of visual concepts and styles is a long-standing goal of generative modeling []. In this regard, recent advancements in pretraining on internet-scale video data [, , , , ] have led to the development of various text-to-video (T2V) generative models such as Sora [] that can generate photo-realistic videos conditioned on a text prompt [, , , , , , ]. Specifically, these models can generate complex scenes (e.g., ‘busy street in Japan’) and realistic motions (e.g., ‘running’, ‘pouring’), making them amenable for understanding and simulating the physical world. Recent efforts [, ] have further utilized text-guided video generation to train agents that can act, plan, and solve goals in the real world. In spite of the strong physical motivations of these works, it remains unclear how well the generated videos from T2V models adhere to the laws of physics.

To evaluate the quality of a T2V generative model, Fréchet video distance (FVD) is traditionally used to measure the similarity between real and generated video distributions [, ]. However, FVD has several limitations for assessing physical commonsense including the requirement for a reference video that is difficult to obtain for novel scenes, bias towards video quality, and failure to detect unrealistic motions [, ]. Similarly, CLIPScore [] measures semantic similarity between generated video frames and the conditioning text in a shared representation space, making it unsuitable for evaluating physical commonsense in generated videos. Moreover, prior work [] introduced a comprehensive benchmark to evaluate various qualities of generated videos (e.g., motion smoothness, background consistency) using existing models, but it does not specifically address the generated videos’ adherence to physical laws. Therefore, existing benchmarks and metrics are either unreliable or lack coverage for holistic evaluation of the physical commonsense capabilities.

To this end, we propose VideoPhy, a dataset designed to evaluate the adherence of generated videos to physical commonsense in real-world activities. Specifically, physical commonsense focuses on the intuitive understanding of the behavior and dynamics of various states of matter (solids, fluids) in the physical world [, , ]. For instance, ‘water pouring into a glass’ will intuitively result in the water level in the glass rising over time. As a result, we rely on human perception and experience in the physical world to assess the adherence of the generated videos to physical laws instead of precise dynamical equations, which are harder to assess. In Figure [2](#S0.F2), we provide qualitative examples to illustrate physical commonsense violations in the videos. Our dataset is constructed through a three-stage pipeline that involves (a) prompting a large language model [] to generate candidate captions that depict interactions between diverse states of matter (e.g., solid-solid, solid-fluid, fluid-fluid), (b) human verification of the generated captions, and (c) annotating the complexity in rendering objects or synthesizing motions described in the captions based on physics simulation.

In total, VideoPhy comprises 688 high-quality, human-verified captions that will be used to generate videos from T2V models. In addition, the dataset consists human-labeled annotations for physical commonsense of the generated videos. Specifically, we acquire generated videos from nine diverse T2V models including open models (e.g., OpenSora [], StableVideoDiffusion [], VideoCrafter2 []) and closed models (e.g., Pika [], Lumiere [] from Google, Gen-2 [] from Runway). Subsequently, we perform human evaluation on the generated videos for semantic adherence to the conditioning text (e.g., do the videos follow the caption?) and physical commonsense (e.g., do the videos follow physical laws intuitively?). Interestingly, we find that the existing T2V generative models severely lack the capability to follow caption accurately and generate videos with physical commonsense. Specifically, the best performing model, Pika, follows the text and generates physically accurate videos for $19.7\%$ of the instances (§[5](#S5)). In Figure [1](#S0.F1), we compare the performance (i.e., accurate semantic adherence and physical commonsense) of various T2V generative models on the VideoPhy dataset, as judged by human annotators.

Although human evaluation of semantic adherence and physical commonsense is reliable, it is both expensive and difficult to scale. To address this challenge, we introduce VideoCon-Physics, a video-language model designed to assess the semantic adherence and physical commonsense of generated videos using user queries grounded in text. Specifically, we fine-tune VideoCon [], a robust semantic adherence evaluator for real videos, on generated videos and human annotations from our VideoPhy dataset. Our results demonstrate that VideoCon-Physics outperforms Gemini-Pro-Vision-1.5 [], showing a $9$ points improvement in semantic adherence and a $15$ points improvement in physical commonsense on unseen prompts. Overall, the VideoPhy dataset aims to bridge the gap in understanding physical commonsense in generated videos and enables scalable testing.

## 2 VideoPhy Dataset

Our dataset, VideoPhy, aims to offer a robust evaluation benchmark for physical commonsense in video generative models. Specifically, the dataset is curated with guidelines to cover (a) a wide range of daily activities and objects in the physical world (e.g., rolling objects, pouring liquid into a glass), (b) physical interactions between various material types (e.g., solid-solid or solid-fluid interactions), and (c) the perceived complexity of rendering objects and motions under graphic simulation. For instance, ketchup, which follows Non-Newtonian fluid dynamics [], is harder to model and simulate than water, which follows Newtonian fluid dynamics, using traditional fluid simulators [].

Under the collection guidelines, we curate a list of text prompts that will be used for conditioning the text-to-video generative models. Specifically, we follow the 3-stage pipeline to create the dataset.

#### LLM-Generated Captions (Stage 1).

Here, we query a large language model, in our case GPT-4 [], to generate a list of $1000$ candidate captions depicting real-world dynamics. As the majority of real-world dynamics involve solids or fluids, we broadly classify those dynamics into three categories: solid-solid interactions, solid-fluid interactions, and fluid-fluid interactions. Specifically, we consider fluid dynamics involving in-viscid and viscous flows—representative examples being water and honey, respectively. On the other hand, we find that solids exhibit more diverse constitutive models, including but not limited to rigid bodies, elastic materials, sands, metals, and snow. In total, we prompt GPT-4 to generate $500$ candidate captions for solid-solid and solid-fluid interactions, and $200$ candidate captions for fluid-fluid interactions. We present the GPT-4 prompts in Appendix [E](#A5).

#### Human Verification (Stage 2).

Since LLM-generated captions may not adhere to our input query, we perform a human verification step to filter bad generations. Specifically, the authors perform human verification to ensure the quality and relevance of the captions, adhering to these criteria: (1) the caption must be clear and understandable (2) the caption should avoid excessive complexity, such as overly varied objects or too intricate dynamics (3) the captions must accurately reflect the intended interaction categories, ensuring, for example, that fluids are indeed described in solid-fluid or fluid-fluid dynamics. To maintain focus on the fundamental interactions among solids and fluids, we also exclude captions involving complex physical phenomena such as phase changes (e.g. ice melting into water) or magnetic effects. Finally, we have $688$ captions where $289$ captions for solid-solid interactions, $291$ for solid-fluid interactions, and $108$ for fluid-fluid interactions, respectively.

#### Difficulty Annotation (Stage 3).

To acquire fine-grained insights into the quality of the video generation, we further annotate our each instance in the dataset with perceived difficulty. Specifically, we ask two experienced graphics researchers (senior Ph.D. students in physics-based simulation) to independently classify each caption as easy (0) or hard (1) based on their perception of the complexity in simulating the objects and motions in the captions using state-of-the-art physics engines [, , , , , ]. Subsequently, the disagreements were discussed to reach a unanimous judgement for less than $5\%$ of the instances. We note that the level of difficulty is evaluated within each category (e.g., solid-solid, solid-fluid, fluid-fluid), and cannot be compared across different categories. We present the examples for generated captions in Table [6](#A3.T6) in Appendix [C](#A3).

Figure: Table 1: Key statistics of the VideoPhy dataset.
Refer to caption: /html/2406.03520/assets/x2.png

#### Data Analysis.

A fine-grained metadata facilitates a comprehensive understanding of the benchmark. Specifically, we present the main statistics of the VideoPhy dataset in Table [1](#S2.T1). Notably, we generate $9000$+ videos for the prompts in the dataset using a diverse range of generative models. In addition, the average caption length is $8.5$ words, indicating that most captions are straightforward and do not complicate our analysis with complex phrasing that could be excessively challenging the generative models. The dataset includes 138 unique actions grounded in our captions. Additionally, Figure [3](#S2.F3) visualizes the root verbs and direct nouns used in the VideoPhy captions, highlighting the diversity of actions and entities depicted. Hence, our dataset encompasses a wide range of visual concepts and actions.

## 3 Evaluation

### 3.1 Metrics

The ability to assess the quality of the generated videos is a challenging task. While humans can evaluate videos across various visual dimensions [, ], we focus primarily on the models’ adherence to the provided text and the incorporation of physical commonsense. These are key objectives that conditional generative models must maximize.

#### Semantic Adherence (SA).

This metric evaluates whether the text caption is semantically grounded in the frames of the generated videos. Specifically, it assesses if the actions, events, entities, and their relationships are perceived to be correctly depicted in the video frames (e.g., water is flowing into the glass in the generated video for the caption ‘water pouring into the glass’). In this work, we annotate the generated videos for semantic adherence, denoted as $\text{SA}=\{0,1\}$. Here, $\text{SA}=0$ indicates that some or all of the caption is not grounded in the generated video.

Physical Commonsense (PC). This metric evaluates whether the depicted actions, and object’s state follow the physics laws in the real-world. For instance, the level of water should increase in the glass as water flows into it, following conversation of mass. In this work, we annotate the physical commonsense of the generated videos, denoted as $\text{PC}=\{0,1\}$. Here, $\text{PC}=1$ indicates that the generated movements and interactions align with intuitive physics that humans acquire with their experience in the real-world. As physical commonsense is entirely grounded in the video, it is independent of the semantic adherence capability of the generated video. In this work, we compute the fraction of the videos for which semantic adherence is high ($\text{SA}=1$), physical commonsense is high ($\text{PC}=1$), and joint performance of these metrics is high ($\text{SA}=1,\text{PC}=1$).

### 3.2 Human Evaluation

We conducted a human evaluation to assess the performance of the generated videos in terms of semantic adherence and physical commonsense using our dataset. Annotations were obtained from a group of qualified Amazon Mechanical Turk (AMT) workers who had passed a qualification test. The workers were compensated at a rate of $\$18$ per hour. In this task, annotators were presented with a caption and the corresponding generated video without any information about the generative model. They were asked to provide a semantic adherence score (0 or 1) and a physical commonsense score (0 or 1) for each instance. Annotators were instructed to treat semantic adherence and physical commonsense as independent metrics and were shown several solved examples by the authors before starting the main annotation task. In some cases, we find that generative models create static scenes instead of video frames with high motion. Here, we ask annotators to judge the physical plausibility of the static scene in the real world (e.g., a static scene of a folded brick does not follow physical commonsense). However, if the static scenes are noisy (e.g., unwanted grainy or speckled patterns), we instruct them to consider it as poor physical commonsense.

In our experiments, the annotators have studied high school-level physics. However, the human annotators were not asked to list the violation of the physics laws since it would make the annotations more time-consuming and expensive. Additionally, the current annotations can be performed by annotators experience in the physical world (e.g., workers know that water flows down from a tap, shape of a wood log will not change while floating on water) instead of advanced education in physics. A screenshot of the human annotation interface is presented in Appendix [F](#A6).

### 3.3 Automatic Evaluation

While the human evaluation is more accurate for model benchmarking, it is time-consuming and expensive to collect at scale. To this end, we evaluate the performance of various zero-shot methods in judging the quality of the generated videos in terms of semantic adherence and physical commonsense. Further, we propose VideoCon-Physics, a capable automatic evaluator on our dataset.

#### Baselines.

Similar to [], we utilize the capability of GPT-4Vision [] to reason over multiple images in a zero-shot manner. Specifically, we prompt the GPT-4V model with the caption and $8$ video frames sampled uniformly from the generated video. Here, we instruct the model to provide the semantic adherence (0 or 1) and physical commonsense score (0 or 1). Since GPT-4V does not process videos natively, we assess the automatic evaluation using Gemini-Pro-Vision-1.5, which can input the caption and the entire generated video. Specifically, we instruct it to provide the semantic adherence (0 or 1) and physical commonsense (0 or 1) of the input video, identical to the GPT-4V analysis. We provide the prompts used in the experiments in Appendix [G](#A7)

Since the previous two models are closed, it is difficult to fine-tune them with custom data. As a result, we use VideoCon, an open generative video-text language model with $7B$ parameters, that is trained on real videos for robust semantic adherence evaluation []. Specifically, we prompt VideoCon to generate a text response (Yes/No) conditioned on the multimodal template $\mathcal{T}_{t}(x)$ for semantic adherence and physical commonsense tasks. Formally,

$$ $\mathcal{T}_{t}(x)=\begin{cases}\mathcal{T}_{SA}(V,C),&t=SA\\ \mathcal{T}_{PC}(V),&t=PC\end{cases}$ (1) $$

where $t$ is either semantic adherence to the caption or physical commonsense task, $C$ is the conditioning caption and $V$ is the generated video for the caption $C$. We provide the multimodal templates ($\mathcal{T}_{SA}(V,C)$, $\mathcal{T}_{PC}(V)$) in Appendix [G](#A7). We compute the score from the VideoCon model $p_{\theta}$:

$$ $s_{\theta}(\mathcal{T}_{t}(x))=\frac{p_{\theta}(\text{{Yes}}|\mathcal{T}_{t}(x))}{p_{\theta}(\text{{Yes}}|\mathcal{T}_{t}(x))+p_{\theta}(\text{{No}}|\mathcal{T}_{t}(x))},$ (2) $$

where $p_{\theta}(\text{{Yes}}|\mathcal{T}_{t}(x))$ is the probability of ‘Yes’ conditioned on $\mathcal{T}_{t}(x)$, and $t\in\{SA,PC\}$. (^1^11As a large video multimodal model, VideoCon predicts a token distribution over the entire token vocabulary conditioned on the multimodal template. Therefore, $p_{\theta}(\text{{Yes}}|\mathcal{T}_{t}(x))+p_{\theta}(\text{{No}}|\mathcal{T}_{t}(x))$ is not equal to 1.)

#### VideoCon-Physics .

Since VideoCon is not trained on the generated video distribution or equipped to judge physical commonsense, it is not expected to perform well in our setup in a zero-shot manner. To this end, we propose VideoCon-Physics, an open-source generative video-text model, that can assess the semantic adherence and physical commonsense of the generated videos. Specifically, we finetune VideoCon by combining the human annotations acquired for the semantic adherence and physical commonsense tasks over the generated videos.(^2^22We note that finetuning separate classifier for semantic adherence and physical commonsense did not provide any additional benefits over a single classifier (VideoCon-Physics) trained in a multi-task manner on the downstream tasks.)

Overall, we evaluate the usefulness of the baseline and the proposed model by computing the AUC-ROC between the human annotations and model predictions for diverse generated videos on the unseen prompts. We will provide more details on the train and test set in §[4.2](#S4.SS2).

## 4 Setup

In this section, we present the list of text-to-video generative models benchmarked on the VideoPhy dataset (§[4.1](#S4.SS1)) and provide further details about the dataset splits (§[4.2](#S4.SS2)).

### 4.1 Text-to-Video Generative Models

We evaluate a diverse range of nine closed and open text-to-video generative models on VideoPhy dataset. The list of the models includes ZeroScope [], LaVIE [], VideoCrafter2 [], OpenSora [], StableVideoDiffusion (SVD)-T2I2V [], Gen-2 (Runway) [], Lumiere-T2V, Lumiere-T2I2V (Google) [], and Pika []. Here, the T2I2V models involve the generation of an image (I) conditioned on the caption (T) followed by video generation (V) conditioned on the generated image. We provide more details about these models in Appendix [D](#A4). While there are various closed models such as Sora [] and Genmo [], we could not get access through their videos due to the lack of API support. We provide inference details in Appendix [J](#A10).

### 4.2 Dataset

As described earlier, we train an automatic evaluation model to enable cheaper and scalable testing of the generated videos on our dataset (§ [3.3](#S3.SS3)). To facilitate this, we split the prompts in the VideoPhy dataset equally into train and test sets. Specifically, we utilize the human annotations on the generated videos for the $344$ prompts in the test set for benchmarking, while the human annotations on the generated videos for the $344$ prompts in the train set are used for training the automatic evaluation model. We ensure that the distribution of the state of matter (solid-solid, solid-fluid, fluid-fluid) and complexity of the captions (easy, hard) is similar in the training and testing.

#### Benchmarking.

Here, we generate one video per test prompt for each T2V generative model in our testbed. Subsequently, we ask three human annotators to judge the semantic adherence and physical commonsense of the generated videos. In our experiments, we report the majority-voted scores from the human annotators. We find that the inter-annotator agreement for semantic adherence and physical commonsense judgment is $75\%$ and $70\%$, respectively. This indicates that the human annotators find the task of judging physical commonsense more subjective than semantic adherence. (^3^33Since most of the generated videos are not perfect, the variations in the annotations result from diverse tolerance for physical laws violations. As the generative models improve, we believe that the human annotations will achieve higher agreement on our dataset.) In total, we collect $18500+$ human annotations across the testing prompts and T2V models.

#### Training set for VideoCon-Physics .

Here, we sample two videos per training prompt for each T2V generative model in our testbed. Specifically, we choose two videos to obtain more data instances for training the automatic evaluation model. Subsequently, we ask one human annotator to judge the semantic adherence and physical commonsense of the generated videos. In total, we collect $12000+$ human annotations, half of them for semantic adherence and the other half for physical commonsense. Specifically, we finetune VideoCon to maximize the log likelihood of Yes/No conditioned on the multimodal template for semantic adherence and physical commonsense tasks (Appendix [G](#A7)). We do not collect three annotations per video as it is financially expensive. In total, we spent $\$2800$ on collecting human annotations for benchmarking and training.

**Table 2: Human evaluation results on the VideoPhy dataset. We report the percentage of testing prompts for which the T2V models generate videos that adhere to the conditioning caption and exhibit physical commonsense. We abbreviate semantic adherence as SA, and physical commonsense as PC. SA, PC indicates the percentage of the instances for which SA=1 and PC=1. Ideally, we want the generative models to maximize the performance on this metric. In the first column, we highlight the overall performance, and the later columns are dedicated to fine-grained performance for the interaction between different states of matter in the prompts.**
|  | Overall (%) | Solid-Solid (%) | Solid-Fluid (%) | Fluid-Fluid (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | SA, PC | SA | PC | SA, PC | SA | PC | SA, PC | SA | PC | SA, PC | SA | PC |
| Open Models |  |  |  |  |  |  |  |  |  |  |  |  |
| VideoCrafter2 [] | 19.0 | 48.5 | 34.6 | 4.9 | 31.5 | 23.8 | 27.4 | 57.5 | 41.8 | 32.7 | 69.1 | 43.6 |
| LaVIE [] | 15.7 | 48.7 | 28.0 | 8.5 | 37.3 | 19.0 | 15.8 | 52.1 | 30.8 | 34.5 | 69.1 | 43.6 |
| SVD-T2I2V [] | 11.9 | 42.4 | 30.8 | 4.2 | 25.9 | 27.3 | 17.1 | 52.7 | 32.9 | 18.2 | 58.2 | 34.5 |
| ZeroScope [] | 11.9 | 30.2 | 32.6 | 6.3 | 17.5 | 22.4 | 14.4 | 40.4 | 37.0 | 20.0 | 36.4 | 47.3 |
| OpenSora [] | 4.9 | 18.0 | 23.5 | 1.4 | 7.7 | 23.8 | 7.5 | 30.1 | 21.9 | 7.3 | 12.7 | 27.3 |
| Closed Models |  |  |  |  |  |  |  |  |  |  |  |  |
| Pika [] | 19.7 | 41.1 | 36.5 | 13.6 | 24.8 | 36.8 | 16.3 | 46.5 | 27.9 | 44.0 | 68.0 | 58.0 |
| Lumiere-T2I2V [] | 12.5 | 48.5 | 25.0 | 8.4 | 37.1 | 25.2 | 17.1 | 59.6 | 26.0 | 10.9 | 49.1 | 21.8 |
| Lumiere-T2V [] | 9.0 | 38.4 | 27.9 | 8.4 | 26.6 | 27.3 | 9.6 | 47.3 | 26.0 | 9.1 | 45.5 | 34.5 |
| Gen-2 (Runway) [] | 7.6 | 26.6 | 27.2 | 4.0 | 8.9 | 37.1 | 8.1 | 38.5 | 18.5 | 15.1 | 37.7 | 26.4 |

## 5 Results

Here, we present the results of the T2V generative models (§[5.1](#S5.SS1)), and establish the effectiveness of the VideoCon-Physics as an automatic evaluator on the VideoPhy dataset (§[3.3](#S3.SS3)).

### 5.1 Performance on VideoPhy Dataset

We compare the performance of the T2V generative models on the VideoPhy dataset using human evaluation in Table [2](#S4.T2). Specifically, we find that the Pika (closed model) and VideoCrafter2 (open model) generates videos that adhere to the caption and follow physics laws ($\text{SA}=1$, $\text{PC}=1$) in $19.7\%$ and $19\%$ of the cases, respectively. This indicates that the video generative models struggle on the VideoPhy dataset, and far from being general-purpose simulators of the physical world.

**Table 3: Fine-grained performance across caption complexity using human evaluation. We find that T2V models struggle more on the harder captions than the easier captions in both the semantic adherence (SA) and physical commonsense (PC) metrics.**
|  | Easy (%) | Hard (%) |  |  |
| --- | --- | --- | --- | --- |
| Model | SA | PC | SA | PC |
| Open Models |  |  |  |  |
| VideoCrafter2 | 53.4 | 38.1 | 42.6 | 30.3 |
| LaVIE | 51.9 | 31.2 | 44.8 | 24.0 |
| SVD-T2I2V | 41.8 | 37.6 | 43.2 | 22.6 |
| ZeroScope | 32.3 | 33.9 | 27.7 | 31.0 |
| OpenSora | 20.1 | 25.4 | 5.2 | 21.3 |
| Closed Model |  |  |  |  |
| Pika | 45.7 | 39.9 | 35.1 | 32.1 |
| Lumiere-T2I2V | 56.6 | 29.1 | 38.7 | 20.0 |
| Lumiere-T2V | 38.6 | 34.9 | 38.1 | 19.4 |
| Gen-2 (Runway) | 26.6 | 31.8 | 26.6 | 21.6 |

Pika stands out as the best model for generating videos that demonstrate physical commonsense, achieving a performance of $36.5\%$, while VideoCrafter2 is a close second at $34.6\%$. VideoCrafter2’s training process involves a mix of low-quality video-text data and high-quality image-text data, suggesting that incorporating high-quality video data could further enhance its performance. Amongst the closed models, Lumiere-T2I2V emerges as the top video generative model for producing videos that accurately follow the conditioning captions, with a performance rate of $56.6\%$. This indicates at the effectiveness of cascaded approach (text-to-image followed by image-to-video) in generating high-quality, text-adherent videos. Conversely, among the open models, OpenSora performs the worst on the VideoPhy dataset, indicating significant potential for the community to improve open-source implementations of Sora.

#### Variation with the states of matter.

We study the variation in the performance of T2V models with the interaction between the diverse states of matter grounded in the captions (e.g., solid-solid) in Table [5.1](#S5.SS1). Interestingly, we find that all the existing T2V models perform the worst on the captions that depict interactions between solid materials (e.g., bottle topples off the table), with the best performing model, Pika, achieving $13.6\%$ on accurate semantic adherence and physical commonsense. Furthermore, we observe that VideoCrafter2 achieves the highest performance in the captions that depict interaction between solid and fluid material types (e.g., a whisk mixes an egg). This indicates that the T2V model performance is greatly influenced by the states of matter involved in a scene, and highlights that model developers can focus on enhancing semantic adherence and physical commonsense for solid-solid interactions.

#### Variation with the complexity.

We analyze the variation in the T2V model performance with the complexity in rendering objects or synthesizing interactions grounded in the caption under physical simulation in Table [3](#S5.T3). We find that the semantic adherence and physical commonsense performance of all the T2V models decreases as the complexity of the captions increases. This indicates that the captions that are harder to simulate physically are also harder to control via conditioning for the T2V generative models. Our analysis thus highlights that the future T2V model development should focus on reducing the gap between the easy and the hard captions from our VideoPhy dataset. We present the results for additional metrics in Table [8](#A9.T8) in Appendix [I](#A9).

**Table 4: Comparison of ROC-AUC for automatic evaluation methods. We find that VideoCon-Physics outperforms diverse baselines, including GPT-4Vision and Gemini-1.5-Pro-Vision, for semantic adherence (SA) and physical commonsense (PC) judgments on the testing prompts.**
| Method | ROC-AUC SA | ROC-AUC PC |
| --- | --- | --- |
| Random | 50 | 50 |
| GPT-4-Vision [] | 53 | 53 |
| Gemini-Pro-Vision-1.5 [] | 73 | 58 |
| VideoCon [] | 65 | 54 |
| VideoCon-Physics (Ours) | 82 | 73 |

### 5.2 VideoCon-Physics : Automatic Evaluator for VideoPhy Dataset

Here, we propose VideoCon-Physics model for scalable and reliable evaluation of semantic adherence and physical commonsense in the generated videos. We compare the ROC-AUC agreement of different automatic evaluators with the human predictions on the testing prompts in Table [4](#S5.T4). We find that the VideoCon-Physics outperforms the zeroshot VideoCon by $17$ points and $19$ points on the semantic adherence and physical commonsense judgment, respectively. This highlights that finetuning with the generated video distribution and human annotations aids in improving the model judgment on the unseen prompts. Further, we notice that the model’s agreement are higher for semantic adherence as compared to the physical commonsense. This indicates that judging physical commonsense is a harder task than judging semantic adherence for VideoCon-Physics.

Interestingly, we observe that the GPT-4-Vision’s judgments are close to random for semantic adherence and physical commonsense on our dataset. This implies that faithful evaluations are hard to obtain from the multi-image reasoning capabilities of the GPT-4-Vision in a zeroshot manner. To address this, we test Gemini-Pro-Vision-1.5 and find that it achieves a good semantic adherence score ($73$ points), however, it is close to random in physical commonsense evaluation ($54$ points). This highlights that the existing multimodal foundation models lack the capability to judge physical commonsense. Overall, our results suggest that VideoCon-Physics is the best automatic evaluator for the VideoPhy dataset. We provide more discussion on the usefulness of VideoCon-Physics in Appendix §[L](#A12).

**Table 5: Performance of VideoCon-Physics on unseen generative model. We train an ablated version of VideoCon-Physics and find that it outperforms the baseline in the semantic adherence (SA) and physical commonsense (PC) judgment averaged over three unseen video models on the testing prompts.**
| Method | SA | PC |
| --- | --- | --- |
| VideoCon [] | 64 | 57 |
| VideoCon-Physics (Ours) | 79 | 72 |

#### VideoCon-Physics generalizes to unseen generative models.

To assess performance on an unseen video distribution, we train an ablated version of VideoCon-Physics on a restricted set of video data. Specifically, we train VideoCon-Physics on human annotations acquired from VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V, and Gen-2, and evaluate it on unseen videos from Lumiere-T2V, Lumiere-T2I2V, and Pika generated for the testing captions. We compare the performance of the zeroshot VideoCon and VideoCon-Physics in Table [5](#S5.T5). We find that VideoCon-Physics outperforms VideoCon by $15$ points and $15$ points on semantic adherence and physical commonsense judgement, respectively. This highlights that VideoCon-Physics can judge semantic adherence and physical commonsense as new T2V generative models are released.

## 6 Qualitative Examples

Figure: Figure 4: Qualitative comparison of Pika with other models. The top row presents the generated videos from the Pika model. (a) For SVD-T2I2V, the apple remains unnaturally still in the flowing water. (b) For Lumiere-T2I2V, a part of the red wine is not falling. (c) For Gen-2, the drumstick deforms over time.
Refer to caption: /html/2406.03520/assets/x3.png

Here, we present some qualitative examples to understand the common failure modes in the generated video regarding poor physical commonsense. Qualitative examples from various T2V generative models are provided in Figure [15](#A14.F15) - [23](#A14.F23) in Appendix [N](#A14). The common failure modes include – (a) Conservation of mass violation: the volume or texture of an object is not consistent over time, (b) Newton’s First Law violation: an object changes its velocity in a balanced state without any external force, (c) Newton’s Second Law violation: an object violates the conversation of momentum, (d) Solid Constitutive Law violation: solids deform in ways that contradict their material properties, e.g., a rigid object deforming over time, (e) Fluid Constitutive Law violation: fluids exhibit unnatural flow motions, and (f) Non-physical penetration: objects unnaturally penetrate each other.

In addition, we analyze some qualitative examples to understand the gap between the top-tier models (Pika and VideoCrafter2) and the other models in our testbed. We present the examples in Figure [4](#S6.F4) and Figure [14](#A13.F14) in Appendix [M](#A13). For instance, we find that SVD-T2I2V is likely to underperform in scenes involving vibrant fluid dynamics. Lumiere-T2I2V performs better than Lumiere-T2V in terms of visual quality, but still lacks a profound understanding of gravity (e.g. in Figure [4](#S6.F4)(b)). Gen-2 sometimes cannot differentiate multiple objects, thus deforming rigid objects in dynamic motions. Additional observations are reported in Appendix [M](#A13). Our analysis highlights the lack of fine-grained physical commonsense understanding that future video modeling research should aim to address.

## 7 Detailed Related Work

#### Video Generation Models.

Recent advancements in video generation models have emerged from two primary architectures: diffusion-based models [, , , , , , , ] and autoregressive modeling-based approaches [, , , ]. Among these, diffusion models have garnered significant attention. The model known as SVD [], built on a Latent Diffusion Model (LDM) [], proposes a three-stage training process for video LDMs: text-to-image pretraining, video pretraining, and video finetuning. Sora [] represents a state-of-the-art in video generation, utilizing a diffusion-transformer architecture with unified training recipes and enhancements in language description processing for video generation. ModelScope [] is also a diffusion-based text-to-video model which combines a VQGAN [], a text-encoder, and a denoising UNet. Another diffusion model, VideoCrafter2 [], leverages low-quality videos and high-quality videos to generate high-quality videos. LaVIE [] is composed of a base text-to-video model, a temporal interpolation model, and a video super-resolution model, indicating that joint image and video training and temporal self-attention with rotary positional embeddings are key components to boost performance. Given the rapid development of video generation technology, an effective evaluation method for the generated videos becomes crucial. Our paper focuses on evaluating text-to-video generation models for their physical commonsense capabilities.

#### Evaluating Video Generation Models.

Traditional evaluation methods for video generation primarily employ metrics such as FVD [] and IS []. However, there is a growing consensus on the need for more comprehensive metrics to assess the performance of video generation models [, , , ]. V-Bench [] offers a detailed benchmark suite that introduces a hierarchical evaluation protocol, breaking down ‘video generation quality’ into various granular perspectives. Another framework, EvalCrafter [], proposes 17 objective metrics. Despite these advancements, existing methods largely overlook the fundamental aspect of physical commonsense. Unlike static images, videos incorporate a temporal dimension, embedding physical commonsense information across frames. Our research dives into the measurement of physical commonsense [] in videos. Additionally, we introduce a VideoCon-Physics auto-evaluator and analyze specific physical laws that are violated in the generated videos through qualitative analysis.

#### Physics Modeling.

Simulating physical behaviors of solids and fluids has always been an important and popular topic in computer graphics. For solid materials, the simplest physical model is the long-established rigid body simulation [], where solids are assumed not to deform. Simulation of deformable solids [], on the other hand, takes into account the strain and stress during deformation. To capture more complicated materials, researchers have been proposing increasingly intricate models for different materials, such as metal [], sand [], and snow []. In contrast, most of the common fluids [] in daily life can be broadly categorized as inviscid [], e.g., water and air, and viscous fluids [, ], e.g., honey and oil. Additionally, an orthogonal research direction is to accurately, efficiently, and robustly model contact and interaction between different materials. These include solid-solid [, ], solid-fluid [, ], and fluid-fluid interactions []. Further, recent advancements in computer vision have started exploring incorporating physics priors into various 3D-aware generation tasks to enhance physical plausibility, such as human animation [, , ] and 3D/4D generation [, , ]. In this work, instead of generating, we focus on identifying whether the generated video adheres to physical laws.

## 8 Conclusion

In this work, we introduce VideoPhy, a first of its kind dataset to assess the physical commonsense in the generated videos. Further, we evaluate a diverse set of T2V models (open and closed models) and found that they significantly lack in the physical commonsense and semantic adherence capabilities. Our dataset unveils that the existing methods are far being general-purpose world simulators. Further, we introduce VideoCon-Physics, an auto-evaluation model that enables cheap and scalable evaluation on our dataset. We believe that our work will serve as the cornerstone in studying physical commonsense for video generative modeling.

## 9 Acknowledgement

Hritik Bansal is supported in part by AFOSR MURI grant FA9550-22-1-0380.

## Appendix A Limitations

In this work, we evaluate the physical commonsense capabilities of T2V generative models. Specifically, we curated the VideoPhy dataset, consisting of 688 captions. We argue that the captions are comprehensive and high-quality after going through our three-stage data curation pipeline. In the future, it will be pertinent to expand the physical commonsense understanding to more branches of physics, including projective geometry. Additionally, we test a diverse set of T2V generative models, including both open and closed models. While it is financially and computationally challenging to evaluate an exhaustive list of models, we have aimed to incorporate models with diverse architectures, training datasets, and inference strategies. In the future, it will be important to gain access to and include new high-performance T2V models in our study.

In addition, we perform human annotations using Amazon Mechanical Turkers (AMT), where most of the workers primarily belong to the US and Canada. Hence, the human annotations in this work do not represent the diverse demographics around the globe. As a result, our human annotations reflect the perceptual biases of the annotators from Western cultures. In the future, it will be pertinent to assess the impact of diverse groups on our human evaluations. Finally, we acknowledge that text-to-video generative models can perpetrate societal biases in their generated content [, ]. It is critical that future work quantifies this bias in the generated videos and provides methods for the safe deployment of the models.

## Appendix B Data Licensing

The VideoPhy dataset comprises videos generated by various T2V (Text-to-Video) generative models, detailed in Section [D](#A4). The licensing terms for these videos will align with those specified by the respective model owners, as cited in this work. The curated captions and human annotations will be licensed under the MIT License.

## Appendix C Example captions in the VideoPhy dataset

We present example captions in our dataset in Table [6](#A3.T6).

**Table 6: Example captions in the VideoPhy dataset. Specifically, we design them to depict the interactions between two states of matter (solid-solid, solid-fluid, fluid-fluid). We further classify the captions as easy or hard based on the modeling and simulation complexity in the computer graphics. We highlight the reasoning behind the easy and hard annotations by our annotators in the yellow.**
| Category | Difficulty | Example Captions |
| --- | --- | --- |
| Solid-Solid | Easy | 1. Bottle topples off the table. (rigid bodies)<br>2. Ball bounces off the floor. (deformable and rigid bodies) |
| Hard | 1. Scrubber scrubs a dirty dish. (complex contacts)<br>2. Scissors trim the paper. (material fracture) |  |
| Solid-Fluid | Easy | 1. Water flows down a circular drain. (contacts with rigid bodies)<br>2. A paint roller coating a wall. (slow/static fluids) |
| Hard | 1. A swimmer splashing in the sea water. (contacts with high-speed)<br>2. A whisk mixes an egg in a bowl. (contacts with high-speed) |  |
| Fluid-Fluid | Easy | 1. Rain splashing on a pond. (mixing of same fluids)<br>2. Sour cream swirls in hot soup. (layering) |
| Hard | 1. Ink spreading in still water. (mixing of different fluids)<br>2. Honey threading slowly into tea. (mixing of different fluids) |  |

## Appendix D Video Generative Models

For the open models, we benchmark Zeroscope [, ], a latent diffusion-based text-to-video model that adapts the text-to-image generative model [] for video generation by training on high-quality video and image data for enhanced visual quality. Further, we benchmark LaVIE [], a cascaded video latent diffusion model instead of a single diffusion model. Specifically, the LaVIE model is trained with a specialized curated dataset for enhanced visual quality and diversity. In addition, we test VideoCrafter2, a latent diffusion T2V model that enhances video generation quality by training on high-quality image-text data []. In our study, we also benchmark OpenSora [], an open-source effort to replicate Sora [], a high-performant closed latent diffusion model that uses diffusion transformers [] for text-to-video generation. Finally, we include StableVideoDiffusion (SVD) [], a latent diffusion model that can generate high resolution videos conditioned on a text or image. Since SVD-I2V (Image-to-Video) is publicly available, we utilize that to generate the videos. Specifically, we utilize SD-XL-Base-1.0 [] to generate the conditioning images from the captions in the VideoPhy dataset. We term the entire pipeline as SVD-T2I2V.

For the closed models, we include Gen-2 [], a closed latent video diffusion model from Runway. In addition, we include Pika [] with undisclosed information about the underlying generative model. Specifically, we wrote a custom API to acquire Gen-2 and Pika videos after paying for their monthly subscription for a total of $\$225$. Finally, we include two versions of the Lumiere [] from Google research. Specifically, Lumiere-T2V generates a video conditioned on the text, while Lumiere-T2I2V generates a video conditioned on an image, that is in-turn generated with the caption using a text-to-image generative model [].

## Appendix E Querying GPT-4 for Prompt Generation

In this section we discuss the prompt we utilized to generate all the prompts including three physical interaction categories: solid-solid, solid-fluid, fluid-fluid for video generation, which is displayed in Table  [5](#A5.F5), Table  [6](#A5.F6) and Table  [7](#A5.F7).

Figure: Figure 5: GPT-4 Prompt to Generate Solid-Solid Captions.

Figure: Figure 6: GPT-4 Prompt to Generate Solid-Fluid Captions.

Figure: Figure 7: GPT-4 Prompt to Generate Fluid-Fluid Captions.

## Appendix F Human Annotation Screenshot

We display the screenshot of our human annotation system in Figure [8](#A6.F8)

Figure: Figure 8: The screenshot of the human annotation interface.
Refer to caption: /html/2406.03520/assets/images/human_annotation_screenshot.png

## Appendix G Multimodal Template for Prompting Models

We present the prompts used for the GPT4V, Gemini-1.5-Pro-Vision, VideoCon baselines, and VideoCon-Physics for semantic adherence evaluation in Figure [9](#A7.F9) and physical commonsense alignment in Figure [10](#A7.F10).

Figure: Figure 9: Template used assessing semantic adherence for a generated video.

Figure: Figure 10: Template for assessing physical commonsense. We note that the physical commonsense is independent of the conditioning caption. Hence, it is not present in this template.

## Appendix H Fine-Grained Statistics of Collections Across Different Physical Interaction Categories

In this section, we visualize the fine-grained statistics of collections across different physical interaction categories (Figure [11](#A8.F11) - Figure [13](#A8.F13)).

Figure: Figure 11: Top 20 most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) in our curated captions that consists of interaction between solid-solid states of matter.
Refer to caption: /html/2406.03520/assets/x4.png

Figure: Figure 12: Top 20 most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) in our curated captions that consists of interaction between solid-fluid states of matter.
Refer to caption: /html/2406.03520/assets/x5.png

Figure: Figure 13: Top 20 most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) in curated captions that consists of interaction between fluid-fluid states of matter.
Refer to caption: /html/2406.03520/assets/x6.png

## Appendix I Fine-Grained Results

In this section, we report the fine-grained performance of semantic adherence and physical commonsense scores from all video generation models and compute the scores across different physical interaction categories (solid-solid, solid-fluid and fluid-fluid), as well as difficulty levels (0 and 1).

**Table 7: Fine-grained performance of T2V models for the interaction between diverse states of matter using human evaluation. Ideally, we want the T2V models to achieve a high score on the SA = 1 and PC = 1 metric while reduce the score on the SA=0 and PC=0, SA=1 and PC=0, and SA=0 and PC=1 metrics.**
| Source | Category | SA (%) | PC (%) | SA=1 and PC=1 (%) | SA=1 and PC=0 (%) | SA=0 and PC=1 (%) | SA=0 and PC=0 (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Open Models |  |  |  |  |  |  |  |
| LaVIE | Fluid-Fluid | 69.1 | 43.6 | 34.5 | 34.5 | 9.1 | 21.8 |
| Solid-Fluid | 52.1 | 30.8 | 15.8 | 36.3 | 15.1 | 32.9 |  |
| Solid-Solid | 37.3 | 19.0 | 8.5 | 28.9 | 10.6 | 52.1 |  |
| OpenSora | Fluid-Fluid | 12.7 | 27.3 | 7.3 | 5.5 | 20.0 | 67.3 |
| Solid-Fluid | 30.1 | 21.9 | 7.5 | 22.6 | 14.4 | 55.5 |  |
| Solid-Solid | 7.7 | 23.8 | 1.4 | 6.3 | 22.4 | 69.9 |  |
| VideoCrafter2 | Fluid-Fluid | 69.1 | 43.6 | 32.7 | 36.4 | 10.9 | 20.0 |
| Solid-Fluid | 57.5 | 41.8 | 27.4 | 30.1 | 14.4 | 28.1 |  |
| Solid-Solid | 31.5 | 23.8 | 4.9 | 26.6 | 18.9 | 49.7 |  |
| SVD-T2I2V | Fluid-Fluid | 58.2 | 34.5 | 18.2 | 40.0 | 16.4 | 25.5 |
| Solid-Fluid | 52.7 | 32.9 | 17.1 | 35.6 | 15.8 | 25.5 |  |
| Solid-Solid | 25.9 | 27.3 | 4.2 | 21.7 | 23.1 | 51.0 |  |
| ZeroScope | Fluid-Fluid | 36.4 | 47.3 | 20.0 | 16.4 | 27.3 | 36.4 |
| Solid-Fluid | 40.4 | 37.0 | 14.4 | 26.0 | 22.6 | 37.0 |  |
| Solid-Solid | 17.5 | 22.4 | 6.3 | 11.2 | 16.1 | 66.4 |  |
| Closed Models |  |  |  |  |  |  |  |
| Gen-2 | Fluid-Fluid | 37.7 | 26.4 | 15.1 | 22.6 | 11.3 | 50.9 |
| Solid-Fluid | 38.5 | 18.5 | 8.1 | 30.4 | 10.4 | 51.1 |  |
| Solid-Solid | 8.9 | 37.1 | 4.0 | 4.8 | 33.1 | 58.1 |  |
| Lumiere-T2V | Fluid-Fluid | 45.4 | 34.5 | 9.1 | 36.4 | 25.5 | 29.1 |
| Solid-Fluid | 47.2 | 26.0 | 9.6 | 37.7 | 16.4 | 36.3 |  |
| Solid-Solid | 26.5 | 27.3 | 8.4 | 18.2 | 18.9 | 54.5 |  |
| Lumiere-T2I2V | Fluid-Fluid | 49.5 | 21.8 | 10.9 | 38.2 | 10.9 | 40.0 |
| Solid-Fluid | 59.6 | 26.0 | 17.1 | 42.5 | 8.9 | 31.5 |  |
| Solid-Solid | 37.1 | 25.2 | 8.4 | 28.7 | 16.8 | 46.2 |  |
| Pika | Fluid-Fluid | 68.0 | 58.0 | 44.0 | 24.0 | 14.0 | 18.0 |
| Solid-Fluid | 46.5 | 27.9 | 16.3 | 30.2 | 11.6 | 41.9 |  |
| Solid-Solid | 24.8 | 36.8 | 13.6 | 11.2 | 23.2 | 52.0 |  |

**Table 8: Fine-grained performance of T2V models for the complexity of the captions using human evaluation. Ideally, we want the T2V models to achieve a high score on the SA = 1 and PC = 1 metric while reduce the score on the SA=0 and PC=0, SA=1 and PC=0, and SA=0 and PC=1 metrics.**
| Source | Category | SA (%) | PC (%) | SA=1 and PC=1 (%) | SA=1 and PC=0 (%) | SA=0 and PC=1 (%) | SA=0 and PC=0 (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Open Models |  |  |  |  |  |  |  |
| LaVIE | EASY | 51.9 | 31.2 | 19.6 | 32.3 | 11.6 | 36.5 |
| HARD | 44.8 | 24.0 | 11.0 | 33.8 | 13.0 | 42.2 |  |
| OpenSora | EASY | 20.1 | 25.4 | 4.8 | 15.3 | 20.6 | 59.3 |
| HARD | 15.5 | 21.3 | 5.2 | 10.3 | 16.1 | 68.4 |  |
| VideoCrafter2 | EASY | 53.4 | 38.1 | 21.2 | 32.3 | 16.9 | 29.6 |
| HARD | 42.6 | 30.3 | 16.1 | 26.5 | 14.2 | 43.2 |  |
| SVD-T2I2V | EASY | 42.0 | 38.0 | 16.0 | 25.0 | 21.0 | 37.0 |
| HARD | 43.0 | 23.0 | 6.0 | 37.0 | 16.0 | 41.0 |  |
| ZeroScope | EASY | 32.3 | 33.9 | 13.8 | 18.5 | 20.1 | 47.6 |
| HARD | 27.7 | 31.0 | 9.7 | 18.1 | 21.3 | 51.0 |  |
| Closed Models |  |  |  |  |  |  |  |
| Gen-2 | EASY | 26.6 | 31.8 | 10.4 | 16.2 | 21.4 | 52.0 |
| HARD | 26.6 | 21.6 | 4.3 | 22.3 | 17.3 | 56.1 |  |
| Lumiere-T2V | EASY | 38.6 | 34.9 | 11.1 | 27.5 | 23.8 | 37.6 |
| HARD | 38.1 | 19.3 | 6.5 | 31.6 | 12.9 | 49.0 |  |
| Lumiere-T2I2V | EASY | 56.6 | 29.1 | 16.4 | 40.2 | 12.7 | 30.7 |
| HARD | 38.7 | 20.0 | 7.7 | 31.0 | 12.3 | 49.0 |  |
| Pika | EASY | 45.7 | 39.9 | 23.7 | 22.0 | 16.2 | 38.2 |
| HARD | 35.1 | 32.1 | 14.5 | 20.6 | 17.6 | 47.3 |  |

## Appendix J Inference Details

We add the inference configurations for different video generation models in Table [9](#A10.T9).

**Table 9: Inference details for models in our testbed. Here, NA indicates that the information is not available for the closed models.**
| Model | Resolution | # of Video Frames | Guidance Scale | Sampling Steps | Noise Scheduler |
| --- | --- | --- | --- | --- | --- |
| Open Models |  |  |  |  |  |
| ZeroScope | 320 $\times$ 576 | 32 | 9 | 50 | DPMSolverMultiStep [] |
| VideoCrafter2 | 320 $\times$ 512 | 32 | 12 | 50 | DDIM [] |
| LaVIE | 320 $\times$ 512 | 32 | 7.5 | 50 | DDPM [] |
| OpenSora | 240 $\times$ 426 | 32 | 7 | 100 | IDDPM [] |
| SVD-T2I2V | 1024 $\times$ 576 | 25 | (1, 3) | 25 | EulerDiscrete [] |
| Closed Models |  |  |  |  |  |
| Lumiere-T2V | 1024 $\times$ 1024 | 80 | 8 | 256 | NA |
| Lumiere-T2I2V | 1024 $\times$ 1024 | 80 | 6 | 256 | NA |
| Gen-2 | 720 $\times$ 1280 | 32 | 8.5 | 100 | NA |
| Pika | 640 $\times$ 1088 | 72 | 12 | NA | NA |

## Appendix K Training Details for VideoCon-Physics

To create VideoCon-Physics, we use low-rank adaptation (LoRA) [] of the VideoCon applied to all the layers of the attention blocks including QKVO, gate, up and down projection matrices. We set the LoRA $r=32$ and $\alpha=32$ and dropout = $0.05$. The finetuning is performed for $5$ epochs using Adam [] optimizer with a linear warmup of $50$ steps followed by linear decay. Similar to [], we chose the peak learning rate as $1e-4$. We utilized $2$ A6000 GPUs with the total batch size of $32$. In addition, we finetune our model with $32$ frames in the video and the frames are resized to $224\times 224$ by image processor. Similar to [, ], we create $32$ segments of the video, and sample the middle frame for each segment.

## Appendix L Applications of VideoCon-Physics

In this work, we propose VideoCon-Physics, an auto-evaluator that judges the semantic adherence and physical commonsense of the generated videos for a given caption. Here, we describe the potential usecases of the model for future work.

#### Video Generative Model Selection:

The ability to perform model verification on downstream tasks cheaply and reliably is critical. In this regard, the model builders can utilize VideoCon-Physics to evaluate their candidate models on the VideoPhy dataset at scale. The top candidate models can then be evaluated with the human workers for more accurate evaluation.

#### Data Filtering:

With the advent of foundation models that are trained on the internet data, high-quality filtering has emerged as a crucial step in the pipeline [, ]. Here, the data builders can utilize VideoCon-Physics to filter low-quality video-text data that lacks in semantic adherence and physical commonsense.

#### Post-training:

Recently, aligning the generative models with human or AI feedback has become pivotal for high-quality generations [, , , , ]. Here, the post-training pipeline of the video generative models can leverage the VideoCon-Physics model as an reward model that provides feedback to the model generated content. Subsequently, this feedback can be utilized to refine the model for better generations.

## Appendix M More Qualitative Examples across Different Models

Here we compare results from VideoCrafter2 with results from other models. We find that LaVIE generates videos that show unnatural motions for the objects. Moreover, the videos tend to possess fast and vibrant dynamics. In addition, we observe that ZeroScope is prone to penetration issues as objects can be mixed with each other. Further, we find that Gen-2 does not understand gravity very well, as objects that should be falling can be either static or even floating upwards, for instance, in Figure [14](#A13.F14) (c).

Figure: Figure 14: Qualitative comparison of VideoCrafter2 with other models. Here, the top row presents the generated videos from the VideoCrafter2 model. (a) LaVIE generated unnatural and inconsistent arm motions here. (b) The plate is deformed and penetrates the coffee beans beneath in the ZeroScope-generated video. (c) Gen-2 is prone to gravity issues. Here the stream of water is floating upwards, instead of flowing downwards.
Refer to caption: /html/2406.03520/assets/x7.png

## Appendix N More Qualitative Examples of Poor Physical Commonsense

We present more examples from each generative model where one or more physical laws are violated in Figure [15](#A14.F15) - Figure [23](#A14.F23).

Figure: Figure 15: Unphysical Generated Examples of LaVIE. (a) Solid Constitutive Laws Violation: the metal spoon should not deform; Nonphysical Penetration: the spoon unnaturally passes through the liquid. (b) Solid Constitutive Laws Violation: the whisk exhibits abnormal shape deformation. (c) Solid Constitutive Laws Violation: the two hands show abnormal shape deformation; Nonphysical Penetration: fingers penetrate each other; Conservation of Mass Violation: the geometry (plus texture) of the two hands are inconsistent over time. (d) Conservation of Mass Violation: the geometry (plus texture) of the soccer is inconsistent over time; Newton’s Second Law Violation: the soccer does not fall under gravity. (e) Conservation of Mass Violation: the volume of yogurt in the cup does not increase as more yogurt is added.
Refer to caption: /html/2406.03520/assets/x8.png

Figure: Figure 16: Unphysical Generated Examples of Gen-2. (a) Conservation of Mass Violation: the volume of juice in the blender increases over time without new substances being added. (b) Solid Constitutive Laws Violation: the metal spoon should not deform. (c) Conservation of Mass Violation: the volume of the book increases over time; Nonphysical Penetration: the fingers pass through the book. (d) Nonphysical Penetration: fingers penetrate into each other. (e) Newton’s Second Law Violation: the flowing water appears to be static, ignoring the effect of gravity.
Refer to caption: /html/2406.03520/assets/x9.png

Figure: Figure 17: Unphysical Generated Examples of VideoCrafter2. (a) Newton’s Second Law Violation: the metal can deforms without being pressed. (b) Newton’s Second Law Violation: Water splashes while the rolling wheel remains static. (c) Newton’s Second Law Violation: the bottle floats in the air, ignoring the effect of gravity; Fluid Constitutive Law Violation: the dripping and flowing of mustard are unnatural. (d) Conservation of Mass Violation: the geometry (plus texture) of the frisbee is not consistent over time. (e) Conservation of Mass Violation: the total volume of milk in the glass does not increase as more milk is poured into; Nonphysical Penetration: milk penetrates the glass.
Refer to caption: /html/2406.03520/assets/x10.png

Figure: Figure 18: Unphysical Generated Examples of ZeroScope. (a) Newton’s Second Law Violation: the motion of the hoverboard does not satisfy the momentum equation. (b) Newton’s Second Law Violation: the motion of the arm of the swimmer is unnatural. (c) Conservation of Mass Violation: the geometry (texture) of the frog is inconsistent over time. (d) Newton’s First Law Violation: the velocity of the fidget spinner changes despite being in a balanced state. (e) Solid Constitutive Laws Violation: the paper is torn apart without external forces but recovers later.
Refer to caption: /html/2406.03520/assets/x11.png

Figure: Figure 19: Unphysical Generated Examples of OpenSora. (a) Solid Constitutive Laws Violation: the metal blender should not deform. (b) Newton’s Second Law Violation: the car moves backward, violating the momentum equation (c) Solid Constitutive Laws Violation: the metal spoon deforms when stirring the cocktail. (d) Newton’s First Law Violation: the coin moves on the ground back and forth without horizontal forces. (e) Conservation of Mass Violation: the left rear wheel disappears over time.
Refer to caption: /html/2406.03520/assets/x12.png

Figure: Figure 20: Unphysical Generated Examples of SVD-T2I2V. (a) Newton’s Second Law Violation: the perfume spreads back and forth, violating the momentum equation. (b) Newton’s Second Law Violation: the water drops float in the air, ignoring gravity. (c) Solid Constitutive Laws Violation: the leg exhibits unnatural deformation. (d) Newton’s Second Law Violation: the water flows upward into the air without external forces. (e) Solid Constitutive Laws Violation: the screwdriver head deforms unnaturally.
Refer to caption: /html/2406.03520/assets/x13.png

Figure: Figure 21: Unphysical Generated Examples of Pika. (a) Solid Constitutive Laws Violation: the fidget spinner should not deform. (b) Solid Constitutive Laws Violation: the necklace should not deform. (c) Conservation of Mass Violation: the volume of cream increases over time without additional input. (d) Fluid constitutive Law Violation: unnatural waves on the sea surface. (e) Solid Constitutive Law Violation: one diving shoe splits into two and detaches from the feet.
Refer to caption: /html/2406.03520/assets/x14.png

Figure: Figure 22: Unphysical Generated Examples of Lumiere-T2V. (a) Conservation of Mass Violation: the vegetable appears on the spoon out of nowhere. (b) Solid Constitutive Laws Violation: the whisk should not deform. (c) Solid Constitutive Laws Violation: the coin splits into two and then merges back into one. (d) Solid Constitutive Laws Violation: the lemon shows an unnatural appearance change; Fluid Constitutive Laws Violation: the lemon juice appears like static glue. (e) Nonphysical Penetration: the tea flows through the cup.
Refer to caption: /html/2406.03520/assets/x15.png

Figure: Figure 23: Unphysical Generated Examples of Lumiere-T2I2V. (a) Solid Constitutive Laws Violation: the drum stick head should not deform (b) Newton’s Second Law Violation: the leaf floats in the air, ignoring gravity. (c) Conservation of Mass Violation: the vegetable appears on the spoon out of nowhere. (d) Nonphysical Penetration: hands penetrate each other. (e) Solid Constitutive Laws Violation: one leg on the skateboard transforms into a person.
Refer to caption: /html/2406.03520/assets/x16.png

## Appendix O Examples from diverse states of matter and complexity

We present a few qualitative examples highlighting instances of good physical commonsense and bad physical commonsense in Figure [24](#A15.F24)-Figure [26](#A15.F26).

Figure: Figure 24: Qualitative examples in the fluid-fluid category. Videos in the left column have a majority PC score of 0, while videos in the right column have a majority PC score of 1. (a) The central ripple does not vanish even in absence of raindrops. (b) The water droplet is floating upwards, defying gravity. (e) The total volume of juice is increasing. (f) The color of the yogurt is not consistent over time.
Refer to caption: /html/2406.03520/assets/x17.png

Figure: Figure 25: Qualitative examples in the solid-solid category. Videos in the left column have a majority PC score of 0, while videos in the right column have a majority PC score of 1. (a) The hands of the clock have illogical motion. (b) One piece of the robe disappears. (e) The geometry and texture of the dart are not consistent over time. (f) The total volume of the sheet of paper is not consistent over time.
Refer to caption: /html/2406.03520/assets/x18.png

Figure: Figure 26: Qualitative examples in the fluid-fluid category. Videos in the left column have a majority PC score of 0, while videos in the right column have a majority PC score of 1. (a) The geometry and color of the duck head changes over time. (b) One chopstick appears from nowhere. (e) One leg appears from nowhere. (f) The geometry and texture of the leaves are not consistent over time.
Refer to caption: /html/2406.03520/assets/x19.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see Weaknesses.
soundness: 3
strengths: The goal of evaluating physical commonsense is quite important for this area. 
The chosen video generators are comprehensive, including open-source models and closed ones. 
The presentation is well-organized and easy to follow, with key numbers regarding evaluations.
summary: This work targets at building a benchmark that can evaluate the physical commonsense for video generation. Multiple physics-related prompts are first generated and evaluated, serving as the input for different video generators. With human evaluation, it is found that both open-source and closed-source, and find that they significantly lack physical commonsense and semantic adherence capabilities. In order to evaluate in scale, an auto-evaluator is trained.
weaknesses: Regarding evaluation, I have several major concerns.

- As mentioned in Sec.3.1, binary feedback (0/1) is used to evaluate semantic adherence and physical commonsense. This discrete value may not reflect and moniter the true capability for different video generators. For example, for a text prompt with 10 physical movements, one generator achieves 8 movements while another is 6. These binary feedback can not tell the gap between two candidates. This example could be too extreme while that could be a weakness of binary value.
- Besides, I am not sure whether the absolute accuracy of physical achievements is a proper metric.  Especially for Fig.1, I believe the relative scores across different generators (like ELO score) make more sense, which also avoid the weakness of binary feedback
- Regarding physical commonsense (PC), it really depends on the text following ability of given generators (semantic adherence in this work).  Joint performance may be one alternative for both text and physics evaluation while the posterior probability may be one perspective for physical commonsense alone.

For auto-evaluator videocon-physics, the fine-tuning details could facilitate the reproduction and transferability from VIDEOCON to other video-language models.

---

confidence: 3
contribution: 4
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 3
strengths: * The paper shifts attention from general visual and semantic quality to the capability of T2V models to simulate real-world physics, addressing a vital aspect of realism in video generation.
* The paper provides detailed insights into different failure modes, guiding future model improvements and research directions.
* The automation pipeline, VideoCon-Physics enables scalable assessment of semantic adherence and physical commonsense in generated videos, which can be useful and meaningful to the research communities on T2V generation.
summary: This paper presents a benchmark designed to evaluate the physical commonsense of videos generated by text-to-video models. It highlights significant gaps in these models' ability to accurately simulate real-world physics and adhere to caption prompts. It introduces VideoPhy, a dataset consisting of real-world interaction prompts, and VideoCon-Physics, an automatic evaluation pipeline. Evaluation shows that even the best models, such as CogVideoX-5B, only achieve a 39.6% adherence rate to physical laws, emphasizing the need for improvements in video generation models.
weaknesses: * Among the T2V models used for comparison, some still frequently fail to reproduce the scenarios specified by the text prompts. For example, in assessing physical reasoning in a scenario where milk is being poured, one needs to verify whether the milk appropriately fills the cup. However, in practice, these models often fail even to generate a video depicting the act of pouring milk. In such cases, the benchmark may be more influenced by the general video generation capabilities of the models rather than their physical commonsense reasoning abilities, as shown in the similar trends observed between Semantic Adherence and Physical Commonsense scores in Table 4. Therefore, it seems necessary to conduct experiments that assess physical commonsense only on generated videos that have appropriate semantic adherence, ensuring that the evaluation focuses on the models' understanding of physical phenomena rather than their basic ability to generate relevant videos.

* Proposed automating evaluations using a video-to-text model relies on a assumption: the V2T model must have a better understanding of physical phenomena than the T2V model. I remain some concerns about the justification of this assumption, as it is essential for the validity of the automated evaluation method. Without depending on this assumption, it is quite reasonable to suggest that the proposed V2T model can evaluate the T2V model because it has been fine-tuned on data containing these physical phenomena. In this context, I'm curious what it would be if the T2V model is similarly fine-tuned on a portion (training split) of the VideoPhy dataset and then generates videos based on prompts from the test split.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: see weaknesses above.

1. Given the challenges around categorization raised above, are there plans to expand or refine this benchmark to incorporate a broader range of interaction types or more nuanced physical properties, beyond basic material categories? 
2. Have the authors considered using more detailed metrics to better decouple the concepts?
soundness: 2
strengths: 1. This paper presents a physical commonsense benchmark that addresses a gap in existing datasets. The field of video generation needs such a benchmark, as t2v models gain popularity partly for their potential as world simulators or physical engines. 
2. The dataset covers a wide range of activities, interactions, and dynamics on various materials as mentioned in line 194/195. The classification of dataset is somehow inspired by graphics field, and the difficulty of simulate those dynamics is also considered, which is a very interesting aspect for constructing the dataset.
3. Experiments are conducted over most of the current t2v models except for those without API support, providing a comprehensive comparison across a wide range of models.
4. This work provides insight and contributes to further improving t2v models.
summary: This paper introduces VideoPHY, a benchmark consisting of 688 captions designed to evaluate text-to-video models on physical commonsense. This work focuses on the real-world activities and interactions, classified by material interactions into three categories: solid-solid, solid-fluid, and fluid-fluid.

The dataset was initially generated using GPT-4 and then refined through manual filtering, and was annotated with binary (0/1) difficulty levels. Evaluation was based on two binary (0/1) metrics Semantic Adherence and Physical Commonsense, assessed by human or VLM. For the VLM, the authors fine-tuned VideoCon on human annotations, creating VideoCon-Physics.

Experiments comparing current SOTA text-to-video models including both open and closed models, indicate that current models struggle to model physical activities well.
weaknesses: 1. I am not an expert in graphics or materials, so I am very uncertain about the category definitions and category ratios. I can get the idea of categorizing based on the state of matter, and solid and liquid are the most common. However, in graphics, rigid bodies, soft bodies, particle systems, fabrics, characters and animals are distinct topics that rely on very different physical models, whereas fluid dynamics, such as inviscid and viscous flows, are comparatively less diverse. If the idea of this benchmark is focusing on physical interactions and dynamics, then categorizing by interaction types or physical properties rather than broad material types can be more informative and nuanced. On the category ratios, as stated in line 179-181 as solids involve more physical constitutive models, we might expect more cases of solid-solid interactions than solid-fluid interactions, yet the sample counts are nearly equal (289 and 291, respectively). 
2. Each sample in the training set for VideoCon-Physics was only labeled by one human annotator, while the author also mentioned while annotating benchmark, the agreement of three human annotators is 75% and 70% on SA and PC. This makes the training set less trustworthy.
3. I am also very skeptical about the motivation of fine-tuning VideoCon for direct evaluation of semantic adherence and physical commonsense. Those two metrics seem to cover a wide range of concepts but only one number between 0 and 1 was given. Directly fine-tuning VideoCon to evaluate especially the physical commonsense without introducing any extra knowledge, reasoning, or explicitly modeling does not make sense to me. The improvement of metrics might result from dataset domain bias, but there is no analysis on that. And from the leader boards of human evaluation and automatic evaluation, the disagreement on models such as Luma Dream Machine and Pika is not negligible. I suggest authors provide more detailed disagreement or evaluation statistics.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see the weakness part for my concerns, but it is just a minor one.
soundness: 4
strengths: 1. The paper is well-written and easy to follow.
2. The paper presents a high-quality benchmark for evaluating physical commonsense for video generation. Careful data curation for text prompts is employed to ensure the quality of the text prompt.
3. The authors provide a comprehensive analysis of the text prompt used for evaluation. The category is balanced.
4. Beyond human evaluation, the authors also provide a model for automatic evaluation by fine-tuning a model for evaluating the physical commonsense of the model. The model eases the use of the benchmark for future research.
5. The authors also will release the generated videos and human evaluation for future research, which will potentially boost the performance of the automatic evaluation model.
summary: The paper presents a benchmark for evaluating physical commonsense for video generation. The benchmark includes 1) high-quality, human-verified captions; 2) automatic evaluation models to evaluate the video generation models; 3) the generated videos using existing methods and human annotations for these videos. Also, the authors reveal some conclusions based on the benchmark results, which could provide insights to the community.
weaknesses: 1. For the VIDEOCON-PHYSICS model, have the authors conducted a human evaluation for the performance? Since the data used for training this model comes from same human annotators, the trained model may be biased. It would be better if the model can provide a human evaluation for the evaluation results generated by the VIDEOCON-PHYSICS model (just for the check of the correctness of the results provided by the VIDEOCON-PHYSICS model)

Meta-review: The paper addresses an important gap in evaluating physical commonsense in text-to-video (T2V) generation models. The proposed VideoPhy benchmark is insightful, covering diverse physical interactions and revealing significant shortcomings in current models. Reviewers appreciated the comprehensive evaluation, clear presentation, and the automation pipeline VideoCon-Physics. Concerns include the limitations of binary feedback, potential biases in annotations, and the need for more nuanced metrics. The authors' revisions addressed these issues effectively, leading to improved reviewer ratings. Overall, this work is a valuable contribution to the community. The AC recommends acceptance.

Decision: Accept (Poster)