OpenWaves: A Large-Scale Anatomically Realistic Ultrasound-CT Dataset for Benchmarking Neural Wave Equation Solvers

Authors: Zhijun Zeng, Youjia Zheng, Hao Hu, Zeyuan Dong, Yihang Zheng, Xinliang Liu, Jinzhuo Wang, Zuoqiang Shi, Linfeng Zhang, Yubing Li, He Sun
Primary area: datasets and benchmarks
Keywords: Computational imaging, Inverse problem, Neural operators, Ultrasound Computed Tomography, Full Waveform Inversion

Abstract:
Accurate and efficient simulation of wave equations is crucial in computational physics, especially for wave imaging applications like ultrasound computed tomography (USCT), which reconstructs tissue properties from scattered waves. Traditional numerical solvers for wave equations are computationally intensive and often unstable, limiting their practical applications for quasi-real-time imaging. Neural operators offer an innovative approach by accelerating PDE solving using neural networks; however, their effectiveness in realistic imaging is constrained by existing datasets that oversimplify real-world complexity. In this paper, we present OpenWaves, a large-scale wave equation dataset designed to bridge the gap between theoretical equations and practical imaging applications. OpenWaves provides over 16 million frequency-domain wave simulations using real USCT configurations, featuring anatomically realistic human breast phantoms across four categories. It enables comprehensive benchmarking of popular neural operators for both forward simulation and inverse imaging tasks, allowing analysis of their performance, scalability, and generalization capabilities. By offering a realistic and extensive dataset, OpenWaves not only serves as a platform for developing innovative neural PDE solvers but also facilitates their deployment in real-world medical imaging problems.

Decision: reject
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730417396237

thread_type:
official_review

<thread_object>
creator: reviewer_cxcq

<content>
summary:
This paper titled: OpenWaves: A Large-Scale Anatomically Realistic Ultrasound-CT Dataset for Benchmarking Neural Wave Equation Solvers presents an large-scale *simulated* dataset for USCT, including the breast image and the simulated wave imaging (from USCT). The dataset simulated 4 different breast regions, and their corresponding wave imaging. The authors thoroughly evaluated a few PDE solver for both the forward approximation and inverse problem.

strengths:
1. I believe this large scale USCT dataset is valuable in terms of USCT breast related inverse problem, as it provided realistic tissue simulation. 
2. I appreciate the authors for the evaluations on multiple algorithms.

weaknesses:
1. From a contribution point of view, this paper is a dataset paper, however, the dataset is purely simulated and all the justifications are not convincing enough to me that this will bring important values. 
a. It is perfectly fine that the paper is simulated, then the point becomes proposing a realistic USCT data generation pipeline. however, this really requires the validation from real clinical data (instead of only simulated data), and also examples with pathologies. 
b. how to proof that your phantom simulation is accurate and more close to the real-breast distribution.

2. Another apparent comparison I would do is (a. training with old datasets, model 1) versus (b. trianing with new datasets, model 2), this also needs real clinical data for justification. And also ablation stuides on how the dataset makes a different, it makes very little sense to train on this distribution and evaluate on the same distribution.

questions:
1. Based on the creation of this dataset, do you have any insights of how to design a better PDE solver specifically for this USCT application.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661282715

thread_type:
official_review

<thread_object>
creator: reviewer_6snf

<content>
summary:
The work introduced the dataset OpenWaves, which provides 16M frequency-domain wave simulations on breast phantom. The dataset can help to evaluate the forward simulation and tomography inversion in ultrasound tomography.

strengths:
1. The authors have conducted 16 million simulations across a substantial number of breast phantoms, requiring significant computational and storage resources. If proven to be high-fidelity, this dataset could save future researchers considerable time and effort in data preparation.

2. The authors provide a thorough comparison of different models in both forward simulation and backward reconstruction, offering a relatively comprehensive evaluation.

weaknesses:
1. Lack of original contribution: This work does not appear to introduce any novel methods in dataset construction for ultrasound tomography. Creating such a dataset fundamentally depends on two aspects: the simulation algorithm and the generation or collection of breast phantoms. However, in this study, both are adopted from existing resources—the VICTRE project for phantom generation and the Convergent Born Series (CBS) algorithm for wave field simulation. Consequently, this dataset could theoretically be reproduced by others with sufficient computational resources, given VICTRE and CBS, without any unique contribution from the authors.

2. Insufficient discussion on parameter selection and real-world distribution: VICTRE requires a range of carefully chosen parameters to generate realistic breast phantoms, such as surface shape, internal compartmentalization, etc. However, this work does not discuss how parameters in OpenWave were selected or how their distributions might mirror real-world variability. Understanding these choices is essential for evaluating the fidelity of the benchmark and whether it can accurately assess the performance of different frameworks.

3. Lack of quality control measures: Although VICTRE enables simulation of breast phantoms, these simulations inherently lack the complexities of real-world samples. The authors do not address how they ensured the quality or realism of the generated phantoms. For instance, was there any subsampling or external validation from clinical experts to assess the fidelity of these simulations? Such quality assessment is crucial for establishing the dataset as a reliable benchmark.

4. Lack of support of breast type: What is the rationale behind the distribution of four breast types?1000:3000:2000:2000. Can authors provide the literature that supports the figure?

questions:
1. The current citation format is structured as though it serves as the subject of a sentence. To improve clarity and consistency, the authors should either use \citep or add parentheses to the current citations.

2. The authors frequently reference the 16M frequency-domain wave simulations to emphasize the dataset's large scale. However, this may be misleading. If my understanding is correct, each breast phantom has 256 sources, and during reconstruction, data from all sources would typically be required. I suggest that the authors represent the dataset size as either 8000 or 8000x8, which would provide a more accurate and meaningful metric for researchers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687682045

thread_type:
official_review

<thread_object>
creator: reviewer_elgk

<content>
summary:
This paper presents a new dataset for ultrasound computed tomography (USCT) that: 1) establishes a connection between a theoretical wave equation and a practical medical imaging application; 2) simulates data based on the Helmholtz equation; and 3) conducts large-scale benchmarking of neural operator and end-to-end baselines.

strengths:
The paper provides a comprehensive evaluation of the simulated data, including experiments on various organs of interest, different imaging models, and various baselines. In addition, the presentation of the paper is clear and easy to follow.

weaknesses:
1) My primary concern pertains to the dataset generation. Despite a thorough review of the introduction and background, I find it unclear why generating this new dataset is particularly challenging. From my understanding, this dataset is generated purely through simulation. As long as one implements the wave equation according to the theoretical framework and generates the digital phantom via simulation tool, reproducing this dataset seems straightforward. From this perspective, the main contribution of this dataset appears to be its potential to save researchers time in regenerating these terabyte-scale data. Could the authors clarify this point?

2) Furthermore, the differences between this dataset and existing datasets are not sufficiently discussed. While the manuscript notes that other datasets employ different imaging models, it would strengthen the work if the authors also discussed additional differences, such as scale, imaging applications, or computational complexity. Comments on the applicability of these datasets in comparison to the proposed dataset would also be beneficial.

3) Since this paper focuses on medical applications of USCT, a purely simulated dataset (both the ground truth and measurements) may be seen as less important, particularly given that medical professionals often prioritize real data. I could not locate the evaluation criteria specifically for the dataset and benchmark track for ICLR. I am concerned whether such a purely simulated dataset meets ICLR's requirements.

questions:
1) Could the authors provide insights into the time complexity involved in generating the entire proposed dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690530681

thread_type:
official_review

<thread_object>
creator: reviewer_ktem

<content>
summary:
This paper introduces a large-scale dataset named OpenWaves, which provides over 16 million frequency-domain wave simulations based on real ultrasound-CT configurations. The dataset is specifically designed to include anatomically realistic human breast phantoms.

strengths:
1. Benchmarking wave simulation and imaging for neural operators represents a significant contribution to the field.

2. The dataset is extensive, with over 16 million simulations, covering a variety of anatomically realistic human breast phantoms.

weaknesses:
1. The mathematical formulation in Section 3.1 requires improvement. Beyond simply explaining variable meanings, the authors should define variables in greater detail, including their range and space. For instance, the symbol $\nabla$ in Eq. 1 is not clearly defined, and the partial derivatives in Eq. 2 are also ambiguous.

2. Although Section 4.3.2 discusses the scaling property, the dataset's efficiency remains somewhat unclear. For example, as noted by the authors, the performance of different neural operators varies with dataset size, yet the underlying rationale and its implications for designing the proposed dataset require clarification.

questions:
1. Do the authors plan to open-source the curated dataset?

2. Have the authors considered comparing the simulated dataset to a smaller real-world dataset to assess the alignment of data distributions with real-world conditions?

3. Could the authors improve the spacing of references?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732125650715

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
\
We thank the reviewers for their constructive feedback and for recognizing the significance of our work, including the importance of the dataset to the field (kTeM, cXcq), its extensive scale (kTeM, 6sNf), comprehensive evaluation (eLgk, 6sNf, cXcq), and clear presentation (eLgk). Below, we address main concerns raised in the reviews. Point-to-point responses are included as a reply to each reviewer, and we also provide additional experimental results in the newly submitted supplementary PDF.
\
\
**1. The key contributions of our work**

&ensp;&ensp;OpenWaves is the first large-scale wave PDE dataset to simulate realistic scenarios with complex scattering media and large wavenumbers, addressing a critical gap in existing datasets. Current datasets oversimplify the problem by assuming small wavenumbers and basic scattering media structures (e.g., layered geometries or Gaussian random fields), resulting in a long-standing yet overlooked issue in the machine learning for PDE community: over-optimistic evaluations and biased models that perform well in toy settings but fail to generalize to real-world challenges.[1] OpenWaves focuses on realistic medical imaging applications, particularly ultrasound computed tomography (USCT) for breast disease diagnosis, offering a benchmark that promotes the development of neural PDE solvers tailored to practical challenges. This dataset is valuable for both the machine learning and medical imaging communities.

&ensp;&ensp;Our work also provides a thorough evaluation of forward and inverse neural PDE baselines, offering insights for designing effective solvers. For instance, multi-grid frameworks are particularly effective at learning high-frequency oscillations, and combining forward neural operators with adjoint optimization demonstrates superior generalization to unseen data compared to direct inverse mapping. These findings provide actionable guidance for future research.
\
\
**2. Data generation using existing tools**

&ensp;&ensp;Utilizing existing simulation tools for data generation is a common practice when building datasets. However, as a dataset & benchmark paper, the processes of generation, standardization, validation, and benchmarking should be recognized as significant contributions to the field, as in prior works [2, 3, 4]. Moreover, OpenWaves goes beyond merely using existing simulation tools: Since VICTRE was originally designed for X-ray mammography, we developed pipelines to assign anatomically accurate sound speed to tissues, modeling realistic acoustic properties of scattering media. Additionally, we incorporated USCT-specific experimental setups into numerical solvers, enhancing the dataset's realism and relevance.
\
\
**3. Generalization to real-world data evaluation**

&ensp;&ensp;Neural PDE solvers trained on OpenWaves generalize well to real-world data. Preliminary results on a public clinical dataset of human breast[5] is included in the supplementary material, confirming our dataset's anatomical fidelity. More detailed quantitative evaluations will be provided in the final version.

&ensp;&ensp;We hope these clarifications and additional results address your concerns. We are happy to discuss any remaining points during the discussion phase. Thank you!
\
\
**Reference:**

[1] Nick McGreivy, Ammar Hakim. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature Machine Intelligence 2024.

[2] Makoto Takamoto et al. PDEBench: An extensive benchmark for scientific machine learning. NeurIPS 2022.

[3] Chengyuan Deng et al. OpenFWI: Large-scale multi-structural benchmark datasets for full waveform inversion. NeurIPS 2022.

[4] Tianlin Liu et al. WaveBench: Benchmarking Data-driven Solvers for Linear Wave Propagation PDEs. TMLR 2024.

[5] Rehman Ali et al. 2-D Slicewise Waveform Inversion of Sound Speed and Acoustic Attenuation for Ring Array Ultrasound Tomography Based on a Block LU Solver. IEEE Transactions on Medical Imaging. 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1733212433306

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
### **Dear Reviewers, ACs, and PCs,**

We sincerely thank all the reviewers for their valuable and constructive feedback, as well as for dedicating their time to reviewing our paper. Based on the insightful suggestions provided during the rebuttal phase, we have conducted a thorough revision addressing the reviewers' key concerns. Below, we summarize these concerns and detail the revisions and updates included in the final submission. We hope this provides the reviewers, ACs, and PCs with a clearer understanding of the progress and outcomes of the rebuttal discussion.

---

### **Concerns**
- **[kTeM]** Mathematical formulation of Ultrasound CT imaging problem.
  - **[Authors]** We add additional clarifications in the response. *[Response to kTeM]*
- **[kTeM]** Scaling property and data efficiency.
  - **[Authors]** We explain in the response. *[Response to kTeM]*
- **[kTeM]** Open Sourcing the dataset.
  - **[Authors]** We explain in the response. *[Response to kTeM]*
- **[eLgk, 6sNf, cXcq]** Main Contribution of Dataset and Benchmark.
  - **[Authors]** We explain in the response. *[Response to eLgk, 6sNf, cXcq] [General Response: The key contributions of our work] [General Response: Data generation using existing tools]*
  - **[Authors]** We conduct additional experiments to demonstrate the usefulness of the proposed dataset in real clinical dataset imaging. *[Supplementary Materials]*
- **[eLgk, cXcq]** Differences from existing datasets.
  - **[Authors]** We explain in the response. *[Response to eLgk, cXcq] [General Response: The key contributions of our work]*
- **[kTeM, eLgk, 6sNf, cXcq]** Alignment of real-world data.
  - **[Authors]** We explain in the response. *[Response to kTeM, eLgk, 6sNf, cXcq] [General Response: Generalization to real-world data evaluation]*
  - **[Authors]** We conducted additional experiments to evaluate the generalization capabilities of different models on clinical datasets. *[Supplementary Materials]*
- **[eLgk]** Time complexity of dataset generation.
  - **[Authors]** We explain in the response. *[Response to eLgk]*
- **[6sNf]** Parameter selection for dataset generation.
  - **[Authors]** We explain in the response. *[Response to 6sNf]*
- **[6sNf]** Quality assessment of generated phantoms.
  - **[Authors]** We explain in the response. *[Response to 6sNf]*
- **[6sNf]** Distribution of breast types.
  - **[Authors]** We provide reference support in the response. *[Response to 6sNf]*
- **[cXcq]** Insights on neural PDE designs.
  - We explain in the response. *[Response to cXcq] [General Response: The key contributions of our work]*

---

### **Revisions**
- **[Overview of the dataset]** We fix a labeling error of the breast type according to concerns raised by Reviewer 6sNf. *[L 227-236, Section3.2.1]*
- **[Supplementary Materials]** We add validation experiments on using different models for real clinical breast datasets.
  - **Reconstruction results:** We successfully reconstruct a clear malignant tumor together with other internal tissue structures, using BFNO with 8 frequencies and 256 transmitters.
  - **Ablation study from different models:** We test another two models, (a) FNO with 8 frequencies and 256 transmitters, and (b) NIO inversion.
  - **Ablation study from different experimental settings:** We employ three observational data settings about frequency and source number, including (a) 8 frequencies and 128 transmitters, (b) 8 frequencies and 64 transmitters, and (c) 3 frequencies and 128 transmitters.

---

### **Other Points**
- **[kTeM]** Spacing of references
  - **[Author]** Citations misused. *[Corrected]*

---

### **Summary**
We sincerely thank the reviewers for their valuable suggestions, which have helped us strengthen our revised submission. We believe the revised submission could be much more robust and contribute to broadening the field of neural PDE and medical imaging, offering valuable insights to the community. We deeply appreciate the efforts of the reviewers, ACs, and PCs during the rebuttal period.

*Best regards,*

*Authors*
</content>
</thread_object>

</thread>
<thread>
creation date:
1733292248646

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear ACs,

Thank you for your efforts in overseeing the review process for our paper. We deeply appreciate the valuable feedback and constructive criticism provided by the reviewers, which have significantly contributed to improving our work.

However, during the rebuttal phase, we observed that both Reviewer kTeM and Reviewer 6sNf did not actively engage in meaningful discussions or address the clarifications and responses we provided. This lack of engagement may limit the extent to which their reviews accurately reflect the quality of our submission.

In light of this, we sincerely request that you take our responses into account and reconsider the value and weight of these two reviewers' comments in the decision-making process. 

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper describes a large dataset of realistic breast phantoms with varying material properties and corresponding high-frequency ultrasound CT measurements. The dataset is designed to develop and benchmark neural operators for solving the associated forward and inverse problems. Most reviewers agreed that this new curated dataset could help push neural operators for wave inversion into clinically relevant territory (where they are currently not). I agree with that. Two reviewers were more critical, and unfortunately several unresolved concerns remain which I believe are important. This is why I am recommending a (weak) rejection (but I would not mind if the paper gets accepted). I think that the addressing the following points would greatly help in the resubmission:

- A clear up-front argument for the importance and uniqueness of OpenWaves and its potential to unlock neural operators for breast cancer screening. There is now a comparison in 2.2 but I recommend moving this very early and maybe evening having a paragraph titles "Why is this dataset needed to make progress on ...". It would be even better if the authors could demo some of the overoptimistic evaluations resulting from earlier datasets; this may not be too hard
- A better delineation in 4.3.2 about what is meant by scaling. Scaling laws for models and data are currently a white-hot research area thanks to LLMs, foundation models, and all that. The related questions revolve around the optimal co-design of datasets and models, and has been the intention of kTeM. Perhaps the authors' intention was to simply show that more data = better performance, scaled by model size, but this is actually an interesting question. At any rate, a delineation of what is meant would be strategically helpful.
- More details about quality control in the main text. In particular, does the breast phantom capture sufficient variation to ensure neural operators which don't remove or hallucinate features? It is great that the authors ran a test on public clinical data, but that should be mentioned prominently in the main text, and there should be a corresponding quantitative / qualitative evaluation at least for some samples (also a comparison to a strong principled iterative inversion). While some neural operator designs may generalize well even if trained of "poor" data (of low variation), some that would perform strongly may need high quality data. A discussion would strengthen the paper.

I hope that the authors can effectively. use these suggestions and the discussions to revise and resubmit their paper.