Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors provide four families of datasets. There is diversity in the dataset sources. Two of the families are based on “math-driven” synthetic data, while a third dataset is based on simulations from a real reservoir. The final family uses an interesting technique to encourage diversity and generalizability: a style network transfers geologic styles onto “natural” images from the COCO dataset. The generalizability and complexity analysis provided by the authors helps enhance the value of the datasets.
strengths: •	There is a diverse set of data. Easy and hard versions are provided.

•	Benchmarks apply four deep learning techniques to the dataset.

•	The authors address generalizability using data based on style transfer. Results include a generalization study by training one dataset and testing on a different dataset.

•	Good explanations and good visuals provide intuition as to the content of the datasets.

•	Results include an evaluation of the complexity of the datasets.

weaknesses: There was a minor type, but this has been corrected.

I gave a Confidence rating of 2, since there may be some FWI domain-specific weaknesses that I am missing due to lack of experience in this area.
correctness: The methodology appears sound and correct.
clarity: Very well written. Clean organization. Even though I am not well-versed in this field, I found the paper easy to follow.
relation to prior work: •	Good description of prior work.

•	Well established how this work differs from previous works.

•	Section 3.3 ties the datasets to several topics of interest.
additional feedback: N/A
review_title: Solid and valuable datasets and benchmarks for Full Waveform Inversion

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors develop an open-source platform containing twelve datasets for full waveform inversion (FWI) and experiment on 4 deep learning methods, 3 supervised and 1 unsupervised. Through this benchmark, the authors try to solve the challenge of lack of large public datasets in the FWI community. The dataset considers diverse domains in geophysics , covers different geological sub-surface structures and contains huge amount of data ranging from 2K to 67K. It also has a dataset for 3D FWI. The paper also provides a generalization study to show the generalization capacity of deep learning methods across different domains.
strengths: 1) Large-scale dataset solving the problem of lack of large-scale public dataset in FWI.
2) Contributions and explanation of concepts are solid.
3) Experiments are clear and extensive.

weaknesses: More experiments from 3D dataset expected.


Authors should include a thorough comparison with pervious FWI datasets, highlighting their contributions and how their dataset is unique from others (preferably in a tabular format). Ignore this comment if that is beyond the scope of this paper.
correctness: Dataset generation concept seems correct.
clarity: Paper is well written and well organized.
relation to prior work: Related to prior work described adequately with proper citations.
additional feedback: The authors have done a good work presenting a large-scale dataset in FWI.
review_title: One of a kind large-scale dataset for full waveform inversion

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Inference of underground structure from seismic measurements is a challenging problem with many important applications. Recently, machine learning techniques have been developed for this inverse problem, and this paper contributes to that body of work.   

The authors present a large dataset containing many simulated seismic recordings from many simulated environments, grouped into 4 classes (or 11 sub-classes, one of which contains a 3D environment and a 2D grid of simulated sensors). 

They also state an intention to release the code for 4 deep learning tools to infer the environment structure from the recorded data (at time of writing the github link in the paper is broken).  For now, they show performance metrics from the 4 tools on each dataset sub-class.

Finally, the authors rank the "complexity" of the datasets by 3 metrics and thereby confirm that the largest inversion errors they report are due to more complex environments. And the authors test "generalization" by training one of their inverse methods on one of the 10 "2D" subclasses of data, and testing performance on the other 9.

All in all, this work would be a strong contribution and aid to the field, if the code is indeed released.  My strongest criticisms are of the writing, which I believe makes this submission much harder to understand and interpret than needed.   

strengths: The simulated datasets and (promised) inversion tools, will be useful for the development and testing on novel inversion algorithms tailored for an applied problem with much active interest.

weaknesses: I see two major weaknesses.  Firstly, the link in the paper for the code is broken. This should be fixed promptly.  If it is not, I would suggest the area chairs consider rejecting. I would downgrade my rating form a 7 to a 5 (marginal reject).

 Secondly, I found the paper very difficult to read. I will say more in the "clarity" section.

correctness: I have no reason to think the work is incorrect.  The results are plausible, as far as I know.
clarity: I found the paper very difficult to read.  I believe the authors have done good work.  It is relevant, (to the best of my knowledge) novel, and timely. But it is not particularly complicated, and the tools are quite standard.  Thus the paper can, and should be, very simply written.

I suggest the following:

- The figures (at least the first appearance of each kind) need axes-labels.  Someone familiar with geophysical inversion will know that the velocity maps are depth-versus-distance and the seismic data are sensor-versus-time.  A Machine Learning enthusiast who encounters this paper will not know this and the authors do not tell them.

- From Equation 1 to Equation 2, $p(r,t)$ has become simply $p$.  A simple acknowledgement that the dependence on location and time are retained but simply not shown will aid a reader. Or, given that $p$ does not appear frequently in the text, just leave the explicit dependence. Also, Equation 2, has dropped the dependence on the source properties $s(r,t)$.  While this is common in geophysical inference (where, the source is often a controlled explosion or someting similar), this should be stated explicitly. In the future work section, the authors might acknowledge that there is a subfield of "passive geophysical inference" that makes use of more complex, and potentially unknown, sources. The tools herein might be applicable to such problems.

- Section 3 begins by naming 4 data groups, and then proceeds over a full paragraph to give details about the groups, many of which are repeated in Table 1.  At no point, in section 3.1, is it ever stated how the groups differ and for what each might be used.  This doesn't come until Section 3.3.  I would strongly recommend starting with the current section 3.3 use-cases and then putting a column of "domain interests" or "applications" in table 1.  That way when the groups are introduced, a reader can easily connect each group to its purpose.

- I would recommend moving section 3.2, on how the velocity maps were synthesized to the supplemental materials. Except for the short mention of the Kimberlina family/Geological resevoir, this section is technical and not relevant to most readers, who will likely be more interested in the inference tools.  

- Fig. 4 is impressive, but  readers would benefit from seeing what more dramatic failures look like. Such as are shown in Figs S6-S9 in the supplemental.  

- It is a lot of work for a reader to connect Table 4 and Table 3 and confirm the (unsurprising) claim: "complex velocity maps should be difficult to invert from the seismic data".  I would suggest three scatter plots of "complexity metric" vs. SSIM, with the different inverse models shown in different colors.   

- The generalization study has quantified things we expect (algorithms perform best on the data sets upon which they were trained; and training on complex and testing on simple is better than vice versa).  But it has not addressed the much more interesting question of real-world relevance.  Which training set should one use to train an inverse model to work on real-world data?  Would our decisions change if we had some prior knowledge on the likely presence/absence of fault-lines, or a crude estimate of sub-surface "complexity"?  If we used multiple inverse models trained on different data synthetic sets and found different sound-speed maps how might we decide which to trust? 

- In Sec 5.3 on future challenges the authors mention computational efficiency.  Up until now this had not been mentioned much in the paper.  If possible adding a metric of computational cost to Table 2 would help a reader recognize and understand the trade-off between performance and cost.

relation to prior work: The authors discussion of previous work, and their relation thereto, seems reasonable to me. But I admit I am not up to date on cutting edge developments of ML in geophysics.

additional feedback: N/A
review_title: If the code was public, and the paper heavily edited, these would be useful tools for a relevant real-world problem.  

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The submission provides benchmark datasets of various subsurface velocity models for inverse problems in seismology. The datasets comprise 12 synthesized velocity models, representing a variety of different styles, complexities and dimensions, and corresponding seismic data, generated from source and receiver surveys placed on the surface of the domains. The problem of Full Waveform Inversion (FWI) is introduced, and ML inversion benchmarks are generated from 4 different Deep Learning methods, including one unsupervised neural network. The performance of the benchmarks is evaluated with three different metrics, capturing numerical as well as structural differences. The submission is accompanied by a user-friendly website (https://openfwi-lanl.github.io/index.html), while the code is not yet published (license approvals still in progress).
strengths: The published dataset, comprising various synthetic subsurface velocity models and generated seismic data, provides a highly valuable open source contribution for researchers and data scientists in seismology and machine learning. Not only does it allow for comparability newly developed methods to baselines, but it also allows for testing of generalization of the methods across distinct geological domain characteristics.

What is highly appreciated is the reporting of negative results as the study demonstrates limitations of the generalization of the inversion methods when increasing complexity in cross-domain experiments.

Regarding the choice of baseline methods, it is highly appreciated that an unsupervised DL-method was included. This has big impact on the applicability of the methods in real world settings. 

The generation of the velocity maps documented in detail. The supplementary material is very comprehensive and helpful to get further information if needed (e.g. on the seismic datasets, on the benchmarks, and on intermediate inversion results). 

Besides the generalization study, the inclusion of a complexity analysis of the velocity maps is an additional detail and makes the study very complete.

weaknesses: Unfortunately, approval for releasing the codes is still pending by the time of this review. This limits the evaluation for accessibility and reproducibility, which is an integral part for this track. Given the user-friendly website, I hope that this can be resolved during the revision period. For the time being, it does not permit a higher rating.

It is mentioned that physics-based FWI suffers from high computation costs. This is not put into relation with the computational costs of the presented DL-based FWI methods (training time of the latter are provided in the Supplementary Materials). Could the authors add a rough comparison?

The study doesn’t discuss robustness against seismic noise of the methods, which has significant impact in real world problems. I acknowledge that the focus of the work is to provide benchmark datasets for FWI, and that field data is to be added in the future, but drawing limitations regarding seismic noise and discussion of uncertainty quantification would increase the impact of the work.

correctness: Both dataset and benchmark pipelines are constructed and evaluated appropriately. Results are reported correctly, including the reporting of negative results (i.e. in the generalization test).
clarity: Overall, the paper is written very clearly. The Additional Feedback section below provides some individual remarks.
relation to prior work: The authors mention existing datasets (which they claim are not publicly released). 
Prior art on FWI is well covered, spanning physics-based, data-driven, and hybrid approaches.
The used methods are appropriately referenced.

additional feedback: The channels (i.e. directionality) of the seismic stations are not well introduced. It can only be inferred from the figures in the supplementary material that 5 channels are being recorded. How are they used in the inversion methods?

It would be very insightful to see how the DL-methods perform compared to conventional physics-based method, which would be great to have included in the benchmark. However, this might be out of scope of the present work.

Will the published code include the data generation algorithms? (Both for velocity model construction as well as for seismic forward modeling.) This could increase impact of the work as it adds flexibility and researchers could generate their own models (e.g. with a source at depth, varying source frequencies, etc.).

Table 1: Add unit of time dimension, and inter-station distance and domain size

Line 180: Should be “l2-norm”
review_title: Review of OpenFWI: Large-scale Multi-structural Benchmark Datasets for Full Waveform Inversion

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposed benchmark datasets for multi-structural seismic full waveform inversion. OpenFWI contains 12 datasets and has following favorable characteristics: (1) Multi-scales (2) Multi-domains (3) Multi-subsurface-complexities. The authors evaluated three representative methods (InversionNet, VelocityGAN, UPFWI) over these datasets. All three methods perform well on some datasets, while the rest may need further improvement. At last, the authors raise three challenges that can be studied with and consider the future advancement as OpenFWI evolves.
strengths: This paper provides many strong mathematical representations to illustrates the data generation process. At the same time, the authors outline five potential topics to support the study motivated by geophysics domain interest. The authors also demonstrates the baseline results and consider three different metrics. 
weaknesses: For benchmark deep learning methods, I think the authors should add more details to support reproducibility.
correctness: The authors provide 12 datasets synthesized from multiple sources. The data generation pipelines sounds correct. The evaluation methods and experiment design appropriate and performed correctly. 
clarity: The paper is well organized and clearly written. Appropriate figures were given to make the paper understood easily. From my point of view, the work is well-done and provides an interesting topic for large-scale multi-structural benchmark datasets for full waveform inversion.  
relation to prior work: This work is a data-driven FWI methods which differs from previous physics-driven approaches.
additional feedback: N/A
review_title: There is a rapidly increasing demand for open geophysics datasets.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a large dataset to enable reproducible research in full waveform inversion (FWI) which is widely used in geophysics applications. Data is available to download, easy to access and code is expected to be released upon approval. The authors also evaluate three network architectures for FWI. Models are also evaluate for generalization and transfer learning between datasets. Detailed results are presented in the paper. 
strengths: The dataset is fairly large, easily accessible. The authors evaluate the application of the dataset to the geophysics domain and present detailed results, making the dataset and baseline implementations good benchmarks for the community to develop and evaluate against. Assuming that the code is also released in a timely manner, this would also enable good experimental reproducibility. 
weaknesses: No particular weakened to point out here.
correctness: Dataset construction is sound and evaluation of benchmarks is thorough.
clarity: Yes
relation to prior work: Prior work and differences are discussed 
additional feedback: n/a
review_title: OpenFWI: Large-scale Multi-structural Benchmark Datasets for Full Waveform Inversion

final_score: rating: null
review_content: Will the benchmark code be available via GitHub ? This link from the dataset website is not valid - https://github.com/margarita-aicyd/OpenFWI.
Thank you. 
review_title: Code release ?

Meta-review: metareview: All reviewers agreed that the proposed benchmark datasets for multi-structural seismic full waveform inversion are solid and valuable, and  there is diversity in the dataset sources. The baselines and metrics are clear. It may well encourage the research in this direction, and also inspire future open-source efforts on AI for science.

It is a clear acceptance. Based on the overall comments from all reviewers, it is recommended for Spotlight.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept