Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: A collection of three classes of AEM problems is released and accompanied by a GitHub repository.
Two problems are completely novel, whereas the third one uses existing resources.
AEM problems were selected based on relevance, representativeness and span of complexity.
Three different DNNs are then exemplarily benchmarked as potential surrogate models on each of the AEM problems.
strengths: * In the field of AEM research, papers that publish code and datasets are apparently rare. Relevance to this community appears to be given.

weaknesses: * The suite appears to be quite limited in the number of problem classes (3) and so does the experimental section (comparing baseline, MLP, Transformer and Mixer).
* It may be useful to further investigate other performance metrics than only the average MSE. If the models are indeed used as surrogate models as described in Section 3.3., other metric such as Spearman's Rho or Kendall's Tau may be of additional interest.
* The analysis of results is solely based on visually inspecting performance, as given in Figure 4 in the form of box plots of the MSE. It could be interesting to additionally perform statistical comparison, for example in the form of critical difference plots.

correctness: * Data appears to be constructed in a sound way.
* The benchmark design is reasonable, although only one performance metric is evaluated (MSE).
clarity: * The paper is written well and can be followed along easily.
* Maybe some more focus could be given to the concept of using different DNNs as surrogate models of the AEM problem.
relation to prior work: * Related work is discussed and focus is given to the lack of publicly available AEM problems.
additional feedback: * I do must admit that I have very little experience regarding the field of artificial electromagnetic materials (AEMs), therefore my review mostly focuses on the general benchmarking part of the paper.
review_title: towards making AEM benchmarking more accessible

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents three benchmarks for artificial electromagnetic materials: one each on metamaterials, nanophotonics, and color filter designs. The datasets and code are openly available.

Edit: I have updated the score after revisions.
strengths: - The paper makes a strong case for the value of the benchmark by pointing out the gaps in the field.
- The study is rigorous and appears well thought-through from a general ML perspective (my knowledge of metamaterials is not sufficient to judge from the domain perspective).
- The software allows for future additions to the benchmark datasets. This is important as the field progresses.
weaknesses: The benchmarks do not contain a simple non-deep learning baseline. It would be great if the authors could add one or comment on why only DL models are suitable for the presented tasks.
correctness: Experiment design is appropriate and has been performed according to well-established standards. Datasets were adopted from previous publications.
clarity: The paper is very clearly written and explains the problems well to non-domain experts.
relation to prior work: The paper discusses an ample body of previous work and justifies its claims. I am not familiar enough with metamaterials though to judge whether something important is missing.
additional feedback: -
review_title: Well thought-through collection of datasets and valuable benchmarks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper is well motivated as the authors point to the lack of shared models and datasets despite the rise in number of papers which apply deep learning to this problem. Application of deep learning to this domain is an instance of avoiding high compute by instead training a surrogate deep learning model on a set of observations. The resulting speedup benefits the design of artificial electromagnetic materials. However without standardized benchmarks, there is no clear route to assessing the generalizability of models. This issue  also hinders reproducibility and limits the impact of improvements in the field. This work aims to introduce a 3 benchmark datasets from within the field of AEMs as well as provide some initial evaluation of model performance. 
strengths: 
- At the time of review, the repository provides a straightforward way to download the datasets. An implementation of the 3 models discussed are also available along with pre-trained weights.
- The choice of datasets appears well motivated and there is a brief discussion of each. The authors have included their selection criteria for the datasets which ensure both relevance and representativeness.  The authors also include significant detail on the data generating process as well as a brief overview of the theoretic background.
- A baseline model is included as representative of prior work, which provides a context for judging the performance of newer models.
weaknesses: 
- There is little to no information on how splitting was conducted for these benchmark datasets, which leaves the reader to assume that a random split was conducted. Information about split sizes and scattering distributions is also missing.
- Baseline model for each dataset deserves more discussion or at least a brief description of the hyperparameters. It is hard to place results in Figure 4 in context. It is unclear how different the Baseline model is compared to the MLP model for example.


correctness: The information presented appears to the best of the reviewer's understanding of the AEM field to be correct. Information about model description is sound.
clarity: Yes the paper is well written.
relation to prior work: There are no existing benchmark datasets for ML in this field. 
additional feedback: - Figure 4
    - the Color dataset results seem to be cropped slightly
- Figure 5:
    - Not sure which best performance means in this case? Would that mean lowest average MSE across datasets? That would just mean that number of parameters are log ratio'd by their performance on the Color dataset, which seems like a weak comparison.
- Supp. Figure 2:
    - missing a-f labels.

Minor Comments

- 3.3: "a parameterized geometry g – which is of order g ∼ 101 – of AEMs as input"
    - slightly confusing. would suggest either using some notation to indicate that the dimensionality of g is on the order of 10s or remove the em dash entirely
- Missing grid search results. The authors state in Section 5 that they performed hyperparameter optimization to find the best performing model using a validation set. It would be helpful to ML practitioners as well as strengthen the paper to include these results.
review_title: DL for AEMs 

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors present datasets for 3 tasks in the design of Artificial Electromagnetic Materials (AEMs)---materials with useful electromagnetic properties based on their nanoscale geometry---and benchmark 3 common deep learning architectures on these tasks. While there has been much interest and progress in using deep learning, or "surrogate models", to approximate these structure-property relationships, there have been almost no datasets made available, making replication of results very difficult. Authors propose to establish a groundwork for this emerging application area by releasing datasets for 3 relevant AEM design problems. Authors benchmark MLPs, MLP-mixers, and transformers on these tasks, with significant modification to the latter two architectures to accept fixed-length feature vectors describing the material geometry.
strengths: * The paper helps bring materials design problems to the attention of the broader ML community in an accessible manner.
* Authors have significant domain knowledge and have provided thorough reasoning for the datasets they chose to develop.
weaknesses: * Since this is a ML conference, it should make sense for the tasks to be broadly presented to the ML community. This can happen if the problem structure is exposed in a way that supports interesting research directions in ML. For example, the presentation of molecules as graph-structured data spurred advances in GNNs and end-to-end learning from 3D structures. However, this paper frames the problem of material property prediction as a mapping from $\mathbb{R}^n \rightarrow \mathbb{R}^m$, with no additional structure, and so (from an ML perspective) is indistinguishable from the countless other application areas that can be so framed. For the same reason that using various classifiers (SVMs, MLPs, ...) with fixed-length molecular fingerprints is more a cheminformatics problem than a ML problem, it's not clear that the formal treatment of materials design in this paper is unifying and/or abstract enough to be an appropriate fit for NeurIPS. 
    - The significance of the paper would be much improved if, for example, the authors leveraged the geometric nature of the input $g$, or the continuous stochastic process nature of the output function $s(\omega)$, or discussed ways of learning representations for a broad class of problems in AEM design.

* The choice of architectures to benchmark is unconvincing. The forced application of Transformers and MLP-mixers to partitions of a fixed-length input vector does not seem well-justified (as the authors themselves note). It would be much more interesting to compare with architectures that take 3D geometry into account, such as 3D CNNs, GNNs, point-cloud networks, etc...  Using such 3D methods seems particularly natural since CEMS involves solving Maxwell's equations in 3D.
    
* To summarize, because of the way the tasks are presented, it's not clear how pursuing further improvement on the datasets would be a fruitful line of ML methods development.
correctness: * How are the datasets split into train/val/test? This is a critical process---which affects the correctness of the submission---that should be described and justified in detail.
* Minor correction: [MLP-mixer](https://arxiv.org/abs/2105.01601) is proposed as an alternative to CNNs, not transformers
* The "performance bias mitigation" seems to be an unconventional restriction that could affect the correctness of results. I am particularly concerned about the asymmetry of the restriction. It doesn't look like it affected any results, but I would suggest standardizing for model size in a more systematic and symmetric manner.
clarity: Some aspects of the paper could be improved in clarity.
* How are the AEM designs in the dataset generated? Are they drawn from a distribution over the design parameters? What type of distribution?
* Authors should much more prominently (at the top of the corresponding sections) state when the datasets are original, when they are sourced from the previous work of the authors, and when they are sourced from other authors.
* The results in Figure 4 should be presented in tabular format for ease of comparison rather than only in a figure.
* What material are the layers of the nanophotonic particles manufactured from? Is this an input into the model?
* While I was generally able to follow how the three types of designs were featurized, the exposition could be clearer.
relation to prior work: No concerns.
additional feedback: The paper is not properly formatted. Authors used the preprint option instead of the submission option, and thus there are no line numbers.
review_title: Interesting application but several aspects of paper lacking

Meta-review: This paper proposes a dataset and a benchmark for materials design. It is an important contribution that would allow ML innovations in this important area of research. The reviewers mostly agree that the paper is well written, introducing material design well to the ML audience. The dataset is well documented and formatted, and appropriate baselines are presented. This paper is an important contribution to multi-disciplinary research in an important area of science, so I recommend acceptance.
Decision: Accept