Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new particle cloud dataset (JetNet) for high-energy physics. The authors set up a benchmark for particle cloud generation with a few physics-inspired and vision-related metrics. As existing point cloud GANs are not suitable for physics applications, they have also developed a new message-passing GAN (MPGAN) and have achieved good empirical performance.
main_review: The paper is well-written and easy to follow. The authors have provided sufficient physics background to help non-physics researchers better understand the problem setup. The topic studied in this paper (using machine learning to advance and accelerate physics research) is very interesting and has a large scientific impact. The authors have promised to release their dataset, which might benefit the researchers in the community a lot.

Apart from these mentioned in the related work (L117-L132), there have been a few other papers related to point cloud generation in the literature [Li et al., ICLR 2019 Workshop; Shu et al., ICCV 2019]. The authors have claimed that existing GANs are inadequate for physics applications. However, without sufficient comparisons with these existing methods, this argument is not very well justified. Besides, the authors have also mentioned in the introduction (L25-L31) that ML-based generative models might accelerate the simulation by five orders of magnitude. However, they have not provided any latency measurement on the hardware to support this claim.

This paper provides a bunch of different metrics for the particle cloud generation. It would be great if the authors could analyze the correlation between different metrics and provide some insights on which metric is the best, which might facilitate the comparisons between different methods in this direction. Besides, it is necessary to provide some qualitative examples to illustrate how the generated samples look like and some discussions on what the target downstream task is. It would be interesting to see how the generated point cloud performs in real physics applications.

**Post-Rebuttal Comments:**
The authors' rebuttal has addressed most of my concerns. I especially appreciate the additional results of TreeGAN and PC-GAN. Therefore, I will raise my score by 1 and vote for accepting this paper.

References:
1. Li et al., "Point Cloud GAN", ICLR 2019 Workshop.
2. Shu et al., "3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions", ICCV 2019.
limitations_and_societal_impact: This paper does not have noticeable negative societal impacts. Besides, the authors have briefly discussed their limitations in Section 5.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to explore an interesting application of existing point cloud generative models to the problem of generation of jets: high energy sprays of particles from elementary particle colliders. Jets are analogous in a way to point clouds with a difference in considered per-object features: per-particle momenta are considered for jets contrary to per-point 3D coordinates for point clouds. With sufficient level of quality such generative models could potentially be used instead of costly Monte Carlo simulations.

The authors explore existing point cloud gan-based generative models and propose an adaptation of existing message-passing networks to be used in generators and discriminators of resulting particle cloud gans. Several metrics are proposed for evaluation, including adaptation of existing point cloud generation task metrics, and the distance between momenta distributions for real and generated jets.
main_review: Pros:
* The proposed application of point cloud generative models is novel and interesting. The authors promise to release the dataset to encourage researchers to contribute to the task.
* Both qualitative and quantitative evaluations show that the proposed message passing generators and discriminators yield better performance compared to straightforward application of point cloud specific alternatives.

Cons:
* The novelty of the approach is somewhat limited, since the overall proposed model is a combination of existing point cloud generative approaches and prior message passing networks. However, given the novel application context I do not think it is a significant drawback.
* Related work section could be improved by incorporation of existing point cloud generative models based on probabilistic modeling [1, 2, 3]. These approaches naturally overcome the issues of models with fixed point cloud size outputs and, in general, might be of interest to the authors.

The overall quality of the submission is sufficient and even though the novelty is limited, I think it is an interesting enough application context to allow it to be published to draw attention to the task.

Questions:
1) What was the number of message passing steps used in the considered architectures?
2) Were there any ablations done for the configuration of the final generators and discriminators in the proposed MPGAN?
3) You mention that one of the problems with application of ML-based generative models to physics in the lack of interpretability. You try to justify the use of such approaches if they would achieve the quality levels comparable to real baselines. Does these baselines give any guarantees on the objectivity of the resulting jet data? In case of GANs there are no such guaranties. Even if the mean metric aggregated over the whole evaluation dataset is relatively close to the aggregated metric calculated on the baseline data, there could be generated samples, which could be completely unrealistic (models are not strictly constrained to exclude them). If applied in the real-life scenarios, how generative approaches can overcome this issue?


[1] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: PointFlow: 3D point cloud generation with continuous normalizing flows. In ICCV’19.

[2] Klokov, R., Boyer, E., Verbeek, J.: Discrete Point Flow Networks for Efficient Point Cloud Generation. In ECCV’20.

[3] Cai, R., Yang, G., Averbuch-Elor, H., Hao, Z., Belongie, S., Snavely, N., Hariharan, B.: Learning Gradient Fields for Shape Generation. In ECCV’20.
limitations_and_societal_impact: The impact and limitations were briefly discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces the new particle cloud dataset JetNet and corresponding evaluation metrics for ML in high energy physics. Several existing point-cloud generative models are evaluated and a new baseline model for the dataset is introduced.
main_review: The introduction of JetNet is well-motivated and tackles a challenging problem in high energy physics.  The proposal of a message passing GAN (MPGAN) designed for this task shows significant improvement upon existing GAN approaches for the generation of point sets. As the authors note, MPGAN is still not sufficient for real-world applications such that JetNet presents a challenging task that might need novel approaches to achieve sufficient precision with a generative model.

The paper is clearly written and structured in a way that nicely introduces the problem setting. The discussion of design choices and merits/limitations of the metrics as well as the evaluation of all the models on three different tasks (gluon, light quarks, and top quarks) is thorough.

Minor issues:
- The comparison focuses very much on GANs. There is also some related work on generating 3d molecules based on autoregressive models and normalizing flows, which deals with similar issues, e.g. variable number of atoms and rotational symmetries (Gebauer et al., NeurIPS 2019, Köhler et al., ICML 2020, Simm et al., ICLR 2021). A discussion on whether these kinds of models might be applicable would be interesting.
- The fonts in the figures is quite small and hard to read (in particular the labels in Fig 2 and labels in Fig. 4.
- It would be nice for the order of gluon, light quarks and top quarks to be consistent across plots and tables.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: All of the reviewers really liked this paper. They especially noted the clarity of exposition and the fact that the authors intend on open sourcing their data. During the rebuttal phase, the authors added new experimental results using TreeGAN that helped to address some concerns of the reviewers. Overall this seems like a timely and interesting contribution to the literature on point clouds and GANs.
Decision: Accept (Poster)