Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to generate training data for symbolic reasoning by training GANs based on Transformers. Specifically, the paper explores two methods --- standard GAN and Wasserstein GAN --- which has a Transformer as an encoder and a decoder, respectively. Training these models on symbolic reasoning data that is randomly generated --- LTL and Symbolic mathematics --- is found to generate high quality formulas.
main_review: ** Update after author responses ** After the author clarified with the format of the data and results, my problems have resolved in part and thus I updated the score. My concern in the motivation of the paper still remains. The paper claims that synthetic dataset generated from rules is limited in the coverage so that it is valuable to construct synthetic dataset from neural models --- however, it is still not clear to me if experiments demonstrate that the resulting data from neural models are shown to have better coverage and higher quality than data from rules.

The model introduced in the paper is well-executed, and based on the generated data attached in the submission, the data quality appears to be good enough. There are still a few concerns I have in the motivation of the problem and experiments.

First, the problem setup is not convincing to me and is not well-motivated in the paper. If data for the symbolic reasoning --- which is not precisely defined in the paper --- can easily be generated at scale using rules, as done for the base data in the paper, what is the reason for training a neural model to generate more data? How is it inherently different from generating more using rules, as data, either generated automatically from rules or generated by neutral models, is equally artificial?

~~Second, although the paper claims that the generated data is high quality, it seems to have quite bad quality to me. I checked Supplementary materials and most formulas contain many repetitions like “&&&&&&&GXaG>!” and are very hard to interpret. It is very difficult to find good examples as listed in Section 4.2.1, so there is a high likelihood that these good examples are cherry-picked.~~

Third, the experiments in Section 4 do not demonstrate that the generated data has high quality and effectively replaces the base data. For example, Table 2 shows that training on generated data achieves performance that is comparable to the model trained on the original data, but not better. This is related to my first point in the motivation of the problem - if the original data can be obtained automatically at scale and generated data from the proposed model is not significantly better than the original data, is there a justification for not using the original data?


summary_of_the_review: The model introduced in the paper is well-executed, and based on the generated data attached in the submission, the data quality appears to be good enough. However, in my opinion, there are more fundamental issues in the motivation of the problem and whether experiments successfully justified the usefulness of the generated data from the proposed model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Generating symbolic reasoning training data using transformer GANs. Use classifier uncertainty in the generator objective to generate samples that are harder for the classifier to solve than the original data. The problem is very interesting, but the empirical results could be improved.
main_review: The problem is interesting and the paper is well motivated.

Abstract: What do they authors mean by “synthetically generated instances are often hard to evaluate in terms of their meaningfulness”?

What do the authors mean by “training on randomly generated data carries the risk of training on meaningless data or the risk of introducing unwanted biases”? Provided that the conclusion follow logically, can the authors elaborate on what they mean here? 

“We show that training directly in the one-hot encoding space is possible when adding Gaussian noise to each position”. Why use Gaussian noise? The authors should be more clear about why they add noise to the real samples? Presumably this is to make distinguishing the real samples from the fake ones non-trivial? This is only explained later on and is a bit confusing.

“on which a classifier can successfully be trained on”. Do you test on an established dataset?

“The generator’s input is a real scalar random value with uniform distribution [0, 1] for each position.” It is not clear here what is mean by “each position”. I assume this is each symbol in the input? But this is not clearly explained.

“The position-wise padding mask is copied from the real data during training, so the lengths of real and generated formulas at the same position in a batch are always identical.” The first time you refer to **the** padding mask it is not clear what this is or where it comes from?

“Still, both generators are able to produce a large fraction of fully correct temporal specifications, which we find surprising” Could there be overfitting? Are the results the same across many runs? Is 0.3 a large fraction? How does this compare to generating examples randomly? This would make a good baseline (even if your models perform worse).

Figure 4: Interesting results.

The satisfiability classifier results are interesting. What would be better tho, is to show that training on the generated data improved results on an established dataset. 

It is interesting that you can learn on data generated using the LTLbase 10k dataset and perform better on the validation set than training directly on the LRLbase (and perform similarly well when training on the whole LTLbase dataset). However, it’s clear that the model has overfit to the LTLbase 10k. How did you decide to stop training? Does this happen for all runs? It would be helpful to see the training and test curves while the model is training. Additionally, it would be good to see a graph with multiple runs.

For all results in the paper, it would be best to perform multiple runs and report the standard deviations. 

In the section titled “GAN with included classifier”: What are you classifying? Are you predicting satisfiability? This is not clear. How do you know the satisfiability of the generated samples? Is there a 50/50 split of satisfiable and unsatisfiable examples?

Are there the same number of training examples in the LTLbase, Uncert-e and mixte-e datasets?

Table 3: Are you training and testing on different splits of the dataset? It would help to add the std? The results somewhat suggest that training to increase uncertainty does produce slightly more challenging problems, but it’s hard to tell because it’s not clear how much results vary between runs and it’s not clear if the differences are statistically significant. 

Table 3: It’s very important to know how train and test samples were split for Uncert-e. Uncert-e being more difficult to classify could also be explained by lack of variation and the samples you tested happening to be in a different mode to those in the training set. What is the average length of problems in each data set?

Is there a qualitative difference between samples generated with and without the uncertainty loss? What makes the problems harder?

Does training on Uncert-e improve performance on LTLbase (more so than training on generated?) Is there a standard dataset that you can show improvement on?

Why are there not more quantitative results on the integration examples? It’s clear that a classifier can already perform very well on the LTL tasks, perhaps it would be easier to see improvements on the function integration task? It does not appear that there are any results for this in the main text? 

General: The paper does not clearly separate LTL results and symbolic math results. This makes some results harder to parse.

General: This model does not generate a supervised training dataset since you still need to use existing algorithms/ programs to compute the labels/targets. This could be a problem for datasets where the solution is intractable and would also suggest that you already have a model capable of solving the problem for which you are generating the data. What is the long term motivation of this work if you either (a) cannot generate labels/targets or (b) can already use existing algorithm to solve these problems.

summary_of_the_review: The problem is really interesting. 


Correctness: 
There are problems with the experimental results. If the authors can add the suggested results and show that their results are statistically significant, I would be very happy to increase my score. 

Novelty: The approach also lacks novelty, only proposing an additional loss which is not clearly described. However, their application is very interesting.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes the use of GANs equipped with Transformers to generate new
instances of the problems in the symbolic reasoning domains, and demonstrated
the usefulness of the idea on two domains: satisfiability prediction of LTL
formulas and mathematical reasoning on integration and ordinary differential
equations.  The experiments show that the trained GAN models can produce
parsable instances on both of the domains and that training on the generated
data can improve the accuracy performance of a classification model for LTL
satisfiability prediction.
main_review: ### Strengths

* Simple idea: using GANs for data augmentation.

* Empirically proving that GANs can generate parsable instances and that GANs
  can generate interesting problems that are hard to solve and contribute to
  improve the performance of classifiers.

### Weaknesses

I have several concerns and questions as follows.

* I'm not entirely sure of how the generated data are labeled.  Is it done by
  the GAN discriminator or by another algorithm as labeling LTL formulas in the
  real dataset with the tool aalta (Appendix B.3)?

  If the latter is employed, it indicates the existence of an algorithm to
  address the task of interest, so I'm wondering why machine learning is applied
  to it.  Perhaps ML models might be expected to answer the problems faster
  than the algorithm, but labeling with the algorithm would be successful only
  on problems that the algorithm can answer in a given time; thus, it seems hard
  to label problems that the algorithm cannot solve in a reasonable time,
  although the ML models should approach such problems.

  For the former, the generated data could be mislabeled.  Then, it is not clear
  for me why training on such data can produce a model with the performance very
  close to the model trained on data with correct labels (Table 2).

* I don't think this is the first work to use GANs for data augmentation.  For
  example, [1] and [2] used GANs for augmenting image data.  Unlike
  these previous works, the paper addresses sequential data that are textual
  representations of mathematical expressions, but lacking the discussion and
  (qualitative) comparison with them makes a challenge and novelty of the paper
  unclear.

  [1] Antreas Antoniou, Amos J. Storkey, Harrison Edwards.
  Data Augmentation Generative Adversarial Networks.

  [2] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger N. Gunn, Alexander Hammers, David Alexander Dickie, Maria del C. Valdés Hernández, Joanna M. Wardlaw, Daniel Rueckert:
  GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks.


* The abstract and introduction seem oversold somewhat in that they don't
  mention that the experiment conducted for symbolic mathematics only
  confirms the GAN model can produce syntactically correct problems; thus, the
  full usefulness of the GAN-based data augmentation is still open in the
  symbolic mathematics domain.  I think it would be nicer to conduct more
  experiments on symbolic mathematics, which would demonstrate the proposed
  approach is useful in broader domains.

* The experiments on satisfiability of LTL formulas look interesting, but I'm
  concerned that it is adequate to be tacked with machine learning.  As written
  in Appendix B.3, there exists a tool for checking satisfiability of LTL
  formulas, and it is used to label the (real) dataset. Then, why do we need the
  trained model for checking satisfiability?  A critical problem with the
  trained model is that its prediction might be wrong.  How can we use a
  satisfiability checker that may produce wrong answers?

  For a similar reason, I'm not sure why machine learning is useful for the
  domain where "data can ... often be labeled automatically" (page 8).

* The paper uses GAN and WGAN, but it does not investigate why they make a
  difference in the experiments (e.g., Table 1).


### Minor comments / questions

P2 "\neg \box (access_p0 \wedge access_p1)"  I think \box (globally) should be
replaced with \diamond (eventually) for correction.

P2 "Mathematical expressions"  What are these?  Are they different from random functions?

P3 "continuous domains were"  where

P4 "we use the alternative generator loss"  Why?

P6 "the origin training data"  original

P6, Table 1:
  For GAN, does the use of \sigmoid_real larger than 0.2 make the fraction of fc larger?

P7 "we combine both critic and classifier into one Transformer encoder"  Why are not they separated?

P8: The result in Table 3 seems peculiar to me.  Why does the model trained on
Mixed-e outperform the model trained on LTLbase even when tested on LTLbase?

### Post-Rebuttal

I would like to thank the authors for the additional comments to answer my questions.
However, I still have two major concerns that make me hesitate to accept the paper.
The first s that I don't still find the task of generating symbolic expressions interesting.
The second is about an application of the approach.  The response from the authors says that neural models may compute solutions faster than classical tools.  I don't disagree with this claim, but the paper doesn't show that the proposed approach is indeed helpful for that task.  I would like to see more discussions and evidence for the story to hold true (e.g., how the augmented data are labeled, whether they can improve the performance of neural models that compute solutions, etc.)
summary_of_the_review: The paper addresses a critical problem in symbolic reasoning domains, and the
experimental results are promising. However, I think it's not ready for
publication because of lacking a discussion for practical settings to use the
proposed GAN-based data augmentation, an evidence to show its generality, and a
comparison with the previous work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors apply (W)GANs with transformer encoders for data-augmentation in two symbolic domains: LTL and function integration. There are many interesting findings, three of which stand out. First, in both domains, the network learns to generate syntactically-correct examples roughly 30% of the time. Second, for LTL, a GAN trained on a dataset of size 10k can produce a much larger dataset, such that training on the new dataset is almost as good as training on the original distribution (when evaluating on the original distribution). Third, by also rewarding the generator for confusing a classifier, they can generate new problems that are harder to classify than those in the original distribution.
main_review: This is a strong paper overall: innovative, well-written, and potentially important.

- I found it awkward that the prose stressed that two domains would be considered, in some cases even listing symbolic mathematics (i.e. function integration) first, even though the integration domain received much shorter shrift and most of the experiments seem to be LTL only. I think the paper would be stronger with two domains throughout rather than one. 

- How many examples were in LTLBase? It might be nice to see an estimate of the effective size of the "Generated" dataset with respect to the original distribution.

- I see that in S4.2 you are starting with only 10k samples from LTLBase, but what are you training on exactly for S4.1? If a larger dataset than in S4.2, did you do a similar check for duplicates between the generated examples and the training dataset? How many (if any) of the 30% that are syntactically valid were seen during training?

- It is not clear to me whether the last paragraph of S4.1 is describing a different training regime than the paragraph preceeding it.

- The first sentence seems more suitable for a blog post than a conference paper; in particular, the claim that deep learning is "on the verge" of something sounds unscientific. Also, the word "transitioning" may not be appropriate, since presumably deep learning will continue to be applied to e.g. image recognition.

- I particularly liked the approach in S4.3 in which an additional objective is introduced that shifts the generated distribution away from the original one. In general, the true objective of data augmentation may be broader than simply modeling the specific distribution of training examples one has at hand. Have the authors considered other "knobs" to add that would allow generating more diverse examples, that may provide useful augmentation either for the original distribution or for out-of-distribution evaluations?

- Minor comment: I did not find that it added much to the paper to consider both GANs and WGANs. I would suggest only discussing WGANs.

summary_of_the_review: This is a strong paper overall: innovative, well-written, and potentially important. I think it will be of interest to anyone applying machine learning in data-sparse symbolic domains.
review_title: null

Meta-review: The paper aims to improve complex reasoning. In this regard, authors identify that acquisition of data for symbolic reasoning domains is a challenge and propose generating the data by GANs. A transformer-based architecture is proposed and trained for LTL and Symbolic mathematics. Experiments show samples generated are of good quality (e.g., correct syntax). We thank the reviewers and authors for engaging in an active discussion. However, the reviewers did not find the task of such data on its own not to be particularly interesting. Also, neither the architecture nor the training algorithm is very novel. If authors could provide a complete story i.e., show the augmented data can improve the performance of neural models that compute solutions, etc., it would make the paper much stronger. Thus, unfortunately I cannot recommend an acceptance of the paper in its current form.
Decision: Reject