Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes ATN to model and identify physical systems. Here, the physical systems are grids consisting of measurement sensors: because the sensor configuration is not perfect, there can be unobservable parts in the grid system. The authors assume that there are physics bases that can represent the output with a linear mapping. Thus, they firstly use a linear LASSO regressor (PNN in the paper) that transforms observable physics bases to the output. The possible remainder (due to the noise or unobservable physics) is modeled via a more complicated neural network (VNN in the paper). These two networks are integrated into a single skip-connection block. They are trained by MSE, with enforcing the PNN part predict the output mostly, to exploit the bias of physics. In addition, the authors suggest the use of siamese networks that measure the distance between PNN and VNN: then an adversarial learning loss is added to guarantee the outputs of PNN and VNN to be far from each other and play their respective roles more concisely. ATN outperforms its counterparts for some physical system datasets.


main_review: Physical system identification is a definitely interesting topic in both the machine learning and physics communities. The authors address such an important issue and validate their proposal empirically. The proposed ATN show remarkable results in terms of model accuracy and unknown parameter estimation, with robustness regarding the sample size and observability. 

However, there are some concerns that should be addressed for clear evaluation of this paper:

* Q1. The authors assume that there are physics bases $z$ that yield a linear relationship $y = Az$. In addition, they assume the unobservable base is also a linear transformation of the observable one, i.e., $z_U = B z_O + \epsilon$, where $\epsilon$ is relatively small noise. I wonder this assumption is general for arbitrary physical systems or just for fine the examples presented in this paper.

* Q2. Because the proposed architecture is complicated, I wonder all the components are truly necessary. The authors claim the usefulness of each component indirectly, by comparing other baselines (e.g., the usefulness of physics bias with comparing ResNet, that of VNN with comparing SINDy, …), However, a more systematic ablation study should be included in the paper (or Appendix if the page is not sufficient), to clearly support the necessity of the complicated ATN structure. 

* Q3. It seems that the following paper is highly relevant: "Y Yin et al. Augmenting physical models with deep networks for complex dynamics forecasting. ICLR 2021." This paper also decomposes the unknown physical system as a known physics part (however whose parameters are unknown) and the neural network part that fills the remainder that cannot be explained by the physics-based one.

* Q4. It seems that the proposed ATN already knows the appropriate physics bases $z = \phi(x)$, referring to the explanation on page 3 (“we fix the parameters from $x_O$ to $z_O$”) and equations (1 – 4) (no $x$ and $\phi$ in the objectives). However, on page 7, the authors state that “ATN and SINDy can both learn physical bases”, which seems that there is a searching procedure of physics bases from a set of predefined operations. Can the authors elaborate it more clearly?

* Q5. There are some inconsistent or confusing notations in the paper, e.g., on page 3 “Correspondingly, we denote $z = [z_1, z_2]^T$, where $z_O$ are physical bases…”, on page 4, $y_O = g_p(z_O) + g_v(g_p(z_O))$ should be $y$ rather than $y_O$ (because $g_v$ is included in the expression, and consistency with (1 – 4))?. In addition, Figure 1 is not clear because all subscripts of variables are 1, 2, … rather than $O$ or $U$.



summary_of_the_review: Currently, I lean towards rejection due to the concerns I raised. I would like to update my evaluation after discussing with the authors and my fellow reviewers.

***
Post-rebuttal:  I appreciate the authors’ clarifications on my questions. I am happy to see the ablation study result, which can support the necessity of the proposed components more clearly.

However, the essential assumptions used in this paper ($y = A \phi(x)$ with known $\phi$ and $z_U = B z_O + \epsilon$) make the proposed model be applicable only for relatively restricted problems. It prevents me from recommending a clear acceptance for this paper. While I slightly tend to accept this paper (thus I updated the review score accordingly), I would not champion this paper for the acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Paper presents a method to push physical quantity to a physical neural network (PNN) and push the ‘additional quantity’ such as noise to a correction term virtual neural network (VNN). Uses a pair of embedding neural network (ENN) to achieve the above.
main_review: Several things need to be clarified before a proper review can be put in place:

#1. Beginning of section 3.1 The author frame the problem as a graph with node set V and edges VxV. Each node in V contains (x,y) with the relationship y = f(x). Does this mean that (x,y) are associated with each node and for different nodes there are different (x,y)?

#2. Then the assumption is made for some nodes being observed and some not observed. V = O union U. The the new symbols are introduced [y_o,y_u] = f( [x_o, x_u]). Seems to me that y_o is a set instead of variables representing one specific node. Otherwise this equation [y_o,y_u] = f( [x_o, x_u]) does not make sense. If so, then the (x,y) in my point #1 are variables, not set and here y_o is a set of variables. Please clarify this point

#3. There could be a typo here: “Correspondingly, we denote z = [z1,z2]⊤, where zO ..”  What are z1 and z2?

#4. \phi the last paragraph of section 3.1 represent the basis coefficients for some known basis functions? Are the values of \phi learned end-to-end in this network? By learning PNN?

#5. Eq(3) the variable Y. Y=0 if sample pair is similar and Y=1 otherwise. In Eq(3) we only pass in one zo, not clear where the ‘pair’ comes from. Perhaps it is due to the confusion I brought up in #2. If zo is a set of variables, then how to apply a function to it? Element wise application?

summary_of_the_review: will do a more proper review after my queries have been addressed. Will change my score after the proper review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The task of identifying a physical system on a graph is addressed. While the main part of the to-be-estimated model is assumed to be linear, the proposed model needs a nonlinear part (which is modeled by a neural net) due to the presence of unobserved nodes. The authors use a combination of a sparse linear model and a neural net, which is basically the same as the model in [Li & Weng, KDD 2021]. They consider some regularization terms to maximize the use of the physics part of the model. They examine the performance of the proposed method on several datasets.
main_review: The motivation and the base problem setting are understandable, and the experiments are done on multiple datasets. However, I found not a few contradicting or unclear statements. Also, the technical novelty is not that significant. Experiments lack important ablation studies. These facts make it difficult to recommend acceptance of the current paper.

Below, I write detailed comments in an unordered way; in my intention, points (2), (3), (7), (8), (9), (10), (11), (12), (13), (14), (15), and (16) are questions, and the others are not necessarily questions (anyway, please correct if something is wrong).

(1) The beginning of the problem formulation in section 3.1 is misleading. Now it says:

> A physical system can be modeled as a directed weighted graph ...

This sounds a bit weird because many physical systems are not usually modeled as a directed graph. Instead, it would be more appropriate to say that you handle specific types of physical systems that are modeled as directed weighted graphs.

(2) Seemingly, at the layer of the problem setting, the main difference from [Li & Weng, 2021] is the presence of a possibly nonlinear function $f$ between $x$ and $y$. Meanwhile, such a difference is actually not valid because now the prior knowledge of the nonlinear functions used in $f$ is assumed as $\phi$. Consequently, in my understanding, there is no effective difference between the problem setting of the current paper and that of [Li & Weng, 2021]. I would like to hear if this understanding is correct or not. Please note that I understand the methods in both papers are more or less different; here I question the difference of the problem setting.

(3) What are "the remaining quantities" that appear at the second line of section 3.3? Do they refer to $x$'s (or $z$'s) on unobserved nodes?

(4) I fully agree with the need for restricting the output of VNN and encouraging PNN to output as much as possible. In fact, similar ideas have been already investigated in the context of physical system learning in literature such as [Yin et al., 2021] and [Takeishi & Kalousis, 2021].

[Yin et al., 2021] Yin et al., Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, in ICLR 2021.

[Takeishi & Kalousis, 2021] Takeishi & Kalousis, Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling, arXiv:2102.13156.

(5) In section 3.3, the skip-connection of the two neural nets is "proposed," while it is just a frequent practice. 

(6) The regularization in equation (1) to encourage $g_p$ to directly approximate $y$ is reasonable. I note that a very similar idea was adopted in [Yin et al. 2021], where they regularize the norm of the non-physics part (i.e., $g_v$ if the notation is aligned to the current paper). I think the two approaches, i.e., minimizing $\Vert y - g_p(z) \Vert$ and minimizing $\Vert g_v(g_p(z)) \Vert$, are almost the same thing eventually after fitting gets enough good. However, in the early phase of training, since the models are not well fitted yet, perhaps there might be some practical difference.

(7) Does $y$ without subscripts in equation (1) mean $[y_\mathcal{O}; y_\mathcal{U}]$, similarly to $z$ and $x$? Maybe I missed something but could not find an exact definition.

(8) The most puzzling thing is that in section 3.4, $g_v$ suddenly starts to take $z_\mathcal{O}$ directly as an argument, whereas it used to take $g_p(z_\mathcal{O})$ as an argument in section 3.3. What are the definitions of the domain and the range of the functions around here?

(9) I could not see the point of introducing matrix $B$ in section 3.4. Does it appear somewhere in the model?

(10) What is "large common knowledge", at the fourth line of the fourth paragraph of section 3.4?

(11) What is the relation between the proposed adversarial loss and the well-known Wasserstein distance based on the Kantorovich-Rubinstein duality (see, e.g., the paper of Wasserstein GAN)? Do you have any observations on the advantages of the proposed loss, compared to the Wasserstein distance?

(12) What is the actual use of $Y$ in equation (3)? In my understanding, the idea here is to maximize dissimilarity between $g_p(z)$ and $g_v(z)$ for each $z$, and thus $Y=1$ is always used. Then, there is no need to consider the case of $Y=0$. Is this correct?

(13) What $\phi$ exactly was given to the proposed method in each experiment?

(14) In the experiment, are the baseline methods also given the knowledge of the base functions, i.e., $\phi$? If not, then the comparison is not really fair. 

(15) The authors say that one of the baselines, PCNN, "doesn't include a clear design of LASSO regularization." However, in my understanding, [Li & Weng 2021] use lasso for initializing the parameters for fully observed nodes. Do you also use the same lasso-based initialization for PCNN?

(16) Seemingly, both the reported test results and the hyperparameter selection are based on cross-validation. Are there two different (and maybe nested) cross-validation? This is an important question. I would like to know the difference between the cross-validation for hyperparameter selection and that for generating the final test results.

(17) There are no ablation studies on the effect of the proposed regularizers (in equations (1) and (3)), which makes it difficult to assess the goodness of the proposed method.

(18) Twin structures of neural networks are mentioned in section 2 and some other occasions, but I could not understand why the proposed method also falls in this category, because nothing seems to be twin in the proposed method (PNN is linear, and VNN is nonlinear).

Below, minor points.

Unnecessary "and" slips in at the third-last line of page 1.

"Target is the identify ..." (the first line of page 2)

$z=[z_1, z_2]^\top$? (the lower half of page 3)

"lower nose level" (the seventh line of page 8)
summary_of_the_review: Technical novelty is limited, and there is no ablation study on the exact proposal of the current paper. Hence, I can hardly recommend acceptance in the current form of the paper.
review_title: null

Meta-review: This paper addresses the identification of physical systems defined on graphs. The authors introduce the Adversarial Twin Neural Network (ATN), which consists in augmenting a simple linear model (PNN) with a virtual neural network (VNN). Some regularization terms are used to enforce maximum prediction from the PNN, and to enforce diverse outputs between PNN and VNN. 

The paper initially received tree rejection recommendations. The main limitations pointed out by reviewers relate to the limited contributions, the limiting assumption of using a linear mode for PNN, the lack of positioning with respect to related works, and clarifications on experiments. The authors' rebuttal answered to some reviewers concerned: Rdem1 increased its grade from 3 to 5, and Rdem1 from 5 to 6 - although not willing to champion the paper. R8dT9, which provided a very detailed review and feedback after rebuttal still voted for rejection, especially because he was not convinced by the positioning with recent related works and the answers on experiments.  

The AC's own readings confirmed the issues essentially raised by R8dT9 and other reviewers. Especially, the AC considers that: 
- The contributions for driving a proper cooperation between the PNN and VNN models are weak, since it reduces to using simple skip connection and adversarial training. 
- The importance of these aspects have not been analysed in depth in the revised version of the paper, neither theoretically nor experimentally: for example, the difference with respect to [Yin+ 2021] for a proper augmentation, the discussion to alternative methods for representing diversity as done in [Rame & Cord 2021], or the positioning with respect to  Wasserstein distance-based objectives.  
- There remains ambiguities in the cross-validation process, which have not been addressed in the rebuttal.

Therefore, the AC recommends rejection.
Decision: Reject