reviews: 
Comments: This work aims to reduce the cost of imposing hard physical constraints in deep neural network training by utilizing Mixture-of-Experts. The proposed approach is scalable and allows multi-GPU training. The improved stability and efficiency of the proposed method over existing approaches are well demonstrated on non-linear systems.
Strengths: (1) The paper is overall well written.

(2) The problem addressed in this paper, which is how to impose physical constraints into large-scale deep neural network training, is of importance.

(3) The improvement of the proposed method over baselines is significant.
Weaknesses: Major: As claimed by the authors in the last paragraph in page 4, the utilizing of Mixture-of-Experts is motivated by two reasons/challenges. (A) In complicated systems, compared with global physical constraints, local constraints are more manageable. (B) The scalability of the non-linear least squares solver impeded the increase in the number of sample points (m), thereby affecting the accuracy of predictions. 

Reason A aligns with the phenomenon we have observed in language tasks [1]: data from different domains can interfere with each other and hinder training. However, the authors did not provide evidence for the occurrence of this phenomenon in dynamic systems, nor did they demonstrate that the improvement of the PI-HC-MoE over PI-HC is due to avoiding such interference. One possible experiment to demonstrate the latter is to randomly assign data points to experts and compare it to the allocation based on spatio-temporal coordinates.

Challenge B can be overcome by utilizing MoE. However, a more straightforward solution will be employing scalable non-linear least squares solvers, e.g. [2][3]. The authors didn’t comment on this line of works. If there are suitable solvers available, I would like to see a comparison between PI-HC and PI-HC-MoE with the same number of data points.

Furthermore, the performance of the baseline appears to be inconsistent with the results reported in [4], e.g., 1D non-linear diffusion-sorption equation results, see [4], Appendix E.

Minor: There are several errors and confusions in the use of mathematical symbols in the text, including but not limited to 
(1) in section 3, first paragraph, line 1, $0$ should be in bold.
(2) in section 3, first paragraph, line 4, the definition of $\mathcal{G}$ and in the second paragraph, line 4, the definition of $f_{\theta}$, other notation should be used to denote the spaces of $\phi$, $u_\theta$ and $\mathbf{b}$.
(3) In section 3, second paragraph, line 5, $f^i: \phi \rightarrow \mathbb{R}$ should be $f^i: \Omega \rightarrow \mathbb{R}$.
(4) In section 3, second paragraph, third to last line, $f_\theta(\phi(x))$ should be $f_\theta(\phi)(x)$.
(5) In page 4, two lines above Property (1), the definition of $S_{\mathbf{b}}$ and $S_\omega$.

Similar errors are repeated throughout the paper and should be corrected.

[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
[2] Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization
[3] DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition
[4] PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good