Learning and reasoning with physical, structural, and symbolic priors
Creators: name: Zheng, Wenqing
Publisher: name: The University of Texas at Austin
Year: 2023
Subjects: Deep learning
Domain specific prior

description: General purpose neural network models such as transformers have achieved remark- able success in solving complicated tasks across multiple domains and modalities. However, in many applications, the need for domain-specific models remains evident. Moreover, large-scale general-purpose models often demand an excessive amount of data to achieve satisfactory generalization. This PhD research aims to address these challenges by focusing on several application domain settings and improving model generalization through the integration of domain-specific priors into the learning and reasoning process. To accomplish this goal, we explore four distinct computational domains: traffic light control, graph cold start recommendation, program synthesis, and optimizer learning. For traffic light control, we tackle the task by incorporating the delayed propagation physical prior into the model architecture. We introduce the Delayed Propagation Transformer (DePT), a transformer-based model that leverages a cone-shaped spatial-temporal attention prior. DePT enables global modeling of CPS by considering the immutable constraints from the physical world, resulting in improved generalization performance compared to state-of-the-art expert methods. In the context of graph cold start recommendation, we address the limitations of Graph Neural Networks (GNNs) under cold start scenarios. Specifically, we introduce Cold Brew, a teacher-student distillation approach that incorporates neighborhood message passing, and quantified the behavior of inductive GNNs through the feature contribution ratio. For program synthesis, we focus on the challenge of generating high-quality code solutions by integrating structured thought processes. We propose ChainCoder, a program synthesis language model that progressively generates Python code in multiple passes, reflecting the “outline-then-detail" paradigm. By decomposing source code into layout frame components and accessory components, ChainCoder incorporates hierarchical generation, syntactic structure priors, and a tailored transformer architecture. In the domain of optimizer learning, we leverage the symbolic regression tool to overcome scalability and interpretability challenges in Learning to Optimize (L2O) models. By introducing a holistic symbolic representation and analysis framework for L2O, we gain insights into learnable optimizers and explicitly develop optimizers in symbolic form. This approach eliminates the scalability limitations associated with numerical rule representation in L2O models and provides interpretability and comparability among different L2O models. This PhD research has explored and developed methods to integrate domain-specific priors into the learning process, both by incorporating them into neural network architecture and by explicitly leveraging symbolic representations. By incorporating these physical, structural, and symbolic priors, we have improved generalization with less data requirements, and demonstrated the potential for more efficient and generalizable learning and reasoning systems.