Physics-informed machine learning for accelerated modelingand optimization of complex systems

Paris Perdikaris
Department of Mechanical Engineering
University of Pennsylvania
email: pgp@seas.upenn.edu

Supported by:

SLAC AI Seminar
October 2, 2020

sha1_base64="aWD19d2etcI9wT8a+lF0v7FOQf0=">AAACCXicdVDLSgMxFM3UV62vqS7dBItQN8NMtdqCi4Ibd1awrdCOJZNm2tBkZkgyShn6BX6DW127E7d+hUv/xEwfYEUPBA7n3Ms9OV7EqFS2/WlklpZXVtey67mNza3tHTO/25RhLDBp4JCF4tZDkjAakIaiipHbSBDEPUZa3vAi9Vv3REgaBjdqFBGXo35AfYqR0lLXzHc4UgOMWHI1Ljr23elR1yzYVrVatssOnJLSnBxXoGPZExTADPWu+dXphTjmJFCYISnbjh0pN0FCUczIONeJJYkQHqI+aWsaIE6km0yij+GhVnrQD4V+gYIT9edGgriUI+7pyTSo/O2l4l9eO1Z+xU1oEMWKBHh6yI8ZVCFMe4A9KghWbKQJwoLqrBAPkEBY6bYWrnh8rDuZfxz+T5oly7Et5/qkUDuftZMF++AAFIEDzkANXII6aAAMHsATeAYvxqPxarwZ79PRjDHb2QMLMD6+AWNRmgE=</latexit>

sha1_base64="aWD19d2etcI9wT8a+lF0v7FOQf0=">AAACCXicdVDLSgMxFM3UV62vqS7dBItQN8NMtdqCi4Ibd1awrdCOJZNm2tBkZkgyShn6BX6DW127E7d+hUv/xEwfYEUPBA7n3Ms9OV7EqFS2/WlklpZXVtey67mNza3tHTO/25RhLDBp4JCF4tZDkjAakIaiipHbSBDEPUZa3vAi9Vv3REgaBjdqFBGXo35AfYqR0lLXzHc4UgOMWHI1Ljr23elR1yzYVrVatssOnJLSnBxXoGPZExTADPWu+dXphTjmJFCYISnbjh0pN0FCUczIONeJJYkQHqI+aWsaIE6km0yij+GhVnrQD4V+gYIT9edGgriUI+7pyTSo/O2l4l9eO1Z+xU1oEMWKBHh6yI8ZVCFMe4A9KghWbKQJwoLqrBAPkEBY6bYWrnh8rDuZfxz+T5oly7Et5/qkUDuftZMF++AAFIEDzkANXII6aAAMHsATeAYvxqPxarwZ79PRjDHb2QMLMD6+AWNRmgE=</latexit><latexit

sha1_base64="aWD19d2etcI9wT8a+lF0v7FOQf0=">AAACCXicdVDLSgMxFM3UV62vqS7dBItQN8NMtdqCi4Ibd1awrdCOJZNm2tBkZkgyShn6BX6DW127E7d+hUv/xEwfYEUPBA7n3Ms9OV7EqFS2/WlklpZXVtey67mNza3tHTO/25RhLDBp4JCF4tZDkjAakIaiipHbSBDEPUZa3vAi9Vv3REgaBjdqFBGXo35AfYqR0lLXzHc4UgOMWHI1Ljr23elR1yzYVrVatssOnJLSnBxXoGPZExTADPWu+dXphTjmJFCYISnbjh0pN0FCUczIONeJJYkQHqI+aWsaIE6km0yij+GhVnrQD4V+gYIT9edGgriUI+7pyTSo/O2l4l9eO1Z+xU1oEMWKBHh6yI8ZVCFMe4A9KghWbKQJwoLqrBAPkEBY6bYWrnh8rDuZfxz+T5oly7Et5/qkUDuftZMF++AAFIEDzkANXII6aAAMHsATeAYvxqPxarwZ79PRjDHb2QMLMD6+AWNRmgE=</latexit><latexit

sha1_base64="aWD19d2etcI9wT8a+lF0v7FOQf0=">AAACCXicdVDLSgMxFM3UV62vqS7dBItQN8NMtdqCi4Ibd1awrdCOJZNm2tBkZkgyShn6BX6DW127E7d+hUv/xEwfYEUPBA7n3Ms9OV7EqFS2/WlklpZXVtey67mNza3tHTO/25RhLDBp4JCF4tZDkjAakIaiipHbSBDEPUZa3vAi9Vv3REgaBjdqFBGXo35AfYqR0lLXzHc4UgOMWHI1Ljr23elR1yzYVrVatssOnJLSnBxXoGPZExTADPWu+dXphTjmJFCYISnbjh0pN0FCUczIONeJJYkQHqI+aWsaIE6km0yij+GhVnrQD4V+gYIT9edGgriUI+7pyTSo/O2l4l9eO1Z+xU1oEMWKBHh6yI8ZVCFMe4A9KghWbKQJwoLqrBAPkEBY6bYWrnh8rDuZfxz+T5oly7Et5/qkUDuftZMF++AAFIEDzkANXII6aAAMHsATeAYvxqPxarwZ79PRjDHb2QMLMD6+AWNRmgE=</latexit><latexit

O(106 ) lines of code, ~20 PhD theses (1990-2010)
32k supercomputer nodes (IBM BG/P, 128k processors)
Finalist for the Gordon-Bell prize in supercomputing (2011, only US entry)
Simulation of one cardiac cycle took ~24hr
4 MW at $0.10/kWh is $400 an hour or about $3.5 million per year.
<latexit*Simulation credit: Leopold Grinberg (IBM)sha1_base64="jA8euuXZ8LO5FBtVh1sQfCwI8II=">AAACDHicdVDLSgMxFM3UV62v8bFzEyxCXThkqsUWXBTcuLOCrYV2LJk0bUMzD5KMUIf5Bb/Bra7diVv/waV/YqYPsKIHAodz7uWeHDfkTCqEPo3MwuLS8kp2Nbe2vrG5ZW7vNGQQCULrJOCBaLpYUs58WldMcdoMBcWey+mtO7xI/dt7KiQL/Bs1Cqnj4b7PeoxgpaWOudf2sBoQzOOrpGCjYxvdFY86Zh5ZlUoJlWw4IcUZOSlD20Jj5MEUtY751e4GJPKorwjHUrZsFConxkIxwmmSa0eShpgMcZ+2NPWxR6UTj9Mn8FArXdgLhH6+gmP150aMPSlHnqsn06zyt5eKf3mtSPXKTsz8MFLUJ5NDvYhDFcC0CthlghLFR5pgIpjOCskAC0yULmzuiuslupPZx+H/pFG0bGTZ16f56vm0nSzYBwegAGxwBqrgEtRAHRDwAJ7AM3gxHo1X4814n4xmjOnOLpiD8fENvX2aqQ==</latexit>

sha1_base64="jA8euuXZ8LO5FBtVh1sQfCwI8II=">AAACDHicdVDLSgMxFM3UV62v8bFzEyxCXThkqsUWXBTcuLOCrYV2LJk0bUMzD5KMUIf5Bb/Bra7diVv/waV/YqYPsKIHAodz7uWeHDfkTCqEPo3MwuLS8kp2Nbe2vrG5ZW7vNGQQCULrJOCBaLpYUs58WldMcdoMBcWey+mtO7xI/dt7KiQL/Bs1Cqnj4b7PeoxgpaWOudf2sBoQzOOrpGCjYxvdFY86Zh5ZlUoJlWw4IcUZOSlD20Jj5MEUtY751e4GJPKorwjHUrZsFConxkIxwmmSa0eShpgMcZ+2NPWxR6UTj9Mn8FArXdgLhH6+gmP150aMPSlHnqsn06zyt5eKf3mtSPXKTsz8MFLUJ5NDvYhDFcC0CthlghLFR5pgIpjOCskAC0yULmzuiuslupPZx+H/pFG0bGTZ16f56vm0nSzYBwegAGxwBqrgEtRAHRDwAJ7AM3gxHo1X4814n4xmjOnOLpiD8fENvX2aqQ==</latexit><latexit

sha1_base64="jA8euuXZ8LO5FBtVh1sQfCwI8II=">AAACDHicdVDLSgMxFM3UV62v8bFzEyxCXThkqsUWXBTcuLOCrYV2LJk0bUMzD5KMUIf5Bb/Bra7diVv/waV/YqYPsKIHAodz7uWeHDfkTCqEPo3MwuLS8kp2Nbe2vrG5ZW7vNGQQCULrJOCBaLpYUs58WldMcdoMBcWey+mtO7xI/dt7KiQL/Bs1Cqnj4b7PeoxgpaWOudf2sBoQzOOrpGCjYxvdFY86Zh5ZlUoJlWw4IcUZOSlD20Jj5MEUtY751e4GJPKorwjHUrZsFConxkIxwmmSa0eShpgMcZ+2NPWxR6UTj9Mn8FArXdgLhH6+gmP150aMPSlHnqsn06zyt5eKf3mtSPXKTsz8MFLUJ5NDvYhDFcC0CthlghLFR5pgIpjOCskAC0yULmzuiuslupPZx+H/pFG0bGTZ16f56vm0nSzYBwegAGxwBqrgEtRAHRDwAJ7AM3gxHo1X4814n4xmjOnOLpiD8fENvX2aqQ==</latexit><latexit

sha1_base64="jA8euuXZ8LO5FBtVh1sQfCwI8II=">AAACDHicdVDLSgMxFM3UV62v8bFzEyxCXThkqsUWXBTcuLOCrYV2LJk0bUMzD5KMUIf5Bb/Bra7diVv/waV/YqYPsKIHAodz7uWeHDfkTCqEPo3MwuLS8kp2Nbe2vrG5ZW7vNGQQCULrJOCBaLpYUs58WldMcdoMBcWey+mtO7xI/dt7KiQL/Bs1Cqnj4b7PeoxgpaWOudf2sBoQzOOrpGCjYxvdFY86Zh5ZlUoJlWw4IcUZOSlD20Jj5MEUtY751e4GJPKorwjHUrZsFConxkIxwmmSa0eShpgMcZ+2NPWxR6UTj9Mn8FArXdgLhH6+gmP150aMPSlHnqsn06zyt5eKf3mtSPXKTsz8MFLUJ5NDvYhDFcC0CthlghLFR5pgIpjOCskAC0yULmzuiuslupPZx+H/pFG0bGTZ16f56vm0nSzYBwegAGxwBqrgEtRAHRDwAJ7AM3gxHo1X4814n4xmjOnOLpiD8fENvX2aqQ==</latexit><latexit

O(10<latexit*Simulation credit: Leopold Grinberg (IBM)

Impressive computation, but still an approximation to the true underlying physics
Validity of results relies on accurate model calibration
Many sources of uncertainty: geometry, IC/BCs, rheology, material properties, etc.

102 ) parameters to be calibrated using clinical datasha1_base64="b22imFUGlXc+S9gzb08KHnbQhhE=">AAAB/XicdVDLSsNAFJ34rPVVdelmsAiuQlIttuKi4MZlBfuANpTJdNIOnUnCzI1QQvAb3Oranbj1W1z6J05fYEUPXDiccy/33uPHgmtwnE9rZXVtfWMzt5Xf3tnd2y8cHDZ1lCjKGjQSkWr7RDPBQ9YADoK1Y8WI9AVr+aObid96YErzKLyHccw8SQYhDzglYKRW15cpZFe9QtGxq9WyU3bxjJQW5LyCXduZoojmqPcKX91+RBPJQqCCaN1xnRi8lCjgVLAs3000iwkdkQHrGBoSybSXTs/N8KlR+jiIlKkQ8FT9OZESqfVY+qZTEhjq395E/MvrJBBUvJSHcQIspLNFQSIwRHjyO+5zxSiIsSGEKm5uxXRIFKFgElra4svMZLJ4HP9PmiXbdWz37qJYu56nk0PH6ASdIRddohq6RXXUQBSN0BN6Ri/Wo/VqvVnvs9YVaz5zhJZgfXwDqHGWeg==</latexit>

sha1_base64="b22imFUGlXc+S9gzb08KHnbQhhE=">AAAB/XicdVDLSsNAFJ34rPVVdelmsAiuQlIttuKi4MZlBfuANpTJdNIOnUnCzI1QQvAb3Oranbj1W1z6J05fYEUPXDiccy/33uPHgmtwnE9rZXVtfWMzt5Xf3tnd2y8cHDZ1lCjKGjQSkWr7RDPBQ9YADoK1Y8WI9AVr+aObid96YErzKLyHccw8SQYhDzglYKRW15cpZFe9QtGxq9WyU3bxjJQW5LyCXduZoojmqPcKX91+RBPJQqCCaN1xnRi8lCjgVLAs3000iwkdkQHrGBoSybSXTs/N8KlR+jiIlKkQ8FT9OZESqfVY+qZTEhjq395E/MvrJBBUvJSHcQIspLNFQSIwRHjyO+5zxSiIsSGEKm5uxXRIFKFgElra4svMZLJ4HP9PmiXbdWz37qJYu56nk0PH6ASdIRddohq6RXXUQBSN0BN6Ri/Wo/VqvVnvs9YVaz5zhJZgfXwDqHGWeg==</latexit><latexit

sha1_base64="b22imFUGlXc+S9gzb08KHnbQhhE=">AAAB/XicdVDLSsNAFJ34rPVVdelmsAiuQlIttuKi4MZlBfuANpTJdNIOnUnCzI1QQvAb3Oranbj1W1z6J05fYEUPXDiccy/33uPHgmtwnE9rZXVtfWMzt5Xf3tnd2y8cHDZ1lCjKGjQSkWr7RDPBQ9YADoK1Y8WI9AVr+aObid96YErzKLyHccw8SQYhDzglYKRW15cpZFe9QtGxq9WyU3bxjJQW5LyCXduZoojmqPcKX91+RBPJQqCCaN1xnRi8lCjgVLAs3000iwkdkQHrGBoSybSXTs/N8KlR+jiIlKkQ8FT9OZESqfVY+qZTEhjq395E/MvrJBBUvJSHcQIspLNFQSIwRHjyO+5zxSiIsSGEKm5uxXRIFKFgElra4svMZLJ4HP9PmiXbdWz37qJYu56nk0PH6ASdIRddohq6RXXUQBSN0BN6Ri/Wo/VqvVnvs9YVaz5zhJZgfXwDqHGWeg==</latexit><latexit

sha1_base64="b22imFUGlXc+S9gzb08KHnbQhhE=">AAAB/XicdVDLSsNAFJ34rPVVdelmsAiuQlIttuKi4MZlBfuANpTJdNIOnUnCzI1QQvAb3Oranbj1W1z6J05fYEUPXDiccy/33uPHgmtwnE9rZXVtfWMzt5Xf3tnd2y8cHDZ1lCjKGjQSkWr7RDPBQ9YADoK1Y8WI9AVr+aObid96YErzKLyHccw8SQYhDzglYKRW15cpZFe9QtGxq9WyU3bxjJQW5LyCXduZoojmqPcKX91+RBPJQqCCaN1xnRi8lCjgVLAs3000iwkdkQHrGBoSybSXTs/N8KlR+jiIlKkQ8FT9OZESqfVY+qZTEhjq395E/MvrJBBUvJSHcQIspLNFQSIwRHjyO+5zxSiIsSGEKm5uxXRIFKFgElra4svMZLJ4HP9PmiXbdWz37qJYu56nk0PH6ASdIRddohq6RXXUQBSN0BN6Ri/Wo/VqvVnvs9YVaz5zhJZgfXwDqHGWeg==</latexit><latexit

sha1_base64="2QhUEt5Uz9AyScala0sksbVV2cM=">AAAB/XicdVDLSgNBEOyNrxhfUY9eBoPgadmNBhPxEPDiMYJ5QLKE2clsMmRmd5mZFcMS/AavevYmXv0Wj/6JkxcY0YKGoqqb7i4/5kxpx/m0Miura+sb2c3c1vbO7l5+/6ChokQSWicRj2TLx4pyFtK6ZprTViwpFj6nTX94PfGb91QqFoV3ehRTT+B+yAJGsDZSs+OL9GF82c0XHLtSKTklF81IcUHOysi1nSkKMEetm//q9CKSCBpqwrFSbdeJtZdiqRnhdJzrJIrGmAxxn7YNDbGgykun547RiVF6KIikqVCjqfpzIsVCqZHwTafAeqB+exPxL6+d6KDspSyME01DMlsUJBzpCE1+Rz0mKdF8ZAgmkplbERlgiYk2CS1t8cXYZLJ4HP1PGkXbdWz39rxQvZqnk4UjOIZTcOECqnADNagDgSE8wTO8WI/Wq/Vmvc9aM9Z85hCWYH18A67Fln4=</latexit>

sha1_base64="2QhUEt5Uz9AyScala0sksbVV2cM=">AAAB/XicdVDLSgNBEOyNrxhfUY9eBoPgadmNBhPxEPDiMYJ5QLKE2clsMmRmd5mZFcMS/AavevYmXv0Wj/6JkxcY0YKGoqqb7i4/5kxpx/m0Miura+sb2c3c1vbO7l5+/6ChokQSWicRj2TLx4pyFtK6ZprTViwpFj6nTX94PfGb91QqFoV3ehRTT+B+yAJGsDZSs+OL9GF82c0XHLtSKTklF81IcUHOysi1nSkKMEetm//q9CKSCBpqwrFSbdeJtZdiqRnhdJzrJIrGmAxxn7YNDbGgykun547RiVF6KIikqVCjqfpzIsVCqZHwTafAeqB+exPxL6+d6KDspSyME01DMlsUJBzpCE1+Rz0mKdF8ZAgmkplbERlgiYk2CS1t8cXYZLJ4HP1PGkXbdWz39rxQvZqnk4UjOIZTcOECqnADNagDgSE8wTO8WI/Wq/Vmvc9aM9Z85hCWYH18A67Fln4=</latexit><latexit

sha1_base64="2QhUEt5Uz9AyScala0sksbVV2cM=">AAAB/XicdVDLSgNBEOyNrxhfUY9eBoPgadmNBhPxEPDiMYJ5QLKE2clsMmRmd5mZFcMS/AavevYmXv0Wj/6JkxcY0YKGoqqb7i4/5kxpx/m0Miura+sb2c3c1vbO7l5+/6ChokQSWicRj2TLx4pyFtK6ZprTViwpFj6nTX94PfGb91QqFoV3ehRTT+B+yAJGsDZSs+OL9GF82c0XHLtSKTklF81IcUHOysi1nSkKMEetm//q9CKSCBpqwrFSbdeJtZdiqRnhdJzrJIrGmAxxn7YNDbGgykun547RiVF6KIikqVCjqfpzIsVCqZHwTafAeqB+exPxL6+d6KDspSyME01DMlsUJBzpCE1+Rz0mKdF8ZAgmkplbERlgiYk2CS1t8cXYZLJ4HP1PGkXbdWz39rxQvZqnk4UjOIZTcOECqnADNagDgSE8wTO8WI/Wq/Vmvc9aM9Z85hCWYH18A67Fln4=</latexit><latexit

sha1_base64="2QhUEt5Uz9AyScala0sksbVV2cM=">AAAB/XicdVDLSgNBEOyNrxhfUY9eBoPgadmNBhPxEPDiMYJ5QLKE2clsMmRmd5mZFcMS/AavevYmXv0Wj/6JkxcY0YKGoqqb7i4/5kxpx/m0Miura+sb2c3c1vbO7l5+/6ChokQSWicRj2TLx4pyFtK6ZprTViwpFj6nTX94PfGb91QqFoV3ehRTT+B+yAJGsDZSs+OL9GF82c0XHLtSKTklF81IcUHOysi1nSkKMEetm//q9CKSCBpqwrFSbdeJtZdiqRnhdJzrJIrGmAxxn7YNDbGgykun547RiVF6KIikqVCjqfpzIsVCqZHwTafAeqB+exPxL6+d6KDspSyME01DMlsUJBzpCE1+Rz0mKdF8ZAgmkplbERlgiYk2CS1t8cXYZLJ4HP1PGkXbdWz39rxQvZqnk4UjOIZTcOECqnADNagDgSE8wTO8WI/Wq/Vmvc9aM9Z85hCWYH18A67Fln4=</latexit><latexit<latexit

sha1_base64="BPIWXyKtrjM+oofb5wAzCfcRVpc=">AAAB/XicdVDLSgNBEJz1GeMr6tHLYBA8LbvRYCIeAl48RjAPSJYwO5kkQ2Zml5leISzBb/CqZ2/i1W/x6J84eYERLWgoqrrp7gpjwQ143qezsrq2vrGZ2cpu7+zu7ecODusmSjRlNRqJSDdDYpjgitWAg2DNWDMiQ8Ea4fBm4jcemDY8UvcwilkgSV/xHqcErNRow4ABuerk8p5bLhe9oo9npLAg5yXsu94UeTRHtZP7ancjmkimgApiTMv3YghSooFTwcbZdmJYTOiQ9FnLUkUkM0E6PXeMT63Sxb1I21KAp+rPiZRIY0YytJ2SwMD89ibiX14rgV4pSLmKE2CKzhb1EoEhwpPfcZdrRkGMLCFUc3srpgOiCQWb0NKWUI5tJovH8f+kXnB9z/XvLvKV63k6GXSMTtAZ8tElqqBbVEU1RNEQPaFn9OI8Oq/Om/M+a11x5jNHaAnOxzduGpZV</latexit>

sha1_base64="BPIWXyKtrjM+oofb5wAzCfcRVpc=">AAAB/XicdVDLSgNBEJz1GeMr6tHLYBA8LbvRYCIeAl48RjAPSJYwO5kkQ2Zml5leISzBb/CqZ2/i1W/x6J84eYERLWgoqrrp7gpjwQ143qezsrq2vrGZ2cpu7+zu7ecODusmSjRlNRqJSDdDYpjgitWAg2DNWDMiQ8Ea4fBm4jcemDY8UvcwilkgSV/xHqcErNRow4ABuerk8p5bLhe9oo9npLAg5yXsu94UeTRHtZP7ancjmkimgApiTMv3YghSooFTwcbZdmJYTOiQ9FnLUkUkM0E6PXeMT63Sxb1I21KAp+rPiZRIY0YytJ2SwMD89ibiX14rgV4pSLmKE2CKzhb1EoEhwpPfcZdrRkGMLCFUc3srpgOiCQWb0NKWUI5tJovH8f+kXnB9z/XvLvKV63k6GXSMTtAZ8tElqqBbVEU1RNEQPaFn9OI8Oq/Om/M+a11x5jNHaAnOxzduGpZV</latexit><latexit

sha1_base64="BPIWXyKtrjM+oofb5wAzCfcRVpc=">AAAB/XicdVDLSgNBEJz1GeMr6tHLYBA8LbvRYCIeAl48RjAPSJYwO5kkQ2Zml5leISzBb/CqZ2/i1W/x6J84eYERLWgoqrrp7gpjwQ143qezsrq2vrGZ2cpu7+zu7ecODusmSjRlNRqJSDdDYpjgitWAg2DNWDMiQ8Ea4fBm4jcemDY8UvcwilkgSV/xHqcErNRow4ABuerk8p5bLhe9oo9npLAg5yXsu94UeTRHtZP7ancjmkimgApiTMv3YghSooFTwcbZdmJYTOiQ9FnLUkUkM0E6PXeMT63Sxb1I21KAp+rPiZRIY0YytJ2SwMD89ibiX14rgV4pSLmKE2CKzhb1EoEhwpPfcZdrRkGMLCFUc3srpgOiCQWb0NKWUI5tJovH8f+kXnB9z/XvLvKV63k6GXSMTtAZ8tElqqBbVEU1RNEQPaFn9OI8Oq/Om/M+a11x5jNHaAnOxzduGpZV</latexit><latexit

sha1_base64="BPIWXyKtrjM+oofb5wAzCfcRVpc=">AAAB/XicdVDLSgNBEJz1GeMr6tHLYBA8LbvRYCIeAl48RjAPSJYwO5kkQ2Zml5leISzBb/CqZ2/i1W/x6J84eYERLWgoqrrp7gpjwQ143qezsrq2vrGZ2cpu7+zu7ecODusmSjRlNRqJSDdDYpjgitWAg2DNWDMiQ8Ea4fBm4jcemDY8UvcwilkgSV/xHqcErNRow4ABuerk8p5bLhe9oo9npLAg5yXsu94UeTRHtZP7ancjmkimgApiTMv3YghSooFTwcbZdmJYTOiQ9FnLUkUkM0E6PXeMT63Sxb1I21KAp+rPiZRIY0YytJ2SwMD89ibiX14rgV4pSLmKE2CKzhb1EoEhwpPfcZdrRkGMLCFUc3srpgOiCQWb0NKWUI5tJovH8f+kXnB9z/XvLvKV63k6GXSMTtAZ8tElqqBbVEU1RNEQPaFn9OI8Oq/Om/M+a11x5jNHaAnOxzduGpZV</latexit><latexit<latexit✓ : Model parameters
x : Model inputs
t : Target quantities of interest<latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit>

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit<latexit✓⇤ = arg min L(✓), L(✓) := ||m✓ (x)sha1_base64="kuqh+IrLCU1vVQjAFvNwAWtLgCk=">AAACP3icdVDLSgMxFM34tr6qLt0Ei9AuLDNV8YGC4EIXLirYB3RKyaRpG0xmhuSOWIb5JD/Cb3Al6MKVO3HrzkxbwYpeSDiccw/33uOFgmuw7SdrYnJqemZ2bj6zsLi0vJJdXavqIFKUVWggAlX3iGaC+6wCHASrh4oR6QlW827OUr12y5TmgX8N/ZA1Jen6vMMpAUO1sueuJNCjRMSXSd6FHgNSODrBrvHA4MOyFQ95o3syvksKeBunCJIfba1szi4eHu7Zew4egtI32DnATtEeVA6NqtzKvrrtgEaS+UAF0brh2CE0Y6KAU8GSjBtpFhJ6Q7qsYaBPJNPNeHBwgrcM08adQJnnAx6wPx0xkVr3pWc60/P0by0l/9IaEXQOmjH3wwiYT4eDOpHAEOA0PdzmilEQfQMIVdzsimmPKELBZDw2xZOJyeT7cPw/qJaKjl10rnZzp8ejdObQBtpEeeSgfXSKLlAZVRBF9+gRPaMX68F6s96tj2HrhDXyrKOxsj6/AKQXsQg=</latexit>

sha1_base64="kuqh+IrLCU1vVQjAFvNwAWtLgCk=">AAACP3icdVDLSgMxFM34tr6qLt0Ei9AuLDNV8YGC4EIXLirYB3RKyaRpG0xmhuSOWIb5JD/Cb3Al6MKVO3HrzkxbwYpeSDiccw/33uOFgmuw7SdrYnJqemZ2bj6zsLi0vJJdXavqIFKUVWggAlX3iGaC+6wCHASrh4oR6QlW827OUr12y5TmgX8N/ZA1Jen6vMMpAUO1sueuJNCjRMSXSd6FHgNSODrBrvHA4MOyFQ95o3syvksKeBunCJIfba1szi4eHu7Zew4egtI32DnATtEeVA6NqtzKvrrtgEaS+UAF0brh2CE0Y6KAU8GSjBtpFhJ6Q7qsYaBPJNPNeHBwgrcM08adQJnnAx6wPx0xkVr3pWc60/P0by0l/9IaEXQOmjH3wwiYT4eDOpHAEOA0PdzmilEQfQMIVdzsimmPKELBZDw2xZOJyeT7cPw/qJaKjl10rnZzp8ejdObQBtpEeeSgfXSKLlAZVRBF9+gRPaMX68F6s96tj2HrhDXyrKOxsj6/AKQXsQg=</latexit><latexit

sha1_base64="kuqh+IrLCU1vVQjAFvNwAWtLgCk=">AAACP3icdVDLSgMxFM34tr6qLt0Ei9AuLDNV8YGC4EIXLirYB3RKyaRpG0xmhuSOWIb5JD/Cb3Al6MKVO3HrzkxbwYpeSDiccw/33uOFgmuw7SdrYnJqemZ2bj6zsLi0vJJdXavqIFKUVWggAlX3iGaC+6wCHASrh4oR6QlW827OUr12y5TmgX8N/ZA1Jen6vMMpAUO1sueuJNCjRMSXSd6FHgNSODrBrvHA4MOyFQ95o3syvksKeBunCJIfba1szi4eHu7Zew4egtI32DnATtEeVA6NqtzKvrrtgEaS+UAF0brh2CE0Y6KAU8GSjBtpFhJ6Q7qsYaBPJNPNeHBwgrcM08adQJnnAx6wPx0xkVr3pWc60/P0by0l/9IaEXQOmjH3wwiYT4eDOpHAEOA0PdzmilEQfQMIVdzsimmPKELBZDw2xZOJyeT7cPw/qJaKjl10rnZzp8ejdObQBtpEeeSgfXSKLlAZVRBF9+gRPaMX68F6s96tj2HrhDXyrKOxsj6/AKQXsQg=</latexit><latexit

sha1_base64="kuqh+IrLCU1vVQjAFvNwAWtLgCk=">AAACP3icdVDLSgMxFM34tr6qLt0Ei9AuLDNV8YGC4EIXLirYB3RKyaRpG0xmhuSOWIb5JD/Cb3Al6MKVO3HrzkxbwYpeSDiccw/33uOFgmuw7SdrYnJqemZ2bj6zsLi0vJJdXavqIFKUVWggAlX3iGaC+6wCHASrh4oR6QlW827OUr12y5TmgX8N/ZA1Jen6vMMpAUO1sueuJNCjRMSXSd6FHgNSODrBrvHA4MOyFQ95o3syvksKeBunCJIfba1szi4eHu7Zew4egtI32DnATtEeVA6NqtzKvrrtgEaS+UAF0brh2CE0Y6KAU8GSjBtpFhJ6Q7qsYaBPJNPNeHBwgrcM08adQJnnAx6wPx0xkVr3pWc60/P0by0l/9IaEXQOmjH3wwiYT4eDOpHAEOA0PdzmilEQfQMIVdzsimmPKELBZDw2xZOJyeT7cPw/qJaKjl10rnZzp8ejdObQBtpEeeSgfXSKLlAZVRBF9+gRPaMX68F6s96tj2HrhDXyrKOxsj6/AKQXsQg=</latexit><latexit✓<latexit*Simulation credit: Leopold Grinberg (IBM)•t||High dimensional optimization• Expensive, black-box loss function• No gradients available

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit>

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit<latexit✓⇤ = arg min L(✓),

Big data vs small data✓

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit>

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexit

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexit

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexitL(✓) : ||m✓ (x)<latexit*Simulation credit: Leopold Grinberg (IBM)t||Dimensionality vs # of observations

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit>

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit

sha1_base64="Xsi+QvFSvEp3ODdqcg18BMkTeMY=">AAACN3icdVBNSyNBEO3x+2vdUY97aQyCgoQZV1FBUdjLHjwoGA1kYqjpdJLG7p6hu0YIw/wcf8T+hr3qwZN4Wfa6/8BOJoKR9UHB470qqurFqRQWg+DJm5icmp6ZnZtfWFz6svzVX1m9sklmGK+xRCamHoPlUmheQ4GS11PDQcWSX8e3Pwb+9R03ViT6EvspbyroatERDNBJLf8kwh5HuMkjsFjQYxqB6dJICU0jKZRA28rLlsKpgD0GMj8rNktta7vlV4Lq4eFesBfSkuy8ke8HNKwGQ1TICOct/yVqJyxTXCOTYG0jDFJs5mBQMMmLhSizPAV2C13ecFSD4raZDx8t6IZT2rSTGFca6VB9P5GDsravYtc5uNV+9Abi/7xGhp2DZi50miHXrFzUySTFhA5So21hOEPZdwSYEe5WynpggKHLdmxLrAqXydvj9HNytVMNg2p4sVs5PRqlM0e+kXWySUKyT07JT3JOaoSRe/KbPJBH75f37P3x/patE95oZo2Mwfv3Cuhsrak=</latexit><latexit<latexit✓⇤ = arg min L(✓),✓

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit>

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexit

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexit

sha1_base64="HzBSSjcGqBosN3s8TU1F89w3+ss=">AAACPnicdVDLSgMxFM3Ud31VXboJFkEXlpmq2IoLQRAXLirYKnRKyaRpG0xmhuSOWIb5Iz/Cb3AnunHjTty6NNMHtKIXEg7n3MO993ih4Bps+8XKTE3PzM7NL2QXl5ZXVnNr6zUdRIqyKg1EoG49opngPqsCB8FuQ8WI9AS78e7OUv3mninNA/8aeiFrSNLxeZtTAoZq5s5dSaBLiYgvkx0XugzI7jF2jQX6H5bNeEAb2ZPxQ7KL93CKIBlra+bydqFcPrQPHTwAxRHYL2GnYPcrj4ZVaebe3VZAI8l8oIJoXXfsEBoxUcCpYEnWjTQLCb0jHVY30CeS6UbcvzfB24Zp4XagzPMB99lxR0yk1j3pmc70Ov1bS8m/tHoE7VIj5n4YAfPpYFA7EhgCnIaHW1wxCqJnAKGKm10x7RJFKJiIJ6Z4MjGZjA7H/4NaseDYBefqIH96MkxnHm2iLbSDHHSETtEFqqAqougRPaNX9GY9WR/Wp/U1aM1YQ88Gmijr+wcLE7DB</latexit><latexitL(✓) : ||m✓ (x)<latexit*Simulation credit: Leopold Grinberg (IBM)t||

Can we solve this problem using surrogate models that are cheaper to compute?

sha1_base64="R2oN10tWoXcKohwnOYpvEhorS1k=">AAACDHicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoAvBghsXLirYBzShTKaTdujkwcyNUEN+wW9wq2t34tZ/cOmfOGmzsK0HLhzOuZd7OF4suALL+jaWlldW19ZLG+XNre2dXXNvv6WiRFLWpJGIZMcjigkesiZwEKwTS0YCT7C2N7rJ/fYjk4pH4QOMY+YGZBByn1MCWuqZh05AYEiJSO+yqgNDBuT0umdWrJo1AV4kdkEqqECjZ/44/YgmAQuBCqJU17ZicFMigVPBsrKTKBYTOiID1tU0JAFTbjpJn+ETrfSxH0k9IeCJ+vciJYFS48DTm3lWNe/l4n9eNwH/0k15GCfAQjp95CcCQ4TzKnCfS0ZBjDUhVHKdFdMhkYSCLmzmixdkuhN7voFF0jqr2VbNvj+v1K+KdkroCB2jKrLRBaqjW9RATUTRE3pBr+jNeDbejQ/jc7q6ZBQ3B2gGxtcvGdubfw==</latexit>

sha1_base64="R2oN10tWoXcKohwnOYpvEhorS1k=">AAACDHicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoAvBghsXLirYBzShTKaTdujkwcyNUEN+wW9wq2t34tZ/cOmfOGmzsK0HLhzOuZd7OF4suALL+jaWlldW19ZLG+XNre2dXXNvv6WiRFLWpJGIZMcjigkesiZwEKwTS0YCT7C2N7rJ/fYjk4pH4QOMY+YGZBByn1MCWuqZh05AYEiJSO+yqgNDBuT0umdWrJo1AV4kdkEqqECjZ/44/YgmAQuBCqJU17ZicFMigVPBsrKTKBYTOiID1tU0JAFTbjpJn+ETrfSxH0k9IeCJ+vciJYFS48DTm3lWNe/l4n9eNwH/0k15GCfAQjp95CcCQ4TzKnCfS0ZBjDUhVHKdFdMhkYSCLmzmixdkuhN7voFF0jqr2VbNvj+v1K+KdkroCB2jKrLRBaqjW9RATUTRE3pBr+jNeDbejQ/jc7q6ZBQ3B2gGxtcvGdubfw==</latexit><latexit

sha1_base64="R2oN10tWoXcKohwnOYpvEhorS1k=">AAACDHicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoAvBghsXLirYBzShTKaTdujkwcyNUEN+wW9wq2t34tZ/cOmfOGmzsK0HLhzOuZd7OF4suALL+jaWlldW19ZLG+XNre2dXXNvv6WiRFLWpJGIZMcjigkesiZwEKwTS0YCT7C2N7rJ/fYjk4pH4QOMY+YGZBByn1MCWuqZh05AYEiJSO+yqgNDBuT0umdWrJo1AV4kdkEqqECjZ/44/YgmAQuBCqJU17ZicFMigVPBsrKTKBYTOiID1tU0JAFTbjpJn+ETrfSxH0k9IeCJ+vciJYFS48DTm3lWNe/l4n9eNwH/0k15GCfAQjp95CcCQ4TzKnCfS0ZBjDUhVHKdFdMhkYSCLmzmixdkuhN7voFF0jqr2VbNvj+v1K+KdkroCB2jKrLRBaqjW9RATUTRE3pBr+jNeDbejQ/jc7q6ZBQ3B2gGxtcvGdubfw==</latexit><latexit

sha1_base64="R2oN10tWoXcKohwnOYpvEhorS1k=">AAACDHicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoAvBghsXLirYBzShTKaTdujkwcyNUEN+wW9wq2t34tZ/cOmfOGmzsK0HLhzOuZd7OF4suALL+jaWlldW19ZLG+XNre2dXXNvv6WiRFLWpJGIZMcjigkesiZwEKwTS0YCT7C2N7rJ/fYjk4pH4QOMY+YGZBByn1MCWuqZh05AYEiJSO+yqgNDBuT0umdWrJo1AV4kdkEqqECjZ/44/YgmAQuBCqJU17ZicFMigVPBsrKTKBYTOiID1tU0JAFTbjpJn+ETrfSxH0k9IeCJ+vciJYFS48DTm3lWNe/l4n9eNwH/0k15GCfAQjp95CcCQ4TzKnCfS0ZBjDUhVHKdFdMhkYSCLmzmixdkuhN7voFF0jqr2VbNvj+v1K+KdkroCB2jKrLRBaqjW9RATUTRE3pBr+jNeDbejQ/jc7q6ZBQ3B2gGxtcvGdubfw==</latexit><latexit<latexit

sha1_base64="uc8a5SpwRKXDwohq4e4Trcz8q3o=">AAACZXicbVHLTuNAEJx4eYQsbxAXDlhESHuKbASCYwQXuIFEAlIcofa4k8xmZmzNtIHIyj/sFf6ML9jf2MnjsA60NFKpqltdXRNnUlgKgs+K92NpeWW1ulb7ub6xubW9s9u2aW44tngqU/MUg0UpNLZIkMSnzCCoWOJjPLye6I8vaKxI9QONMuwq6GvRExzIUe2IBkjwvF0PGsG0/K8gnIM6m9fd807lNkpSnivUxCVY2wmDjLoFGBJc4rgW5RYz4EPoY8dBDQptt5jaHfsnjkn8Xmrc0+RP2f8nClDWjlTsOhXQwC5qE/I7rZNT77JbCJ3lhJrPFvVy6VPqT273E2GQkxw5ANwI59XnAzDAySVU2hKr0g3F28x6rURmYuLfNWp85alSoJMi4grMcFxEidD94jwcL8hvJfncyS78cDHqr6B92giDRnh/Vm9ezb+hyg7ZMfvFQnbBmuyG3bEW4+w3+8Pe2Uflr7fh7XsHs1avMp/ZY6Xyjv4BjQi8BQ==</latexit>

sha1_base64="uc8a5SpwRKXDwohq4e4Trcz8q3o=">AAACZXicbVHLTuNAEJx4eYQsbxAXDlhESHuKbASCYwQXuIFEAlIcofa4k8xmZmzNtIHIyj/sFf6ML9jf2MnjsA60NFKpqltdXRNnUlgKgs+K92NpeWW1ulb7ub6xubW9s9u2aW44tngqU/MUg0UpNLZIkMSnzCCoWOJjPLye6I8vaKxI9QONMuwq6GvRExzIUe2IBkjwvF0PGsG0/K8gnIM6m9fd807lNkpSnivUxCVY2wmDjLoFGBJc4rgW5RYz4EPoY8dBDQptt5jaHfsnjkn8Xmrc0+RP2f8nClDWjlTsOhXQwC5qE/I7rZNT77JbCJ3lhJrPFvVy6VPqT273E2GQkxw5ANwI59XnAzDAySVU2hKr0g3F28x6rURmYuLfNWp85alSoJMi4grMcFxEidD94jwcL8hvJfncyS78cDHqr6B92giDRnh/Vm9ezb+hyg7ZMfvFQnbBmuyG3bEW4+w3+8Pe2Uflr7fh7XsHs1avMp/ZY6Xyjv4BjQi8BQ==</latexit><latexit

sha1_base64="uc8a5SpwRKXDwohq4e4Trcz8q3o=">AAACZXicbVHLTuNAEJx4eYQsbxAXDlhESHuKbASCYwQXuIFEAlIcofa4k8xmZmzNtIHIyj/sFf6ML9jf2MnjsA60NFKpqltdXRNnUlgKgs+K92NpeWW1ulb7ub6xubW9s9u2aW44tngqU/MUg0UpNLZIkMSnzCCoWOJjPLye6I8vaKxI9QONMuwq6GvRExzIUe2IBkjwvF0PGsG0/K8gnIM6m9fd807lNkpSnivUxCVY2wmDjLoFGBJc4rgW5RYz4EPoY8dBDQptt5jaHfsnjkn8Xmrc0+RP2f8nClDWjlTsOhXQwC5qE/I7rZNT77JbCJ3lhJrPFvVy6VPqT273E2GQkxw5ANwI59XnAzDAySVU2hKr0g3F28x6rURmYuLfNWp85alSoJMi4grMcFxEidD94jwcL8hvJfncyS78cDHqr6B92giDRnh/Vm9ezb+hyg7ZMfvFQnbBmuyG3bEW4+w3+8Pe2Uflr7fh7XsHs1avMp/ZY6Xyjv4BjQi8BQ==</latexit><latexit

sha1_base64="uc8a5SpwRKXDwohq4e4Trcz8q3o=">AAACZXicbVHLTuNAEJx4eYQsbxAXDlhESHuKbASCYwQXuIFEAlIcofa4k8xmZmzNtIHIyj/sFf6ML9jf2MnjsA60NFKpqltdXRNnUlgKgs+K92NpeWW1ulb7ub6xubW9s9u2aW44tngqU/MUg0UpNLZIkMSnzCCoWOJjPLye6I8vaKxI9QONMuwq6GvRExzIUe2IBkjwvF0PGsG0/K8gnIM6m9fd807lNkpSnivUxCVY2wmDjLoFGBJc4rgW5RYz4EPoY8dBDQptt5jaHfsnjkn8Xmrc0+RP2f8nClDWjlTsOhXQwC5qE/I7rZNT77JbCJ3lhJrPFvVy6VPqT273E2GQkxw5ANwI59XnAzDAySVU2hKr0g3F28x6rURmYuLfNWp85alSoJMi4grMcFxEidD94jwcL8hvJfncyS78cDHqr6B92giDRnh/Vm9ezb+hyg7ZMfvFQnbBmuyG3bEW4+w3+8Pe2Uflr7fh7XsHs1avMp/ZY6Xyjv4BjQi8BQ==</latexit><latexit

latexit<sha1_base64="j2QAVhFPV/vezrD7zkL/qWl6MyI=">AAACdHicbVFNT9tAEN2Ylo+0pQGOcLAaIdFLZKMiekTl0ko9gEQAKY6i8XqSrLK7tnbHLdHKv4Ur/KT+kZ67TnLAoSOt9PTejObN27SQwlIU/WkFG2/ebm5t77Tfvf+w+7Gzt39r89Jw7PNc5uY+BYtSaOyTIIn3hUFQqcS7dHZZ63e/0FiR6xuaFzhUMNFiLDiQp0adg0QBTTlI97M6SWiKBJ9HnW7UixYVvgbxCnTZqq5Ge60fSZbzUqEmLsHaQRwVNHRgSHCJVTspLRbAZzDBgYcaFNqhW7ivwmPPZOE4N/5pChfsywkHytq5Sn1n7dWuazX5P21Q0vjr0AldlISaLxeNSxlSHtZRhJkwyEnOPQBuhPca8ikY4OQDa2xJVeMG97C03m6Qhaj9+0aNv3muFOjMJVyBmVUuyYSeuLO4WpMfGvKZl3348XrUr8HtaS+OevH1l+7Ft9U3bLND9omdsJidswv2nV2xPuNszh7ZE3tu/Q2Ogm5wvGwNWquZA9aooPcPu3fBgw==</latexit>

sha1_base64="j2QAVhFPV/vezrD7zkL/qWl6MyI=">AAACdHicbVFNT9tAEN2Ylo+0pQGOcLAaIdFLZKMiekTl0ko9gEQAKY6i8XqSrLK7tnbHLdHKv4Ur/KT+kZ67TnLAoSOt9PTejObN27SQwlIU/WkFG2/ebm5t77Tfvf+w+7Gzt39r89Jw7PNc5uY+BYtSaOyTIIn3hUFQqcS7dHZZ63e/0FiR6xuaFzhUMNFiLDiQp0adg0QBTTlI97M6SWiKBJ9HnW7UixYVvgbxCnTZqq5Ge60fSZbzUqEmLsHaQRwVNHRgSHCJVTspLRbAZzDBgYcaFNqhW7ivwmPPZOE4N/5pChfsywkHytq5Sn1n7dWuazX5P21Q0vjr0AldlISaLxeNSxlSHtZRhJkwyEnOPQBuhPca8ikY4OQDa2xJVeMG97C03m6Qhaj9+0aNv3muFOjMJVyBmVUuyYSeuLO4WpMfGvKZl3348XrUr8HtaS+OevH1l+7Ft9U3bLND9omdsJidswv2nV2xPuNszh7ZE3tu/Q2Ogm5wvGwNWquZA9aooPcPu3fBgw==</latexit><latexit

sha1_base64="j2QAVhFPV/vezrD7zkL/qWl6MyI=">AAACdHicbVFNT9tAEN2Ylo+0pQGOcLAaIdFLZKMiekTl0ko9gEQAKY6i8XqSrLK7tnbHLdHKv4Ur/KT+kZ67TnLAoSOt9PTejObN27SQwlIU/WkFG2/ebm5t77Tfvf+w+7Gzt39r89Jw7PNc5uY+BYtSaOyTIIn3hUFQqcS7dHZZ63e/0FiR6xuaFzhUMNFiLDiQp0adg0QBTTlI97M6SWiKBJ9HnW7UixYVvgbxCnTZqq5Ge60fSZbzUqEmLsHaQRwVNHRgSHCJVTspLRbAZzDBgYcaFNqhW7ivwmPPZOE4N/5pChfsywkHytq5Sn1n7dWuazX5P21Q0vjr0AldlISaLxeNSxlSHtZRhJkwyEnOPQBuhPca8ikY4OQDa2xJVeMG97C03m6Qhaj9+0aNv3muFOjMJVyBmVUuyYSeuLO4WpMfGvKZl3348XrUr8HtaS+OevH1l+7Ft9U3bLND9omdsJidswv2nV2xPuNszh7ZE3tu/Q2Ogm5wvGwNWquZA9aooPcPu3fBgw==</latexit><latexit

sha1_base64="j2QAVhFPV/vezrD7zkL/qWl6MyI=">AAACdHicbVFNT9tAEN2Ylo+0pQGOcLAaIdFLZKMiekTl0ko9gEQAKY6i8XqSrLK7tnbHLdHKv4Ur/KT+kZ67TnLAoSOt9PTejObN27SQwlIU/WkFG2/ebm5t77Tfvf+w+7Gzt39r89Jw7PNc5uY+BYtSaOyTIIn3hUFQqcS7dHZZ63e/0FiR6xuaFzhUMNFiLDiQp0adg0QBTTlI97M6SWiKBJ9HnW7UixYVvgbxCnTZqq5Ge60fSZbzUqEmLsHaQRwVNHRgSHCJVTspLRbAZzDBgYcaFNqhW7ivwmPPZOE4N/5pChfsywkHytq5Sn1n7dWuazX5P21Q0vjr0AldlISaLxeNSxlSHtZRhJkwyEnOPQBuhPca8ikY4OQDa2xJVeMG97C03m6Qhaj9+0aNv3muFOjMJVyBmVUuyYSeuLO4WpMfGvKZl3348XrUr8HtaS+OevH1l+7Ft9U3bLND9omdsJidswv2nV2xPuNszh7ZE3tu/Q2Ogm5wvGwNWquZA9aooPcPu3fBgw==</latexit><latexitL(✓)<latexit

Optimization of expensive black-box functions
Situation

We have a few function evaluations

✓

L(✓)?
Where is
of of
Where
is thetheminimum
minimumf?
Where should the take the next evaluation?

Where should we take our next evaluation?

How can we distill useful information from low-fidelity
models to accelerate this process?

Data-driven modeling with Gaussian processes

tart with a multivariate Gaussian:faces
p(f , f , · · · , f , f
|
{z
} |

,

1

2

s

s+1 , fs+2 , · · · , fN ) ⇠ N (µ, K).

{z

Priorsfover
functions: fBA}0f ⇠ GP(µ(x),K(x,x ; ✓))µAµ=µB

KAA KABand K =KBA KBBcovariance function

nalisation property:constant

expression
2
0

Marginalization:
linearPD

Covariance functions
S fromNDa GP prior
Samples
p

2
0
x
x
d
d
d=1 d
p(fA , fB ) ⇠ N (µ, K). Then:
Neural02 pZpolynomial(x · x + 0 )networkspp2
rp(fA , fexponential,
K)p(fA ) = squaredB )dfB = N (µexp(
AA
A
Infinite
2)2`fB
Duality⇣ p ⌘⌫⇣ p limit
⌘
p
p
2⌫
2⌫
1
Matérn
r
K
⌫
1
⌫
2
(⌫)
`
` r
p
p
….i.e. y = f(x)r+ ε, f~GP(μ,Σ)
exponential
exp( ` )ocesesfully non-parametric!
pprGaussian
Kernel-exponential
exp(`)
k et al., 2015].
processes p
machinesp2r
Posterior is also
Gaussian:rational
quadratic(1 + 2↵`2 ) ↵Bayesianp functions>
0
Covariance
1 p
2x̃ ⌃x̃
inferencesinp(fA , fB ) ⇠ Nneural(µ, K).networkThen:(1+2x̃> ⌃x̃)(1+2x̃0> ⌃x̃0 )Posterior is also Gaussian!
tions
Processes
ate theinear
dictions

1
p(fA |fB4.2)=
N (µA +
µB ), KAAAB KBB (fB
Examplesof K
Covariance
Functionscovariance function

85

S ND
pcovariances
Table 4.1: Summary of several commonly-used
covariance functions. The
20
1 constantν=1/2for machine
Rasmussen,are
C.E. andWilliamseither
C. Gaussian
learning
or as a function of r = |x x0 |. Two
writtenas aprocesses
functionof0 x andx ,(2006)
ν=2expression

1
KAB KBBKBA )Multi-fidelity modelingMulti-fidelity modelingMulti-fidelity modelingMulti-fidelity modelingMulti-fidelity modelingMulti-fidelity observations:yL = fL (xL ) + ✏L

yH = fH (xH ) + ✏H
Probabilistic model:fH (x) = ⇢fL (x) + (x)fL (x) ⇠ GP(0, kL (x, x0 ; ✓L ))(x) ⇠ GP(0, kH (x, x0 ; ✓H ))✏L ⇠ N (0, ✏2L I)✏H ⇠ N (0,

2
✏H I)

Training:✓yL⇢kL (xL , x0H ; ✓L )0kL (xL , x0L ; ✓L ) + ✏2L I
y=
⇠N
,
yH
⇢2 kL (xH , x0H ; ✓L ) + kH (xH , x0H ; ✓H ) + ✏2H I⇢kL (xH , x0L ; ✓L )
0

xL
1 T
1
NL + NH2
2
1
X=
log 2⇡
log p(y|X, ✓L , ✓H , ⇢, ✏L , ✏H ) = log |K| + y K yxH222Prediction:p(f (x⇤ )|y, X, x⇤ ) ⇠ N (f (x⇤ )|µ(x⇤ ),2◆(x⇤ ))µ(x⇤ ) = k(x⇤ , X)K 1 y(x⇤ ) = k(x⇤ , x⇤ )k(x⇤ , X)K 1 k(X, x⇤ )M.C Kennedy, and A. O'Hagan. Predicting the output from a complex computer code when fast approximations are available, 2000.
Demo code: https://github.com/PredictiveIntelligenceLab/GPTutorial

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

Bayesian optimization

Sample at the locations that minimize the lower super-quintile risk confidence bound of the posterior:xn+1 = arg min µ(x)
x2Rd

1

(
1

(↵))
(x)
↵

021304-8

Perdikaris, Grinberg, and Karniadakis

Phys. Fluids 28

Example application: Calibration of blood flow simulations

Goal: Calibrate a large-scale flow solver to match observed clinical data.

Outflow boundary
conditions?

Rheology/
constitutive laws?

0.4
0.3
0.2
0.1
0

Elastic
moduli?

Geometry?

-0.1
10×10 11Models (e.g. 3D vs 1D,
continuum vs atomistic, etc.)

5

4

6

8
×10 10Decreased the relative error to 1e-3 after 3
iterations of BO, mainly sampling the lowest
fidelity (cheapest) solver.

FIG.
5. MaN-MeN
coupling: (a) Flow dynamics in the meso-vascular regime are modeled using nonline
Multi-fidelity
approach:modelsin fractal arterial
trees attachedto eachof theterminal
the parent, patient-specific 3D d1.) 3D Navier-Stokes(spectral/hp
elements,
rigid
artery)- highvessels
fidelityofO(hrs)numberof arteries
in theresulting
MaN-MeN
as a fidelity
functionof the cut-o↵ radius r cut of the2.) Non-linear
1D-FSI(DG,
compliantartery) -topology
intermediateO(mins)Exchanged
quantities
at the MaN-MeN
interfaceinaccurate
and asynchronous
communication( t33.) Linearized
1D-FSI solver
around a (possibly)
referencetime-steppingstate - low fidelityO(s)n : flow-rate at the 3D outlet at tiof the 3D-FSI solver, t 1D : time step size of the 1D-FSI solver, Q 3D

P. Perdikaris, and G.E Karniadakis. "Model inversion via multi-fidelity Bayesian optimization." J. R. Soc. Interface (2016)

duExample application:
Assessing the risk of drug-induced
cardiac arrhythmias

8;PU[LY]HSSV^MPKLSP[`+,*.
SLM[]LU[YPJSLYPNO[]LU[YPJSLNL = 400

SV^ÄKLSP[`+JHISLTVKLSLUKVTPK^HSS

100ms

baseline

100ms

ranolazine

Q S

QT interval

165ms
R
T
Q S

240ms

250ms

QT interval

300ms
R T
Q S

QT interval

345ms
R
T
Q S

460ms

505ms

QT interval

I

I

I

I

I

I

I *H3 I 5H3 I [V I 2Z I 2 I 5HW I 2Y\UJLY[HPU[`X\HU[PMPJH[PVU

8;PU[LY]HSLWPJHYKP\T

160ms
R T

LSLTLU[HY`LMMLJ[NH = 45I

6 8;PU[LY]HS

T\S[PMPKLSP[`.H\ZZPHUWYVJLZZYLNYLZZPVU

KY\NPUK\JLKPVUJOHUULSISVJRZLUZP[P]P[`HUHS`ZPZ

OPNOMPKLSP[`+,*.

OPNOÄKLSP[`+OLHY[TVKLSKY\NJVUJLU[YH[PVU

1050msR T
Q S

QT interval

1050msR
T
Q S

2050msQ

2050msQT interval

Q3050ms
R T

4050ms-100 potential q +70

3050msRT

4050ms-100 potential q +70

We demonstrated that drugs that block a specificpotassium current most drastically prolong the
QT interval and are therefore associated with thehighest risk of torsades de pointes.

Costabal, F. S., Matsuno, K.,Yao, J., Perdikaris, P., & Kuhl, E. (2019). Machine learning in drug development: Characterizing the effect of 30 drugs
on the QT interval using Gaussian process regression, sensitivity analysis, and uncertainty quantification. Computer Methods in Applied
Mechanics and Engineering, 348, 313-333.

Example application:
Shape optimization ofsuper-cavitating hydrofoils

min R0.9

x2X

✓

CD
(x; ⇠)
CL◆

s.t 0.23  R0.5 (CL (x; ⇠))  0.26

R0.5 (Ix (x; ⇠))R0.5 (Tp (x; ⇠))8.1 · 10 60.00132

35 design variables
B-MF: 735 LF + 120 HF runs
GA-LF: ~50,000 LF runs

Bonfiglio, L., Perdikaris, P., Brizzolara, S., & Karniadakis, G. E. (2018). Multi-fidelity optimization of super-cavitating hydrofoils. Computer Methods
in Applied Mechanics and Engineering.

sha1_base64="xt0dI6m/ziP1aDa4mpgz6FGMVoI=">AAADBXicbVFNbxMxEPUuXyV8pXDkYhEhNUJEu4i2UIRUCQ5wKxJpK8VRNOv1JlZsr2V7IZHlM3+EKzfEld/Bv8H5KJCUkSw9vzczfp4ptODWZdmvJL1y9dr1Gzs3W7du37l7r717/9TWjaGsT2tRm/MCLBNcsb7jTrBzbRjIQrCzYvpmoZ99YsbyWn10c82GEsaKV5yCi9So/ZUU0s/CyKsnecCvMQEzJpIrIrjkzo78Sg8EhJ7A3ur2ikhwEwrCvw1dfBSrZLOWuvgpJlPQGojlY3lR0fVk6dUbVgZMKgPU6z/NL5LCkpNNJP427IYwaneyXrYMfBnka9BB6zgZ7SbvSVnTRjLlqABrB3mm3dCDcZwKFlqksUwDncKYDSJUIJkd+qXFgB9HpsRVbeJRDi/Zfys8SGvnsoiZiznYbW1B/k8bNK56MfRc6cYxRVcPVY3ArsaL3eCSG0admEcA1PDoFdMJxFm5uMGNVwq58Qc/W1lvbZCaL/zHRMU+01pKUGVcgwQzDZ6UXI39fh625NmGvB/lOPx8e9SXwemzXp718g/PO8cH6zXsoIfoEdpDOTpEx+gdOkF9RJNWkiUvk6P0S/ot/Z7+WKWmybrmAdqI9Odvb235vw==</latexit>

sha1_base64="xt0dI6m/ziP1aDa4mpgz6FGMVoI=">AAADBXicbVFNbxMxEPUuXyV8pXDkYhEhNUJEu4i2UIRUCQ5wKxJpK8VRNOv1JlZsr2V7IZHlM3+EKzfEld/Bv8H5KJCUkSw9vzczfp4ptODWZdmvJL1y9dr1Gzs3W7du37l7r717/9TWjaGsT2tRm/MCLBNcsb7jTrBzbRjIQrCzYvpmoZ99YsbyWn10c82GEsaKV5yCi9So/ZUU0s/CyKsnecCvMQEzJpIrIrjkzo78Sg8EhJ7A3ur2ikhwEwrCvw1dfBSrZLOWuvgpJlPQGojlY3lR0fVk6dUbVgZMKgPU6z/NL5LCkpNNJP427IYwaneyXrYMfBnka9BB6zgZ7SbvSVnTRjLlqABrB3mm3dCDcZwKFlqksUwDncKYDSJUIJkd+qXFgB9HpsRVbeJRDi/Zfys8SGvnsoiZiznYbW1B/k8bNK56MfRc6cYxRVcPVY3ArsaL3eCSG0admEcA1PDoFdMJxFm5uMGNVwq58Qc/W1lvbZCaL/zHRMU+01pKUGVcgwQzDZ6UXI39fh625NmGvB/lOPx8e9SXwemzXp718g/PO8cH6zXsoIfoEdpDOTpEx+gdOkF9RJNWkiUvk6P0S/ot/Z7+WKWmybrmAdqI9Odvb235vw==</latexit><latexit

sha1_base64="xt0dI6m/ziP1aDa4mpgz6FGMVoI=">AAADBXicbVFNbxMxEPUuXyV8pXDkYhEhNUJEu4i2UIRUCQ5wKxJpK8VRNOv1JlZsr2V7IZHlM3+EKzfEld/Bv8H5KJCUkSw9vzczfp4ptODWZdmvJL1y9dr1Gzs3W7du37l7r717/9TWjaGsT2tRm/MCLBNcsb7jTrBzbRjIQrCzYvpmoZ99YsbyWn10c82GEsaKV5yCi9So/ZUU0s/CyKsnecCvMQEzJpIrIrjkzo78Sg8EhJ7A3ur2ikhwEwrCvw1dfBSrZLOWuvgpJlPQGojlY3lR0fVk6dUbVgZMKgPU6z/NL5LCkpNNJP427IYwaneyXrYMfBnka9BB6zgZ7SbvSVnTRjLlqABrB3mm3dCDcZwKFlqksUwDncKYDSJUIJkd+qXFgB9HpsRVbeJRDi/Zfys8SGvnsoiZiznYbW1B/k8bNK56MfRc6cYxRVcPVY3ArsaL3eCSG0admEcA1PDoFdMJxFm5uMGNVwq58Qc/W1lvbZCaL/zHRMU+01pKUGVcgwQzDZ6UXI39fh625NmGvB/lOPx8e9SXwemzXp718g/PO8cH6zXsoIfoEdpDOTpEx+gdOkF9RJNWkiUvk6P0S/ot/Z7+WKWmybrmAdqI9Odvb235vw==</latexit><latexit

sha1_base64="xt0dI6m/ziP1aDa4mpgz6FGMVoI=">AAADBXicbVFNbxMxEPUuXyV8pXDkYhEhNUJEu4i2UIRUCQ5wKxJpK8VRNOv1JlZsr2V7IZHlM3+EKzfEld/Bv8H5KJCUkSw9vzczfp4ptODWZdmvJL1y9dr1Gzs3W7du37l7r717/9TWjaGsT2tRm/MCLBNcsb7jTrBzbRjIQrCzYvpmoZ99YsbyWn10c82GEsaKV5yCi9So/ZUU0s/CyKsnecCvMQEzJpIrIrjkzo78Sg8EhJ7A3ur2ikhwEwrCvw1dfBSrZLOWuvgpJlPQGojlY3lR0fVk6dUbVgZMKgPU6z/NL5LCkpNNJP427IYwaneyXrYMfBnka9BB6zgZ7SbvSVnTRjLlqABrB3mm3dCDcZwKFlqksUwDncKYDSJUIJkd+qXFgB9HpsRVbeJRDi/Zfys8SGvnsoiZiznYbW1B/k8bNK56MfRc6cYxRVcPVY3ArsaL3eCSG0admEcA1PDoFdMJxFm5uMGNVwq58Qc/W1lvbZCaL/zHRMU+01pKUGVcgwQzDZ6UXI39fh625NmGvB/lOPx8e9SXwemzXp718g/PO8cH6zXsoIfoEdpDOTpEx+gdOkF9RJNWkiUvk6P0S/ot/Z7+WKWmybrmAdqI9Odvb235vw==</latexit><latexit<latexit

Oakley–O’Hagan function [36](3.9a)
f (x) = 5 + x1 + x2 + 2 cos(x1 ) + 2 sin(x2 )with x 2 [ 4, 4]2 and px (x) = N (0, I), and the 2-D Michalewicz function

Likelihood-weighted Bayesian optimization(3.9b)
f (x) =

with x 2 [0, ⇡]2 and p
x (x) = N (0+⇡/2, 0.1I). For these functions, Figure 3.1 makes it visually

Key idea: Account for the importance of the output relative to the input.

sin(x1 ) sin20 (x21 /⇡)We summarize our algorithm for computation of likelihood-weighted acquisition functions(f) 10-D Michalewicz functionin Algorithm 3.1. We note that by virtue of (3.7), the GMM step in Algorithm 3.1 is optional
Blanchard, A., & Sapsis,T. (2020). Bayesian Optimization with Output-Weighted Importancefor US-LW. However, it is useful to compare the performance of US-LW with andwithout
theFigure2: Performanceof EI, PI, IVR(-BO), IVR-LW(BO) and LCB(-LW) for si

Sampling. arXiv preprint arXiv:2004.10599.
sin(x2 ) sin20 (2x22 /⇡)clear that the likelihood ratio gives more emphasis to the regions where
the magnitude of thepx (x)
objectivefunctionis unusuallylargeD)and :=
promotes
xn+1= arg
min ↵(x;µ(x) points
 that(x)have a large impact on thexpµ (µ(x))
output despite their probabilityof occurrence not being the largest.
Figure 3.1 also showsthat w(x) can be approximated satisfactorily with a small number of Gaussian mixtures, a
key prerequisitee.g.
for preserving
algorithm efficiency.
likelihood-weighted
LCB acquisition function(a) Oakley–O’Hagan function

Figure 3.1. From left to right: Contour plots of the objective function f (x), the input pdf px (x), thelikelihood ratio w(x) with f in place of µ in (3.1), and the GMM approximation of the likelihood ratio wGMM (x)with two Gaussian mixtures.

• Currently limited to scalar-output GP priors and kernels withtest functions.

(d) 2-D Michalewicz function(d) 2-D Michalewicz function(e) 6-D Hartmann function• The likelihood ratio guides the search algorithm towards

regions of the input space where the objective function
assumes abnormally large (or small) values.

• Largest gains achieved when the objective function assumes

rare and extreme (i.e., abnormally large or small) output
values, i.e. the conditional pdf of the output p is heavy- tailed.

(b) Michalewicz function
f|x(e) 6-D Hartmann function

analytical gradients (RBF).

sha1_base64="Z1WdGrmgzCROJfEkOnxPhT4/F6M=">AAACZ3icbVFdS8MwFE3r15wfmwoi+BIdgk+jFb8eRV/0SQWng3VImqYzLElLkupG6J/wVf+YP8F/Ybr1wU4vBA7n3Ms99yRMGVXa874cd25+YXGptlxfWV1bbzQ3Nh9VkklMOjhhieyGSBFGBeloqhnpppIgHjLyFA6vCv3plUhFE/GgxynpczQQNKYYaUt1TRDG8PI2f262vLY3KfgX+CVogbLunjecmyBKcMaJ0JghpXq+l+q+QVJTzEheDzJFUoSHaEB6FgrEieqbieEcHlgmgnEi7RMaTtjfEwZxpcY8tJ0c6Rc1qxXkf1ov0/F531CRZpoIPF0UZwzqBBbXw4hKgjUbW4CwpNYrxC9IIqxtRpUtIa/cYEZT6/UKmdLCv20U5A0nnCMRmQBzJIe5CSIqBubEz2fkUUU+sbIN35+N+i94PGr7Xtu/P25dnJbfUAO7YB8cAh+cgQtwDe5AB2DAwDv4AJ/Ot9twt92daavrlDNboFLu3g+NBbxo</latexit>

sha1_base64="Z1WdGrmgzCROJfEkOnxPhT4/F6M=">AAACZ3icbVFdS8MwFE3r15wfmwoi+BIdgk+jFb8eRV/0SQWng3VImqYzLElLkupG6J/wVf+YP8F/Ybr1wU4vBA7n3Ms99yRMGVXa874cd25+YXGptlxfWV1bbzQ3Nh9VkklMOjhhieyGSBFGBeloqhnpppIgHjLyFA6vCv3plUhFE/GgxynpczQQNKYYaUt1TRDG8PI2f262vLY3KfgX+CVogbLunjecmyBKcMaJ0JghpXq+l+q+QVJTzEheDzJFUoSHaEB6FgrEieqbieEcHlgmgnEi7RMaTtjfEwZxpcY8tJ0c6Rc1qxXkf1ov0/F531CRZpoIPF0UZwzqBBbXw4hKgjUbW4CwpNYrxC9IIqxtRpUtIa/cYEZT6/UKmdLCv20U5A0nnCMRmQBzJIe5CSIqBubEz2fkUUU+sbIN35+N+i94PGr7Xtu/P25dnJbfUAO7YB8cAh+cgQtwDe5AB2DAwDv4AJ/Ot9twt92daavrlDNboFLu3g+NBbxo</latexit><latexit

sha1_base64="Z1WdGrmgzCROJfEkOnxPhT4/F6M=">AAACZ3icbVFdS8MwFE3r15wfmwoi+BIdgk+jFb8eRV/0SQWng3VImqYzLElLkupG6J/wVf+YP8F/Ybr1wU4vBA7n3Ms99yRMGVXa874cd25+YXGptlxfWV1bbzQ3Nh9VkklMOjhhieyGSBFGBeloqhnpppIgHjLyFA6vCv3plUhFE/GgxynpczQQNKYYaUt1TRDG8PI2f262vLY3KfgX+CVogbLunjecmyBKcMaJ0JghpXq+l+q+QVJTzEheDzJFUoSHaEB6FgrEieqbieEcHlgmgnEi7RMaTtjfEwZxpcY8tJ0c6Rc1qxXkf1ov0/F531CRZpoIPF0UZwzqBBbXw4hKgjUbW4CwpNYrxC9IIqxtRpUtIa/cYEZT6/UKmdLCv20U5A0nnCMRmQBzJIe5CSIqBubEz2fkUUU+sbIN35+N+i94PGr7Xtu/P25dnJbfUAO7YB8cAh+cgQtwDe5AB2DAwDv4AJ/Ot9twt92daavrlDNboFLu3g+NBbxo</latexit><latexit

sha1_base64="Z1WdGrmgzCROJfEkOnxPhT4/F6M=">AAACZ3icbVFdS8MwFE3r15wfmwoi+BIdgk+jFb8eRV/0SQWng3VImqYzLElLkupG6J/wVf+YP8F/Ybr1wU4vBA7n3Ms99yRMGVXa874cd25+YXGptlxfWV1bbzQ3Nh9VkklMOjhhieyGSBFGBeloqhnpppIgHjLyFA6vCv3plUhFE/GgxynpczQQNKYYaUt1TRDG8PI2f262vLY3KfgX+CVogbLunjecmyBKcMaJ0JghpXq+l+q+QVJTzEheDzJFUoSHaEB6FgrEieqbieEcHlgmgnEi7RMaTtjfEwZxpcY8tJ0c6Rc1qxXkf1ov0/F531CRZpoIPF0UZwzqBBbXw4hKgjUbW4CwpNYrxC9IIqxtRpUtIa/cYEZT6/UKmdLCv20U5A0nnCMRmQBzJIe5CSIqBubEz2fkUUU+sbIN35+N+i94PGr7Xtu/P25dnJbfUAO7YB8cAh+cgQtwDe5AB2DAwDv4AJ/Ot9twt92daavrlDNboFLu3g+NBbxo</latexit><latexit<latexit

Integrated softwareBO

A Bayesian optimization library in JAX

https://github.com/PredictiveIntelligenceLab/JAX-BO

Key features:• Automatic differentiation• Vectorization/Parallelization• XLA/JIT compilation to GPUs/TPUs• Support for arbitrary GP priors and deep neural networks• Support for fully Bayesian inference with HMC-NUTS sampling (via NumPyro)• Support for multi-fidelity GPs with heterogeneous inputs• Support for likelihood-weighted acquisition functions for Bayesian optimization

Currently under final rounds of development - - let me know if you’d like to try it out and contribute!

gather samples by probing a cheaper but inaccurate (low-fidelity) wind prediction
could this noisy dataThe
be basic
combined
to3.accurately
the bridge
displacemenbuilding
blocksof estimate
the Bayesian
approachadoptedSolution
methodology
What is the uncertainty/error
associatedwith thisestimation?
Howis canwe besregressive stochastic
schemes[19,21].
This choice
motivated
another observationof the windforce? andQuoting[4], “once to
we allow
tractability
properties,
their Diaconis
natural extension
mult
The basicbuilding
blocks of thethe
Bayesia
Low-fidelity sources can bethings,
inaccurate,
but theyneed
toprovide
be strongly
correlated
the high-fidelity
data.
it becomesnaturaltotakea Bayesian
approach”.
ticular,GPsa flexible
priorwith
distributionfunction
regressive
stochastic
schemes over
[19,21].
This creturns robust posteriorvarianceestimatesand
which
tractability
properties,
theirenable
naturaladap
exAutomating simulation pipelines
ischallenging(mesh
generation,
pre/post-processing,etc.)
3. Solution methodology
we propose is summarized
in Fig.
1 andaisflexible
outlinedfoll
M. Raissiprioretinal.the/distrib
Journalticular, GPs
provideInspired by [19,21],
we willpresent
the framework
considereturnsrobustposteriorvariance estimates
Failed/unstable experiments/simulations
cantointroduce
anddata approachbias.
multiplelevelsisstraightforward.
Let us startwith
the isau
The basic building
blocksoutliersofthe Bayesian
adoptedhere
are1Gaussia
weproposesummarized
in Fig.
anddiscretization uncertaintyisusing
Gaussian
processprio
u 1 (xschemes) are two[19,21].
independent
Gaussian
processes[17–19,21]
wit
regressive stochastic
Thischoiceis motivatedbywillthe
BayesianthenoInspiredby[19,21],
we
presentinwhich
probabilistic
machinelearning
andclassical
Dealing with heterogeneoustractability
inputs is notstraightforward.
properties,
extension
tocovariancethesettings
Here, andg 1 (x,theirx# ; θ1natural), g 2to(x,multiple
x# ; θ2 ) are
functions,θLet
1 , θ2ulevelsis multi-fidelity
straightforward.
for
Bayesianreasoning
andSec.
computation.
In
ticular, GPs provide
a platform
flexible
prior
distributionoverfunctions,
and,
more
impo
parameter
to belearned
from
the
data
(see
3.1).
Then,
on
u 1 (x) are two independent Gaussian proces
Scaling to high-dimensions andlarge
data-sets
is
(to
some
extend)possible,not straightforward.
inferenceframework
thatbut
learningfrom (nois# ;enables# ; θ ) refinement
returns robust posterior variance estimatesenableadaptiveaHere, gwhich
(
x
,
x
θ
),
g
(
x
,
x
are
covarian1
12
2#
we propose is summarized
the following.
u (x)in∼ Fig.
GP (10and,parameterg (xis, xoutlined; θ)),
to beinlearned
from the data (seePrior specification/kernel design
playsamassively
importantrole.
Inspired by [19,21], 2.
weProblem
will present
the framework considering two-levels ofsetup# ; θ )# , and θ = (θ , θ
withg (x, x# ; θ) = ρ 2 Let
g 1 (xus, ux#(start;xθ)1∼)+g(x,
x
to multiple levels
is straightforward.
with
the
mod
2
1
GP (0, g (xauto-regressive, 2x ; θ)),How to best incorporate domain
knowledge
and physicalconstraints?
integralsofGaussiana Gaussian
process[17–19,21]
are still Gaussianu 1 (x) are
two independent
processeswith δ2 (processes.
x) ∼ GP (0,Th
g2
We
considergenerallinearintegro-differential
equHere, g 1 (x, x# ; θ1 ), g 2 (x, x# ; θ2 ) are covariance
theirwith g (x,functions,x# ; θ) = ρθ21g,1θ(2x,denotex# ; θ1 ) +g 2 (xhype, x# ;
parameter to be learnedf (from
Then, onecan triviallyof 3.1).
a Gaussian
process
are still obta
Gaux) ∼theGP data(0,integralsk((seex, x#Sec.
; θ)),

The fine prints…

•
•
•
•
•
•
•

e.g. linear constraints:

Lx u (x) = f (x),#u (x) ∼ GPwith(0, g (wherex, x# ; θ)),
f
(
x
)
∼
GP
(
0
,
k
(
x
,
x
; θ)),
x is a D-dimensional vector that
includes spa

equation,) represenk(x(,xx, #x; θ;1θ)=gL2with; θ).
with g (x, x# ; θ) = ρ 2 g 1unknown) +solution(xxL, xx##g;toθ(2x),the,x andθ = (θ1 ,and
θ2 , ρf).(xThekey obcomplex,to processes.
evaluate, “black-box”
function
integrals of a Gaussian aprocess
are expensive
still Gaussian
Therefore, given
that#
Similarly,
arrive at
auto-regressive(x) =f1
the weoutcomeof thea costly
the
ofρan
k
(x, xexperiment,; θ) = Lx Lstructure(x, output
x# ; fθ).
x# g# consequentlyxx), xare
two
independentdata
Gaussianxu
availability ofhigh-fidelity
for f L processes, denoted
f (x) ∼ GPL(0, 1k((limited; θ)),##
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017). Inferring solutionsof differential
equations
using
noisy
JournalatstFurthermore,
forcases,! =Similarly,
1,we
2,
k!may
(xwe
, xmulti-fidelity;arriveθ! ) =
Ldata.
Laccessg ! (x, ofxto; θ!supplem).
xthe
x# auto-regressive
In
many
also
have
Computational Physics, 335, 736-746.
Lx uof) are consequently
two independent
with
1 (x
increasing level
fidelity,
and generatingdata{x ! , y#Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017). Machine learning of linear differentialFurthermore,equations using Gaussian
processes.
Journaloffor ! = 1, 2, k! (x, x ; θ! ) = Lx L#

Computational Physics, 348, 683-693.

#

Such data may come from simplified computer modk(x, x# ; θ) = Lx Lx# g (x, x# ; θ).
we also have a small set of data on the solution u

Solving PDEs via active learning

@2u
@2u
+
= f (x1 , x2 )
2
2
@x1
@x2

Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017). Inferring solutions of differential equations using noisy multi-fidelity data. Journal ofComputational Physics, 335, 736-746.

see also a visualization of a neural network in Figure 1. Commonly used activation
functions include the logistic sigmoid 1/(1 + e x ), the hyperbolic tangent (tanh), andthe rectified linear unit (ReLU, max{x, 0}).

Physics of AI: Two schools of thought
1. Physics is implicitly
baked in specializedneural architectures withstrong inductive biases(e.g. invariance to simple
group symmetries).

2.2. Physics-informed neural networks for solving PDEs. We consider
the following PDE parameterized by for the solution u(x) with x = (x1 , . . . , xd )defined on a domain ⌦ ⇢ Rd :
◆
✓
@u
@2u
@2u
@u
,...,
;
,...,
;...;
= 0, x 2 ⌦,
(2.1)
f x;
@x1
@xd @x1 @x1@x1 @xd
1

3

2

4

5

with suitable boundary conditions

6

1

B(u, x) = 0

3

on
4

@⌦,
5

7

2

6

where B(u, x) could be Dirichlet, Neumann,
Robin,or Atperiodic
boundary
conditions.
F IGURE
5. Top left:
level ` = 1 n3 aggregates
informationfrom {n4 , n5 } andn2 aggregates
information{n5 , nS.
At ` = 2, n1 collects
this summary informa6 }. (2018).
*figuresfrom
Kondor,
R.,
Son,
H.
T.,
Pan,
H.,
Anderson,
B.,&
Trivedi,Covariant
For time-dependent problems, we considertiontime
specialof tox,andfrom n3 tandas
n2 . aBottomleft: Thiscomponent
graph is not isomorphic
the topone,n2 at ` = 1 will be identical. Therefore, at ` = 2, n1
butarXiv
the activationsof n3 andarXiv:1801.02144.
compositional
networks
forlearning
graphs.
preprint⌦ contains the temporal domain. The initialcondition
be simply
treatedasn5 andawill get thesame inputs fromcan
its neighbors,
irrespective ofwhether or not
n7 are the same node or not. Right: Aggregation at different levels. For keeping
special type of Dirichlet boundary conditionon thethe figure
legible spatio-temporal
only the neighborhood arounddomain.
one node in higher levels ismarked.

PDE( )

@2
@x2

x
t

..
.

..
.

û

I

T

Minimize ⇤
û(x,
t)
g
(x,
t)
Loss
✓ the permutatio
D
Definition 7. We say that ni is a first order covariant node in a comp-net if underof its receptive field Pi by any ⇡ 2 S|Pi| , its activation trasforms as fi 7! P⇡ fi .

@@nBC & ICT

4.2.@ SûECOND ORDER COVARIANT COMP - NETSb@n (x, t)gR (u, x, t)

It is easy to verify that given any representation (Rg )g2G of a group G, the matrices (Rg ⌦ Rg )g2G
also furnish a representation of G. Thus, one step up in the hierarchy from P⇡ –covariant comp-net
2
are P⇡ ⌦ P⇡ –covariant comp-nets, where the fi feature vectors are now |Pi | dimensional vector
that transform
N under permutations of the internal ordering by ⇡ as fi 7! (P⇡ ⌦ P⇡ )fi .

u
1 XL(✓) :=[uf (x )] +
N
|
{z
}

1

, then the action
If we reshape fi into a matrix Fi 2 R|Pi |⇥|Pi |2@u@2u >Fig. 1. Schematic of a PINN for solving the di↵usioni equation✓ @ti F=✓i 7! P⇡ Fi2P⇡with mixed boundary@xis equivalent
to@uP⇡ ⌦P⇡ acting on fi . In the following, we will prefer this more intuitive matrix viewu andconditions (BC) u(x, t) = gD (x, t) on D ⇢ since@⌦(x, t)that=featuregR (u,
x, t)⇢ way@⌦.express
Therelationships
initial betwee
R thisit i=1clearly@n
expresses
vectors
thatontransform
different constituentsof the receptive
field. TNote,
in particular,
thattwo
if wesets
define of
A#Pi as thdenotethecondition (IC) is treated as a special type of theboundary
conditions.
Tf andb Physics
regularization
restriction of theadjacency
matrixtoP(i.e.,
if
P
=
(e
,
.
.
.
,
e
)
then
[A#
]
=
A
i
i
p1
pm
pa ,pb ), thePi a,bDatafitresidual points for the equation and BC/IC. A# transforms
exactly
as
F
does
in
the
equationabove.
iPi
sha1_base64="1HGZtmjbtzdRdyLbi+i+HT/d8uc=">AAADuHicbVJbb9MwFE5WLqPcNnjkxWJC2kSpknbAmFQxDSR4QKhM7CI1WXCck9Sa7US+oBXLv4/fsH+D2xVEth3Jysl3jr9z+Zw3jCodRRfhSufW7Tt3V+917z94+Ojx2vqTI1UbSeCQ1KyWJzlWwKiAQ001g5NGAuY5g+P87MM8fvwTpKK1+K5nDaQcV4KWlGDtoWztIuFYTwlm9ovbTPQUNN5CuyOUGFGAzCUmYJPSf2zs7NfMuEQZnjDKqVaZpaPYnS5gNElybo3L6Ksys5dEntBj5x7bSk8HLks0nGv7EWuMSqodenlzlYT59gvs/nV24CbXOLfSv3zj6UxRopCEyjAs6a/FZA5laxtRP1oYuu7ES2cjWNo4Ww9/J0VNDAehCcNKTeKo0anFUlPCwHUTo6DB5AxXMPGuwBxUahcaOPTCIwUqa+mP0GiB/n/DYq7UjOc+cz6XuhqbgzfFJkaXO6mlojEaBLksVBqGdI3mgqKCSiCazbyDiaS+V0Sm2O9Se9lbVXLemsESLAiw9lx4u8ENyJ6n15iN7BsqemiHCtfjWFZUjKK+/+umtoKag5azNqWRzKFuC5pPpOuaqXYmZlXtu53ygd9sNymg9C94sTU7BiEOoHD24NO+s/HrYS/qRTcl7TMDy6woinvR0J93set66eOrQl93jgb9eNgffNve2NtfPoLV4FnwPNgM4uBtsBd8DsbBYUDC9yGEIqw7u50fnapDL1NXwuWdp0HLOvIP9X86QQ==</latexit>

Psichogios & Ungar, 1992
Lagaris et. al., 1998
Raissi et. al., 2019
Lu et. al., 2019
Zhu et. al., 2019

NN(x, t; ✓)The representation theory of symmetric groups is a rich subject that goes beyond the scope of thpresent paper (Sagan, 2001). However, there is one particular representation of Sm that is likel
familiar even to non-algebraists, the so-called fdefining representation, given by the P⇡ 2 Rn⇥n2permutation @matricesû@ û
⇢
1 if ⇡(j) = i@t@x2[P⇡ ]i,j =0 otherwise.
It is easy to verify that P⇡2 ⇡1 = P⇡2 P⇡1 for any ⇡1 ,⇡2 2 Sm , so {P⇡ }⇡2Sm is indeed a representation of Sm . If the transformation rules of the fi activations in a given comp-net are dictated by thi
representation, then each fi must necessarily be a |Pi | dimensional vector, and intuitively each com
ponent of fi carries information related to one specific atom in the receptive field, or the interactioof that specific atom with all the others. We call this case first order permutation covariance.

<latexit

2. Physics is explicitly
imposed by constraining
the output of conventional
neural architectures withweak inductive biases.

@@t

Proposition 3. If for any ⇡ 2 Sm , the f 7! R⇡ (f ) map appearing in Definition 6 is linear, then th
corresponding {R⇡ }⇡2Sm matrices form a representation of Sm .

8

R[f (x)]| {z }

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit>

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit<latexit

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit>

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit<latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit>

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit<latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit>

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit<latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit>

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit<latexit

sha1_base64="k4K1V+WdTsjhgiTKcGzkuIuAYBU=">AAAC2nicbVFLbxMxEPYur1JeAY5cLFKkRIJotwfgWAmQyqUKEmkrJVE068wmVvxYPDZtFOXCDXHlz3GEX4Kd5kBbRrI872/mm6pRknxR/MryGzdv3b6zc3f33v0HDx+1Hj85JhucwIGwyrrTCgiVNDjw0is8bRyCrhSeVIt3KX7yFR1Jaz77ZYNjDTMjaynAR9ek9WdkrDRTNJ7350uSgl5JU1unccoNBgcqfv7MugXxTv/j0RF1OUjNwfOYh85JM+PAhTVemmADcQU+dauDEQmC740qvQrrTvrO1y99d4/7eSwHJwmJA0UTOVkVNunexnacluRRc1tzY03aDRxvwHkZ55nKOgJHjGTgl7DZJE33/kM3VaR2MzSYZk+bTFrtoldshF9Xyq3SZlvpT1q/R1Mrgo4QQgHRsCwaP14leKFwvTsKhA2IBcxwGFUDGmm82txizV9EzzThxhdp2Hj/rViBJlrqKmZq8HO6GkvO/8WGwddvxytpmhD5FRdAdVCJsHTYSItD4dUyKiCcjLNyMQcHwsfzX0Kp9DpyUl5l4LpyvN8ri175ab998HrLzg57xp6zDivZG3bADlmfDZjIDjOTnWXn+Sj/ln/Pf1yk5tm25im7JPnPvzcg5cE=</latexit>

sha1_base64="k4K1V+WdTsjhgiTKcGzkuIuAYBU=">AAAC2nicbVFLbxMxEPYur1JeAY5cLFKkRIJotwfgWAmQyqUKEmkrJVE068wmVvxYPDZtFOXCDXHlz3GEX4Kd5kBbRrI872/mm6pRknxR/MryGzdv3b6zc3f33v0HDx+1Hj85JhucwIGwyrrTCgiVNDjw0is8bRyCrhSeVIt3KX7yFR1Jaz77ZYNjDTMjaynAR9ek9WdkrDRTNJ7350uSgl5JU1unccoNBgcqfv7MugXxTv/j0RF1OUjNwfOYh85JM+PAhTVemmADcQU+dauDEQmC740qvQrrTvrO1y99d4/7eSwHJwmJA0UTOVkVNunexnacluRRc1tzY03aDRxvwHkZ55nKOgJHjGTgl7DZJE33/kM3VaR2MzSYZk+bTFrtoldshF9Xyq3SZlvpT1q/R1Mrgo4QQgHRsCwaP14leKFwvTsKhA2IBcxwGFUDGmm82txizV9EzzThxhdp2Hj/rViBJlrqKmZq8HO6GkvO/8WGwddvxytpmhD5FRdAdVCJsHTYSItD4dUyKiCcjLNyMQcHwsfzX0Kp9DpyUl5l4LpyvN8ri175ab998HrLzg57xp6zDivZG3bADlmfDZjIDjOTnWXn+Sj/ln/Pf1yk5tm25im7JPnPvzcg5cE=</latexit><latexit

sha1_base64="k4K1V+WdTsjhgiTKcGzkuIuAYBU=">AAAC2nicbVFLbxMxEPYur1JeAY5cLFKkRIJotwfgWAmQyqUKEmkrJVE068wmVvxYPDZtFOXCDXHlz3GEX4Kd5kBbRrI872/mm6pRknxR/MryGzdv3b6zc3f33v0HDx+1Hj85JhucwIGwyrrTCgiVNDjw0is8bRyCrhSeVIt3KX7yFR1Jaz77ZYNjDTMjaynAR9ek9WdkrDRTNJ7350uSgl5JU1unccoNBgcqfv7MugXxTv/j0RF1OUjNwfOYh85JM+PAhTVemmADcQU+dauDEQmC740qvQrrTvrO1y99d4/7eSwHJwmJA0UTOVkVNunexnacluRRc1tzY03aDRxvwHkZ55nKOgJHjGTgl7DZJE33/kM3VaR2MzSYZk+bTFrtoldshF9Xyq3SZlvpT1q/R1Mrgo4QQgHRsCwaP14leKFwvTsKhA2IBcxwGFUDGmm82txizV9EzzThxhdp2Hj/rViBJlrqKmZq8HO6GkvO/8WGwddvxytpmhD5FRdAdVCJsHTYSItD4dUyKiCcjLNyMQcHwsfzX0Kp9DpyUl5l4LpyvN8ri175ab998HrLzg57xp6zDivZG3bADlmfDZjIDjOTnWXn+Sj/ln/Pf1yk5tm25im7JPnPvzcg5cE=</latexit><latexit

sha1_base64="k4K1V+WdTsjhgiTKcGzkuIuAYBU=">AAAC2nicbVFLbxMxEPYur1JeAY5cLFKkRIJotwfgWAmQyqUKEmkrJVE068wmVvxYPDZtFOXCDXHlz3GEX4Kd5kBbRrI872/mm6pRknxR/MryGzdv3b6zc3f33v0HDx+1Hj85JhucwIGwyrrTCgiVNDjw0is8bRyCrhSeVIt3KX7yFR1Jaz77ZYNjDTMjaynAR9ek9WdkrDRTNJ7350uSgl5JU1unccoNBgcqfv7MugXxTv/j0RF1OUjNwfOYh85JM+PAhTVemmADcQU+dauDEQmC740qvQrrTvrO1y99d4/7eSwHJwmJA0UTOVkVNunexnacluRRc1tzY03aDRxvwHkZ55nKOgJHjGTgl7DZJE33/kM3VaR2MzSYZk+bTFrtoldshF9Xyq3SZlvpT1q/R1Mrgo4QQgHRsCwaP14leKFwvTsKhA2IBcxwGFUDGmm82txizV9EzzThxhdp2Hj/rViBJlrqKmZq8HO6GkvO/8WGwddvxytpmhD5FRdAdVCJsHTYSItD4dUyKiCcjLNyMQcHwsfzX0Kp9DpyUl5l4LpyvN8ri175ab998HrLzg57xp6zDivZG3bADlmfDZjIDjOTnWXn+Sj/ln/Pf1yk5tm25im7JPnPvzcg5cE=</latexit><latexit<latexit

General formulation of PINNs

Physics-informed neural networks (PINNs) aim at inferring a continuous latentfunction u(x, t) that arises as the solution to a system of nonlinear partial
di↵erential equations (PDE) of the general form
ut + Nx [u] = 0, x 2 ⌦, t 2 [0, T ]u(x, 0) = h(x), x 2 ⌦u(x, t) = g(x, t), t 2 [0, T ], x 2 @⌦We proceed by approximating u(x, t) by a deep neural network f✓ (x, t), anddefine the residual of the PDE asr✓ (x, t) :=Data fit@f✓ (x, t) + Nx [f✓ (x, t)]@t

The corresponding loss function is given by

Training via stochastic gradient descent:L(✓) := Lu (✓) +| {z }
PDE residual

Lr (✓)| {z }✓n+1 = ✓n+ Lu0 (✓) + Lub (✓)| {z } | {z }ICs fit⌘r✓ L(✓n )BCs fit*all gradients are computed
via automatic differentiation

Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward
and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707.
Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998). Artificial neural networks for solving ordinary and partial differential equations.
Psichogios, D. C., & Ungar, L. H. (1992). A hybrid neural network-first principles approach to process modeling.

reads as
is notoriously
hard to resolve by classical numerical methods. In one spacedimension, the Burger’s
conditions
ut + uuxequation(0.01/⇡)ualong
0, withx 2 [Dirichlet
1, 1], t 2 boundary[0, 1],(3)xx =Physics-informed
NeuralNetworksu(0, x) = sin(⇡x),
reads asExample: Burgers’ equation in 1Du(t, 1) = u(t, 1) = 0.

ut + uux (0.01/⇡)uxx = 0, x 2 [ 1, 1],
Let us define f (t, x) to be given byu(0, x) = sin(⇡x),
f := ut + uux (0.01/⇡)uxx ,u(t, 1) = u(t, 1) = 0.

t 2 [0, 1],(3)

and proceed by approximating u(t, x) by a deep neural network. To highlightthe simplicity
this idea we have included a Python code
Let us definef (t, x) intoimplementing
be given bysnippet using Tensorflow [16]; currently one of the most popular and well
documented open source libraries for machine learning computations. To
ut +defined
uux as(0.01/⇡)uxx ,this end, u(t, x) canfbe:=
simplydef u(t, x):
and proceeduby
approximating u(t, x) by a deep neural network. To highlight= neural_net(tf.concat([t,x],1), weights, biases)
the simplicity
in implementing
this idea we have included a Python code
returnusnippet using Tensorflow [16]; currently one of the most popular and well
Correspondingly, the physics informed neural network f (t, x) takes the form
documented open source libraries for machine learning computations. To
f(t, x):this end, defu(t,x) can be simply defined asu = u(t, x)u_t = tf.gradients(u, t)[0]
def u(t, x):u_x = tf.gradients(u, x)[0]u = neural_net(tf.concat([t,x],1),
weights, biases)
u_xx = tf.gradients(u_x, x)[0]
return fu = u_t + u*u_x - (0.01/tf.pi)*u_xx
return f

Physics-informed Neural Networksu(t, x)
1.0
0.5

x

0.75
0.50
0.25
0.00
0.25
0.50
0.75

Data (100 points)

0.0
0.5
1.0
0.0

0.2

0.4

0.6

0.8

t
t = 0.25

t = 0.50011u(t, x)1u(t, x)u(t, x)1

t = 0.75

0

1
1

0

1

0

1
1

x

0

1

x
Exact

1

0

1

x
Prediction

Figure 1: Burgers’ equation: Top: Predicted solution u(t, x) along with the initial andboundary training data. In addition we are using 10,000 collocation points generated usinga Latin Hypercube Sampling strategy. Bottom: Comparison of the predicted and exact
solutions corresponding to the three temporal snapshots depicted by the white verticallines in the top panel. The relative L2 error for this case is 6.7 · 10 4 . Model training tookapproximately 60 seconds on a single NVIDIA Titan X GPU card.

Physics-informed neural networks

Raissi, M.,Yazdani, A., & Karniadakis, G. E. (2020). Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science.

sha1_base64="KXuXnC9hW9r5uIhQr2bMaIVKYY4=">AAAC83icbVJLb9NAEF6bVwmvAEcuI1KqAGlk5wCoUqVKFKkHJIpE2kpxiMabSbzKeu3uriHBypkrnLkhrvwgjvwOLmycUPXBrFb69pv5ZkYzG+dSGBsEvzz/0uUrV6+tXa/duHnr9p363XsHJis0py7PZKaPYjQkhaKuFVbSUa4J01jSYTx5ufAffiBtRKbe2VlO/RTHSowER+uoQf1PpDKhhqRsZGlq41H5aoppLmlrDnsk0yST9hPQcVHFg1DQ2W3XopjGQpUoxVg9mddgYRsQ7ZK0CEVz2po9hqcwed/599iG4wq0IHJnyUUuWfQmpTFubUNzM2xB6MioyrZxoktO6ZbCSpejtgLlKgHUIlLDk34+JqQJ1pf9rIMwYBOC15hL5ARZThptptuDeiNoB5XBRRCuQIOtbH9Q/x0NM16kblpcojG9MMhtv1y0wiXNa1FhKEc+wTH1HFSYkumX1ZLm8MgxQxhl2l1loWJPK0pMjZmlsYtM0SbmvG9B/s/XK+zoRb8UKi8sKb4sNCok2AwWG4eh0MStnDmAXAvXK/AENXLr/sWZKnE6dzMJz0/gIjjotMOgHb7tNHaeraazxh6wh6zJQvac7bA9ts+6jHvoffa+eF/9wv/mf/d/LEN9b6W5z86Y//MvKgzivg==</latexit>

sha1_base64="KXuXnC9hW9r5uIhQr2bMaIVKYY4=">AAAC83icbVJLb9NAEF6bVwmvAEcuI1KqAGlk5wCoUqVKFKkHJIpE2kpxiMabSbzKeu3uriHBypkrnLkhrvwgjvwOLmycUPXBrFb69pv5ZkYzG+dSGBsEvzz/0uUrV6+tXa/duHnr9p363XsHJis0py7PZKaPYjQkhaKuFVbSUa4J01jSYTx5ufAffiBtRKbe2VlO/RTHSowER+uoQf1PpDKhhqRsZGlq41H5aoppLmlrDnsk0yST9hPQcVHFg1DQ2W3XopjGQpUoxVg9mddgYRsQ7ZK0CEVz2po9hqcwed/599iG4wq0IHJnyUUuWfQmpTFubUNzM2xB6MioyrZxoktO6ZbCSpejtgLlKgHUIlLDk34+JqQJ1pf9rIMwYBOC15hL5ARZThptptuDeiNoB5XBRRCuQIOtbH9Q/x0NM16kblpcojG9MMhtv1y0wiXNa1FhKEc+wTH1HFSYkumX1ZLm8MgxQxhl2l1loWJPK0pMjZmlsYtM0SbmvG9B/s/XK+zoRb8UKi8sKb4sNCok2AwWG4eh0MStnDmAXAvXK/AENXLr/sWZKnE6dzMJz0/gIjjotMOgHb7tNHaeraazxh6wh6zJQvac7bA9ts+6jHvoffa+eF/9wv/mf/d/LEN9b6W5z86Y//MvKgzivg==</latexit><latexit

sha1_base64="KXuXnC9hW9r5uIhQr2bMaIVKYY4=">AAAC83icbVJLb9NAEF6bVwmvAEcuI1KqAGlk5wCoUqVKFKkHJIpE2kpxiMabSbzKeu3uriHBypkrnLkhrvwgjvwOLmycUPXBrFb69pv5ZkYzG+dSGBsEvzz/0uUrV6+tXa/duHnr9p363XsHJis0py7PZKaPYjQkhaKuFVbSUa4J01jSYTx5ufAffiBtRKbe2VlO/RTHSowER+uoQf1PpDKhhqRsZGlq41H5aoppLmlrDnsk0yST9hPQcVHFg1DQ2W3XopjGQpUoxVg9mddgYRsQ7ZK0CEVz2po9hqcwed/599iG4wq0IHJnyUUuWfQmpTFubUNzM2xB6MioyrZxoktO6ZbCSpejtgLlKgHUIlLDk34+JqQJ1pf9rIMwYBOC15hL5ARZThptptuDeiNoB5XBRRCuQIOtbH9Q/x0NM16kblpcojG9MMhtv1y0wiXNa1FhKEc+wTH1HFSYkumX1ZLm8MgxQxhl2l1loWJPK0pMjZmlsYtM0SbmvG9B/s/XK+zoRb8UKi8sKb4sNCok2AwWG4eh0MStnDmAXAvXK/AENXLr/sWZKnE6dzMJz0/gIjjotMOgHb7tNHaeraazxh6wh6zJQvac7bA9ts+6jHvoffa+eF/9wv/mf/d/LEN9b6W5z86Y//MvKgzivg==</latexit><latexit

sha1_base64="KXuXnC9hW9r5uIhQr2bMaIVKYY4=">AAAC83icbVJLb9NAEF6bVwmvAEcuI1KqAGlk5wCoUqVKFKkHJIpE2kpxiMabSbzKeu3uriHBypkrnLkhrvwgjvwOLmycUPXBrFb69pv5ZkYzG+dSGBsEvzz/0uUrV6+tXa/duHnr9p363XsHJis0py7PZKaPYjQkhaKuFVbSUa4J01jSYTx5ufAffiBtRKbe2VlO/RTHSowER+uoQf1PpDKhhqRsZGlq41H5aoppLmlrDnsk0yST9hPQcVHFg1DQ2W3XopjGQpUoxVg9mddgYRsQ7ZK0CEVz2po9hqcwed/599iG4wq0IHJnyUUuWfQmpTFubUNzM2xB6MioyrZxoktO6ZbCSpejtgLlKgHUIlLDk34+JqQJ1pf9rIMwYBOC15hL5ARZThptptuDeiNoB5XBRRCuQIOtbH9Q/x0NM16kblpcojG9MMhtv1y0wiXNa1FhKEc+wTH1HFSYkumX1ZLm8MgxQxhl2l1loWJPK0pMjZmlsYtM0SbmvG9B/s/XK+zoRb8UKi8sKb4sNCok2AwWG4eh0MStnDmAXAvXK/AENXLr/sWZKnE6dzMJz0/gIjjotMOgHb7tNHaeraazxh6wh6zJQvac7bA9ts+6jHvoffa+eF/9wv/mf/d/LEN9b6W5z86Y//MvKgzivg==</latexit><latexit<latexit

sha1_base64="1HGZtmjbtzdRdyLbi+i+HT/d8uc=">AAADuHicbVJbb9MwFE5WLqPcNnjkxWJC2kSpknbAmFQxDSR4QKhM7CI1WXCck9Sa7US+oBXLv4/fsH+D2xVEth3Jysl3jr9z+Zw3jCodRRfhSufW7Tt3V+917z94+Ojx2vqTI1UbSeCQ1KyWJzlWwKiAQ001g5NGAuY5g+P87MM8fvwTpKK1+K5nDaQcV4KWlGDtoWztIuFYTwlm9ovbTPQUNN5CuyOUGFGAzCUmYJPSf2zs7NfMuEQZnjDKqVaZpaPYnS5gNElybo3L6Ksys5dEntBj5x7bSk8HLks0nGv7EWuMSqodenlzlYT59gvs/nV24CbXOLfSv3zj6UxRopCEyjAs6a/FZA5laxtRP1oYuu7ES2cjWNo4Ww9/J0VNDAehCcNKTeKo0anFUlPCwHUTo6DB5AxXMPGuwBxUahcaOPTCIwUqa+mP0GiB/n/DYq7UjOc+cz6XuhqbgzfFJkaXO6mlojEaBLksVBqGdI3mgqKCSiCazbyDiaS+V0Sm2O9Se9lbVXLemsESLAiw9lx4u8ENyJ6n15iN7BsqemiHCtfjWFZUjKK+/+umtoKag5azNqWRzKFuC5pPpOuaqXYmZlXtu53ygd9sNymg9C94sTU7BiEOoHD24NO+s/HrYS/qRTcl7TMDy6woinvR0J93set66eOrQl93jgb9eNgffNve2NtfPoLV4FnwPNgM4uBtsBd8DsbBYUDC9yGEIqw7u50fnapDL1NXwuWdp0HLOvIP9X86QQ==</latexit><latexit

Physics as a prior in deep learning

u
1 X
1
2
R[f✓ (x)]L(✓) :=[ui f✓ (xi )] +Nu i=1| {z }|{z} Physics regularization

N

where
Data fit

Recent results showcase remarkable promise, but failure looms even for the simplest problems…

Example: Helmholtz equation in 2D.
u(x, y) + k 2 u(x, y) = q(x, y), (x, y) 2 ⌦ := ( 1, 1)u(x, y) = h(x, y), (x, y) 2 @⌦is the Laplace operator.

20% error in the prediction of a fully-connected, 4-layer deep PINN modelAn “unconventional” regularizer/prior that requires
us to revisit standard deep learning practices:• loss function• network initialization
Despite some empirical success, we are still lacking• data normalization
a rigorous mathematical understanding on when,• optimization
why and how physics-informed constraints can be• network architecture
effective in regularizing deep learning models.

Understanding the training dynamics of PINNs
Key concepts:• Explore the connection between deep neural networks and kernel regression methods toelucidate the training dynamics of deep learning models.
• At initialization, fully-connected networks are equivalent to Gaussian processes in the infinitewidth limit.
• The evolution of an infinite-width network during training can also be described by anotherkernel, the so-called Neural Tangent Kernel (NTK).

Lee, J., Bahri,Y., Novak, R., Schoenholz, S. S., Pennington, J., & Sohl-Dickstein, J. (2017). Deep neural networks as Gaussian processes.
Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural
information processing systems (pp. 8571-8580).

NeuralTangent
KernelPINNs
NeuralTangent
Kernelfor for
PINNs
Neural Tangent Kernel for PINNs

PINNs converge to GPs at the infinite-width limit for linear PDEs

Model
Problem:Poisson’s
equation
Model
Problem:
1D 1D
Poisson’s
equation
Model Problem: 1D Poisson’s equationf (x),8x 2
8x[0,21][0, 1]
= f=
(x),
uxxu(x)xx (x)
uxx (x) = f (x), 8x 2 [0, 1]u(x)= g(x),x =x0,=1.0, 1.
u(x)= g(x),
u(x) = g(x), x = 0, 1.

(27)(27)(27)

WeWe
proceedby byapproximating
the thesolutionu(x)u(x)by abyfully-connectedneuralproceedapproximating
solutiona fully-connectedneuralWe proceed by approximating the solution u(x) by a fully-connected neuralnetwork
denotedbyu(x,u(x,✓)withwithonehidden
hiddenlayer:
network
denotedbyu(x,✓) withone
hiddenlayer:
network
denotedby✓)onelayer:
1 1 (1) (1)(0) (0)(0)(0) (0)(1) (1)
1(1)W· (W
(0)
(1)
p
W
x
+
b
)
+
b
u(x,
✓)
=
p
·
(W
x
+
b
)
u(x,
✓)
=
u(x, ✓) = p W · (W x + b ) + b + b
NN N
Then
Then
Then

(28)(28)(28)

Neural Tangent Kernel for PINNs

h h h Kernel for PINNs
NeuralTangent
11 1 (1)
(0)
(0)
(0)

W(1)

·(1)¨ (W(0)

x(0)
+(0)
b

(0)
) (0) W

(x,
✓)=
= pp
W W· ¨ (W(x,x + bx +) b W)
uuxx
xxu
(x,
✓)
=p
· ¨ (W
xx ✓)
NN N

i i

i

(0) (0) (0)(0)W(29)(29)
W

W

W

(29)

Theorem: Assume that the activation function is smooth and has a
Theorem: Assume that the activation function is smooth and has a
induction this result can• Bynetwork
bounded second order derivative ¨ . Then for a fully-connected neuralbounded second order derivative ¨ . Then for a fully-connected neural network
be generalized to arbitraryofonehiddenlayer
atinitialization,of one hidden layer at initialization,
deep fully-connectedarchitectures and any linear
D
(1)
0
D
(1)
0
u(x,
✓) !
(30)
u(x, ✓) !
GP(0,⌃ GP(0,(x, x ))⌃ (x, x ))(30)
PDE operator.
D (1)
D
0
✓) !
⌃(1)
(x,
x
)),
(31)
uxx (x, ✓)uxx
!(x,
GP(0,
⌃xxGP(0,(x, x0 )),(31)xx• Opens up a path for
as N !
wheremeansDconvergencein distribution
andstudying approximation
as 1,
N!
1,Dwheremeans convergencein distributionandihaccuracy and convergence
i
h
0
4
0
(1)
0
4
(x,
x
)
=
¨
(ux
+
v)¨
(ux
+
v)
. (ux0 + v)
(32).
u
⌃(1)
E
rate of
PINNs in the
xx
x
)
=
¨
(ux
+
v)¨
(32)
u
⌃xx (x,Eu,v⇠N (0,1)
u,v⇠N (0,1)
associated RKHS.

TheNeural
NeuralTangent
Tangent Kernel
Kernel(NTK)of PINNs
NeuralTangent
Kernel
for
PINNs
for
PINNs
Neural Tangent Kernel for PINNs

NeuralTangent
Kernelfor for
PINNs
NeuralTangent
KernelPINNsConsider thethegradientflow:
Considergradientflow:
Considerthegradient flow:
Considerflow:
Considerthe
thegradientgradientflow:
d✓d✓= rL(✓),
d✓d✓ d✓(24) (24)=rL(✓),= =rL(✓),(24) (24)= rL(✓),(24dt
rL(✓),
dt
dt
dtN
dt i
i )} b , {xi , f (xi )}Nr , we haveN
N
Lemma:
Given
the
data
points
{x
,
g(x
N
i
i
i
i
Nr i Nr
b
i ,)}
i,i=1
i ,Nfrbr (xi )}b f (x
i
i
i
r
i=1
b
b
Lemma: Giventhedata
points
{x
,
g(x
{x
,
)}
,
we
have
N
Lemma:
Given
the
data
points
{x
g(x
)}
,
{x
, havewehaveN
Lemma:
Given
the data{xbbi=1
, {x
fwe
(x
)}i=1
, we haver, g(xi )}i b,r)}i )}rr ,i=1i=1b{xi ,points
b i=1
r
r
b
b
r
"
#
i=1
Lemma:
Given
the
data
points
g(x
,
{x
f
(x
,
##   #rr i=1bb i=1du(x,✓(t))""" du(x"b ,✓(t))K(t), ✓(t)) (t)g(x g(x# b ,✓(t))
b ,✓(t)) du(x
u(x
(t)
K
(t)
,
✓(t))
uu
ur
b
b)
du(x


K
u(x
K
)
b
dt
uu
ur
b
b
du(x
,✓(t))
dtb ,✓(t)) ==· Ku(x, ) (25)
K
u(x
(t)
(t)
,
✓(t))
g(x
)
K
u(x
(t)
K
(t)
,
✓(t))
g(x
uu
ur
b
b
·
,
(25)
uu
ur
b
b
dLu(x
r
K
(t)
K
(t)
,
✓(t))
g(x
)
dt
dt
dLu(x
,✓(t))uu(t)ur(t) Lu(xbr , ✓(t))rdt
K(t)f) (xbr ) , (25)
K
Lu(x
=
·
, (25
ru
rr
=
·
,
(25)
(t)
K
,
✓(t))
f
(x
K
ru
rr
r
r
dLu(x
,✓(t))=·dLu(x,✓(t))r
dt
r
dt r ,✓(t))
dLu(x
(t)
(t)r , ✓(t)), ✓(t))Lu(xfr(x(t) KK
K(t) K
fr )(xr ) f (xr )Kruru(t)
Lu(x
ru
rr
rr
r , ✓(t))
(t)
K
Lu(x
rr
dt
dt
dt
Nbb⇥Nb , K (t) N
Nr b ⇥Nr , and
T T
NR
b ⇥N
b ⇥N
(t)==KK
(t)
and
K
(t)
2
2
R
where
K
(t)
(t)
and
K
(t)
2
R
,
K
(t)
2
R
, and
where Kruru
uu
ur
uu
ur
urur
T
N
⇥N
Nb ⇥Nr , and
b
b
T
N
⇥N
N
⇥N
T
N
⇥N
N
⇥N
r
b
b
b
r
(t)
=
K
(t)
and
K
(t)
2
R
,
K
(t)
2
R
where
K
b
b
b
N
⇥N
N
⇥N
ru
uuby ,byur , and
r
r r=
r K
(t)
K
(t)
and
K
(t)
2R
Rgiven
, ur
K(t)
, and
where
K
ur
(t)
K
(t)
K
2 R2 RwhereKuu
ur (t)
ruR
uu
22ru
R
whose
(i,andj)-thentry
is2given
K
(t)
whose
(i,
j)-th
entry
is
Krrrr(t)ur(t)
urNr ⇥Nr whose (i, j)-th entry is given by
N
N
⇥N
r
r
r
r
(t)
2
R
K
rr
(t)22 R
R
whose
entry
is is
given
Krrrr(t)whose(i,
(i,
j)-th
entry
given
K
j byj byE
Dj)-th
i
E
D
i
du(x
,
✓(t))
du(x
du(x
,
✓(t))
,
✓(t))
du(x
b , ✓(t))
b
b
bj
j
E
D
(K
,
)ij)(t)
=
i
uuuu
E
D
(K
,
(t)
=
i
j
ij
E
Ddu(x
du(x
,
✓(t))
i
,
✓(t))
du(x
du(x
, ✓(t)), ,✓(t))
d✓
d✓
b d✓
b
du(x
,
✓(t))
✓(t))
du(x
d✓
b
b
b
b
(K
,
)
(t)
=
uu iji(Kuu,, j j E E(t) D=uu))ij
ij(t)
(K
=du(x
D
id✓
d✓
d✓ (26)dLu(xd✓r , ✓(t))
,
✓(t))
du(x
b , ✓(t))
,
✓(t))
dLu(x
d✓
d✓
r
b
(K
,D , i)ij)(t)
=
(26)
urur
j
E
j
(K
E
i
ij (t) = D
du(x
d✓ d✓
d✓
dLu(x
Er , ✓(t)) (26)Ddu(xi, ,✓(t))
dLu(xr , j✓(t))
b , ✓(t))
d✓
b
(26
✓(t))
du(x
,
✓(t))
dLu(x
r
(K
j
(K
,
)
(t)
=
E
D
ur )iji (t) b=
ur
ij
(26)
jd✓
, ✓(t))dL(xE d✓
D dL(x(Kur )ij (t) =dL(x, rd✓i , ✓(t))
r , ✓(t))
d✓
,
✓(t))
dL(x
r d✓
(Krr )ij (t) =,jE(Krr )ij (t) = D d✓ rid✓ D,
ji , ✓(t))
E
d✓
dL(x
,
✓(t))
dL(x
dL(xd✓dL(xr E
jr , ✓(t))
Dr d✓ir , ✓(t))
dLu(x,r , ✓(t))(Krr )ij (t)(K,= rr )dLu(xij (t) =r , ✓(t))
d✓(Krr )ij (t) =,d✓ d✓d✓d✓d✓*this ODE system holds for any differential operator and any neural network architecture.

Neural Tangent Kernel for PINNs

u(x) = g(x),x = 0, 1.

The NTK of PINNs and its evolution under gradient descent
Solution: u(x) = sin(x)Model Problem: 1D Poisson’s
Neuralequation
Tangent Kernel for PINNs

One hidden layer

Neural Tangent Kernel for PINNsf (x), 8x 2
[0, 1] for PINNsNeural Tangent Kernel foruNeuralxx (x) = Tangent
Kernel(27)u(x) = g(x), x = 0, 1.
Neural Tangent Kernel for PINNs

RecallModel
Problem:
Numerical Experiment 2: 1D Poisson’s equation
Recall#"ent
Kernelfor
PINNs
We proceed bydu(xapproximatingthe solution
u(x)Problem:by a fully-connectedneuralModelb ,✓(t))
#
"
u(x
Kur
(x)
= f (x),
8x 2 [0,1] b , ✓(t)) g(xb )uxxK
 with (t)uu (t)
dt u(x, ✓)networkdu(xdenotedone
hiddenlayer:b ,✓(t)) by=uu (t) Kur (t), 2 [0,(x)=1]f (x), 8x 2 [0,uxxKu(xb·, ✓(t))g(x) = f f(x),
(x)
8x
u
b
xx
dtduxx (xr ,✓(t))
(t)
K
(t)
(x
,
✓(t))
(x
)
K
u
r
,
u(x) ru= g(x), · rrx = 0, 1. xx r
duxx (xr ,✓(t)) dt =(xr , ✓(t)) u(x)f (xKr )g(x), u(x)1 ru (t)(1) Krr (t)(0) uxx(0)= f (x), 8x 2=
x = 0,=
1. g(x),x = 0, 1.
dt [0, 1]
(1)
p
W
·
(W
x
+
b
)
+
b
(28)
u(x,
✓)
=
Loss function:
For a physics-informed
networkwith one hidden layer atN(x), xTheorem:= 0, 1.
Lossfunction:
Theorem:
For a physics-informed
networkwithonehiddenlayer
atthe
Solution:u(x)=sin(x)
(✓)
+
L
(✓)
L(✓)
=
L
initialization,
and
in
the
limit
as
the
layer’s
width
N
!
1,
NTK K(t) of Three(b)hidden layersrb(a)NeuralTangent
Kernel
for
PINNs
Then
(✓) NTK+ Lr (✓)
Lbthe
initialization,and in the limit as the layer’s width L(✓)
N !=1,
K(t) ofthe PINNs model definedinequationabove
converges
inprobability
to a
Nb
Nr
X
X
h
Nb iequation):inaboveFigure:converges
probability
to 2a (a) (b) The
1 equation
1PoissonNrrelative change of pa(1Din✓)the PINNs model defined
i
i
2
i
X
X
1
deterministic
limiting
i.e,
1
(0)
(0)
(0) 1
(0)
= (1) ·kernel,
|u(x
g(x
+
|r(x
,i ✓)|(29). a fully-connectedi 2
i2 network
rtrainingb ,x✓)
b )|
p
W
(x,
✓)
=
¨
(W
+
b
)
W
W
u
PINNs
K
obtainedbyneural
xx
=
|u(x
,
✓)
g(x
)|
+
|r(x
,
✓)|
.
deterministic
limiting
kernel,i.e,
NCombine
2
2
r
r
b
b
X
N i 2i=11
2i=1(10,#100, 500) via 10, 000
2 iterations of full-batch"widths
i 2
diﬀerent
i=1
i=1
#
"
✓) g(xb )| +  |r(x
.
(1)
(1)
#
" (1)
r , ✓)| K(1)⇥(0)a fully-connectedK⇥ur
5 . (c)
(1)
(1)
uu
uu
ur (0)For the 2model
problem,withneuralnetworkof⇤ onehiddenTheeigenvaluesof (33)theNTKnetwork
K at initializatio
rate
of
10
KK(0)
⇥
K
(0)
⇥
uu
ur
uu (0)
ur
K
⇥
(0)
K
(0)
⇥
i=1
⇤
For
the
model
problem,withafully-connectedneuralof one hidd
=
:=
K
,
!
uu
ur
uu
ur
⇤
(1)
(1)
=
K
,
N
!
1
K(0)K(0)
= =
!
:=
K
,
(33)
!
(0) loss
K
(0)
K(0)
(1)
(1)10,
ru
rr
(n
=
000,
width=(1)(1)⇥⇥
layer, consider
minimizing
function
gradient
descent
withan by gradient descent with
(0)
Kofrr
Kru
ruby
rrthe500).
⇥
⇥
Kthe
(0)
Kru (0)layer,
considerminimizing
loss
functionaboveully-connectedneuralnetworkone
hiddenru
rr above
rr
⇥ru
⇥
rr
infinitesimallysmalldescentlearningrate.
> 0 satisfying
someinfinitesimallylearning rate.
Forsuitable
any T > 0 satisfying some suitables function
above by gradient
with
an For any T small
withany
assumptions,
we have. For
T > 0 satisfying
some suitable assumptions, we have
kK(t)

lim
sup kK(t)K(0)k0 sup kK(t) K(0)k(34) 2 = 0
lim kK(t)2 =sup
K(0)k2 =0
N !1 t2[0,T ]N !1] t2[0,T(a)(b) quickly
K(0)k2 = 0 N !1 t2[0,T(34)]*in practice the NTK
lim(3

Figure: neural(1D Poisson(a) (b) toTherelative changeof paa deterministic
kernelWe conclude that, for suﬃcient wide fully-connected
networkequation):of one converges
PINNs K obtained by training
a fully-connected
that
remains constantneuralduringnetwork
hidden layer at any random initialization, the NTK of PINNs satisfies
diﬀerent widths (10, 100, 500)training
via 10,for
000anyiterationsof full-batchneuralarchitecture
5 . (c) The eigenvalues of the NTK K at initializatio
rate
of
10
⇤
K(t) ⇡ K(0) ⇡ K , 8t 2 [0, T ]and any PDE operator.
(n = 10, 000, width= 500).

Neural Tangent Kernel for PINNs) = sin(⇡x)

PINNs is equivalent to kernel regression

Model Problem: 1D Poisson’s equation

Neural Tangent= f (x), for
8x 2PINNs[0, 1]
uxx (x) Kernel(27)

Neuralu(x) Tangent= g(x), xKernel= 0, 1. for PINNs

re, we obtain" proceed bywe
# obtainWe
approximating
thesolution u(x) by a fully-connected neural
Furthermore,du(xb ,✓(t))
network dt
denotedhiddenlayer: g(xb )" by u(x, ✓) #with oneu(xb ,✓(t))=b ,✓(t))K(t) ·
duxx (xr ,✓(t)) du(xu(xb , ✓(t))
f (xr )g(xb )uK(t)dtxx (x·r , ✓(t))
1
=
dt
(1)
du
xx (x
· (W (0)x+) + b(1) f (xr )
(28)
u(x,
✓)r ,✓(t))=p W(xbr(0), ✓(t))uxxdtNu(xb ,✓(t)) g(xb )⇤⇡ K ·u(xb , ✓(t)) g(x.b )⇤⇡ uxx
K(x·r , ✓(t)) f (xr ).
.
Then
h

1
(1)
then
on: If K ⇤ is invertible,

uxx (xr , ✓(t))

(0)
(0)
(0)
⇤
p
W
(x,
✓)
=
·
¨
(W
x
+
b
)
W
u
xx
Assumption: If KN is invertible, then
)

⇣ (b)
⌘ 

f (xr ).
W (0)



i

(29) NTK spectrum at initialization, and after
10,000 gradient descent steps(c)⇣ ⇤⌘ b ) g(x )xtest , ✓(t))
g(x⇤u(xtest , ✓(t))
⇤
⇤ ⇤1
K
t
I ⇤ ) e1 I e · K t · , b , (35) (35)) test (K⇡ Ktest (K⇡Kf (xof, ✓(t))
uxx (xtest(a)(xr )
n(xequation):(b) The relative change✓ and the NTK of
test , ✓(t))
r )fparametersd Lettingbytraining
a fully-connected neural network with• one
hidden layer andt ! 1 gives
From our experience, the NTK of!
1 gives
0, 100,10, 000 iterations offull-batch
gradient
with ahence
learnin
PINNsdescentis always degenerate, 500) via
u(xtest , ✓(1))⇤⇤g(x1 ) g(xb )we (36)may
notlastbe ableto casually
u(x
,
✓(1))
test
b·
(K
)
⇡
K
.
⇤
⇤ testat1 initialization
The eigenvalues
of
the
NTK
K
and
at
the
step
of trainin
Ktest (K ) ·
(36)
✓(1))
uxx (xtest ,⇡f .(xr )perform kernel regression predictions
(x
,
✓(1))
)
u
f
(x
xx
test
r
h= 500).
in practice.

• Additional regularization may help in
obtaining an invertible NTK.

mwill“spectralbias"[29, 41,
32]
referstotheorya wellknown
pathology
that
prevents
deep
fully- autilizethedeveloped
theoryinvestigate
whetherphysics-informed
neuralnetworks
are networks
In thissection,we will utilize
the todeveloped
to investigate
whether physics-informed
neuralIn term
this section,
we Thewill term
utilizethe 32]developed
theoryto32]
investigate
whetherphysics-informed
neuralnetworks
earninghigh-frequency
functions.
spectrally
biased.
“spectral
bias"
refers toa well known
pathologydeep
that
prevents
deepareful
The“spectral
bias"
[29,
41,
refers[29,to a41,well
known
pathology
that prevents
fully-

Spectralbias
intoPINNs
spectrally biased.
The
term“spectralbias"[29, 41,functions.
32]
refersa well known pathology that prevents deep fullyconnected
networks
fromlearning
high-frequency
fromlearning
high-frequency
functions.

arelychangesduringtraining,
we may rewrite
equation 4.9 asconnected
networks
from learning
high-frequency
functions.
eigenvaluesof the NTK• The 4.9
Since"barely
the NTK
of PINNs
barelychangesduringtraining,equation
we may 4.9
rewriteequationas
INNschangesduringtraining,
wemay
rewriteas#
Since the NTK of PINNs barely changes during training, we may rewrite equationcharacterize
4.9 ashow fast the absolute"#du(x,✓(t))"b# " du(xb ,✓(t))
# b , ✓(t))  g(xb )u(xerror decreases.
dtdu(xb ,✓(t))⇡
g(xtrainingu(xdu(x ,✓(t))
b , ✓(t))b),(5.1) (5
K(0)
dt
,
✓(t))
g(x
)
u(x
g(x
)
b⇡ K(0) u(xbb , ✓(t)),
duxx (xr ,✓(t))
b
dt
dt
(x
,✓(t))
du
(x
,
✓(t))
f
(x
)
u
xx
r xx
(5.1) function(5.1)⇡r K(0)uxx (xrr ,,✓(t)) f (xr,)
duxx (xr ,✓(t)) ⇡ duK(0)(x ,✓(t))
Componentsof the target
b

• r)
f (x
that correspond to NTK eigenvectors
whichleads totowith larger eigenvalues will be learned
which leads
"
#

#
" "
# # " du(xfaster.
⌘⇣du(x,✓(t))
b ,✓(t))

⌘
⇣

b
)
g(x
⌘⌘⇡ g(x⇣ ⇣ dtdt
K(0)t g(xb ) bdu(xbdu(x,✓(t))b ,✓(t))
K(0)t
.
(5
·
I
e
)
)
g(x
duxx
(x,✓(t))
b
.
(5.2)
·
⇡
I
e
K(0)t
r ,✓(t))
b
K(0)t
dt
)
f
(x
For
fully-connected
networks,
the
du
(x
r
dt (x ,✓(t))⇡ ⇡I Ixx e e. . f (xr )•r(5.2) (5.2)
·
·
du
dt
xx
r
)
ff(x
duxx (xr ,✓(t))
dt
r
(xr )
eigenvectors corresponding to higher
Spectral
Bias
in
PINNs
dtSpectral
Bias
in
PINNs
dt
As
in remarkremark3.3,
3.3,thetheNTKNTKofofPINNs
PINNsisisalso
alsopositive
positivesemi-definite.
we
its
decompositi
thespectral
NTKdecomposition
matrixAs mentioned
mentioned insemi-definite.
So So
weeigenvalues
cancantaketakeitsofspectral
mark3.3,=thePINNs
also
positivematrix
semi-definite.
So
we cantakewhose
itswhose
spectraldecomposition
K(0)QTTNTK⇤Q,of
where
Qisisisananorthogonal
matrixand⇤isadiagonal
matrixentriestheeigenvalues
generally
exhibitlower
frequencies.
⇤Q,
where
Q
orthogonal
and
⇤
is
a
diagonal
matrixentriesare aretheeigenvalues
i0
i
T
T
,here
the
NTK
of
PINNs
is
also
positivesemi-definite.
So
we
can
take
its
spectraldecomposition
Recallis positive
semi-definite,
so
==
⇤Q,
Recallpositive thesemi-definite,so
Qmatrix⇤Q,where
whereQentries
Qisisananare the eigenvalues i 0
Q K(0)
is an
orthogonal
matrixanderror⇤is isK(0)aK(0)
diagonal
whoseof K(0).
Consequently,
thetraining
trainingerror
isgivengiven
byConsequently,
byQ
In practice,
we observe thatithe 0•isantheorthogonal
matrixdiagonal
matrixwhose
entriesaretheeigenvalues
orthogonal
matrixerror
andisis andaa diagonal
whose
entriesaretheorthogonal
and⇤diagonal
matrixwhose
entries
are
the
ntly,
trainingis⇤by⇤ is"given##amatrix
"
 
 

⇣
⌘
du(x
⇣
⌘
eigenvaluesof the NTK of PINNsdu(xb ,✓(t))b ,✓(t))eeigenvalues
training
error
given
by
eigenvalues
0is#ofK(0).
Consequently,
g(x
)
)
)
g(x
g(x
"
g(x
)
)
)
g(x
g(x
of
K(0).
Consequently,
i
b
b
b
b
b
b
K(0)t
K(0)t



dtdt
⇡
· ·
decay
rapidly.
This results in⇣⌘⇡ I I e edu(xb ,✓(t))
du
(x
duxx
(x
r ,✓(t))xxr ,✓(t))
)
)
)
f
(x
f
(x
f
(x
)
)
)
f
(x
f
(x
f
(x
r
r
r
g(x
)
)
)
g(x
g(x
r
r
r
# 
bdtdt
b 
b
K(0)t  
" du(xdt #extremely slow convergence to the⇡Ie·,✓(t))⌘⇣⌘⇣⇣⌘
du(xb ,✓(t))
duxx (xrbb ,✓(t))
fg(x(xbbr)))
fg(xg(x)b ) r ) ) g(x
g(x
g(x
)g(x
r ) b )high-frequency components of theb(xbf)b(x
K(0)t
)
g(x
dt
K(0)t · g(x
dt
b
b
K(0)t
g(x
⇡
I
e
dt
g(x
dt du
b) b)
·f· (x⇡ I I eeT T ⇤t⇤t,✓(t))
duxx
xx (x
r
⇡
(x
,✓(t))
r, target
⇡
Q
e
Q
·
)
)
)
f
(x
f
(x
,
⇡f (x
Q
·fr (xrr )r r(xuxx (xr ,✓(t))dt) er ) Q
)f(x
fr(x
f (x
rf
function.
)
)
)
f
(x
f
(x
)
r
r
dt
r
)
g(x

b
T
⇤t 
dt
,
⇡ Q
e
Q ·g(x
)

b
T
⇤t
g(x
)
)
f
(x
b r,
⇤tQ ·
which is⇡which
is equivalentequivalenttotoQ · f (xg(x⇡ Q
QTTee ⇤t,))br
f (xr ) ! !,#Q"" Q e
#

 
⇡
·

du(x
,✓(t))b ,✓(t))
to
du(x
)
f
(x
b
g(x
g(xg(xb) b)b ) br)
⇤t ⇤t
g(x
dtdt
⇡
e
. .
(5.3)(5Q·
Q
which isis equivalentequivalentto!
#Q duduxxxx(x(x
"
⇡
e
Q
·
r ,✓(t))
f
(x
f
(x
)
)
which
to
,✓(t))r )
rf (xfr(xr )
du(xb ,✓(t))
r
dt
g(xb )##  g(xb )dt
"
!!
⇤t 
dt
"
⇡
e
.
(5.3)
Q
·
Q
du(x
,✓(t))b ,✓(t))dudu(xxx (xbr,✓(t))
g(x
g(x
)
)
)
)
f
(x
f
(x
b b) rdt
g(xbb r) !⇡ e ⇤t
g(x
#
" Q
⇤t
Q
·
Q
..
(39)
dt
dt


du
(x
,✓(t))
Q
·
⇡
e
(39)
xx
r
)
)
ff(x
f
(x
du(xbdu,✓(t))xx (xr ,✓(t))
r
r
7
f (xg(xdtr) b)g(x(xb )r )
⇤t 7
dt
dt
dt

Q

xx

dt

duxx (xr ,✓(t))

r

dt
dt

f (xr )f (xuxx (xr , ✓(t)) uxx(xr ,)✓(t))

⇡

e

Q·

f (xr ).

(5.3)

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit>

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit

sha1_base64="wymmur7ludfDE4q0FDa/gRgGDF4=">AAACinicbVFNa9wwEJXdr3Sbttv2WCiiS+m2JYu9hyZpKATaQw89pJBNAuvFjOVZr4g8MpIcWMz+ify7Hvsvcqy860CTdEDo6c0bzegpq5S0Lop+B+G9+w8ePtp63Huy/fTZ8/6LlydW10bgRGilzVkGFpUknDjpFJ5VBqHMFJ5m59/a/OkFGis1HbtlhbMSCpJzKcB5Ku1fJqQl5UiOHxuQJKngFxK4dVoswDopeGEgl60gRyv8/qWXZFhIakDJgj6ueolboIO0oU/xin/l3ZH4Dk884AlBpiDd0DwpwS0EqObnanit/NBLkPLr+9L+IBpF6+B3QdyBAeviKO3/SXIt6tLPJhRYO42jys0aMH56hX6+2mIF4hwKnHpIUKKdNWvvVvydZ3I+18Yv/8Y1+29FA6W1yzLzynZyezvXkv/LTWs335s1kqraIYlNo3mtuNO8/QieS4PCqaUHIIxsnfaOGxDOf9eNLlnZehLfduAuOBmP4mgU/xoPDj937myx1+wtG7KY7bJD9oMdsQkT7Cp4E7wPhuF2OA73w4ONNAy6mlfsRoTf/wJcl8ac</latexit><latexit<latexit

Another key observation
hereis that thegradienteigenvaluesof Krr are much greater than Kuu , namely Krr dominates Kuu
Training via
stochastic
descent:
by definition 5.1. As a consequence, the PDE residual converges much faster than*all
fitting
the PDEareboundary
conditions,
gradients
computed
which may prevent the network from approximating
authors’
experience,high frequency✓n correct⌘r✓ solution.
L(✓n ) From thevia✓n+1 =theautomatic
differentiation
functions typically lead to high eigenvalues in Krr , but in some cases Kuu can dominate Krr . We believe that such a
discrepancy between Kuu and Krr is one of the key fundamental reasons behind why PINNs can often fail to train andsi,yieldM., Perdikaris,
P., &Karniadakis, G.
(2019).
networks:
A deep learning
framework
for solving techniqueforwardaccuratepredictions.
InE.light
ofPhysics-informed
this evidence,neuralin thenext section,
we describe
a practicalto address thisinverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707.
pathology by appropriately assigning weights to the different terms in a PINNs loss function.
sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit>

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit

sha1_base64="Zt/H4maZJMObw7Ak7O3ilQRIvyw=">AAADWXichVJdaxQxFM3sqF3Hr6199CW4CBVhmemDSkEotoKCDyt028LOsiaZO7OhmWTIR3EZ5vf5G8Qnf4Cv+mpmO5R+rHohcHLvObnJyaWV4MbG8begF966fWejfze6d//Bw0eDzcdHRjnNYMKUUPqEEgOCS5hYbgWcVBpISQUc09P9tn58BtpwJQ/tsoJZSQrJc86I9an5ZvA5lYrLDKTFhwvATGkNplIy47LAQhmDcydZS8bc4IKfgcR0GaUUCi5rInghmwj7SEtiF4yI+mOzndoFWPJ8903q/NGaasKgvkSY1+6C5DephS8W1wfEEpxzi5vmxV+Feo1wfPAO+1vzzBHxT7Gb13Gz5oAP++a/jb2WrtO+vdBib6V0JQUdpSCzzpz5YBiP4lXgmyDpwBB1MZ4PfqSZYq70X8IEMWaaxJWd1URbzgQ0UeoMVISdkgKmHkpSgpnVq1Fo8DOfyXCutF/+S1fZy4qalMYsS+qZ7ePM9VqbXFebOpu/ntVcVs6CZOeNciewVbidK5xxDcyKpQeEae7vitmCeButn74rXWjZepJcd+AmONoZJfEo+bQz3HvZudNHT9BTtI0S9ArtofdojCaIBV+Dn8Gv4HfvexiE/TA6p/aCTrOFrkS49QcieBnC</latexit><latexit<latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit>

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit

sha1_base64="yY03LYg6yHAHT3t1MtH47E/vel8=">AAACcnicbVHRShwxFM2MVldbddQ3fYkuBUVZZqS0RRAWfPGpWHBV2BmWO9mMG0xmhuSOdDvkQ33Tf+gHNLMdpLpeCJyck5N7c5KWUhgMw0fPX1j8sLTcWVn9+GltfSPY3Lo2RaUZH7BCFvo2BcOlyPkABUp+W2oOKpX8Jr0/b/SbB66NKPIrnJY8UXCXi0wwQEeNgt9xqmptR3WME45gD5r9L3tM8ZCentE408DquASNAqR9QRRtNudxliMaK8AJA1n/aC6dCXb43tlkFHTDXjgrOg+iFnRJW5ej4DkeF6xSPEcmwZhhFJaY1M1ETHK7GleGl8Du4Y4PHcxBcZPUs4ws/eyYMc0K7VaOdMb+76hBGTNVqTvZvMC81RryPW1YYfY9qUVeVshz9q9RVrmECtoETsdCc4Zy6gAwLdyslE3AxYruW151SZV1mURvE5gH1ye9KOxFP790+1/bdDpkl+yTAxKRb6RPLsglGRBGnrwlb8MLvD/+jr/nt1H6XuvZJq/KP/4L3xG/1g==</latexit><latexit<latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit>

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit

sha1_base64="g3Odr6CDVtYyAkpjPR+5gn1uHt4=">AAACh3icbVFLa9tAEF6pryR9Oe0xl6F2IIXgSjmkOaa0hR5dqOOAZcxqNbIXr3bFPpoYod/Q39djf0TvHSs+5NGBZb6Z+YaZ/SavlXQ+SX5H8aPHT54+29nde/7i5avXvf03F84EK3AsjDL2MucOldQ49tIrvKwt8ipXOMlXnzf1yU+0Thr9w69rnFV8oWUpBfeUmvd+ZdpIXaD2MEGorRGIBeRr4DUF17Iinl7AIMurJrRHG3fdHvv3g44DBWINGoPlipy/MnYFg3LeZH6Jnt/mHwPXBfFLWhSoChadLAL1mbKLR1++AnfzXj8ZJp3BQ5BuQZ9tbTTv/ckKI0JFPxCKOzdNk9rPGm69FArbvSw4rLlY8QVOCWpeoZs1nXAtHFKmgNJYeqRAl73d0fDKuXWVE5OEWLr7tU3yf7Vp8OXZrJG6Dh61uBlUBgXewOYKUEiLwqs1AS6spF1BLLnlwtOt7kzJq5Y0Se8r8BBcnAzTZJh+P+mfn27V2WEH7B07Yin7yM7ZNzZiYybY3+ggGkSH8W78IT6Nz26ocbTtecvuWPzpH/1Vw+k=</latexit><latexit<latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit>

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit

sha1_base64="UohXjcXIJLY/cMMVIp1KZsCa4JM=">AAADBHicdVJNb9QwEHXCR8sW6BaOXEasQK1YrRKESi+VKnHhBEXqtpXiKHK83qxV24lsB3UV5drf0CucuSGu/A+O/BOcj8NuW8ayNDNv5j177LQQ3Ngg+OP59+4/eLix+Wiw9fjJ0+3hzrNTk5easinNRa7PU2KY4IpNLbeCnReaEZkKdpZefGjws69MG56rE7ssWCxJpvicU2JdKtnxtnDKMq4q49RsPYDGXuNUVmWdWHgDWBK7oERUn+qkavKXdR11eAyHEIwBuwUdApgrwJ8ly8gYbBtFruIkBozXqHe7+jEEe45k0Yd7HdnlCs1/G23TmK2EXe+q6B1HK4i2nIievGXGTM362yfDUTAJWoPbTtg7I9TbcTL8i2c5LSVTlgpiTBQGhY2rRoQKVg9waVhB6AXJWORcRSQzcdW+Wg2vXGYG81y7rSy02dWOikhjljJ1lc0bmJtYk7wLi0o7P4grrorSMkU7oXkpwObQfAGYcc2oFUvnEKq5OyvQBdGEWvdR1lRS2cwkvDmB287p20kYTMIv70ZH+/10NtEL9BLtohC9R0foIzpGU0Q97V1737zv/pX/w//p/+pKfa/veY7WzP/9D9v16Dw=</latexit><latexit<latexit

the overall convergence rate of the u
total
training error is characterized by the eigenvalues of Kuu and Krr together.
t + Nx [u] = 0, x 2 ⌦, t 2 [0, T ]Meanwhile, the separate training error of u(xb , ✓) and uxx (xr , ✓) is determined by the eigenvalues of Kuu and Krr ,u(x,0) = h(x),x2Discrepancy
convergencerate
in PINNs
respectively. The
above observationof
motivates
us to givethe⌦following
definition.loss functionsu(x, t) =g(x,matrixt), t K2 [0,T n⇥n], x, the2 @⌦kernel2R
average convergence rate c is defined as theDefinition 5.1. For a positive semi-definite
mean of all its eigenvalues i ’s, i.e.
Pn
T r(K)
We proceed by approximating u(x, t)
neural network f✓ (x, t), and
i=1byi a deep
=
.
(5.4)
c=
define the residual of the PDE asnn
In particular, for any two kernel matrices K1 , K2 with average convergence rate c1 and c2 respectively, we say that@
K1 dominates K2 if c1c2 .
r✓ (x, t) := f✓ (x, t) + Nx [f✓ (x, t)]@t
Beyondthe spectralof fully-connected
PINNs
also
sufferfroma remarkableAs a concrete
example,we trainbias
a fully-connectedneural architectures,
network with one
hiddenlayer
and 100
neurons
to solve theThecorresponding
losssolutionfunctionis givenmodel problem
7.1
with a fabricatedu(x)=for different
frequency
amplitudes
a. Figure 1 shows
discrepancy
of convergencerate
in sin(a⇡x)
thebydifferent
componentsof theirloss function:
all eigenvalues of K, Kuu and Krr at initialization in descending order. As with conventional deep fully-connected
A near
PREPRINT - J ULY 30, 20
(✓)
+
L
(✓)
+
L
(✓)
+
L
(✓)
L(✓)
:= LNTK
networks, the eigenvalues of the
PINNs’
decay
rapidly
and
most
of
the
eigenvalues
arezero. Moreover, the
u
r
u0
ub
| {z } for different| {z } frequency| {z functions} | {z(different}
distribution of eigenvalues of K looks similara), which may heuristically
Data fit
PDE residualICs fitBCs fitexplain that PINNs tend to learn all frequencies almost simultaneously, as observed in Lu et. al. [43].

6
Practical insights

Thesection,
eigenvaluesof Krr are
much
greater
than
KThe
Kand
5.1. foruu
rr dominates
uu by
this
we problem
considergerenalPDEsof theform
3.1, namely- 3.2 byKleveraging
theuuNTK
theorydeveloped
PINNs.
•InFigure
1: Model(1D
Poissonequation):eigenvaluesof K, K
Kdefinition
initializationin descendrr atweu(x) bya fully-connectedneuralWe
approximate
the fabricatedlatent solution
order
for different
solutionsu(x)= sin(a⇡x) wherea network= 1, 2, 4.u(x, ✓) with multiple hidden layers, andconsequence,theminimizing
PDE residual
converges composite
much fasterfitting the PDE boundary conditions,•trainAsitsa parameters✓ bythe following
lossthanfunction
which may prevent the network from approximating the correct solution

trainparameters✓ by minimizing
the following
lossleveraging
function the NTK theory we developed for PINN
n thisits
section,we consider
gerenal PDEsof the formcomposite
3.1 - 3.2 byby +a fully-connectedneural networku(x,for✓) withmultiple hidden layers,(6.
anWe approximate the Fromlatent solution
L(✓)
=u(x)
Lb to
(✓)
Lr (✓)
theorypractice:
Adaptive
training
PINNs
rain its parameters ✓ by minimizing the following
composite loss function N
N
b
r
X
X
b
r
i
i 2
i
2
=Lb (✓) + L|u(x
,
✓)
g(x
)|
+
|r(x
,
✓)|
,
(6.
L(✓) =
(✓)
(6.1
r
b
b
r
2Nb i=1
2Nr i=1
Nb
Nr
X
X
b
r
- J ULY 30, 2020
ii 2
i A PREPRINT
2
=
|u(x
,
✓)
g(x
)|
+
|r(x
,
✓)|
,
(6.2
Practical insight: Adjusting2N
the learning-rate(or,equivalently,
thebatch-size)
canresolvethebbr
2Nr i=1b i=1
discrepancy in convergence rate
8 of the total training error.
Algorithm 1: Adaptive weights for physics-informed neural networks
Consider a physics-informed neural network u(x, ✓) with 8
parameters ✓ and a loss function
L(✓) :=b Lb (✓) +r Lr (✓),

where Lr (✓) denotes the PDE residual loss and Lr (✓) corresponds to boundary conditions. b = r = 1 are freeparameters used to overcome the discrepancy between Kuu and Krr . Then use S steps of a gradient descentalgorithm to update the parameters ✓ as:for n = 1, . . . , S do(a) Compute b and r by
PNr +Nb
T r(K(n))
i (n)
i=1
(6.5)
=
b = PNbuu (n)
T
r(K
(n))
uu
i=1 i
PNr +Nb
T r(K(n))
i (n)
i=1
(6.6)
=
=
PNr rrr
T r(Krr (n))
i=1 i (n)rrwhere i (n), uu
i and i (n) are eigenvalues of K(n), Kuu (n), Krr (n) at n-th iteration.
(b) Update the parameters ✓ via gradient descent✓n+1 = ✓n
end⌘r✓ L(✓n )Wang, S.,Yu, X., & Perdikaris, P. (2020).When and why PINNs fail to train: A neural tangent kernel perspective.

(6.7)

equation

Numerical studies: Wave propagation

nt a study that demonstrates the effectivenss of Algorithm 1 in a practical problem fordels face severe diffuculties. To this end, we consider a one-dimensional wave equation
1] taking the the form

A PREPRINT - J ULY 30, 202

A PREPRINT - J ULYA PREPRINT - J ULY 30, 20
Here we treat the temporal coordinate t as an additional spatial coordinate in x and then the initial condition 7.7 can b
4uxxboundary(x, t) = 0,condition(x, t) 2 7.6,
(0, 1)namely⇥ (0, 1)
(7.5)
utt (x, t)in the
ncludedu(0, t) = u(1, t) = 0, t 2 [0, 1]
(7.6)
u(x)
=
g(x),
x
2
@⌦
Here we treat the temporal coordinate t as an additional spatial coordinate in x and then the initial condition 7.7 can
1ncludedinthe boundary
condition
Now
approximate
solutionxu7.6,a1]5-layer deep fully-connected networku(x, ✓) with 500 neurons per hiddeu(x,we0) =sin(⇡x)+ thesin(4⇡x),2by[0,namely(7.7)
ayer, where x = (x,2t). Then we can formulatea “physics-informed"
function byu(x)= g(x), x 2 @⌦loss
ut (x, 0) = 0, x 2 [0, 1].
(7.8)+ L ut u(✓)(7.1
L(✓) = Ltheu (✓)
r (✓) deep fully-connected
Now we approximate
solutionby+a L
5-layer
network u(x, ✓) with 500 neurons per hidd(b)[45], thesolution
given weby can formulate a “physics-informed"
Nut
ayer,
where
x =u(x,(x, t)t).isThenloss function(a) by Nr
N
u
X
X
X
u
u
r
t
i
i
2
i
2
i
2 by trainin
One-dimensionalwave
equation:(a)The
predicted
solutionversustheexact
solution
1L
(✓)
+
(✓)
+
L
(✓)
(7.1
L(✓)
=
L
|u(x
,
✓)
g(x
)|
+
|u
(x
,
✓)|
+
|Lu(x
,
✓)|
,
(7.1
u
u
r
t
t u
u
ut(7.9)rsin(4⇡x)cos(8⇡t).
u(x, t) = sin(⇡x) cos(2⇡t)+
2Nufive
2Nuneurons
2Nr i=1
per
layer
using
the
Adam optimizer wi
neural network with
t i=1
2 hidden layers and 500
i=1
N
u
N
N
u
r
t
X
X
X
2
u
u
r
7] after
000 =iterations.
The
relativeL
error
is4.518e01.
(b)The
predicted
t
i
i
2
i
2
where
the80,
hyper-parameters,,areinitialized
to
1,
the
batch
sizes
are
set
to
N
= Nutisolution=✓)|N2r, = versus
300,(7.1
anuru |Lu(x|u(xuut , ✓)g(xu )| +|ut (xut , ✓)| +
,
r 2
2N
2N
= @tt 4@the
Here 2Nall
training
data
areuniformely
sampling
insidethecomputational
domain
at
each
u
u
r
yLtraining
networkusing
Algorithm
1
after
80,
000
iterations.
The
relative
L gradeint
error isdesce
1.7
xx . samet i=1
12 i=1i=1
To investigate
the reason
behindPINN’s failure
forthethisstandard
example,we compute
its
NTK
and
track
it duringtraining.
teration.
The
networku(x,✓)
isinitializedusing
Glorotscheme
[46]
and
then
trainedbyminimizin
where the hyper-parameters u , ut , r are initialized to 1, the batch sizes are set to Nu = Nut = Nr = 300, aloss function
via stochastic gradient descent using the Adam optimizer with default settings [47]. Figure 6
Lhe=above@tt 4@
xx . Here all training data are uniformely sampling inside the computational domain at each gradeint desce
provides a comparison between the predicted solution against the ground truth obtained
after 80, 000 training iteration
teration. The network u(x, ✓) is initialized using the standard Glorot scheme(b)[46] andthen trained by minimizi
Clearly the original PINN model fails to approximate the ground truth solution and the relative L2 error is above 40%
he above loss function via stochastic gradientdescent usingwave
theequation:
Adam (a)optimizer
withdefault
Figure 6: One-dimensional
The predicted
solutionversussettings
the exact [47].
solutionFigure
by traini
500 neurons
per layerusingtheAdam optimizerw
connected
neuralnetwork
withwefive computehidden layers
andprovidesa comparison
betweenthe predicted
solutionagainstthegroundtruth
obtained
after
80,
000
training
iteratio
To explore
the reason behind
PINN’sfailurefor
thisexample,
its
and
track
it
duringtraining.
Simil
2 NTK
settings [47] after 80, 000 iterations. The relative L error is 4.518e 01. (b) The2predicted solution versus
2
Clearly
the original
PINN
fails tosolution
approximate
thesamesolutionthe000relativeo the proof
of Lemma
3.1,model
the corresponding
canbegroundderivedfromthe lossfunction7.10 LTheerror
byNTK
training
thenetworktruth
using
Algorithm
1 and
after
80,
iterations.
relativeisLabove
error is 40
1.7
23
du(xu ,✓(t))"we compute its NTK#and track it during training. Simi
To explore the reason behind PINN’s failureforthisexample,u(xb , ✓(t)) g(xb )dt
67
dut (xut ,✓(t))NTK canf be ·derived
o the proof of Lemma 3.1, the corresponding
loss function
7.10(xut ,the✓(t))
utfrom
:=
K(t)
,
(7.1
4
5
2
3
dt
du(xur,✓(t)),✓(t))dLu(x" Lu(xr , ✓(t))#
u(xb , ✓(t)) g(xb )6 dut (xdt7,✓(t))utf ·
ut (xut , ✓(t))
:=
K(t)
,
(7.1
where
4
5
dt
3
2 dLu(x,✓(t))uLu(xr , ✓(t))
r(t)
J
u
Nu dt
⇤
7 ⇥ T
6 ut
T
T
f
K(t) = 2
where
5 · Ju (t), Jut (t), Jr (t) ,4 Nut Jut (t)3
u
r

ij

d✓

d✓

ncludedthe boundary
Now
we in
approximate
solutionu7.6,
by namely
a 5-layer
deep fully-connected network u(x, ✓) with 500 neurons per hidde⌧thecondition
i
j
⇥
⇤
dLu(x
✓(t))
r , ✓(t))
T x = (x, t).
ayer,
where
Thenr ,wecan, dLu(xformulatea “physics-informed"
loss function byu(x)=. g(x),x 2 @⌦
Numerical
studies:Wave
propagation= Jr (t)Jr (t) ij =d✓d✓
+ L ut u(✓)(7.1
L(✓) = Ltheu (✓)
r (✓) deep fully-connected network u(x, ✓) with 500 neurons per hidd
Now we approximate
solutionby+a L
5-layeralues of Ku , Kut and Kr at initialization and the last step of gradient descent are
Nut
ayer,
where
(x, t).
Then
can fixed
formulatea eigenvalues“physics-informed"
loss function by X
Nu weNAr PREPRINT - J ULY 30
observedthat xthe=NTK
does
notremainandallmove“outward"
X
X
FigureEigenvaluesofsnaps
u 7: One-dimensional
ut
i
iwave
2 equation:i
2 Ku ,rKut and Kr i at different
2
+
L
+
Lrg(x
(✓)
(7.1
L(✓)
=sorted
Lstatic|u(x, ✓)+|ut (x|Lu(xr , ✓)| ,(7.1
Kuut )|
dominate
Ku duringtraining.
nd then remain
almost
such
that
u (✓)
u
r and
t (✓)
uK
ut , ✓)| +indescendingorder.
2Nconverge
2N
2Nr i=1u i=1ut i=1 conditions,ut (xut , ✓) and Lu(xr , ✓))much
faster
than
the
loss
of
boundary
N
ut
Nu
Nr
X
X
X
discrepancy
intheconvergence
rate
of
each
different
term
in
the
loss,
causing
thisur Nu = Nu i = N2r = 300, ant the batch isizes are
i 2
2 set towhere the hyper-parameters, uit ,, ✓)initialized
tou1,
ur areg(x
=
|u(x
)|
+
|u
(x
,
✓)|
+
|Lu(xtr , ✓)| ,(7.1
t1 withuthe same networkuut theToverifyourhypothesis,
we
also
train
using
Algorithm
2Nut i=1
2Nr domain
L = Adaptive@tt 4@xxweights:. Here 2N
all utraining
data are uniformely sampling
inside the computational
at each gradeint desce
i=1
i=1
le
for hyper-parameters, ut✓)
andis initializedurteration.
The
networku(x,using the standard
Glorotscheme
[46]
and
then
trainedby minimizin
where the hyper-parameters,,areinitialized
to
1,
the
batch
sizes
are
set
to
N
=
N
=
N
u T r(Kut rr)uut
r = 300, aT r(K)+
u ) + T r(Kustochastic
t
he
above
loss
functionvia
gradient
descent
using
the
Adam
optimizer
withdefault
settings[47].
Figure6(7.13)
L = @utt = 4@xx . Here Tallr(Ktraining
data
areuniformely
sampling
insidethecomputational
domainat
eachgradeint
desceu ) the predicted solution against the ground truth obtained after 80, 000 training iteration
provides a comparison between
teration. The network u(x, ✓) is initialized using the standard Glorot scheme [46] and then trainedby minimizi
2
Clearly the Toriginal
PINN
model
fails
to
approximate
thegroundtruth
solutionandtherelativeL
error
is above 40%r(Ku ) + T r(Kut ) + T r(Kr )he above
loss
functionvia
stochastic
gradient
descent
using
the
Adam
optimizer
withdefault
settings[47].
Figure(7.14)
ut =Tbetweenr(KutPINN’s) the predictedprovidesa comparison
solutionagainst wethe compute
ground truth
obtained
after it
80,during
000 training
To explore
the reason behindfailure forthis example,
its NTK
and track
training.iteratio
Simil
2
T r(K
Tfailsr(KtoToriginalr(Ku ) +PINN
uther ) approximate
Clearly
the
model
the
groundtruth
solutionandtherelativeLerror is above 40
o the proof
of
Lemma
3.1,
corresponding
NTK
can
be
derived
from
the
loss
function
7.10
t) +
.
(7.15)2
3r =T r(K )du(xu ,✓(t))"we compute its NTK#and track it during training. Simi
To explore the reason behindr PINN’s failureforthisexample,u(xcomparedb , ✓(t)) tog(xb )dt extra computational costs
ghts every 1, 000 training iterations, hence
the
6
7
du
(x
,✓(t))tut
f be ·derived
oligible.
the proof
of
Lemma
3.1,
the
corresponding
NTK
canloss function
7.10
(xone
✓(t))
utfrom
:=
K(t)
,
(7.1
4
5
ut ,the
The results of this experimentin3Figure 6b, from whichcan2 are showndt
du(xur,✓(t)),✓(t))" Lu(x#r , ✓(t))
n obtained using the proposed adaptivedLu(xtraining
scheme achieves
excellentagreement
u(x
,
✓(t))
g(x
)
(a)
b
b
dt
(a)
(b)
6 du
7 hyper-parameters
ive L2 error is 1.73e 3. To quantify
thet (xeffectof the,andut ,✓(t))
uut, ✓(t))
f
(x
u
:=
K(t)
·
,
(7.1
where
4
5or without the hyper-parameterstut
dt with3
he eigenvalues of Ku , Kut and Kr2multiplieddLu(xur ,✓(t)) wave equation: (a)Lu(xFigure8b,8:theOne-dimensionalTheevolutionr , ✓(t)) of hyper-parameters u , ut and ru (t)of the
As it can be seen in Figure
discrepancy
convergence
rateofdifferent
Nu Jdt⇥ neural⇤500 neurons per layer using Algorithm 1. (bfive-layer
deep
fully-connected
network
with
7
6
u
T
T
T
t
f
s is considerably resolved.
Furthermore,
Figure8a·presents
thechangeof(t)weights
J
(t)
J
(t),
J
(t),
J
,of training.
K(t)
=
where
5
4
u
t
u
u
r
N
t
u
3
2
K
,
K
,
K
and
K
,
K
,
K
at
last
step
t
u
u
r
u
u
u
u
r
r
t
t
remainur almost fixedt whileu , ut increase rapidly and thenr is near 1 for all time.
J
(t)
u
r
all training process using Algorithm 1 N
isNurstable.
⇤
7 ⇥ T
6
ut
⌧
T
T
f
(t),iJ, r✓(t))(t) , du(xj , ✓(t))
K(t) = 4 Nu ⇥Jut (t)5 · J⇤u (t), Judu(xtuuT[Ku (t)]ij = rt J
,
(t)J
(t)
=
u
u
ij
15d✓
d✓
14
Nr Jr (t)
⌧⌧
i i
j j
⇥⇥
⇤
⇤
,
✓(t))
du(x
du(x
(x
,
✓(t))
du(x
du
t
u ut
u , ✓(t))
TTut , ✓(t))[Ku (t)], ,(t)= JJuu(t)J(t)ij ij==uu
ijij=
t (t)]
t (t)J
t
d✓d✓
d✓ d✓
⌧⌧
i i , ✓(t)) du(xj j , ✓(t))
⇤
⇥⇥
⇤
(x
du
dLu(x
dLu(x
,
✓(t))
, ✓(t))
t
(b)
r ut
ru
T T
t

The fine prints…
• Fully-connected PINNs not only suffer from spectral bias, but also from a remarkable discrepancy
of convergence rate in the different components of their loss function.

• This can be partly mitigated by adaptive training procedures.
• PINNs are currently far more effective for inverse (rather than forward) problems, indicating thatthe distribution of the observed data plays a profound role in their trainability.

• The application of PINNs still remains challenging in problems involving coupled physics and multiscale dynamics.

• Need for novel architectures that overcome spectral bias and training convergence pathologies.
• Need for stable and effective optimization algorithms for PINNs, and multi-task learning in general.
• Predictive uncertainty estimates can be facilitated via an (expensive) Bayesian formulation, although
averaging across an ensemble of neural nets still remains the most effective solution in practice.

Forward vs Inverse problems

Forward problem for 1D Poisson equation

Inverse problem for 1D Poisson equation
The data distribution plays a profound role in shaping the NTK spectrum, and, hence, the trainability of PINNs!
Basri, R., Galun, M., Geifman, A., Jacobs, D., Kasten,Y., & Kritchman, S. (2020). Frequency bias in neural networks for input of non-uniform density. arXiv
preprint arXiv:2003.04560.

sha1_base64="4KtAu64KkWkbXG865tS/kQr92L8=">AAACj3icbVFNaxsxENVuvxL3I057zGWoaXBax6x8aEIhxdAe2lNTqJOA5RqtPN4Ia7UbSVuyLP4d+W059ocUql070CR9QvDmzTzNMIpzJa2LousgfPDw0eMnG5utp8+ev9hqb788sVlhBI5EpjJzFnOLSmocOekUnuUGeRorPI0Xn+r86S80Vmb6hytznKQ80XIuBXdemravWIyJ1BVXMtFvly2osQvsMyrHoehe9so9eAeLn4Ob4AguGtID5s9KY1ID+5Ziwj8cQXef9oB6kTWv7dY+qI3MSt3lUwosl3DpC1bxoInLvRZDPbsZZNruRP2oAdwndE06ZI3jafs3m2WiSFE7obi1YxrlblJx46RQuGyxwmLOxYInOPZU8xTtpGoWuIQ3XpnBPDP+ageN+q+j4qm1ZRr7ypS7c3s3V4v/y40LNz+cVFLnhUMtVo3mhQKXQf0bMJMGhVOlJ1wY6WcFcc4NF87/2a0ucVrvhN7dwH1yMujTqE+/DzrD9+vtbJAd8pp0CSUHZEi+kGMyIoL8CTpBL9gPt8OD8GM4XJWGwdrzitxC+PUvx2q8WA==</latexit>

sha1_base64="4KtAu64KkWkbXG865tS/kQr92L8=">AAACj3icbVFNaxsxENVuvxL3I057zGWoaXBax6x8aEIhxdAe2lNTqJOA5RqtPN4Ia7UbSVuyLP4d+W059ocUql070CR9QvDmzTzNMIpzJa2LousgfPDw0eMnG5utp8+ev9hqb788sVlhBI5EpjJzFnOLSmocOekUnuUGeRorPI0Xn+r86S80Vmb6hytznKQ80XIuBXdemravWIyJ1BVXMtFvly2osQvsMyrHoehe9so9eAeLn4Ob4AguGtID5s9KY1ID+5Ziwj8cQXef9oB6kTWv7dY+qI3MSt3lUwosl3DpC1bxoInLvRZDPbsZZNruRP2oAdwndE06ZI3jafs3m2WiSFE7obi1YxrlblJx46RQuGyxwmLOxYInOPZU8xTtpGoWuIQ3XpnBPDP+ageN+q+j4qm1ZRr7ypS7c3s3V4v/y40LNz+cVFLnhUMtVo3mhQKXQf0bMJMGhVOlJ1wY6WcFcc4NF87/2a0ucVrvhN7dwH1yMujTqE+/DzrD9+vtbJAd8pp0CSUHZEi+kGMyIoL8CTpBL9gPt8OD8GM4XJWGwdrzitxC+PUvx2q8WA==</latexit><latexit

sha1_base64="4KtAu64KkWkbXG865tS/kQr92L8=">AAACj3icbVFNaxsxENVuvxL3I057zGWoaXBax6x8aEIhxdAe2lNTqJOA5RqtPN4Ia7UbSVuyLP4d+W059ocUql070CR9QvDmzTzNMIpzJa2LousgfPDw0eMnG5utp8+ev9hqb788sVlhBI5EpjJzFnOLSmocOekUnuUGeRorPI0Xn+r86S80Vmb6hytznKQ80XIuBXdemravWIyJ1BVXMtFvly2osQvsMyrHoehe9so9eAeLn4Ob4AguGtID5s9KY1ID+5Ziwj8cQXef9oB6kTWv7dY+qI3MSt3lUwosl3DpC1bxoInLvRZDPbsZZNruRP2oAdwndE06ZI3jafs3m2WiSFE7obi1YxrlblJx46RQuGyxwmLOxYInOPZU8xTtpGoWuIQ3XpnBPDP+ageN+q+j4qm1ZRr7ypS7c3s3V4v/y40LNz+cVFLnhUMtVo3mhQKXQf0bMJMGhVOlJ1wY6WcFcc4NF87/2a0ucVrvhN7dwH1yMujTqE+/DzrD9+vtbJAd8pp0CSUHZEi+kGMyIoL8CTpBL9gPt8OD8GM4XJWGwdrzitxC+PUvx2q8WA==</latexit><latexit

sha1_base64="4KtAu64KkWkbXG865tS/kQr92L8=">AAACj3icbVFNaxsxENVuvxL3I057zGWoaXBax6x8aEIhxdAe2lNTqJOA5RqtPN4Ia7UbSVuyLP4d+W059ocUql070CR9QvDmzTzNMIpzJa2LousgfPDw0eMnG5utp8+ev9hqb788sVlhBI5EpjJzFnOLSmocOekUnuUGeRorPI0Xn+r86S80Vmb6hytznKQ80XIuBXdemravWIyJ1BVXMtFvly2osQvsMyrHoehe9so9eAeLn4Ob4AguGtID5s9KY1ID+5Ziwj8cQXef9oB6kTWv7dY+qI3MSt3lUwosl3DpC1bxoInLvRZDPbsZZNruRP2oAdwndE06ZI3jafs3m2WiSFE7obi1YxrlblJx46RQuGyxwmLOxYInOPZU8xTtpGoWuIQ3XpnBPDP+ageN+q+j4qm1ZRr7ypS7c3s3V4v/y40LNz+cVFLnhUMtVo3mhQKXQf0bMJMGhVOlJ1wY6WcFcc4NF87/2a0ucVrvhN7dwH1yMujTqE+/DzrD9+vtbJAd8pp0CSUHZEi+kGMyIoL8CTpBL9gPt8OD8GM4XJWGwdrzitxC+PUvx2q8WA==</latexit><latexitu(x, y) + k 2 u(x, y) = q(x, y), (x, y) 2 ⌦ := ( 1, 1)u(x, y) = sin(a1 ⇡x) sin(a2 ⇡y)<latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit>

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="xCjSDZOkD67p5+gCODhbtZtsFjA=">AAACMHicbVDLSgNBEJz1GeMr6tHLYBA8hV0PGjwJKuSoYFTIhjA76TWD81hnesW45Ff8CL/Bq571JF79CicxB18FA0VVN9VTSSaFwzB8DSYmp6ZnZktz5fmFxaXlysrqmTO55dDkRhp7kTAHUmhookAJF5kFphIJ58nVwdA/vwHrhNGn2M+grdilFqngDL3UqdRjbYTugkYaI9xikhZHt0xlEvYGNI5pA6TqGYl3FK7z0Q4Vmm4fljuValgLR6B/STQmVTLGcafyHncNz5WP4pI514rCDNsFsyi4hEE5zh1kjF+xS2h5qpkC1y5GPxzQTa90aWqsf/7Ukfp9o2DKub5K/KRi2HO/vaH4n9fKMa23C6GzHEHzr6A0lxQNHdZFu8ICR9n3hHEr/K2U95hlHH2pP1ISNfCdRL8b+EvOtmtRWItOwur+zridElknG2SLRGSX7JMGOSZNwsk9eSRP5Dl4CF6Ct+D9a3QiGO+skR8IPj4B87+p7w==</latexit><latexit

sha1_base64="/KTAU+7Th9A77+GeYSQmGBmGpoU=">AAACJXicbVDLThtBEOwlIRgDieGaywgLiZO1ywXEKVJA4ghSjC15LWt23ItHzGOZ6Y0wK/8KH8E3cIVzcor4E8aPA5iUNFKpalrVXVmhpKc4/hutfPq8+mWttl7f2Nz6+q2xvXnpbekEtoVV1nUz7lFJg22SpLBbOOQ6U9jJrn9O/c5vdF5a84vGBfY1vzIyl4JTkAaNo9RYaYZoiKWEt5Tl1ekt14XC4wlLU3aGSo+sojuGN+VshknDDk7qg0YzbsUzsI8kWZAmLHA+aLykQytKHaKE4t73krigfsUdSaFwUk9LjwUX1/wKe4EartH3q9mFE7YXlCHLrQsvrDpT305UXHs/1ln4qTmN/LI3Ff/n9UrKj/qVNEVJaMQ8KC8VI8umdbGhdChIjQPhwsmwKxMj7rigUOq7lExPQifJcgMfyeVBK4lbyUUMNfgOu7APCRzCDziDc2iDgHt4hCd4jh6iP9G/eXsr0aLGHXiH6OUVr5moZA==</latexit><latexit

sha1_base64="/KTAU+7Th9A77+GeYSQmGBmGpoU=">AAACJXicbVDLThtBEOwlIRgDieGaywgLiZO1ywXEKVJA4ghSjC15LWt23ItHzGOZ6Y0wK/8KH8E3cIVzcor4E8aPA5iUNFKpalrVXVmhpKc4/hutfPq8+mWttl7f2Nz6+q2xvXnpbekEtoVV1nUz7lFJg22SpLBbOOQ6U9jJrn9O/c5vdF5a84vGBfY1vzIyl4JTkAaNo9RYaYZoiKWEt5Tl1ekt14XC4wlLU3aGSo+sojuGN+VshknDDk7qg0YzbsUzsI8kWZAmLHA+aLykQytKHaKE4t73krigfsUdSaFwUk9LjwUX1/wKe4EartH3q9mFE7YXlCHLrQsvrDpT305UXHs/1ln4qTmN/LI3Ff/n9UrKj/qVNEVJaMQ8KC8VI8umdbGhdChIjQPhwsmwKxMj7rigUOq7lExPQifJcgMfyeVBK4lbyUUMNfgOu7APCRzCDziDc2iDgHt4hCd4jh6iP9G/eXsr0aLGHXiH6OUVr5moZA==</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="xCjSDZOkD67p5+gCODhbtZtsFjA=">AAACMHicbVDLSgNBEJz1GeMr6tHLYBA8hV0PGjwJKuSoYFTIhjA76TWD81hnesW45Ff8CL/Bq571JF79CicxB18FA0VVN9VTSSaFwzB8DSYmp6ZnZktz5fmFxaXlysrqmTO55dDkRhp7kTAHUmhookAJF5kFphIJ58nVwdA/vwHrhNGn2M+grdilFqngDL3UqdRjbYTugkYaI9xikhZHt0xlEvYGNI5pA6TqGYl3FK7z0Q4Vmm4fljuValgLR6B/STQmVTLGcafyHncNz5WP4pI514rCDNsFsyi4hEE5zh1kjF+xS2h5qpkC1y5GPxzQTa90aWqsf/7Ukfp9o2DKub5K/KRi2HO/vaH4n9fKMa23C6GzHEHzr6A0lxQNHdZFu8ICR9n3hHEr/K2U95hlHH2pP1ISNfCdRL8b+EvOtmtRWItOwur+zridElknG2SLRGSX7JMGOSZNwsk9eSRP5Dl4CF6Ct+D9a3QiGO+skR8IPj4B87+p7w==</latexit><latexit

sha1_base64="/KTAU+7Th9A77+GeYSQmGBmGpoU=">AAACJXicbVDLThtBEOwlIRgDieGaywgLiZO1ywXEKVJA4ghSjC15LWt23ItHzGOZ6Y0wK/8KH8E3cIVzcor4E8aPA5iUNFKpalrVXVmhpKc4/hutfPq8+mWttl7f2Nz6+q2xvXnpbekEtoVV1nUz7lFJg22SpLBbOOQ6U9jJrn9O/c5vdF5a84vGBfY1vzIyl4JTkAaNo9RYaYZoiKWEt5Tl1ekt14XC4wlLU3aGSo+sojuGN+VshknDDk7qg0YzbsUzsI8kWZAmLHA+aLykQytKHaKE4t73krigfsUdSaFwUk9LjwUX1/wKe4EartH3q9mFE7YXlCHLrQsvrDpT305UXHs/1ln4qTmN/LI3Ff/n9UrKj/qVNEVJaMQ8KC8VI8umdbGhdChIjQPhwsmwKxMj7rigUOq7lExPQifJcgMfyeVBK4lbyUUMNfgOu7APCRzCDziDc2iDgHt4hCd4jh6iP9G/eXsr0aLGHXiH6OUVr5moZA==</latexit><latexit

sha1_base64="/KTAU+7Th9A77+GeYSQmGBmGpoU=">AAACJXicbVDLThtBEOwlIRgDieGaywgLiZO1ywXEKVJA4ghSjC15LWt23ItHzGOZ6Y0wK/8KH8E3cIVzcor4E8aPA5iUNFKpalrVXVmhpKc4/hutfPq8+mWttl7f2Nz6+q2xvXnpbekEtoVV1nUz7lFJg22SpLBbOOQ6U9jJrn9O/c5vdF5a84vGBfY1vzIyl4JTkAaNo9RYaYZoiKWEt5Tl1ekt14XC4wlLU3aGSo+sojuGN+VshknDDk7qg0YzbsUzsI8kWZAmLHA+aLykQytKHaKE4t73krigfsUdSaFwUk9LjwUX1/wKe4EartH3q9mFE7YXlCHLrQsvrDpT305UXHs/1ln4qTmN/LI3Ff/n9UrKj/qVNEVJaMQ8KC8VI8umdbGhdChIjQPhwsmwKxMj7rigUOq7lExPQifJcgMfyeVBK4lbyUUMNfgOu7APCRzCDziDc2iDgHt4hCd4jh6iP9G/eXsr0aLGHXiH6OUVr5moZA==</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

sha1_base64="KGV2VlciVeJQkEKG+iansQ1w/5g=">AAACMHicbVDLSgNBEJz1bXxFPXoZDIKnsJuDiidBhRwjGBWyIcxOes3gPNaZXjEu+RU/wm/wqmc9iVe/wknMwVfBQFHVTfVUkknhMAxfg4nJqemZ2bn50sLi0vJKeXXtzJnccmhyI429SJgDKTQ0UaCEi8wCU4mE8+TqcOif34B1wuhT7GfQVuxSi1Rwhl7qlPdibYTugkYaI9xikhbHt0xlEvYHNI5pHaTqGYl3FK7z0Q4VmtaOSp1yJayGI9C/JBqTChmj0Sm/x13Dc+WjuGTOtaIww3bBLAouYVCKcwcZ41fsElqeaqbAtYvRDwd0yytdmhrrnz91pH7fKJhyrq8SP6kY9txvbyj+57VyTPfahdBZjqD5V1CaS4qGDuuiXWGBo+x7wrgV/lbKe8wyjr7UHymJGvhOot8N/CVntWoUVqOTWuVgZ9zOHNkgm2SbRGSXHJA6aZAm4eSePJIn8hw8BC/BW/D+NToRjHfWyQ8EH5/0X6nx</latexit><latexit

Example:Helmholtz equation in 2D<latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit>

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="lSH/DjRp0wdENNYPoJ3NPNZ+2rE=">AAAEOHicjVNba9RAFE6zXup6a/XRl4OtUBGXpA8qBbHgSwUpFXuDZl1OJifpoZNJmJnUrmH/jj/C3+Crgm/6JL76C5xkt9V2q3ggyZlz/c53JnEp2dgg+DLjdy5cvHR59kr36rXrN27Ozd/aNkWlBW2JQhZ6N0ZDkhVtWbaSdktNmMeSduKD541/55C04UJt2mFJ/RwzxSkLtM40mPefRapglZCykaUjG6f1a8tpqsiYlRF0o5gyVjVbyvkdjbrgJGpOcBIGRQp2nyDTmLCrA6ks3oJABTGB2EeNwpJ22QnEwzZSos7IWCDOSB2irKipsRgpjCUOIhdi8c0yRDnafYGyfjlaGhvvL/a6EankN54p9GvDsnCxhs1o5R/oNx0MczIBHTmmDbA6Z5BkqDBn0Y658WJ93fT+WkYSJgZsAZzHKFEJN/JxMQNJpVllkBcJSbAaWTXHyjTvk5YJGeG+/9EjcRYWlbRMY+THFVuQcMg4VdVR3NI4qNWDcARPYXJU8BAip8CpDZzD/0BNbWAwtxD0glZgWgknyoI3kY3B3LcoKUSVOzxCojF7YVDafo3aspDNRitDJYoDzGjPqY58Mv26vekjuOcsCaSFdo+bp7X+mVFjbswwj11kA96c9TXG83x7lU2f9GtWZWVJiXGjtJIN0c1v49jWJKwcOgWFZof199U+3SXOG07CswxMK9vLvTDoha+ChdVHE3ZmvTveXW/JC73H3qq35m14W57w3/sf/U/+586HztfO986Pcag/M8m57Z2Szs9fE35mqQ==</latexit><latexit

sha1_base64="iF2uLayh3ljg38D9dAWHLLLOcik=">AAAELXicjVPNbtNAEN46/JRQoOXKZUSLVISI7F5AlRBIXIqEqiL6J8UhGq/HzqrrtbW7Lg1WXoeH4Bm4gsQNToi3YO2kKWkCYiTb87cz33yzjgopjPX9b0te68rVa9eXb7Rvrty6fWd1beXQ5KXmdMBzmevjCA1JoejACivpuNCEWSTpKDp5WcePTkkbkat9Oyyol2GqRCI4Wufqr3nPQ5ULFZOyoaUzGyXVWyuSRJEx2yNohxGlQlXCUiY+0KgNTsLagmka5AnYAUGqMRauDiQyfw8cFUQEfIAauSXtTscQDZtMiTolY4FESuoUZUl1jY1QYSSxH7oUi++2IMzQDjjK6vVoc+x8uNFph6TiCzxz6HeGRe5yjTCj7X+g33cwzHQCOnNMGxBqwSDxUGEmeDPm3qvdXdP5axlJGBuwOYgsQomKu5HPixmISy1UClkekwSrUajaLE39nraMyXD3/Y8esfMIXkoraIz8vGIDEk4FzlV1FDc09iv1KBjBM5iYCh5D6BSY2cAC/vtqbgP91XW/4zcC80owUdbZRPb6qz/COOdl5vBwicZ0A7+wvQq1FVzWGy0NFchPMKWuUx35ZHpVc9NH8MB5Ykhy7R43T+P980SFmTHDLHKZNXhzOVY7F8W6pU2e9iqhitKS4uNGSSlrouvfxrGtiVs5dApyLRzWi6s92yXKak6CywzMK4dbncDvBG98tszusftskwXsCXvBdtgeO2Dc++h99r54X1ufWt9bP8fseUsTGu+yGWn9+g2pTmUU</latexit><latexit

sha1_base64="iF2uLayh3ljg38D9dAWHLLLOcik=">AAAELXicjVPNbtNAEN46/JRQoOXKZUSLVISI7F5AlRBIXIqEqiL6J8UhGq/HzqrrtbW7Lg1WXoeH4Bm4gsQNToi3YO2kKWkCYiTb87cz33yzjgopjPX9b0te68rVa9eXb7Rvrty6fWd1beXQ5KXmdMBzmevjCA1JoejACivpuNCEWSTpKDp5WcePTkkbkat9Oyyol2GqRCI4Wufqr3nPQ5ULFZOyoaUzGyXVWyuSRJEx2yNohxGlQlXCUiY+0KgNTsLagmka5AnYAUGqMRauDiQyfw8cFUQEfIAauSXtTscQDZtMiTolY4FESuoUZUl1jY1QYSSxH7oUi++2IMzQDjjK6vVoc+x8uNFph6TiCzxz6HeGRe5yjTCj7X+g33cwzHQCOnNMGxBqwSDxUGEmeDPm3qvdXdP5axlJGBuwOYgsQomKu5HPixmISy1UClkekwSrUajaLE39nraMyXD3/Y8esfMIXkoraIz8vGIDEk4FzlV1FDc09iv1KBjBM5iYCh5D6BSY2cAC/vtqbgP91XW/4zcC80owUdbZRPb6qz/COOdl5vBwicZ0A7+wvQq1FVzWGy0NFchPMKWuUx35ZHpVc9NH8MB5Ykhy7R43T+P980SFmTHDLHKZNXhzOVY7F8W6pU2e9iqhitKS4uNGSSlrouvfxrGtiVs5dApyLRzWi6s92yXKak6CywzMK4dbncDvBG98tszusftskwXsCXvBdtgeO2Dc++h99r54X1ufWt9bP8fseUsTGu+yGWn9+g2pTmUU</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit

sha1_base64="+QviZ669r/DvDjbhp9ddS+MzXvk=">AAAEOHicjVNLb9NAEHYdHiU82sKRy4gWqQgRxT0AqoSoxKVIqCqiL6kO0Xg9dkZdr63ddWmw8nf4EfwGriBxgxPiyi9g7aQtbQpiJNuz8/zmm3VUSDa22/0647cuXb5ydfZa+/qNm7fm5hdu75i81IK2RS5zvRehIcmKti1bSXuFJswiSbvRwYvav3tI2nCutuywoF6GqeKEBVpn6i/4z0OVs4pJ2dDSkY2S6o3lJFFkzOoI2mFEKauKLWX8nkZtcBLWJzgJgzwBOyBINcbs6kAi83cgUEFEIAaoUVjSLjuGaNhEStQpGQvEKalDlCXVNZZChZHEfuhCLL5dgTBDOxAoq1ej5bHxwVKnHZKKT/FMoV8fFrmLNWxGq/9Av+VgmJMJ6MgxbYDVBYPEQ4UZi2bMzZcbG6bz1zKSMDZgc+AsQolKuJGPixmIS80qhSyPSYLVyKo+lqZ+n7SMyQj3/Y8esbOwKKVlGiM/rtiAhEPGqaqO4obGfqUeBiN4BpOjgkcQOgXObOAC/vtqagP9+cVup9sITCvBRFn0JrLZn/8exrkoM4dHSDRmP+gWtlehtixkvdHSUIHiAFPad6ojn0yvam76CO47SwxJrt3j5mmsf2ZUmBkzzCIXWYM353218SLffmmTp72KVVFaUmLcKCllTXT92zi2NQkrh05BodlhPb3aZ7tEWc1JcJ6BaWVnpRN0O8HrlcW1xxN2Zr273j1v2Qu8J96at+5tetue8D/4n/zP/pfWx9a31o/Wz3GoPzPJueOdkdav3xQeZqs=</latexit><latexit<latexit

sha1_base64="IQA0O+h+cYQt8Seo1TG4V+P0o1g=">AAACZHicdVFNa9tAFFypbZo6TqI29FQoS0zBOdhIpbS5GAy99NBCCrUdsBzxtFrFS3ZXYvcpYIT+ZG459tLf0GPXH4XaaQYWhpk3vMdsWkphMQzvPf/J02d7z/dftA7ah0fHwctXY1tUhvERK2RhLlOwXArNRyhQ8svScFCp5JP05vPSn9xyY0Whf+Ci5DMF11rkggE6KQlknBtgdRbjnCM0dYYNHdBerCGVkKxVGivAOQNZf20S012LZ7RHY1uppBaDqLn6Rh+PiL+RJOiE/XAF+pBEG9IhG1wkwc84K1iluEYmwdppFJY4q8GgYJI3rbiyvAR2A9d86qgGxe2sXrXS0HdOyWheGPc00pX6b6IGZe1CpW5yeazd9Zbi/7xphfn5rBa6rJBrtl6UV5JiQZcV00wYzlAuHAFmhLuVsjm4mtF9xNaWVDWuk2i3gYdk/L4fhf3o+4fO8OOmnX3yhpySLonIJzIkX8gFGRFG7shvz/N875ff9k/81+tR39tkTsgW/Ld/AJx3uXM=</latexit>

sha1_base64="IQA0O+h+cYQt8Seo1TG4V+P0o1g=">AAACZHicdVFNa9tAFFypbZo6TqI29FQoS0zBOdhIpbS5GAy99NBCCrUdsBzxtFrFS3ZXYvcpYIT+ZG459tLf0GPXH4XaaQYWhpk3vMdsWkphMQzvPf/J02d7z/dftA7ah0fHwctXY1tUhvERK2RhLlOwXArNRyhQ8svScFCp5JP05vPSn9xyY0Whf+Ci5DMF11rkggE6KQlknBtgdRbjnCM0dYYNHdBerCGVkKxVGivAOQNZf20S012LZ7RHY1uppBaDqLn6Rh+PiL+RJOiE/XAF+pBEG9IhG1wkwc84K1iluEYmwdppFJY4q8GgYJI3rbiyvAR2A9d86qgGxe2sXrXS0HdOyWheGPc00pX6b6IGZe1CpW5yeazd9Zbi/7xphfn5rBa6rJBrtl6UV5JiQZcV00wYzlAuHAFmhLuVsjm4mtF9xNaWVDWuk2i3gYdk/L4fhf3o+4fO8OOmnX3yhpySLonIJzIkX8gFGRFG7shvz/N875ff9k/81+tR39tkTsgW/Ld/AJx3uXM=</latexit><latexit

sha1_base64="IQA0O+h+cYQt8Seo1TG4V+P0o1g=">AAACZHicdVFNa9tAFFypbZo6TqI29FQoS0zBOdhIpbS5GAy99NBCCrUdsBzxtFrFS3ZXYvcpYIT+ZG459tLf0GPXH4XaaQYWhpk3vMdsWkphMQzvPf/J02d7z/dftA7ah0fHwctXY1tUhvERK2RhLlOwXArNRyhQ8svScFCp5JP05vPSn9xyY0Whf+Ci5DMF11rkggE6KQlknBtgdRbjnCM0dYYNHdBerCGVkKxVGivAOQNZf20S012LZ7RHY1uppBaDqLn6Rh+PiL+RJOiE/XAF+pBEG9IhG1wkwc84K1iluEYmwdppFJY4q8GgYJI3rbiyvAR2A9d86qgGxe2sXrXS0HdOyWheGPc00pX6b6IGZe1CpW5yeazd9Zbi/7xphfn5rBa6rJBrtl6UV5JiQZcV00wYzlAuHAFmhLuVsjm4mtF9xNaWVDWuk2i3gYdk/L4fhf3o+4fO8OOmnX3yhpySLonIJzIkX8gFGRFG7shvz/N875ff9k/81+tR39tkTsgW/Ld/AJx3uXM=</latexit><latexit

sha1_base64="IQA0O+h+cYQt8Seo1TG4V+P0o1g=">AAACZHicdVFNa9tAFFypbZo6TqI29FQoS0zBOdhIpbS5GAy99NBCCrUdsBzxtFrFS3ZXYvcpYIT+ZG459tLf0GPXH4XaaQYWhpk3vMdsWkphMQzvPf/J02d7z/dftA7ah0fHwctXY1tUhvERK2RhLlOwXArNRyhQ8svScFCp5JP05vPSn9xyY0Whf+Ci5DMF11rkggE6KQlknBtgdRbjnCM0dYYNHdBerCGVkKxVGivAOQNZf20S012LZ7RHY1uppBaDqLn6Rh+PiL+RJOiE/XAF+pBEG9IhG1wkwc84K1iluEYmwdppFJY4q8GgYJI3rbiyvAR2A9d86qgGxe2sXrXS0HdOyWheGPc00pX6b6IGZe1CpW5yeazd9Zbi/7xphfn5rBa6rJBrtl6UV5JiQZcV00wYzlAuHAFmhLuVsjm4mtF9xNaWVDWuk2i3gYdk/L4fhf3o+4fO8OOmnX3yhpySLonIJzIkX8gFGRFG7shvz/N875ff9k/81+tR39tkTsgW/Ld/AJx3uXM=</latexit><latexitSti↵ness:<latexit

Stiffness in the gradient flow dynamicsd✓=dtr✓ Lr (✓)
M
X
i=1

*training via gradient descent (aka Forward Euler) introduces
severe restrictions on the learning rate (η<2/λmax).
r✓ Li (✓)

• Sti↵ness of the gradient flow can be characterized by the largest eigenvalueof r2✓ L(✓).

Hypothesis:• The sti↵ness exists in the gradient flow dynamics of PINNs.

• The sti↵ness leads to imbalanced gradients during model training using
gradient descent.
Largest eigenvalue of the Hessianduring the training of a 4-layer PINN.

• The sti↵ness leads to difficulties in training PINNs via gradient descent✓n+1 = ✓n ⌘r✓ L(✓n ).

Wang, S.,Teng,Y., & Perdikaris, P. (2020).
Understanding and mitigating gradientpathologies in physics-informed neural networks.

3.2. advances
The KdV equation
RecentAs a mathematical model of waves on shallow water surfaces one could
consider the Korteweg-de Vries (KdV) equation. The KdV equation reads as

Discovery of ODEs
Exact Dynamics

Discovery of PDEs
ut =Learned Dynamics
50

uux

uxxx .

Exact Dynamics

50

(6)

Learned Dynamics

20

20

x

x

x

2.0 (6) using
To obtain a set of training data 2.0
we simulate the KdV equation
10
10
conventional
spectral methods. In1.5particular,
we start from an1.5initial conz
z
25
25
1.0
1.0
0 =
dition u(0, x)
sin(⇡x/20), x 20.5[ 20,020] and assume periodic
0.5 boundary
conditions.10We integrate equation 0.0(6) up10to the final time t = 40.0.0We use the
0
0
0.5
0.5
Chebfun package
[43]
with
a
spectralFourier
discretization
with512 modes
40
40
20
1.0
20
0
10 explicit
20
30Runge-Kutta
40
0
10
20
30
40with time-stepand a fourth-ordertemporal
integrator
20
20
0
0
0
0
t
y
y
40
40
size 10 4 . The solution
is saved u(t,everyt = 0.2 tot give us a total of 201
20
20
x)
x
x
Raissi,M.equation:(2018).
Deep
HiddenModels:Deep
1.0
FigureOut
3:
The
A solution
the KdVPhysics
equation(leftpanel)
iscomparedtosnapshots.
ofKdV
this
data-set,
weto
generatea smaller
training
subset,scat0.75
Data (200 points)
thecorresponding
solutionofthelearnedpartialdi↵erential
equation(rightpanel).
The
0.50
Figure2:
LorenzSystem:
The
exact
phase
portraitoftheLorenzsystem(left
panel)
is0.5
LearningofNonlinear
PartialDifferential
Equations.
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2018). Multisteptered inidentified
space systemand time,
randomly
10000
points from
0.25data
correctly bycapturesthe form ofsub-sampling
the dynamics and accurately
reproducescompared to the corresponding phase portrait of the learned dynamics (right panel).
0.00
0.0
2preprint arXiv:1801.06637.
arXiv
Neural Networks for Data-driven Discovery of Nonlinear time t =the0solution
a relative-error of 6.28e-02.
shouldbe emphasized
that0.25
the training
to t with= 26.8.
InL otherwords,Itwearesub-samplingfrom the origdata
arecollected
inroughly
two-thirdsofthedomainbetween
times
t
=
0
and
0.50 t = 26.8
0.5
Dynamical Systems. arXiv preprint
0.75from
inal dataset
onlyby inof thedomainfromtime
represented
the thewhite training
vertical lines.portion
The algorithm
is thusextrapolatingtime t = 0 to
1.0
2
The Lorenz system has a positive Lyapunov exponent, and small di↵eron thewetrainingof the1.0domain
3.78e-02. N as at = 26.8
onwards.
The training
relative
L -error0.0the
0.2
0.4
0.6
0.8
t = 26.8.
Given
data,
areportion
interested
in islearning
ences between the exact and learned models grow exponentially, even thought PDEs6
Stochastic
the attractor remains
intact. This behavior is evident
in figure 3, as we com- function of the solution u and its derivatives up to the 3rd order ; i.e.,
High-dimensionalPDEsu(t, x)u(t, x)= 0.25t =the0.50learned partial tdi↵erential= 0.75fectiveness of tour
approach, we solve
equation
111(7) using the PINNs algorithm [34]. We assume periodic boundary condiut condition= N (u,asuxthe, uone). generate the original(7)
xx , uxxxto
tions and the same initialused
0
0 of the learned partial
dataset.0 The resulting solutiondi↵erential equation as
We represent
theexact
solution
u ofbythe
a 5-layer
deepareneuralnetwork
50 neurons
well as thesolution
KdV equation
depictedin figurewith
3. Thisfigureindicatesthat our algorithm
is N
capableidentifyingwith
the 2 hidden
1 we let1
per hidden
layer.
Furthermore,
to beofa1accurately
neuralnetwork2
underlying1 partial
equationof1 6.28e-02.
1 L -error
10 di↵erential
1
0
0 with1 a relative
layers and100
neuronsperhiddenlayer.
These
two
networks
trained by
x in roughlyarexx
It should be highlighted
that the training
data are collected
twoExact
Prediction
Twoband (3).
minimizing
sumof squared
errors
To illustrate
the efthirdstheof thedomainbetweentimes
tloss= 0 ofandequationt =std26.8.
The algorithmisof u(t,x)thus extrapolating from timeVariancet = 26.8
onwards.
The corresponding relative
1.0
2
L -error on the training portion of the domain is 3.78e-02.
0.8
u(t, x)pare the exact versus the predicted trajectories. Small discrepancies due tofinite accuracy in the predicted dynamics lead to large errors in the forecasted time-series after t > 4, despite the fact that the bi-stable structure of
the attractor is well captured (see figure 2).

x3.3. Fluid flow behind a cylinder
In this example we collect data for the fluid flow past a cylinder (see figure 4) at Reynolds number 100 using direct numerical simulations of the twodimensional Navier-Stokes equations. In particular, following the problem
setup presented in [23] and [24], we simulate the Navier-Stokes equations describing the two-dimensional fluid flow past a circular cylinder at Reynolds
number 100 using the Immersed Boundary Projection Method [25, 26]. This
0.5
6
A detailed study of the choice of the order is provided in section0.63.1 for the Burgers’
approach utilizes a multi-domain scheme with four nested domains, each suc0.0
To testthe algorithm even further, let us change the initial condition
to
0.4
cessive grid being twice as large as the previous one. Length and time are equation.
0.5
non-dimensionalized so that the cylinder has unit diameter and the flow hascos( ⇡x/20)
and solve the KdV (6) using the conventional spectral
0.2 method
unit velocity. Data is collected on the finest domain with dimensions 9 ⇥ 4 atoutlined1.0above. We compare the resulting solution to the one obtained by
0.0
0.4
0.6
0.8
1.0
a grid resolution of 449 ⇥ 199. The flow solver uses a 3rd-order Runge Kutta
solving thelearned 0.2
partialdi↵erential
equation
using
the
PINNs
algo1st order
2nd
order(5)
3rd order
4th
order

Raissi, M. (2018). Forward-backward stochastic neuralnetworks: Deep learning of
10 high-dimensional partial
differential equations. arXiv preprint arXiv:
1804.07010.

trithm [34]. 2 It is worth emphasizing that the algorithm is trained on the
Relative
L -error
1.14e+00
1.29e-02
3.42e-02
5.54e-02
Figure5: Burgers
equationwith 3noisy
data:
Mean
of p✓ (u|x,alongwith thedatasetdepicted
inPerdikaris,
figureandis Top:being
testedon t,az),di↵erent
dataset as
Yang,
Y.,
&
P.
(2019).
Adversarial
uncertainty
location of the training data {(xi , ti ), ui }, i = 1, . . . , Nu . Middle: Prediction and predictive
shown
in figureThe
surprising
figureuncertainty
at tquantification
=4.
0.25,
t = 0.5
and t =in
0.75.
Bottom:reported
Variance ofin(u|x,t, z).4 strongly indi2result
neuralTable 2: Burgers’ equation:
Relative
Lphysics-informed-errorbetweenp✓ solutionsof the Burgers’ equacates that the algorithm is accurately learning the underlying partial di↵er-

networks.
Journal of equation
Computational
Physics. of the highest ordertion and the learned
partial di↵erential
as a function

Recent advances
Fractional PDEs

Surrogate modeling & high-dimensional UQ(a) GRF KLE512, test
1. Y., Zabaras, N., (b)GRF KLE512,
test
Zhu,
Koutsourelakis,
P. S.,& 2.Perdikaris, P. (2019).
Figure 1: fPINNs for solving integral, diﬀerential, and integro-diﬀerential equations. Here wechoose specific integro-diﬀerential operators in the form of time- and/or space- fractional derivatives. fPINNs can incorporate both fractional-order and integer-order operators. In the PDE
shown in the figure, f (·) is a function of operators. The abbreviations “SM” and “AD” represent
spectral methods and automatic diﬀerentiation, respectively.

Pang, G., Lu, L., & Karniadakis, G. E. (2018). fpinns: Fractional
physics-informed neural networks. arXiv preprint arXiv:
1811.08967.

Physics-constrained deep learning for high-dimensionalsurrogate modeling and uncertainty quantification withoutlabeled data. Journal of Computational Physics, 394, 56-81.

In this paper, we focus on the NN approaches due to the high expressive power of NNs infunction approximation [24, 25, 26, 27]. In particular, we concentrate on physics-informed neural
10
L. LU, X. MENG, Z. MAO, AND G. E. KARNIADAKIS
networks (PINNs) [28, 29, 30, 1], which belong to the second aforementioned category. The recent
applications of PINNs include (1) inferring the velocity and pressure fields from the concentration field of a passive[31], y)and=(2)q(y|x)q(x)identifying thez ⇠scalarp(z)in solving the Navier-Stokes equations
x, y ⇠ q(x,Differential
Boundary/initial
distributed parameters of stochastic PDEs [21]. However, PINNs, despite their high flexibility,
Geometry
Neural net
yz
equationscannotbe directly applied to the solution of fractional PDEs, because the classical chain rule,2
conditions
which works rather eﬃciently in forward and backward propagation for NN, is not even valid in
fractional calculus. We could consider a fractional version of chain rule, but it is in the form of
an infinite series, and hence it is computationally prohibitive. To overcome this diﬃculty here
data.PDE or= off✓fractional(x, z) PINNs (fPINNs). Specifically, we proTraining dataModel
we propose an alternative method in the yformpose fPINNs for solving integral, diﬀerential, and integro-diﬀerential equations, and more(c)generally
Channelized, test 1.
(d)data.TimePDE
Channelized, test 2.
fPINNs can handle both fractional-order and integer-order operators. We employ the automatic
diﬀerentiation technique to analytically derive the integer-order derivatives of NN output, while
Figureexamples of the PCS under the mixed
residual loss. (a) and (b) are
Model.train(...,
we approximate the fractional derivatives numerically using standard methods
for8:
thePrediction
numerical
Model.predict(...)
Model.compile(...)
discretization of fractional operators;
in2Fig.
There arexfor
threethe PCS trained with 8192 samples of GRF
test1.
results
KLE512; (c) and (d) are 2
callbacks=...)
z1an illustrative schematic is shown
Latent space
Physicalspaceattractive features of fPINNs.

Multi-fidelity modeling for stochastic systems

Integrated software

test results for the PCS trained with 4096 samples of channelized fields.

(1) They have superior
accuracy
black-box
forcingy = f✓ (x,z), z for⇠ p(z), andy ⇠noisyp✓ (y|x,z) terms. When theforcing term is simply measured at scattered spatio-temporal points, interpolation has tobe1:performed
standardsurrogatesnumericalusing
methods
but this generative
may introduce
large We
interpolation
FigureBuilding using
probabilistic
conditionalmodels:assumeerrorsforsparsemeasurements.
In
contrast,
fPINNscanbypasstheforcing
terminterpolationthat each observed data pair in the physical space (x, y) is generated by a deterministic
and instead construct the equation residual at these measurement points. Numerical results
nonlineartransformation of the inputs x and a set of latent variables z, i.e. y = f✓ (x, z).
show that fPINNs can achieve higher solution accuracy for sparse measurements for bothThis construction generalizes the classical observation model used in regression, namely
forward and inverse problems. Additionally, the noise in the data can be naturally taken intowhich
can beregularization
viewed as a techniques,
simplified caseadditive noisey = faccount✓ (x) + ✏,byemployingsuch correspondingas L1 , L2 and Lto1an
regularization[32],

Lu, L., Meng, X., Mao, Z., & Karniadakis, G. E. (2019).
DeepXDE: A deep learning library for solving differential
Conditional deep surrogate models for stochastic, high-dimensional,
arXiv preprint arXiv:1907.04502.
and multi-fidelity systems. Computational Mechanics,
1-18. from GRF KLE512,equations.
ber of samples
and compare its predictive performanceFig. 5. Flowchart of DeepXDE corresponding to Procedure 3.1. The white boxes define thePDE problem and the training hyperparameters. The blue boxes combine the PDE problem andtraining hyperparameters in the white boxes. The orange boxes are the three steps (from right toVarying the number of training
inputs. We train the PCS with di↵erent numleft) to solve the PDE.

against the DDS in Fig. 9. From the figure, the relative L2 error decreases

sha1_base64="DR/WgnSLdvkNopgdt+amb4M190Y=">AAAEy3icpVNLb9NAEHabACW8WjhyGVGDUtRGdoV4CVBRLxyKKFIfSNnUWq/H8SrrtfGu26SOj/xBbhz5J6zjNH2kggNz8Xg+7zcz3+f1U8GVdpxfC4uN5o2bt5Zut+7cvXf/wfLKwwOV5BnDfZaIJPvmU4WCS9zXXAv8lmZIY1/goT/YrvDDY8wUT+SeHqXYi2lf8pAzqk3JW1n8SWTCZYBSE41D7YfFR6XyOK1g9bZsER/7XBZcY8xPsWyBCVK9gR16BdERalq2h2vwHnLzIJgqLgzzBcheh5MIMwR7htZghQFXQEHFSaIjCHPJqsYQYGgWCoBLsLvOutuzO3Ch9d6EDodGH3XOCh8cG1TOItAR1WCP59vBhjsGIvC7YZqCNlAZGJIxCTPKCpLSTHMqjgYwf7ycwTA8GpRk7BU7R4TLUI9KeHdOuQ5hkgEVAmQiNyT2jdrHaLbR2McM7IHdaRGUwbmsrZkNcIjAqAQVJSf1ImceUMH78nltwTMzr6S+oNPhgMRUR4yKYqf0itzzy3YNrF0est59czYqEBYkGubYrrHxChH59xzZX2b40t4+erEGZ/3/c5xazalA3vKq03EmAfOJO01WrWnsesu/SZCwPDbqM0GV6rpOqntF5TQTxhySK0wpG9A+dk0qaYyqV0wuXwlPTSWY+B0mxr1J9eKJgsZKjWLffFlJo65iVfE6rJvr8HWv4DLNNUpWNwpzATqB6iZDwDNkWoxMQlnGzazAImr+YW3u+6Uuflxp4l5VYD452Oy4Tsf9urm69XKqzpL12HpitS3XemVtWZ+sXWvfYo03Da8RNXjzc1M1T5vj+tPFhemZR9alaP74A9Xklis=</latexit>

sha1_base64="DR/WgnSLdvkNopgdt+amb4M190Y=">AAAEy3icpVNLb9NAEHabACW8WjhyGVGDUtRGdoV4CVBRLxyKKFIfSNnUWq/H8SrrtfGu26SOj/xBbhz5J6zjNH2kggNz8Xg+7zcz3+f1U8GVdpxfC4uN5o2bt5Zut+7cvXf/wfLKwwOV5BnDfZaIJPvmU4WCS9zXXAv8lmZIY1/goT/YrvDDY8wUT+SeHqXYi2lf8pAzqk3JW1n8SWTCZYBSE41D7YfFR6XyOK1g9bZsER/7XBZcY8xPsWyBCVK9gR16BdERalq2h2vwHnLzIJgqLgzzBcheh5MIMwR7htZghQFXQEHFSaIjCHPJqsYQYGgWCoBLsLvOutuzO3Ch9d6EDodGH3XOCh8cG1TOItAR1WCP59vBhjsGIvC7YZqCNlAZGJIxCTPKCpLSTHMqjgYwf7ycwTA8GpRk7BU7R4TLUI9KeHdOuQ5hkgEVAmQiNyT2jdrHaLbR2McM7IHdaRGUwbmsrZkNcIjAqAQVJSf1ImceUMH78nltwTMzr6S+oNPhgMRUR4yKYqf0itzzy3YNrF0est59czYqEBYkGubYrrHxChH59xzZX2b40t4+erEGZ/3/c5xazalA3vKq03EmAfOJO01WrWnsesu/SZCwPDbqM0GV6rpOqntF5TQTxhySK0wpG9A+dk0qaYyqV0wuXwlPTSWY+B0mxr1J9eKJgsZKjWLffFlJo65iVfE6rJvr8HWv4DLNNUpWNwpzATqB6iZDwDNkWoxMQlnGzazAImr+YW3u+6Uuflxp4l5VYD452Oy4Tsf9urm69XKqzpL12HpitS3XemVtWZ+sXWvfYo03Da8RNXjzc1M1T5vj+tPFhemZR9alaP74A9Xklis=</latexit><latexit

sha1_base64="DR/WgnSLdvkNopgdt+amb4M190Y=">AAAEy3icpVNLb9NAEHabACW8WjhyGVGDUtRGdoV4CVBRLxyKKFIfSNnUWq/H8SrrtfGu26SOj/xBbhz5J6zjNH2kggNz8Xg+7zcz3+f1U8GVdpxfC4uN5o2bt5Zut+7cvXf/wfLKwwOV5BnDfZaIJPvmU4WCS9zXXAv8lmZIY1/goT/YrvDDY8wUT+SeHqXYi2lf8pAzqk3JW1n8SWTCZYBSE41D7YfFR6XyOK1g9bZsER/7XBZcY8xPsWyBCVK9gR16BdERalq2h2vwHnLzIJgqLgzzBcheh5MIMwR7htZghQFXQEHFSaIjCHPJqsYQYGgWCoBLsLvOutuzO3Ch9d6EDodGH3XOCh8cG1TOItAR1WCP59vBhjsGIvC7YZqCNlAZGJIxCTPKCpLSTHMqjgYwf7ycwTA8GpRk7BU7R4TLUI9KeHdOuQ5hkgEVAmQiNyT2jdrHaLbR2McM7IHdaRGUwbmsrZkNcIjAqAQVJSf1ImceUMH78nltwTMzr6S+oNPhgMRUR4yKYqf0itzzy3YNrF0est59czYqEBYkGubYrrHxChH59xzZX2b40t4+erEGZ/3/c5xazalA3vKq03EmAfOJO01WrWnsesu/SZCwPDbqM0GV6rpOqntF5TQTxhySK0wpG9A+dk0qaYyqV0wuXwlPTSWY+B0mxr1J9eKJgsZKjWLffFlJo65iVfE6rJvr8HWv4DLNNUpWNwpzATqB6iZDwDNkWoxMQlnGzazAImr+YW3u+6Uuflxp4l5VYD452Oy4Tsf9urm69XKqzpL12HpitS3XemVtWZ+sXWvfYo03Da8RNXjzc1M1T5vj+tPFhemZR9alaP74A9Xklis=</latexit><latexit

sha1_base64="DR/WgnSLdvkNopgdt+amb4M190Y=">AAAEy3icpVNLb9NAEHabACW8WjhyGVGDUtRGdoV4CVBRLxyKKFIfSNnUWq/H8SrrtfGu26SOj/xBbhz5J6zjNH2kggNz8Xg+7zcz3+f1U8GVdpxfC4uN5o2bt5Zut+7cvXf/wfLKwwOV5BnDfZaIJPvmU4WCS9zXXAv8lmZIY1/goT/YrvDDY8wUT+SeHqXYi2lf8pAzqk3JW1n8SWTCZYBSE41D7YfFR6XyOK1g9bZsER/7XBZcY8xPsWyBCVK9gR16BdERalq2h2vwHnLzIJgqLgzzBcheh5MIMwR7htZghQFXQEHFSaIjCHPJqsYQYGgWCoBLsLvOutuzO3Ch9d6EDodGH3XOCh8cG1TOItAR1WCP59vBhjsGIvC7YZqCNlAZGJIxCTPKCpLSTHMqjgYwf7ycwTA8GpRk7BU7R4TLUI9KeHdOuQ5hkgEVAmQiNyT2jdrHaLbR2McM7IHdaRGUwbmsrZkNcIjAqAQVJSf1ImceUMH78nltwTMzr6S+oNPhgMRUR4yKYqf0itzzy3YNrF0est59czYqEBYkGubYrrHxChH59xzZX2b40t4+erEGZ/3/c5xazalA3vKq03EmAfOJO01WrWnsesu/SZCwPDbqM0GV6rpOqntF5TQTxhySK0wpG9A+dk0qaYyqV0wuXwlPTSWY+B0mxr1J9eKJgsZKjWLffFlJo65iVfE6rJvr8HWv4DLNNUpWNwpzATqB6iZDwDNkWoxMQlnGzazAImr+YW3u+6Uuflxp4l5VYD452Oy4Tsf9urm69XKqzpL12HpitS3XemVtWZ+sXWvfYo03Da8RNXjzc1M1T5vj+tPFhemZR9alaP74A9Xklis=</latexit><latexit<latexit

sha1_base64="BCHfkHhSUTJdYiX29FqHnbinqv0=">AAAEiHicbVNbb9MwFM7aAqPcOniEhyPWTS2MKukDA6RKExsSDxMa0m5S00aO46TWHDvYzpYqym/g9/HIn+AZp4m6dduJopycy/cdfcf2E0aVtu0/a41m68HDR+uP20+ePnv+orPx8lSJVGJyggUT8txHijDKyYmmmpHzRBIU+4yc+Rf7Zf7skkhFBT/W84RMYhRxGlKMtAl5G43fLheUB4Tr3NUk036Yf8tQnDDypYAjQZUSHMivdFEPlINzULRdn0SU54jRiL8r2lDaNoB7QJhGkPayPowgMp8dcM2TgWsax/aOMwHXbVfVddXsumpkw2IEGeeAeACFCTmDtkt4sKRaTguHREOqAAuuaEAkkAxhDUqwtJxUgQhBzwiEQsbQrclcRXlvP+t3B3A8IxyuCGDEDQwBQBAQkgAnqUTMfPSVkBfQDT3XwGhkALqgBaAkkSKjsdGDR1AhdwftG4MZZGBCKQhTjivVFET00vD583ulcw3aDCOWHxa9iq0P26PVhCeXqferiTz1/Ou+Wt+lGRw3lAjnTpH/MIVGgzT2cjpyiukiMA69vGouepnnT2kfPpRbWbiT6dDQ3QCQtwFkMa7SboKkpohNh8XSh8z8wQqBrAii2jUEqxv2Opv2wF4Y3HWc2tm0ajvyOn/dQOA0NtJjhpQaO3aiJ3k5AGbEnBiz3QThCxSRsXE5ioma5IubU8CWiQTlGTGvWd0ierMjR7FS89jItlUqrm7nyuB9uXGqw0+TnPIk1YTjiihMWXmCymsIAZUEazY3DsKSmlkBz5DRUZvLusLix6Umzm0F7jqnw4FjD5yfw829j7U669Zr663Vsxxr19qzvltH1omFG/+ab5pbze1Wu2W3dlufq9LGWt3zylqx1tf/3o5sWA==</latexit>

sha1_base64="BCHfkHhSUTJdYiX29FqHnbinqv0=">AAAEiHicbVNbb9MwFM7aAqPcOniEhyPWTS2MKukDA6RKExsSDxMa0m5S00aO46TWHDvYzpYqym/g9/HIn+AZp4m6dduJopycy/cdfcf2E0aVtu0/a41m68HDR+uP20+ePnv+orPx8lSJVGJyggUT8txHijDKyYmmmpHzRBIU+4yc+Rf7Zf7skkhFBT/W84RMYhRxGlKMtAl5G43fLheUB4Tr3NUk036Yf8tQnDDypYAjQZUSHMivdFEPlINzULRdn0SU54jRiL8r2lDaNoB7QJhGkPayPowgMp8dcM2TgWsax/aOMwHXbVfVddXsumpkw2IEGeeAeACFCTmDtkt4sKRaTguHREOqAAuuaEAkkAxhDUqwtJxUgQhBzwiEQsbQrclcRXlvP+t3B3A8IxyuCGDEDQwBQBAQkgAnqUTMfPSVkBfQDT3XwGhkALqgBaAkkSKjsdGDR1AhdwftG4MZZGBCKQhTjivVFET00vD583ulcw3aDCOWHxa9iq0P26PVhCeXqferiTz1/Ou+Wt+lGRw3lAjnTpH/MIVGgzT2cjpyiukiMA69vGouepnnT2kfPpRbWbiT6dDQ3QCQtwFkMa7SboKkpohNh8XSh8z8wQqBrAii2jUEqxv2Opv2wF4Y3HWc2tm0ajvyOn/dQOA0NtJjhpQaO3aiJ3k5AGbEnBiz3QThCxSRsXE5ioma5IubU8CWiQTlGTGvWd0ierMjR7FS89jItlUqrm7nyuB9uXGqw0+TnPIk1YTjiihMWXmCymsIAZUEazY3DsKSmlkBz5DRUZvLusLix6Umzm0F7jqnw4FjD5yfw829j7U669Zr663Vsxxr19qzvltH1omFG/+ab5pbze1Wu2W3dlufq9LGWt3zylqx1tf/3o5sWA==</latexit><latexit

sha1_base64="BCHfkHhSUTJdYiX29FqHnbinqv0=">AAAEiHicbVNbb9MwFM7aAqPcOniEhyPWTS2MKukDA6RKExsSDxMa0m5S00aO46TWHDvYzpYqym/g9/HIn+AZp4m6dduJopycy/cdfcf2E0aVtu0/a41m68HDR+uP20+ePnv+orPx8lSJVGJyggUT8txHijDKyYmmmpHzRBIU+4yc+Rf7Zf7skkhFBT/W84RMYhRxGlKMtAl5G43fLheUB4Tr3NUk036Yf8tQnDDypYAjQZUSHMivdFEPlINzULRdn0SU54jRiL8r2lDaNoB7QJhGkPayPowgMp8dcM2TgWsax/aOMwHXbVfVddXsumpkw2IEGeeAeACFCTmDtkt4sKRaTguHREOqAAuuaEAkkAxhDUqwtJxUgQhBzwiEQsbQrclcRXlvP+t3B3A8IxyuCGDEDQwBQBAQkgAnqUTMfPSVkBfQDT3XwGhkALqgBaAkkSKjsdGDR1AhdwftG4MZZGBCKQhTjivVFET00vD583ulcw3aDCOWHxa9iq0P26PVhCeXqferiTz1/Ou+Wt+lGRw3lAjnTpH/MIVGgzT2cjpyiukiMA69vGouepnnT2kfPpRbWbiT6dDQ3QCQtwFkMa7SboKkpohNh8XSh8z8wQqBrAii2jUEqxv2Opv2wF4Y3HWc2tm0ajvyOn/dQOA0NtJjhpQaO3aiJ3k5AGbEnBiz3QThCxSRsXE5ioma5IubU8CWiQTlGTGvWd0ierMjR7FS89jItlUqrm7nyuB9uXGqw0+TnPIk1YTjiihMWXmCymsIAZUEazY3DsKSmlkBz5DRUZvLusLix6Umzm0F7jqnw4FjD5yfw829j7U669Zr663Vsxxr19qzvltH1omFG/+ab5pbze1Wu2W3dlufq9LGWt3zylqx1tf/3o5sWA==</latexit><latexit

sha1_base64="BCHfkHhSUTJdYiX29FqHnbinqv0=">AAAEiHicbVNbb9MwFM7aAqPcOniEhyPWTS2MKukDA6RKExsSDxMa0m5S00aO46TWHDvYzpYqym/g9/HIn+AZp4m6dduJopycy/cdfcf2E0aVtu0/a41m68HDR+uP20+ePnv+orPx8lSJVGJyggUT8txHijDKyYmmmpHzRBIU+4yc+Rf7Zf7skkhFBT/W84RMYhRxGlKMtAl5G43fLheUB4Tr3NUk036Yf8tQnDDypYAjQZUSHMivdFEPlINzULRdn0SU54jRiL8r2lDaNoB7QJhGkPayPowgMp8dcM2TgWsax/aOMwHXbVfVddXsumpkw2IEGeeAeACFCTmDtkt4sKRaTguHREOqAAuuaEAkkAxhDUqwtJxUgQhBzwiEQsbQrclcRXlvP+t3B3A8IxyuCGDEDQwBQBAQkgAnqUTMfPSVkBfQDT3XwGhkALqgBaAkkSKjsdGDR1AhdwftG4MZZGBCKQhTjivVFET00vD583ulcw3aDCOWHxa9iq0P26PVhCeXqferiTz1/Ou+Wt+lGRw3lAjnTpH/MIVGgzT2cjpyiukiMA69vGouepnnT2kfPpRbWbiT6dDQ3QCQtwFkMa7SboKkpohNh8XSh8z8wQqBrAii2jUEqxv2Opv2wF4Y3HWc2tm0ajvyOn/dQOA0NtJjhpQaO3aiJ3k5AGbEnBiz3QThCxSRsXE5ioma5IubU8CWiQTlGTGvWd0ierMjR7FS89jItlUqrm7nyuB9uXGqw0+TnPIk1YTjiihMWXmCymsIAZUEazY3DsKSmlkBz5DRUZvLusLix6Umzm0F7jqnw4FjD5yfw829j7U669Zr663Vsxxr19qzvltH1omFG/+ab5pbze1Wu2W3dlufq9LGWt3zylqx1tf/3o5sWA==</latexit><latexit<latexit

Gradient analysis for PINNs

Example: Poisson equation in 1Du(x) = g(x), x 2 [0, 1]

u(x) = h(x), x = 0 and x = 1.

Let us consider exact solutions of the form u(x) = sin(Cx). Then we can use adeep neural network f✓ (x) to approximating u(x).
The loss function is given byL(✓) = Lr (✓) + Lub (✓)
N

b
1 X[f✓ (xib )=Nb i=1r
2X
1@h(xib )]2 +[ 2 f✓ (xir )Nr i=1 @x• There exists ✏ > 0 such that |✏✓ (x)non-negative integer k.
Nkr✓ Lub (✓)kL1  2✏ · kr✓ ✏✓ (x)kL1

Wang, S.,Teng,Y., & Perdikaris, P. (2020). Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv
preprint arXiv:2001.04536.

kr✓ Lr (✓)kL1  O(C 4 ) · ✏ · kr✓ ✏✓ (x)kL1
g(xir )]2 .

Assumptions:• f✓ (x) = u(x)✏✓ (x), where ✏✓ (x) is a smooth function defined in [0, 1].
@ k ✏✓ (x)1|  ✏ and k @xk kL1 < ✏, for all

We can show that

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit>

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="Q0kk8KfurYuDN297vQQjiRRHsF8=">AAAF3HicvVTNT9RAFC+wq7h+gR69vMhiFlk22z2oISHBqAkxhmDCV8KwzXT6ujthOi2dKVBKb96MV/83j8Z/xGm3EBZW48l32H19n7/5vTfjRoIr3e3+mJqeqdXv3J2917j/4OGjx3PzT3ZVmMQMd1gownjfpQoFl7ijuRa4H8VIA1fgnnv0rvDvnWCseCi3dRrhYUAHkvucUW1MzvzMLyJDLj2UGojGM+362YczGkQCV3PYQBEMQ6HPAY+TMgO4hN77Ngxi6vEiyUPFiv8k8qjG1QZxccBlRgUfyJd5A4wQPURNnUwu2zmsXX5KWAFiFCCSuoI6IzOQgOohoyL7lLcuI5dM1q20UohAX7f+XMKJrxVZ/ktcljju9YYk5oOhXmoQlN7VYd5GkUi5HIBCFkoPwtjDGLZpalgGPIuoLHgGHYKpAyJUCvxEspK35u2DLTWBauOomjbhFIFRCWoYnpoKVE8i88XE05XcLhXUTPbmBYUj2sjFPzFu4pxevwetFbsgzo8py+w860FeTU0lgZPxNTvvb5oxmI3zqMP7MaTmtweGQXID7BjFEwHfiPhvoMu+l8Db4zP3zWhVGCA0M+L6kBYb3Eoduw3EC7WCdupsmkEqczuUXy5Hk1xUoRWatbJvVX/NbrbhdIixqXiJwC72+BiuEPVG34SVHcZ9m2ZpTC7yAcoTKhJUEPqmVMlOvzeBn2bHmVvodrqlwG3FrpQFq5ItZ+6nORxLAnOxmaBKHdjdSB9mNNacCcwbJFEYUXZEB3hgVEkDVIdZ+RrlsGgsHhS8+aF5GErr9YyMBkqlgWF8sQCpbvoK4yTfQaL9N4cZl1GiUbJRIz8R5X0zTxt4PEamRWoUymJusAIbUrMD2jyAY13cIDec2DcZuK3s9jp2t2N/7i6sv6rYmbWeWc+tlmVbr611a8PasnYsVvtYi2pp7bzer3+pf61/G4VOT1U5T60xqX//DSbo7Wk=</latexit><latexit

sha1_base64="HeGDDKpiR++nnynWRGPZH19Vv6I=">AAAF0XicvZTfT9NAHMALbIoTFXz15RsZZpOxrH3RkJBo1IQYQzABJOG25nr9drtwvZbeFSilb74Z/0Afjf+I124QBtP45D1s335/3+e+d14suNK93o+5+YVa/d79xQeNh0uPHj9ZXlk6UFGaMNxnkYiSQ48qFFzivuZa4GGcIA09gV+843el/cspJopHck9nMfZDOpQ84Ixqo3JXFn4RGXHpo9RANJ5rL8g/nNMwFrhZwDaKcBQJfQF4klYRwCU47zswTKjPyyAfFSv/09inGjcbxMMhlzkVfChfFg0wi+gRaurmct0uYOvqU8IGECMAkdQT1B2rgYRUjxgV+aeideXZNlF3wqpFBAa69ecUbnIjyfpf/PLU9W4WJAkfjnS7QVD615t5G8ci43IIClkkfYgSHxPYo5mhDHgeU1lyBh2ByQMiUgqCVLKKW/PuxtpNoNoYJkWbcIbAqAQ1is5MBqpnwXwxc3cV23aJZra1KBGOsZHLfyJu/Fxn4EBrwy7BBQlluV3kDhSTU1Np6OZ8yy4GO+YYzMT51OWDBDLz64AhSG41O4V4ZsO3PP5b01Xdq8Y702cemKNVUYjQzIkXQFZOcCtz7Q4QP9IKOpm7Yw5Smduhgmo4muRy4jrpZquqO8m/ZTc7cDbCxGS86sAu5/gErjtyxt+EVRWmbTtmaEws8iHKUypSVBAFJlVFZ+DM4NPsusurvW6vWnBXsCfCqjVZu+7yT7M5lobmYjNBlTqye7Hu5zTRnAksGiRVGFN2TId4ZERJQ1T9vHqNClgzGh9KbkFkHoZKezMip6FSWWiIr5VNqtu2UjnLdpTq4HU/5zJONUo2LhSkorpv5mkDnyfItMiMQFnCTa/ARtTMgDYP4FQVLywME/s2gbvCgdO1e137c89atJ5Zz62WZVuvrDfWtrVr7Vus9rEW17LaRX1Q/1r/NqY3PzfB+NSaWvXvvwHV/Owu</latexit><latexit

sha1_base64="HeGDDKpiR++nnynWRGPZH19Vv6I=">AAAF0XicvZTfT9NAHMALbIoTFXz15RsZZpOxrH3RkJBo1IQYQzABJOG25nr9drtwvZbeFSilb74Z/0Afjf+I124QBtP45D1s335/3+e+d14suNK93o+5+YVa/d79xQeNh0uPHj9ZXlk6UFGaMNxnkYiSQ48qFFzivuZa4GGcIA09gV+843el/cspJopHck9nMfZDOpQ84Ixqo3JXFn4RGXHpo9RANJ5rL8g/nNMwFrhZwDaKcBQJfQF4klYRwCU47zswTKjPyyAfFSv/09inGjcbxMMhlzkVfChfFg0wi+gRaurmct0uYOvqU8IGECMAkdQT1B2rgYRUjxgV+aeideXZNlF3wqpFBAa69ecUbnIjyfpf/PLU9W4WJAkfjnS7QVD615t5G8ci43IIClkkfYgSHxPYo5mhDHgeU1lyBh2ByQMiUgqCVLKKW/PuxtpNoNoYJkWbcIbAqAQ1is5MBqpnwXwxc3cV23aJZra1KBGOsZHLfyJu/Fxn4EBrwy7BBQlluV3kDhSTU1Np6OZ8yy4GO+YYzMT51OWDBDLz64AhSG41O4V4ZsO3PP5b01Xdq8Y702cemKNVUYjQzIkXQFZOcCtz7Q4QP9IKOpm7Yw5Smduhgmo4muRy4jrpZquqO8m/ZTc7cDbCxGS86sAu5/gErjtyxt+EVRWmbTtmaEws8iHKUypSVBAFJlVFZ+DM4NPsusurvW6vWnBXsCfCqjVZu+7yT7M5lobmYjNBlTqye7Hu5zTRnAksGiRVGFN2TId4ZERJQ1T9vHqNClgzGh9KbkFkHoZKezMip6FSWWiIr5VNqtu2UjnLdpTq4HU/5zJONUo2LhSkorpv5mkDnyfItMiMQFnCTa/ARtTMgDYP4FQVLywME/s2gbvCgdO1e137c89atJ5Zz62WZVuvrDfWtrVr7Vus9rEW17LaRX1Q/1r/NqY3PzfB+NSaWvXvvwHV/Owu</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

sha1_base64="U9ccMZ4StGYa602dbJm0dXZihrs=">AAAF3HicvVRLb9NAEHZpAiW8WjhyGdEUpRCq2AdAlSoVARJCqAKpL6nbWOv1OFmxXrvedVvX9Y0b4sp/44j4I6wdp2qagDgxh2Q8z2+/mV0vFlzpXu/H3LX5RvP6jYWbrVu379y9t7h0f1dFacJwh0UiSvY9qlBwiTuaa4H7cYI09ATueZ9fl/69Y0wUj+S2zmI8DOlA8oAzqo3JXZr/RWTEpY9SA9F4qr0gf3tKw1jgegHvUITDSOgzwKO0ygAuwXnThUFCfV4m+ahY+Z/GPtW43iIeDrjMqeAD+aRogRGih6ipm8undgEb408Jz4AYBYiknqDuyAwkpHrIqMg/FJ1x5KrJmkqrhAgMdOfPJdzkUpGnf4nLU9e73JAkfDDUqy2C0r84zKs4FhmXA1DIIulDlPiYwDbNDMuApzGVJc+gIzB1QERKQZBKVvHWnj7YahuoNo66aRtOEBiVoIbRialA9SwyH888XcXtaknNbG9RUjiijZz/E+MmznX6DnSe2SVxQUJZbhe5A0U9NZWGbs437KK/ZcZgNs6nLu8nkJlfBwyD5ArYCYpnAr4S8d9AV33HwLuTMw/MaFUUIrRz4gWQlRvcyVy7C8SPtIJu5m6ZQSpzO1RQLUebnNehNZqNqm9df8Nud+FkiImpOEZgl3t8BBeInNE3YVWHSd+WWRqTi3yA8piKFBVEgSlVsdN3ZvDTXnMXl3trvUpgWrFrZdmq5aO7+NMcjqWhudhMUKUO7F6sD3OaaM4EFi2SKowp+0wHeGBUSUNUh3n1GhWwYiw+lLwFkXkYKuvljJyGSmWhYXylBKmu+krjLN9BqoOXhzmXcapRslGjIBXVfTNPG/g8QaZFZhTKEm6wAhtSswPaPIATXbywMJzYVxmYVnadNbu3Zn9yljef1+wsWA+tR1bHsq0X1qb1zvpo7Vis8b4RN7LGWbPf/NL82vw2Cr02V+c8sCak+f03J4jtaw==</latexit><latexit

for some y = (y1 , . . . , yN ) satisfying kyk22 =2
N are eigenvalues of r✓ L.
<latexit

Stiffness in the gradient flow dynamics

Example: Helmholtz equation in 2D, gradient descent update:✓n+1 = ✓n

Lr (✓n+1 )Lub (✓n+1 )⌘r✓ L(✓n ) = ✓nLarge Hessian eigenvalues indicate
stiffness in the gradient flow dynamics⌘ (r✓ Lr (✓n ) + r✓ Lub (✓n ))Applying second order Taylor expansion to the loss function L(✓) at ✓n we canshow thatN
X
1
Lr (✓n ) = ⌘kr✓ L(✓n )k22 ( 1 + ⌘
2 i=1
r 2
i yi )NX
1
Lub (✓n ) = ⌘kr✓ L(✓n )k22 ( 1 + ⌘
2 i=1

P 2
yi = 1, whereub 2
i yi ),1 
2  ··· 

Even full-batch gradient descent can get trappedin limit cycles and does not guarantee amonotonic decrease of the loss.

aplace operator
andpathologies
in physics-informed
neural networks

Gradient
pathologiesq(x,in y)physics-informed
neural networks=r the Helmholtz
equation:u + k u = q(x, y) (x, y) 2 [ 1, 1]

sha1_base64="WR9eDXVnr+RxrfJ6IzzKc6++bb0=">AAAClnicdVFRa9RAEN7EqrWteuqL4MviUagKR1ILFkEp1tI++HCFXlu4HGGymfSWbjZhd1Y8Qv6K/8tH/4mb61Hanh1Y+Hbmm292v8lqJS1F0Z8gfLDy8NHj1Sdr6xtPnz3vvXh5aitnBI5EpSpznoFFJTWOSJLC89oglJnCs+xyv6uf/URjZaVPaFbjpIQLLQspgHwq7f1OSqCpANX8aLcSmiLBu89feKK8RA5pzBOnczSZAYHNDW7amGu+vySEv6gZfj/gBq3MHai25R+uZbbvl3Fptiz0bd/yQlLbpr1+NIjmwZdBvAB9tohh2vub5JVwJWoSCqwdx1FNkwYMSaGwXUucxRrEJVzg2EMNJdpJM/ex5Zs+k/OiMv5o4vPszY4GSmtnZeaZ3Rfs3VqX/F9t7KjYnTRS145Qi6tBhVOcKt4thefSoCA18wCEkf6tXEzBm0V+dbemZGXnSXzXgWVwuj2Io0F8vNPf2124s8resLdsi8XsE9tjR2zIRkwEK8H74GOwE74Ov4YH4eEVNQwWPa/YrQiH/wBpc8zB</latexit>

sha1_base64="WR9eDXVnr+RxrfJ6IzzKc6++bb0=">AAAClnicdVFRa9RAEN7EqrWteuqL4MviUagKR1ILFkEp1tI++HCFXlu4HGGymfSWbjZhd1Y8Qv6K/8tH/4mb61Hanh1Y+Hbmm292v8lqJS1F0Z8gfLDy8NHj1Sdr6xtPnz3vvXh5aitnBI5EpSpznoFFJTWOSJLC89oglJnCs+xyv6uf/URjZaVPaFbjpIQLLQspgHwq7f1OSqCpANX8aLcSmiLBu89feKK8RA5pzBOnczSZAYHNDW7amGu+vySEv6gZfj/gBq3MHai25R+uZbbvl3Fptiz0bd/yQlLbpr1+NIjmwZdBvAB9tohh2vub5JVwJWoSCqwdx1FNkwYMSaGwXUucxRrEJVzg2EMNJdpJM/ex5Zs+k/OiMv5o4vPszY4GSmtnZeaZ3Rfs3VqX/F9t7KjYnTRS145Qi6tBhVOcKt4thefSoCA18wCEkf6tXEzBm0V+dbemZGXnSXzXgWVwuj2Io0F8vNPf2124s8resLdsi8XsE9tjR2zIRkwEK8H74GOwE74Ov4YH4eEVNQwWPa/YrQiH/wBpc8zB</latexit><latexit

sha1_base64="WR9eDXVnr+RxrfJ6IzzKc6++bb0=">AAAClnicdVFRa9RAEN7EqrWteuqL4MviUagKR1ILFkEp1tI++HCFXlu4HGGymfSWbjZhd1Y8Qv6K/8tH/4mb61Hanh1Y+Hbmm292v8lqJS1F0Z8gfLDy8NHj1Sdr6xtPnz3vvXh5aitnBI5EpSpznoFFJTWOSJLC89oglJnCs+xyv6uf/URjZaVPaFbjpIQLLQspgHwq7f1OSqCpANX8aLcSmiLBu89feKK8RA5pzBOnczSZAYHNDW7amGu+vySEv6gZfj/gBq3MHai25R+uZbbvl3Fptiz0bd/yQlLbpr1+NIjmwZdBvAB9tohh2vub5JVwJWoSCqwdx1FNkwYMSaGwXUucxRrEJVzg2EMNJdpJM/ex5Zs+k/OiMv5o4vPszY4GSmtnZeaZ3Rfs3VqX/F9t7KjYnTRS145Qi6tBhVOcKt4thefSoCA18wCEkf6tXEzBm0V+dbemZGXnSXzXgWVwuj2Io0F8vNPf2124s8resLdsi8XsE9tjR2zIRkwEK8H74GOwE74Ov4YH4eEVNQwWPa/YrQiH/wBpc8zB</latexit><latexitu(x, y) = (x + y) sin(⇡x) sin(6⇡y)

Loss function:
A PREPRINT - O CTOBER 27, 2019 (5L(✓):= 1 - O CTOBER
Lr (✓) 27, +A PREPRINT2019
2 Lub (✓)
| {z }| {z }(6sha1_base64="WR9eDXVnr+RxrfJ6IzzKc6++bb0=">AAAClnicdVFRa9RAEN7EqrWteuqL4MviUagKR1ILFkEp1tI++HCFXlu4HGGymfSWbjZhd1Y8Qv6K/8tH/4mb61Hanh1Y+Hbmm292v8lqJS1F0Z8gfLDy8NHj1Sdr6xtPnz3vvXh5aitnBI5EpSpznoFFJTWOSJLC89oglJnCs+xyv6uf/URjZaVPaFbjpIQLLQspgHwq7f1OSqCpANX8aLcSmiLBu89feKK8RA5pzBOnczSZAYHNDW7amGu+vySEv6gZfj/gBq3MHai25R+uZbbvl3Fptiz0bd/yQlLbpr1+NIjmwZdBvAB9tohh2vub5JVwJWoSCqwdx1FNkwYMSaGwXUucxRrEJVzg2EMNJdpJM/ex5Zs+k/OiMv5o4vPszY4GSmtnZeaZ3Rfs3VqX/F9t7KjYnTRS145Qi6tBhVOcKt4thefSoCA18wCEkf6tXEzBm0V+dbemZGXnSXzXgWVwuj2Io0F8vNPf2124s8resLdsi8XsE9tjR2zIRkwEK8H74GOwE74Ov4YH4eEVNQwWPa/YrQiH/wBpc8zB</latexit><latexit(2D Helmholtz
aplaceoperator andequation):

2<latexit

A simplebenchmarkanalyticalsolutionisoundary function h(x, y) from the exact
solution.
q(x,
y) =

PDE residualBCs fitnding meansquareisloss isanalyticalsolution
=+
My)
SE
SEf
b + M sin(6⇡y)
u(x,M
y)SE
= (x
sin(⇡x)(7(6

oundaryy) fromthex)exact
solution.
to learnfunction
the latenth(x,solutionu(t,of theHelmholtz equation (4) by training the network using throrlossmeanof (5).
And lossthen iswe choose to represent the latent solution using a 5-layer feed forward neura
nding
square0 neurons per layer and a hyperbolic tangent activation function. From the figure, we find that ou
M SE
SEb + M3SE
(7
3: 1.565970e-02
hiddenf layers 50 units
not learn the latent solution u(xx=correctly.
1 ,Figure2)MPrediction of a fully connected
4-layer deep physics-informed neural network (0.5% relative error)
Figure 3: 1.565970e-02 3 hidden layers 50 units

ason why this model collapses, inspired by the paper [understanding the difficulty of learning a fee
to learn the latent solution u(t, x) of the Helmholtz equation (4) by training the network using the
etwork], we are trying to investigate the gradients of weights during the training. Rather than trackin
ror loss of (5). And then we choose to represent the latent solution using a 5-layer feed forward neura
he mean squared loss function M SE, we track the gradients of M SEb and M SEf with respect to th
0 neurons per layer and a hyperbolic tangent activation function. From the figure, we find that ou
ayer respectively. From the figure, we find the gradients of M SEb in each layer is much less than th
not learn the latent solution u(x1 , x2 ) correctly.
Ef in each layer.
eason why this model collapses, inspired by the paper [understanding the difficulty of learning a feeda partial differential equation, if we don’t give any other restriction such as boundary condition oetwork], we are trying to investigate the gradients of weights during the training. Rather than tracking
the equation may have infinitely many solutions. So if the gradients of M SE is very small, thehe mean squared loss function M SE, we track the gradients of M SEb and M SEbf with respect to themay have some trouble in fitting the boudary conditions. While the gradients of M SE is large, thayer respectively. From the figure, we find the gradients of M SEb in each layer is muchfless than theanlearn
any solutions that satisfy the equation. As a result, our PINN model may not learn thEfeasilyin eachlayer.
Histograms of back-propagated gradients∇θ ℒiterations
we expected.
Figure 4: 40000
ub(θ), ∇θ ℒr(θ) at each hidden layersha1_base64="kN7yos2tYlGg5rAYgg97N26tOt4=">AAADHnicfZJLaxRBEMd7xleMr40evTQugYiwzCQBgyAEE0HBwwpuEthZhpqemmyTngfd1eIyzDfwQ/gZvOrZm3jVo9/Ens0ashlNQcO/6/Hr6upOKiUNBcEvz79y9dr1Gys3V2/dvnP3Xm/t/oEprRY4EqUq9VECBpUscESSFB5VGiFPFB4mJ3tt/PA9aiPL4h3NKpzkcFzITAog54rXvPUoB5oKUPWbZiOiKRI8fvacR8oxUohDHtkiRZ1oEFify41re5bvNhHhB6r3gYBnkpqGPzlDbP4fobuI4f5LrtHI1IJawmxd0kkcdEGv90ynle3LGEmX8eIvI+71g0EwN94V4UL02cKGce93lJbC5liQUGDMOAwqmtSgSQqFzWpkDVYgTuAYx04WkKOZ1PP3bPi686Q8K7VbBfG593xFDbkxszxxme0VzMVY6/xXbGwp25nUsqgsYSFOD8qs4lTy9nPwVGoUpGZOgNDS9crFFNywyH2hpVOSvJ1JeHECXXGwOQiDQfh2u7+7s5jOCnvIHrENFrKnbJe9YkM2YsL76H32vnhf/U/+N/+7/+M01fcWNQ/Ykvk//wAvQwR0</latexit>

sha1_base64="kN7yos2tYlGg5rAYgg97N26tOt4=">AAADHnicfZJLaxRBEMd7xleMr40evTQugYiwzCQBgyAEE0HBwwpuEthZhpqemmyTngfd1eIyzDfwQ/gZvOrZm3jVo9/Ens0ashlNQcO/6/Hr6upOKiUNBcEvz79y9dr1Gys3V2/dvnP3Xm/t/oEprRY4EqUq9VECBpUscESSFB5VGiFPFB4mJ3tt/PA9aiPL4h3NKpzkcFzITAog54rXvPUoB5oKUPWbZiOiKRI8fvacR8oxUohDHtkiRZ1oEFify41re5bvNhHhB6r3gYBnkpqGPzlDbP4fobuI4f5LrtHI1IJawmxd0kkcdEGv90ynle3LGEmX8eIvI+71g0EwN94V4UL02cKGce93lJbC5liQUGDMOAwqmtSgSQqFzWpkDVYgTuAYx04WkKOZ1PP3bPi686Q8K7VbBfG593xFDbkxszxxme0VzMVY6/xXbGwp25nUsqgsYSFOD8qs4lTy9nPwVGoUpGZOgNDS9crFFNywyH2hpVOSvJ1JeHECXXGwOQiDQfh2u7+7s5jOCnvIHrENFrKnbJe9YkM2YsL76H32vnhf/U/+N/+7/+M01fcWNQ/Ykvk//wAvQwR0</latexit><latexit

sha1_base64="kN7yos2tYlGg5rAYgg97N26tOt4=">AAADHnicfZJLaxRBEMd7xleMr40evTQugYiwzCQBgyAEE0HBwwpuEthZhpqemmyTngfd1eIyzDfwQ/gZvOrZm3jVo9/Ens0ashlNQcO/6/Hr6upOKiUNBcEvz79y9dr1Gys3V2/dvnP3Xm/t/oEprRY4EqUq9VECBpUscESSFB5VGiFPFB4mJ3tt/PA9aiPL4h3NKpzkcFzITAog54rXvPUoB5oKUPWbZiOiKRI8fvacR8oxUohDHtkiRZ1oEFify41re5bvNhHhB6r3gYBnkpqGPzlDbP4fobuI4f5LrtHI1IJawmxd0kkcdEGv90ynle3LGEmX8eIvI+71g0EwN94V4UL02cKGce93lJbC5liQUGDMOAwqmtSgSQqFzWpkDVYgTuAYx04WkKOZ1PP3bPi686Q8K7VbBfG593xFDbkxszxxme0VzMVY6/xXbGwp25nUsqgsYSFOD8qs4lTy9nPwVGoUpGZOgNDS9crFFNywyH2hpVOSvJ1JeHECXXGwOQiDQfh2u7+7s5jOCnvIHrENFrKnbJe9YkM2YsL76H32vnhf/U/+N/+7/+M01fcWNQ/Ykvk//wAvQwR0</latexit><latexit

sha1_base64="kN7yos2tYlGg5rAYgg97N26tOt4=">AAADHnicfZJLaxRBEMd7xleMr40evTQugYiwzCQBgyAEE0HBwwpuEthZhpqemmyTngfd1eIyzDfwQ/gZvOrZm3jVo9/Ens0ashlNQcO/6/Hr6upOKiUNBcEvz79y9dr1Gys3V2/dvnP3Xm/t/oEprRY4EqUq9VECBpUscESSFB5VGiFPFB4mJ3tt/PA9aiPL4h3NKpzkcFzITAog54rXvPUoB5oKUPWbZiOiKRI8fvacR8oxUohDHtkiRZ1oEFify41re5bvNhHhB6r3gYBnkpqGPzlDbP4fobuI4f5LrtHI1IJawmxd0kkcdEGv90ynle3LGEmX8eIvI+71g0EwN94V4UL02cKGce93lJbC5liQUGDMOAwqmtSgSQqFzWpkDVYgTuAYx04WkKOZ1PP3bPi686Q8K7VbBfG593xFDbkxszxxme0VzMVY6/xXbGwp25nUsqgsYSFOD8qs4lTy9nPwVGoUpGZOgNDS9crFFNywyH2hpVOSvJ1JeHECXXGwOQiDQfh2u7+7s5jOCnvIHrENFrKnbJe9YkM2YsL76H32vnhf/U/+N/+7/+M01fcWNQ/Ykvk//wAvQwR0</latexit><latexit<latexit…but how to choose the weights/learning rates?

L(✓) :=
1 Lu (✓) + 2

Data fit| {z }
PDE residual

Lr (✓)| {z }+ 3 Lu0 (✓) + 4 Lub (✓)| {z }| {z }
ICs fitBCs fitAdaptive moment estimation
Algorithm 1: Adam, our proposed algorithm for stochastic optimization. See section 2 for details,
and for a slightly more efficient (but less clear) order of computation. gt2 indicates the elementwisesquare gt gt . Good default settings for the tested machine learning problems are ↵ = 0.001,
8
t
t
=
0.9,
=
0.999
and
✏
=
10
.
All
operations
onvectorsareelement-wise.
With
and
1
2
1
2
we denote 1 and 2 to the power t.
Require: ↵: Stepsize
Require: 1 , 2 2 [0, 1): Exponential decay rates for the moment estimates
Require: f (✓): Stochastic objective function with parameters ✓Require: ✓0 : Initial parameter vectorm0
0 (Initialize 1st moment vector)
v0
0 (Initialize 2nd moment vector)t
0 (Initialize timestep)
while ✓t not converged dot
t+1
gtr✓ ft (✓t 1 ) (Get gradients w.r.t. stochastic objective at timestep t)
mt
1 · mt 1 + (11 ) · gt (Update biased first moment estimate)2vt
2 · vt 1 + (12 ) · gt (Update biased second raw moment estimate)tmbt
mt /(11 ) (Compute bias-corrected first moment estimate)tvbt
vt /(1bias-corrected second raw moment estimate)2 ) (Computep✓t✓t 1 ↵ · mb t /( vbt + ✏) (Update parameters)
end while
return ✓t (Resulting parameters)…i.e. use the gradient statistics during training to adaptively adjust the learning rate.

automatically tune the i weights by utilizing the back-propagated gradient statistics during model training, such thatthe interplay between all terms in equation 37 is appropriately balanced.

A learning rate annealing algorithm for PINNs

Algorithm 1: Learning rate annealing for physics-informed neural networks
Consider a physics-informed neural network f✓ (x) with parameters ✓ and a loss function
L(✓) := Lr (✓) +

M
X
i=1

i Li (✓),

where Lr (✓) denotes the PDE residual loss, the Li (✓) correspond to data-fit terms (e.g., measurements, initial orboundary conditions, etc.), and i = 1, i = 1, . . . , M are free parameters used to balance the interplay between the
different loss terms. Then use S steps of a gradient descent algorithm to update the parameters ✓ as:for n = 1, . . . , S do(a) Compute ˆ i by
ˆ i = max✓ {|r✓ Lr (✓n )|} , i = 1, . . . , M,|r✓ i Li (✓n )|(40)

where |r✓ i Li (✓n )| denotes the mean of |r✓ i Li (✓n )| with respect to parameters ✓.
(b) Update the weights i using a moving average of the form↵) i + ↵ ˆ i , i = 1, . . . , M.

i = (1(41)(c) Update the parameters ✓ via gradient descent✓n+1 = ✓n⌘r✓ Lr (✓n )

⌘

M
X
i=1

i r✓ Li (✓n )

(42)

end
The recommended hyper-parameter values are: ⌘ = 10 3 and ↵ = 0.1.

Wang, S.,Teng,Y., & Perdikaris, P. (2020). Understanding and mitigating gradient pathologies in physics-informed neural networks. arXivˆ i byThe
proposedalgorithm
ischaracterizedbytwo
steps.
First,
wecomputeinstantaneous
valuesfortheconstants
preprint arXiv:2001.04536.

sha1_base64="XkLqpElICXtOAmoiFp5O+pHyz2c=">AAACeHicfVHLSiNBFK1uHd+PqOjGTTHBGUUM3SLoRgi4caHgwESFdKa5XamYwqrqpuq2EJqG+c1ZuvEjXFlJ2rfMhYLDeXAvp5JMCotB8M/zJya/TU3PzM7NLywuLddWVi9tmhvGWyyVqblOwHIpNG+hQMmvM8NBJZJfJbcnQ/3qjhsrUv0bBxnvKLjRoicYoKPi2t8I+xwhLvRuWNIfx5RWhKZ7NHKARhoSCfGYppEC7DOQxVkZm+1n786L2eYqLsRxWP45/09SvCbjWj1oBKOhn0FYgTqp5iKu3UfdlOWKa2QSrG2HQYadAgwKJnk5F+WWZ8Bu4Ya3HdSguO0Uo6ZKuuWYLu2lxj2NdMS+TRSgrB2oxDmH59qP2pD8Smvn2DvqFEJnOXLNxot6uaSY0mHttCsMZygHDgAzwt1KWR8MMHSf825LokrXSfixgc/gcr8RBo3w10G92azamSGb5DvZJiE5JE1ySi5IizDy4C14696G9+hT/6e/M7b6XpVZI+/G338COIS/Dg==</latexit>

sha1_base64="XkLqpElICXtOAmoiFp5O+pHyz2c=">AAACeHicfVHLSiNBFK1uHd+PqOjGTTHBGUUM3SLoRgi4caHgwESFdKa5XamYwqrqpuq2EJqG+c1ZuvEjXFlJ2rfMhYLDeXAvp5JMCotB8M/zJya/TU3PzM7NLywuLddWVi9tmhvGWyyVqblOwHIpNG+hQMmvM8NBJZJfJbcnQ/3qjhsrUv0bBxnvKLjRoicYoKPi2t8I+xwhLvRuWNIfx5RWhKZ7NHKARhoSCfGYppEC7DOQxVkZm+1n786L2eYqLsRxWP45/09SvCbjWj1oBKOhn0FYgTqp5iKu3UfdlOWKa2QSrG2HQYadAgwKJnk5F+WWZ8Bu4Ya3HdSguO0Uo6ZKuuWYLu2lxj2NdMS+TRSgrB2oxDmH59qP2pD8Smvn2DvqFEJnOXLNxot6uaSY0mHttCsMZygHDgAzwt1KWR8MMHSf825LokrXSfixgc/gcr8RBo3w10G92azamSGb5DvZJiE5JE1ySi5IizDy4C14696G9+hT/6e/M7b6XpVZI+/G338COIS/Dg==</latexit><latexit

sha1_base64="XkLqpElICXtOAmoiFp5O+pHyz2c=">AAACeHicfVHLSiNBFK1uHd+PqOjGTTHBGUUM3SLoRgi4caHgwESFdKa5XamYwqrqpuq2EJqG+c1ZuvEjXFlJ2rfMhYLDeXAvp5JMCotB8M/zJya/TU3PzM7NLywuLddWVi9tmhvGWyyVqblOwHIpNG+hQMmvM8NBJZJfJbcnQ/3qjhsrUv0bBxnvKLjRoicYoKPi2t8I+xwhLvRuWNIfx5RWhKZ7NHKARhoSCfGYppEC7DOQxVkZm+1n786L2eYqLsRxWP45/09SvCbjWj1oBKOhn0FYgTqp5iKu3UfdlOWKa2QSrG2HQYadAgwKJnk5F+WWZ8Bu4Ya3HdSguO0Uo6ZKuuWYLu2lxj2NdMS+TRSgrB2oxDmH59qP2pD8Smvn2DvqFEJnOXLNxot6uaSY0mHttCsMZygHDgAzwt1KWR8MMHSf825LokrXSfixgc/gcr8RBo3w10G92azamSGb5DvZJiE5JE1ySi5IizDy4C14696G9+hT/6e/M7b6XpVZI+/G338COIS/Dg==</latexit><latexit

Relative prediction error (L2 norm) averaged over 10 independent
Stiffnessin theflowbenchmark.
dynamics.
trialsfor the2DgradientHelmholtz
sha1_base64="XkLqpElICXtOAmoiFp5O+pHyz2c=">AAACeHicfVHLSiNBFK1uHd+PqOjGTTHBGUUM3SLoRgi4caHgwESFdKa5XamYwqrqpuq2EJqG+c1ZuvEjXFlJ2rfMhYLDeXAvp5JMCotB8M/zJya/TU3PzM7NLywuLddWVi9tmhvGWyyVqblOwHIpNG+hQMmvM8NBJZJfJbcnQ/3qjhsrUv0bBxnvKLjRoicYoKPi2t8I+xwhLvRuWNIfx5RWhKZ7NHKARhoSCfGYppEC7DOQxVkZm+1n786L2eYqLsRxWP45/09SvCbjWj1oBKOhn0FYgTqp5iKu3UfdlOWKa2QSrG2HQYadAgwKJnk5F+WWZ8Bu4Ya3HdSguO0Uo6ZKuuWYLu2lxj2NdMS+TRSgrB2oxDmH59qP2pD8Smvn2DvqFEJnOXLNxot6uaSY0mHttCsMZygHDgAzwt1KWR8MMHSf825LokrXSfixgc/gcr8RBo3w10G92azamSGb5DvZJiE5JE1ySi5IizDy4C14696G9+hT/6e/M7b6XpVZI+/G338COIS/Dg==</latexit><latexit

sha1_base64="WdiPXTitr9wg3BacjjicO/0NvfQ=">AAACZHicdVHPSxtBGJ3dqk1Tq9HQkyCDQbAHw64U9CIEvPTQQgQThWy6fDs7awZnZpeZb4Ww7D/prUcv/Rt67OSHoFEfDDze+x7fx5ukkMJiEPzx/A9r6xsfG5+anze/bG23dnaHNi8N4wOWy9zcJGC5FJoPUKDkN4XhoBLJr5O7i5l/fc+NFbm+wmnBxwputcgEA3RS3JJRZoBVaYQTjlBXKdb0nB5HGhIJ8UKlkQKcMJDVzzo2RwvxGz2mkS1VXInzsP79i74fEU+RuNUJusEc9DUJl6RDlujHrccozVmpuEYmwdpRGBQ4rsCgYJLXzai0vAB2B7d85KgGxe24mrdS00OnpDTLjXsa6Vx9nqhAWTtViZucHWtXvZn4ljcqMTsbV0IXJXLNFouyUlLM6aximgrDGcqpI8CMcLdSNgFXM7qPeLElUbXrJFxt4DUZnnTDoBtefu/0est2GmSPHJAjEpJT0iM/SJ8MCCMP5J/neb7319/02/7XxajvLTNt8gL+/n+febl9</latexit>

sha1_base64="WdiPXTitr9wg3BacjjicO/0NvfQ=">AAACZHicdVHPSxtBGJ3dqk1Tq9HQkyCDQbAHw64U9CIEvPTQQgQThWy6fDs7awZnZpeZb4Ww7D/prUcv/Rt67OSHoFEfDDze+x7fx5ukkMJiEPzx/A9r6xsfG5+anze/bG23dnaHNi8N4wOWy9zcJGC5FJoPUKDkN4XhoBLJr5O7i5l/fc+NFbm+wmnBxwputcgEA3RS3JJRZoBVaYQTjlBXKdb0nB5HGhIJ8UKlkQKcMJDVzzo2RwvxGz2mkS1VXInzsP79i74fEU+RuNUJusEc9DUJl6RDlujHrccozVmpuEYmwdpRGBQ4rsCgYJLXzai0vAB2B7d85KgGxe24mrdS00OnpDTLjXsa6Vx9nqhAWTtViZucHWtXvZn4ljcqMTsbV0IXJXLNFouyUlLM6aximgrDGcqpI8CMcLdSNgFXM7qPeLElUbXrJFxt4DUZnnTDoBtefu/0est2GmSPHJAjEpJT0iM/SJ8MCCMP5J/neb7319/02/7XxajvLTNt8gL+/n+febl9</latexit><latexit

sha1_base64="WdiPXTitr9wg3BacjjicO/0NvfQ=">AAACZHicdVHPSxtBGJ3dqk1Tq9HQkyCDQbAHw64U9CIEvPTQQgQThWy6fDs7awZnZpeZb4Ww7D/prUcv/Rt67OSHoFEfDDze+x7fx5ukkMJiEPzx/A9r6xsfG5+anze/bG23dnaHNi8N4wOWy9zcJGC5FJoPUKDkN4XhoBLJr5O7i5l/fc+NFbm+wmnBxwputcgEA3RS3JJRZoBVaYQTjlBXKdb0nB5HGhIJ8UKlkQKcMJDVzzo2RwvxGz2mkS1VXInzsP79i74fEU+RuNUJusEc9DUJl6RDlujHrccozVmpuEYmwdpRGBQ4rsCgYJLXzai0vAB2B7d85KgGxe24mrdS00OnpDTLjXsa6Vx9nqhAWTtViZucHWtXvZn4ljcqMTsbV0IXJXLNFouyUlLM6aximgrDGcqpI8CMcLdSNgFXM7qPeLElUbXrJFxt4DUZnnTDoBtefu/0est2GmSPHJAjEpJT0iM/SJ8MCCMP5J/neb7319/02/7XxajvLTNt8gL+/n+febl9</latexit><latexit

sha1_base64="WdiPXTitr9wg3BacjjicO/0NvfQ=">AAACZHicdVHPSxtBGJ3dqk1Tq9HQkyCDQbAHw64U9CIEvPTQQgQThWy6fDs7awZnZpeZb4Ww7D/prUcv/Rt67OSHoFEfDDze+x7fx5ukkMJiEPzx/A9r6xsfG5+anze/bG23dnaHNi8N4wOWy9zcJGC5FJoPUKDkN4XhoBLJr5O7i5l/fc+NFbm+wmnBxwputcgEA3RS3JJRZoBVaYQTjlBXKdb0nB5HGhIJ8UKlkQKcMJDVzzo2RwvxGz2mkS1VXInzsP79i74fEU+RuNUJusEc9DUJl6RDlujHrccozVmpuEYmwdpRGBQ4rsCgYJLXzai0vAB2B7d85KgGxe24mrdS00OnpDTLjXsa6Vx9nqhAWTtViZucHWtXvZn4ljcqMTsbV0IXJXLNFouyUlLM6aximgrDGcqpI8CMcLdSNgFXM7qPeLElUbXrJFxt4DUZnnTDoBtefu/0est2GmSPHJAjEpJT0iM/SJ8MCCMP5J/neb7319/02/7XxajvLTNt8gL+/n+febl9</latexit><latexit<latexit

3.25E-03
1.15E-01MXd✓ layers
30 units / 7 hidden
3.10E-01
7.86E-03= r✓ Lr (✓)
r✓ Li (✓)
dt layers
50 units / 7 hidden
1.98E-01
3.66E-03
i=1
MX
100 units / 7 hidden layers
8.14E-02
2.57E-03✓n+1 = ✓n ⌘r✓ Lr (✓n ) ⌘r✓ Li (✓n )i=1<latexit

sha1_base64="1HGZtmjbtzdRdyLbi+i+HT/d8uc=">AAADuHicbVJbb9MwFE5WLqPcNnjkxWJC2kSpknbAmFQxDSR4QKhM7CI1WXCck9Sa7US+oBXLv4/fsH+D2xVEth3Jysl3jr9z+Zw3jCodRRfhSufW7Tt3V+917z94+Ojx2vqTI1UbSeCQ1KyWJzlWwKiAQ001g5NGAuY5g+P87MM8fvwTpKK1+K5nDaQcV4KWlGDtoWztIuFYTwlm9ovbTPQUNN5CuyOUGFGAzCUmYJPSf2zs7NfMuEQZnjDKqVaZpaPYnS5gNElybo3L6Ksys5dEntBj5x7bSk8HLks0nGv7EWuMSqodenlzlYT59gvs/nV24CbXOLfSv3zj6UxRopCEyjAs6a/FZA5laxtRP1oYuu7ES2cjWNo4Ww9/J0VNDAehCcNKTeKo0anFUlPCwHUTo6DB5AxXMPGuwBxUahcaOPTCIwUqa+mP0GiB/n/DYq7UjOc+cz6XuhqbgzfFJkaXO6mlojEaBLksVBqGdI3mgqKCSiCazbyDiaS+V0Sm2O9Se9lbVXLemsESLAiw9lx4u8ENyJ6n15iN7BsqemiHCtfjWFZUjKK+/+umtoKag5azNqWRzKFuC5pPpOuaqXYmZlXtu53ygd9sNymg9C94sTU7BiEOoHD24NO+s/HrYS/qRTcl7TMDy6woinvR0J93set66eOrQl93jgb9eNgffNve2NtfPoLV4FnwPNgM4uBtsBd8DsbBYUDC9yGEIqw7u50fnapDL1NXwuWdp0HLOvIP9X86QQ==</latexit><latexit

Soft physics-informed
Systematic comparison
learning, a recap

Nu et. al., 2019)
M1: Baseline PINN model (Raissi
X
1
2
M2: PINN withL(✓)
the proposed
learning
annealing:=[ui rate
f✓ (x
)]
+
i

30 units / 3 hidden layers

Nu i=1|
Architecture
Data fit{z}

M1
Physics
regularization

2.44E-01

R[f✓ (x)]| {z }

1

100 units / 5 hidden layers
M2

3.98E-02

50 units/ 3 hidden layers that requires us to
1.06E-01
1.58E-02
An “unconventional”
regularizer/priorrevisit standard deeplearning practices:square residual, variational principle, Hamiltonian, etc.?)• loss functions100(e.g.,units /(e.g.,
3 hiddenlayers
9.07E-02
2.39E-03
Glorot,adaptive?)• network initialization
PDE solution
bounds?)• normalization30(e.g.,unitszero-mean/unit-variance,/ 5 hidden layers
2.47E-01
8.91E-03• optimization (e.g., Adam, adaptive learning rates, proximal algorithms, meta-learning?)
fully
connected, residual/recurrent/convolutional
layers,attention?)
50 units / 5(e.g.,hiddenlayers
1.40E-01
8.08E-03• network architectureAn improved neural architecture
1

1

2

2

U = φ(W "x + b ), V = φ(W "x + b )A PREPRINT - O CTOBE

H (1) = φ(W z,1 "x + bz,1 )Original
Adaptive
ADGM
Adaptive ADGM(k)z,k (k)z,k= φ(W3.98e-02
H +5.31e-02b ), k = 1, .2.56e-03..,L
3 hidden layers Z2.44e-01(k+1)
3 hidden layers H1.06e-01
1.81e-03= (11.58e-02− Z (k) ) "2.46e-02
U + Z (k) " V,k = 1, . . . , L/ 3 hidden layers 9.07e-02 2.39e-03 1.17e-02
1.28e-03
(L+1)
f
(x;
θ)
=
W
H
b
5 hidden layers 2.47e-01 8.91e-03 +4.12e-02
1.96e-03
5 hidden layers 1.40e-01 8.08e-03 1.97e-02
1.86e-03/ 5 hidden layers
1.15e-01
3.25e-03 1.08e-02
1.22e-03xU
7 hidden layers 3.10e-01 7.86e-03 3.17e-02
1.98e-03
7 hidden layers 1.98e-01 3.66e-03 2.37e-02
1.54e-03(2)...
xfθ (x)
H
H (1)
H (L)
/ 7 hidden layers
8.14e-02
2.57e-03
9.36e-03
1.40e-03xV
notes the data points anddenotes element-wise multiplication. The parameters are✓ = {W 1 , b1 , W 2 , b2 , (W z,l , bz,l )L
l=1 , W, b}r of units
in each layer is M andKey points:: RM ! RM is a nonlinear activation function.

• Account for multiplicative interactions of the inputs, similar to attention mechanisms.
ts • Residual connections improve resilience against vanishing gradient pathologies.

Systematic comparison
M1: Baseline PINN model (Raissi et. al., 2019)
M2: PINN with the proposed learning rate annealing
M3: PINN with the proposed neural architecture
M4: PINN with the proposed learning rate annealing and improved neural architecture
Architecture

M1

M2

M3

M4

30 units / 3 hidden layers

2.44E-01

3.98E-02

5.31E-02

2.56E-03

50 units / 3 hidden layers

1.06E-01

1.58E-02

2.46E-02

1.81E-03

100 units / 3 hidden layers

9.07E-02

2.39E-03

1.17E-02

1.28E-03

30 units / 5 hidden layers

2.47E-01

8.91E-03

4.12E-02

1.96E-03

50 units / 5 hidden layers

1.40E-01

8.08E-03

1.97E-02

1.86E-03

100 units / 5 hidden layers

1.15E-01

3.25E-03

1.08E-02

1.22E-03

30 units / 7 hidden layers

3.10E-01

7.86E-03

3.17E-02

1.98E-03

50 units / 7 hidden layers

1.98E-01

3.66E-03

2.37E-02

1.54E-03

100 units / 7 hidden layers

8.14E-02

2.57E-03

9.36E-03

1.40E-03

Relative prediction error (L2 norm) averaged over 10 independent
trials for the 2D Helmholtz benchmark.

mphasize the abilitywithof our
proposedmethods and neural network architecture to deal with different ty
the initialconditions
nearity in the governing partial differential equations, let us consider the Klein Gordon Equation along withu(x, 0) = g1 (x), x 2 ⌦conditions:utt + ↵uxx + u + uk Klein= f (x, t),Gordon(x, t) 2 ⌦ equation⇥ [0, T ]ditions

ut (x, 0) = g2 (x), x 2 ⌦utt + ↵uxx + u + uk = f (x, t), (x, t) 2 ⌦ ⇥ [0, T ]and Dirichlet boundary condition
u(x, 0) = g1 (x), x 2 ⌦the initial conditionsu( x, t) = h(x, t), (x, t) 2 @⌦ ⇥ [0, T ]ut (x, 0) = g2 (x),
x 0)
2⌦
u(x,
g1 (x), kx=22⌦when we have quadratic nonlinearity and kwhere ↵, , and k are known =constants,
dary condition
nonlinearity. f, g1 , g2 andh arefunctions,ut (x,
0) known= g2 (x),x 2 ⌦and the function u is unknown.

uThe= h(x, t), (x,t) 2 @⌦⇥ [0, T ] used in mathematical models for problems in ma
Klein-Gordon
equationis frequently( x, t)Dirichlet boundary conditionengineering,particularly
in quantum
field theoryand
andkrelativistic
quantummechanics.
k are known constants,
k = 2 whenwehavequadratic
nonlinearity
=
3
when
we
have
cubic
u( x, t) = h(x, t), (x, t) 2 @⌦ ⇥ [0, T ]g2 and h are known functions, and the function u is unknown.
3.3 Example(Lid-Driven
CavityFlow)quadratic nonlinearity and k = 3 when we hav
e ↵, , and k are known
constants,
k = 2 when
we haveequation is frequently used in mathematical models for problems in many fields of science andnearity.
g1 , g2 and
h are
known
functions, importantand the mechanics.
function
u is unknown.
ularly inf,
quantumfield
and relativistic
Thetheory
lid-driven
cavity is anquantum
benchmark problem within the field of computational flvalidating
computational
methods.
While theboundary
relatively
simple,the
Klein-Gordon equation
is frequently
used inmathematicalmodelsfor conditions
problems arein many
fieldsof scien
quite
interesting
andcomplex.
we consider
a 2Dmechanics.
lid driven cavity problem, which has t
d-Driven
Cavity Flow)neering,
particularly
in quantum
field
theoryand Here
relativistic
quantum
1(Ucomputational· )U + p =0 dynamics (CFD) forin (0, T ) ⇥ ⌦U@
ity is an important benchmark problem within
fluid
t the field+of
Example (Lid-Driven Cavity Flow)Re
ational methods. While the boundary conditions are relatively simple, the flow features created are
U = 0,problem, which has the following form:in (0, T ) ⇥ ⌦nd complex. Here we consider a 2D lid driven·cavity
id-driven cavity is an important benchmark
withinU (t,problemx, y) = (1,0) the field of computational fluid dynamics
on (0, T ) (CF⇥1
ating computational methods. While the boundary conditions are relatively simple, the flow features crea
+ (U · )U + p = 0(18)on (0, T ) ⇥@t U
U (t, x, y) = (0, 0) in (0, T ) ⇥ ⌦interestingReand complex. Here we consider a 2D lid driven cavity problem, which has the following form
U (0, x, y) = (0, 0)in (0, T ) ⇥ ⌦· U = 0,(19)in ⌦1(U1)· ⇥ )U
+ is ap 2D= 0square cavityin (0,T ) ⇥ ⌦time.
U
@t0)
where
⌦ =+(0,(0, 1)
andT >1 0 thesimulation
And U = (u,
U
(t,
x,
y)
=
(1,
on
(0,
T
)
⇥
(20)
Top: Imbalanced gradientsinadense,
5-layerdeep
physics-informed
neuralnetwork
lead
toRepressure respectively.
and the1 is the top boundary and 0 denotes the other three sides.
U
(t,
x,
y)
=
(0,
0)
on (0, T ) ⇥ 0 in (0, T ) ⇥ ⌦ (21)considerable prediction(6.7%).
· U =errors0,
U (0, x,y) = (0,0) x, y) = (1,in ⌦Upredictions(t,(0,annealingT ) ⇥ 1(22)Bottom:Accurate
can0)
be obtained using the proposed
learning onrate
and improved
9
neuralprediction
error:⇥ (0,1) isarchitecture
a 2D squarecavityT 0)> 0 thesimulation
time.0.1%).
And U = (u, v) andareTtheU (t,strategyx, y) and=(relative(0,
onp(0,) ⇥velocity0sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit>

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="QZQWJEytCgdXpKY6oAbZtXxasyA=">AAADLHicfVLLitRAFK3E19i+enTp5mJj06NOU1FQEYQBBV3p+OiZga6mqVTfzhRTVYmpikwT8iN+hN/gVtduRFy48TusTiLMQ/pCyOE8csOpijMlraP0RxCeOXvu/IW1i51Ll69cvdZdv75j0yIXOBKpSvO9mFtU0uDISadwL8uR61jhbnzwbKnvfsTcytS8d4sMJ5onRs6l4M5T0/XgAYsxkabkSibmTtUBP0zxGFWJH57Aq3cN1WexLouKiVnqmOGx4mXDVHAXGgIy2AQ2z7koo6p8ixV7jspxaHzwFCj0+8zhoSulgQrYa42Jl1m7oPlIveFoZnWo9g2Wr8Nqw/sH0T2gG7XWxtI69oJrzafRiiBdFaQdhmb2r6Rpt0eHtB44DaIW9Eg729PubzZLRaHROKG4teOIZm5S8txJobDqsMJixsUBT3DsoeEa7aSsT7eC256ZwTzN/WMc1OzRRMm1tQsde6fmbt+e1Jbk/7Rx4eaPJ77WrHBoRLNoXihwKSyvCsxkjsKphQdc5NL/K4h97s/X+Qt1bEusl51EJxs4DXbuDyM6jN7Q3tbDtp01cpPcIgMSkUdki7wk22RERPAp+BJ8Db6Fn8Pv4c/wV2MNgzZzgxyb8M9fZUP2vA==</latexit><latexit

sha1_base64="X6cVB9dyMeKJ4rK8MeMKzxAI0Vg=">AAADIXicfVLLihQxFE2Vr7Edtcetm4uDTY+PJqULRRAEBV3p+OiZgU7T3Erf7gmTpMpKSqYp6kf8CL/Bra7diLjwT0xXtTAPmQshh3PuyQ0nSXOtnOf8ZxSfO3/h4qW1y50r61evXe9urO+4rCwkDWWms2IvRUdaWRp65TXt5QWhSTXtpgfPl/ruJyqcyuwHv8hpbHBu1UxJ9IGabEQPRUpzZSvUam7v1B0IJTSmpCv6+ARev2+pnkhNVdZCTjMvLKYaq5ap4S60BORwH8SsQFkldfWOavGCtEdo++ApcOj1hKdDXykLNYg3huZBFqsB7SHNhKOes01NX3+5HdZbob+f3AO+1WgrW9bYXqIxOEnOMPKzjLwjyE7/hTTpbvIBbwpOg2QFNtmqtifdP2KaydKQ9VKjc6OE535cYeGV1FR3ROkoR3mAcxoFaNGQG1fN69ZwOzBTmGVFWNZDwx51VGicW5g0dBr0++6ktiT/p41KP3s8DrHmpScr20GzUoPPYPlVYKoKkl4vAkBZqHBXkPsY3teHD3VsSmqWmSQnEzgNdh4MEj5I3nK2xm6yW6zPEvaIPWOv2DYbMhl9jr5G36Lv8Zf4R/yrTS+OVjHeYMcq/v0XzY31NA==</latexit><latexit

sha1_base64="X6cVB9dyMeKJ4rK8MeMKzxAI0Vg=">AAADIXicfVLLihQxFE2Vr7Edtcetm4uDTY+PJqULRRAEBV3p+OiZgU7T3Erf7gmTpMpKSqYp6kf8CL/Bra7diLjwT0xXtTAPmQshh3PuyQ0nSXOtnOf8ZxSfO3/h4qW1y50r61evXe9urO+4rCwkDWWms2IvRUdaWRp65TXt5QWhSTXtpgfPl/ruJyqcyuwHv8hpbHBu1UxJ9IGabEQPRUpzZSvUam7v1B0IJTSmpCv6+ARev2+pnkhNVdZCTjMvLKYaq5ap4S60BORwH8SsQFkldfWOavGCtEdo++ApcOj1hKdDXykLNYg3huZBFqsB7SHNhKOes01NX3+5HdZbob+f3AO+1WgrW9bYXqIxOEnOMPKzjLwjyE7/hTTpbvIBbwpOg2QFNtmqtifdP2KaydKQ9VKjc6OE535cYeGV1FR3ROkoR3mAcxoFaNGQG1fN69ZwOzBTmGVFWNZDwx51VGicW5g0dBr0++6ktiT/p41KP3s8DrHmpScr20GzUoPPYPlVYKoKkl4vAkBZqHBXkPsY3teHD3VsSmqWmSQnEzgNdh4MEj5I3nK2xm6yW6zPEvaIPWOv2DYbMhl9jr5G36Lv8Zf4R/yrTS+OVjHeYMcq/v0XzY31NA==</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit<latexit

Flow in a lid-driven cavity

u · ru + rp
1
u=0
Re

r·u=0
u(x) = (1, 0)
u(x) = (0, 0)

Model M1

Model M2

in ⌦

in ⌦
on 1
on 0

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit>

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="QZQWJEytCgdXpKY6oAbZtXxasyA=">AAADLHicfVLLitRAFK3E19i+enTp5mJj06NOU1FQEYQBBV3p+OiZga6mqVTfzhRTVYmpikwT8iN+hN/gVtduRFy48TusTiLMQ/pCyOE8csOpijMlraP0RxCeOXvu/IW1i51Ll69cvdZdv75j0yIXOBKpSvO9mFtU0uDISadwL8uR61jhbnzwbKnvfsTcytS8d4sMJ5onRs6l4M5T0/XgAYsxkabkSibmTtUBP0zxGFWJH57Aq3cN1WexLouKiVnqmOGx4mXDVHAXGgIy2AQ2z7koo6p8ixV7jspxaHzwFCj0+8zhoSulgQrYa42Jl1m7oPlIveFoZnWo9g2Wr8Nqw/sH0T2gG7XWxtI69oJrzafRiiBdFaQdhmb2r6Rpt0eHtB44DaIW9Eg729PubzZLRaHROKG4teOIZm5S8txJobDqsMJixsUBT3DsoeEa7aSsT7eC256ZwTzN/WMc1OzRRMm1tQsde6fmbt+e1Jbk/7Rx4eaPJ77WrHBoRLNoXihwKSyvCsxkjsKphQdc5NL/K4h97s/X+Qt1bEusl51EJxs4DXbuDyM6jN7Q3tbDtp01cpPcIgMSkUdki7wk22RERPAp+BJ8Db6Fn8Pv4c/wV2MNgzZzgxyb8M9fZUP2vA==</latexit><latexit

sha1_base64="X6cVB9dyMeKJ4rK8MeMKzxAI0Vg=">AAADIXicfVLLihQxFE2Vr7Edtcetm4uDTY+PJqULRRAEBV3p+OiZgU7T3Erf7gmTpMpKSqYp6kf8CL/Bra7diLjwT0xXtTAPmQshh3PuyQ0nSXOtnOf8ZxSfO3/h4qW1y50r61evXe9urO+4rCwkDWWms2IvRUdaWRp65TXt5QWhSTXtpgfPl/ruJyqcyuwHv8hpbHBu1UxJ9IGabEQPRUpzZSvUam7v1B0IJTSmpCv6+ARev2+pnkhNVdZCTjMvLKYaq5ap4S60BORwH8SsQFkldfWOavGCtEdo++ApcOj1hKdDXykLNYg3huZBFqsB7SHNhKOes01NX3+5HdZbob+f3AO+1WgrW9bYXqIxOEnOMPKzjLwjyE7/hTTpbvIBbwpOg2QFNtmqtifdP2KaydKQ9VKjc6OE535cYeGV1FR3ROkoR3mAcxoFaNGQG1fN69ZwOzBTmGVFWNZDwx51VGicW5g0dBr0++6ktiT/p41KP3s8DrHmpScr20GzUoPPYPlVYKoKkl4vAkBZqHBXkPsY3teHD3VsSmqWmSQnEzgNdh4MEj5I3nK2xm6yW6zPEvaIPWOv2DYbMhl9jr5G36Lv8Zf4R/yrTS+OVjHeYMcq/v0XzY31NA==</latexit><latexit

sha1_base64="X6cVB9dyMeKJ4rK8MeMKzxAI0Vg=">AAADIXicfVLLihQxFE2Vr7Edtcetm4uDTY+PJqULRRAEBV3p+OiZgU7T3Erf7gmTpMpKSqYp6kf8CL/Bra7diLjwT0xXtTAPmQshh3PuyQ0nSXOtnOf8ZxSfO3/h4qW1y50r61evXe9urO+4rCwkDWWms2IvRUdaWRp65TXt5QWhSTXtpgfPl/ruJyqcyuwHv8hpbHBu1UxJ9IGabEQPRUpzZSvUam7v1B0IJTSmpCv6+ARev2+pnkhNVdZCTjMvLKYaq5ap4S60BORwH8SsQFkldfWOavGCtEdo++ApcOj1hKdDXykLNYg3huZBFqsB7SHNhKOes01NX3+5HdZbob+f3AO+1WgrW9bYXqIxOEnOMPKzjLwjyE7/hTTpbvIBbwpOg2QFNtmqtifdP2KaydKQ9VKjc6OE535cYeGV1FR3ROkoR3mAcxoFaNGQG1fN69ZwOzBTmGVFWNZDwx51VGicW5g0dBr0++6ktiT/p41KP3s8DrHmpScr20GzUoPPYPlVYKoKkl4vAkBZqHBXkPsY3teHD3VsSmqWmSQnEzgNdh4MEj5I3nK2xm6yW6zPEvaIPWOv2DYbMhl9jr5G36Lv8Zf4R/yrTS+OVjHeYMcq/v0XzY31NA==</latexit><latexit

sha1_base64="i21blzHBwl/OPIo/vlvVfBB8eO0=">AAAB6HicbZBLSwMxFIXv1FetVevaTbAIrsqMG10KblxWsA9oh5LJZNrQPIbkjlKG/gG3rt2J/8ml/8T0sbDWA4HDOQn35ktyKRyG4VdQ2dnd2z+oHtaO6rXjk9NGvetMYRnvMCON7SfUcSk076BAyfu55VQlkveS6f2i7z1z64TRTzjLeazoWItMMIo+ao8azbAVLkW2TbQ2TVhr1PgepoYVimtkkjo3iMIc45JaFEzyeW1YOJ5TNqVjPvBWU8VdXC7XnJNLn6QkM9YfjWSZ/n5RUuXcTCX+pqI4cX+7RfhfNygwu41LofMCuWarQVkhCRqy+DNJheUM5cwbyqzwuxI2oZYy9GQ2piRq7pFEfwFsm+51Kwpb0WMIVTiHC7iCCG7gDh6gDR1gkMIrvAUvwXvwsUJXCdYMz2BDwecPfSOQvw==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit

sha1_base64="tCax8gHh1Gu0yb5jHs4Nqmux85o=">AAADLHicfVLLitRAFK3E19i+enTp5mJj0+OjSUZQEYQBBV3p+OiZga6mqVRuMsVUVWKqItOE/Igf4Te41bUbERdu/A6rkwjzkL4QcjiP3HCqolwKY4Pgh+efOXvu/IW1i71Ll69cvdZfv75jsrLgOOGZzIq9iBmUQuPECitxLy+QqUjibnTwbKnvfsTCiEy/t4scZ4qlWiSCM+uo+br3gEaYCl0xKVJ9p+6BGypZhLLCD0/g1buWGtJIVWVNeZxZqlkkWdUyNdyFloAc7gNNCsarsK7eYk2fo7QMWh88hQCGQ2rx0FZCQw30tcLUybRb0H6k2XA0szrU+EbL12G94fyj8B4EG43WxbIm9oIpxebhimCwKhj0KOr4X0nz/iAYB83AaRB2YEC62Z73f9M446VCbblkxkzDILezihVWcIl1j5YGc8YPWIpTBzVTaGZVc7o13HZMDElWuEdbaNijiYopYxYqck7F7L45qS3J/2nT0iaPZ67WvLSoebsoKSXYDJZXBWJRILdy4QDjhXD/CnyfufO17kId2xKpZSfhyQZOg53NcRiMwzebg62HXTtr5Ca5RUYkJI/IFnlJtsmEcO+T98X76n3zP/vf/Z/+r9bqe13mBjk2/p+/ZeP2vg==</latexit><latexit<latexit

Flow in a lid-driven cavity

u · ru + rp
1
u=0
Re

r·u=0
u(x) = (1, 0)
u(x) = (0, 0)

Model M3

Model M4

in ⌦

in ⌦
on 1
on 0